Characterization of Excess Risk for Locally Strongly Convex Population Risk
Anonymous Author(s)
AfﬁliationAddress
Abstract
We establish upper bounds for the expected excess risk of models trained byproper iterative algorithms which approximate the local minima.
Unlike the resultsbuilt upon the strong globally strongly convexity or global growth conditionse.g.
, PL-inequality, we only require the population risk to be locally stronglyconvex around its local minima.
Concretely, our bound under convex problemsis of order ˜ O
(1 /n ) .
For non-convex problems with d model parameters such thatd/n is smaller than a threshold independent of n , the order of ˜ O (1 /n ) can bemaintained if the empirical risk has no spurious local minima with high probability.
Moreover, the bound for non-convex problem becomes ˜ O
(1 / √ n ) without suchassumption.
Our results are derived via algorithmic stability and characterizationof the empirical risk’s landscape.
Compared with the existing algorithmic stabilitybased results, our bounds are dimensional insensitive and without restrictions onthe algorithm’s
implementation, learning rate, and the number of iterations.
Ourbounds underscore that with locally strongly convex population risk, the modelstrained by any proper iterative algorithm can generalize well, even for non-convexproblems, and d is large.
Introduction
The core problem in machine learning is obtaining a model that generalizes well on unseen test data.
The excess risk decides the model’s performance on these unseen data and it can be decomposed into optimization and generalization errors.
The tool of algorithmic stability [9, 10] has been provento be a suitable tool for exploring the excess risk.
Roughly speaking, the output of a stable algorithmis robust to a slight change in the algorithm’s input, i e., training set.
The output of a stable algorithmhas been proved to have controlled excess risk in [9], and the result has been further developedunder some speciﬁc algorithms [32, 79, 14, 16, 50, 20] e.g., stochastic gradient descent [61] (SGD).
However, these results have some limitations.
The results in [79, 14, 50, 47] are obtained under theassumption of either global strong convexity or global growth conditions (PL-inequality [42]).
Onthe other hand, the results in [32, 20] are only applicable to a speciﬁc algorithm, i.e., SGD, and theirbounds of generalization error diverge across training which is inconsistent with the observation that“train longer, generalize better” [34].
To improve these, we provide a uniﬁed analysis of the expected excess risk for a generic class ofiterative algorithms without any strong global conditions, i.e., global strong convexity or globalgrowth conditions in [79, 14, 50].
Concretely, we substitute the strong global conditions with weakerlocal strong convexity (see Section 2) of population risk around its local minima.
The substitution isbased on the fact that the nice strong convexity property can be locally (though not globally) satisﬁedby many important problems, e.g., PCA [30], ICA [27], and matrix completion [28].
We derive ourresults via algorithmic stability and characterize the empirical risk’s landscape.
For both convex and non-convex problems, our results can be applied to any proper algorithms that approximate localminima.
Moreover, our generalization upper bounds do not diverge with the number of training steps.
Technically, we upper bound both generalization and optimization errors to control the excess risk.
We ﬁrst show a fact that the locally strongly convexity around the local minima of population risk(population local minima) can be generalized to the local minima of empirical risk (empirical localminima), and the empirical local minima would concentrate around population local minima.
Then forconvex problems, we establish the generalization upper bound of the iterates of any proper algorithmvia algorithmic stability by leveraging the facts of iterates will converge to empirical local minima,which concentrate around population local minima.
For non-convex problems, our generalizationerror analysis includes three steps. 1)
By applying similar arguments under the convex problem, weupper bound the generalization error of those empirical local minima around population local minima.2)
Then, we prove that, with high probability, there are no extra empirical local minima except forthose concentrated around population local minima with guaranteed generalization capability. 3)
Finally, we extrapolate the upper bound of the generalization error to the iterates obtained by theproper algorithm as they converge to empirical local minima.
After controlling the generalization error, the excess risk is directly implied by characterizing theoptimization error.
By the proved local strong convexity of empirical risk and the convergence resultsof proper algorithms, the optimization error can be controlled as in [12, 29, 64, 27, 37].
Concretely, we establish an upper bound of order ˜ O
(1 /n )
( ˜ O ( · ) deﬁned in Section 2) for the expectedexcess risk of iterates obtained by any proper algorithm under convex problems.
Here n is thenumber of training samples.
For non-convex problems with d parameters of model, we establishan upper bound of order ˜ O
(1 / √ n + exp( − n ( c 1 − d/n )) where c 1 is a constant independent ofn and d .
Noticeably, the exponential term in the bound can be ignored when d/n ≤ c 1 , then ourbound becomes ˜ O (1 / √ n ) .
The bound can be applied to high-dimensional problems such that d isin the same order of n .
The result signiﬁcantly improves the classical one of order O ( (cid:112) d/n ) [63],which has polynomial dependence on d .
Moreover, our bound of order ˜ O
(1 / √ n ) can be improvedto ˜ O
(1 /n )
if the empirical risk has no spurious local minima with high probability, which can besatisﬁed for many important non-convex problems [30, 28, 1].
Our upper bounds to the excess risk underscore that, for both convex and non-convex problemssatisfying our regularity conditions, the model trained by an algorithm can generalize on test dataeven when d is large.
Our improvements over existing classical results are summarized as follows.
• For convex problems, our bound improves the standard upper bound of the expected excess risk in the order of O ( (cid:112) 1 /n )
[32] to ˜ O
(1 /n ) , under an extra locally strongly convex assumption.• For non-convex problems, we relax the dimensional-dependence in the standard excess risk boundof order
O ( (cid:112) d/n ) [63], under local strong convexity assumption.
• In contrast to the existing algorithmic stability based works [32, 79, 14], our results can be applied to any algorithms that approximate local minima without restrictions on the implementation ofalgorithms, learning rate, and the number of iterations.
Preliminaries2.1 Notations and Assumptions
In this subsection, we collect our (mostly standard) notations and assumptions.
We use (cid:107) · (cid:107) todenote (cid:96) 2 -norm for vectors and spectral norm for matrices.
B p ( w , r ) is (cid:96) p -ball with radius raround w ∈ R d .
Let dataset { z 1 , · · · , z n , z (cid:48) 1 , · · · , z (cid:48) n } be 2 n i.i.d samples from an unknowndistribution, and S = { z 1 , · · · , z n } is the training set, S i
= { z 1 , · · · , z i − 1 , z (cid:48)
i , z i +1 , · · · , z n } and
S (cid:48) =
S 1 .
Throughout this paper, we assume without further mention that the loss function f ( w , z ) isdifferentiable w.r.t.
to parameter w for any z , 0 ≤ f ( w , z ) ≤ M , and the parameter space W ⊆ R dis a convex compact set.
Thus (cid:107) w 1 − w 2 (cid:107) ≤ D for w 1 , w 2 ∈ W and some positive constant D .
The population risk is R ( w ) =
E z
[ f ( w , z )] and its empirical counterpart on the training set Sis R S ( w ) = n − 1 (cid:80) ni
=1 f ( w , z i ) .
Let w ∗ S ∈ arg
min w R S ( w ) and w ∗ ∈ arg
min w R ( w ) , Theprojection operator P W ( · ) is deﬁned as P W ( v ) =
arg min w ∈W {(cid:107) w − v (cid:107)} .
During our analysis, the order of sample size n can go to inﬁnity, and d can diverge to inﬁnity with n .
But we assume theother quantities are universal constant independent of n .
The symbol O ( · ) is the order of a number,while ˜ O ( · ) hides a poly-logarithmic factor in the number of model parameters d .
The following twoassumptions on loss function f ( w , z ) are imposed on the population risk.
Assumption 1 (Smoothness) .
For 0 ≤ j ≤ 2 , each z and any w 1 , w 2 ∈ W ,
[Equation]
where ∇ j f ( w , z ) are respectively loss function, gradient, and Hessian at w for j = 0 , 1 , 2 .
Assumption 2 (Non-Degenerate Local Minima) .
For w ∗ local in the set of local minima of populationrisk R ( w ) , ∇ 2 R ( w ∗ local ) (cid:23) λ > 0 , i.e., ∇ 2 R ( w ∗ local ) − λ I d is a semi-positive deﬁnite matrix.
Assumption 1 says that the loss function should be smooth enough, which is a mild assumptionand has been adopted in [32, 81, 30].
Assumption 1 and 2 together imply that the population riskis locally strongly convex around its local minima.
The rationale behind the imposed local strongconvexity is as follows.
Though the strong global conditions (e.g., global strong convexity) in[32, 79, 14, 16, 50, 20] do not hold in many problems, the weaker locally strongly convex conditionholds for many important problems, e.g., generalized linear regression [49], robust regression [49], PCA [30], ICA [27], and matrix completion [28].
2.2 Stability and Generalization
Deﬁnition 1 (Proper Algorithm) .
The algorithm A is proper if it approximates local minima 1 ofempirical risk R S ( w ) .
This is a rough deﬁnition of the discussed proper algorithm.
The sense in which algorithms approximate local minima will be made clear in our formal theoretical results.
Let A ( S ) be the parametersobtained by an algorithm A , e.g., SGD, on the training set S .
The performance of model on unseendata is determined by the excess risk deﬁned as follows.
Deﬁnition (Excess Risk) .
The excess risk is deﬁned as R ( A ( S ))
− inf w R ( w ) .
The excess risk is the gap of population risk between the current model and the optimal one.
In thispaper, we study the expected excess risk E A , S [ R ( A ( S ))
− inf w R ( w )] where E A , S [ · ] means theexpectation is taken over the randomized algorithm A and the training set S .
We may neglect thesubscript if there is no obfuscation.
Since R S ( w ∗ S ) ≤ R S ( w ∗ ) , we have the following decomposition.
[Equation]
The expected excess risk is upper bounded by the sum of optimization error E opt and generalizationerror E gen .
E opt is decided by the convergence rate of the algorithm A [12, 29].
The generalizationerror E gen can be controlled by algorithmic stability [9] as follows.
Deﬁnition 2.
Let S and S (cid:48) be two training sets, an algorithm A is (cid:15) -uniformly stable, if
[Equation]
The (cid:15) -uniformly stable is different from the one in [32], which does not take expectation over trainingsets S and S (cid:48) .
The next theorem proofed in Appendix A shows that the uniform stability implies theexpected generalization of the model, i.e., E gen ≤ (cid:15) stab .
It’s idea is similar to the ones in [9, 32, 14].
Theorem 1.
If A is (cid:15) -uniformly stable, then
[Equation]
Please note that all the analysis in this paper is applicable to the practically infeasible empirical riskminimization “algorithm” such that A ( S ) =
w ∗ S .
However, to make our results more practical, wesuppose A as iterative algorithms in the sequel.
For any given iterative algorithm A , let w t and w (cid:48) tdenote the output of the algorithm when A is iterated t steps on the training set S and S (cid:48) respectively.
Excess Risk under Convex Problems
In this section, we propose upper bounds of the expected excess risk for convex problems.
We imposethe following convexity assumption throughout this section.
Assumption 3 (Convexity) .
For each z and any w 1 , w 2 ∈ W , f ( w , z ) satisﬁes
[Equation]
3.1 Generalization Error under Convex Problems
As we have discussed, in the existing literature [32, 79, 14, 16, 50, 20], researchers have explored theexcess risk via the algorithmic stability to control the error generalization.
However, the obtainedgeneralization upper bounds of order O (1 /n ) in [32, 79, 14, 50] are built upon the strong assumptionsof either global strong convexity or global growth conditions, e.g., PL-inequality [42].
On the otherhand, the generalization upper bounds in [32, 20] are only applied to SGD, and they diverge asthe number of iterations grows.
For example, Theorem 3.8 in [32] establishes an upper bound2 L 20 (cid:80) t − 1 k =
η k /n to the algorithmic stability of SGD with learning rate η k , which diverges whent → ∞ , as the convergence of SGD requires (cid:80) ∞ k =0 η k = ∞ [8].
Thus the bound can not explain theobservation that the generalization error of SGD trained model converges to a constant [8, 34].
To mitigate the drawbacks in the existing literature, we propose the following new upper boundof algorithmic stability (Theorem 2).
Our bound can be applied on the top of any proper algorithm deﬁned in Deﬁnition 1, and it remains small for an arbitrary number of iterations as longas the sample size n is large.
Under convexity Assumption 3, the proper algorithm means that 144 E [ R S ( w t ) − R S ( w ∗ S )] → 0 as t → ∞ .
Our theorem is based on the following intuition.
Due to thelocally strongly convex property discussed after Assumption 2, there exists (with high probability)the unique global minimum w ∗ S of R S ( · ) and w ∗ S (cid:48) of R S (cid:48) ( · ) that concentrate around the unique (theuniqueness is from Assumption 2) population global minimum w ∗ .
Then, the provable convergenceresults of w t → w ∗ S and w (cid:48) t → w ∗ S (cid:48) imply the algorithmic stability (see Lemma 3 in Appendix).
Theorem 2.
Under Assumption 1-3,
[Equation]
[Equation]
where (cid:15) stab ( t ) =
E S , S (cid:48)
[sup z | E A [ f ( w t , z ) − f ( w (cid:48) t , z )] | ] is the stability of w t , and (cid:15) ( t ) =
151 E [ R S ( w t ) − R S ( w ∗ S )] , w ∗ S is the global minimum of R S ( · ) .
The proof of this theorem is in Appendix B.1.
The expected generalization error of w t is upperbounded by the right hand side of (6) due to Theorem 1.
Compared with the existing result [32], theextra term related to (cid:112) (cid:15) ( t ) in our bound originates from our proof technique, and it seems to beunavoidable according to [63].
Since for proper algorithms, e.g., GD and SGD, (cid:15) ( t ) → 0 as t → ∞the leading term of the upper bound (6) is C ∗ log d/n = ˜
O (1 /n ) with C [Equation]
3200 L 0 L 21 D/λ 2 .
In summary, the local strong convexity (Assumption 2) enables us to establish an algorithmic stabilitybased generalization bound (6).
The bound improves the classical result of SGD 2 L 2 0 (cid:80) t − 1 k =
η k /n in[32] as it can be applied to any proper algorithm with any learning rate and number of iterations.
3.2 Excess Risk Under Convex Problems
According to (2), we can upper bound the expected excess risk by combining the generalization upperbound (6) with the convergence results in convex optimization.
Theorem 3.
For w ∗ S ∈ arg min w R S ( w ) , and w ∗ ∈ arg
min w R ( w ) , under Assumption 1-3,
[Equation]
where (cid:15) ( t ) =
E [ R S ( w t ) − R S ( w ∗ S )] .
This theorem provides an upper bound of the expected excess risk.
The bound decreases with thenumber of training steps t , and is of order ˜ O
(1 /n )
if t is sufﬁciently large.
Comparison.
Under the extra local strong convexity assumption, our result signiﬁcantly improvesthe bound of order O (1 / √ n ) in [32].
On the other hand, our bound matches (in order) the resultunder strongly convex problem [63, 81].
It seems our result has a worse dependence on the strongconvex parameter λ , i.e., from 1 /λ to 1 /λ 4 .
The worse dependence is acceptable as local strongconvexity is weaker than strong convexity.
Moreover, our bound is not necessarily weaker comparedto the current results [63, 81] under global strongly convex problem.
This is because λ in our boundis the local strongly convex parameter restricted around the minimum point, which is larger than theglobal one over the whole parameter space appears in [81].
Improving the dependence on λ withoutsacriﬁcing the order of n seems to be infeasible based on our techniques 2 .
It might be a meaningfultopic to be explored in the future.
Finally, our result has no conﬂict with the lower bound for generalconvex problem in the order of O ( (cid:112) d/n ) [25].
This is because Assumption 1 and 2 restrict ourresult to a smaller class of distributions and functions, which rules out the counter-examples in [25].
To make our results concrete, we apply them to GD and SGD as examples.
Note that R S ( w ) =n − 1 (cid:80) ni
=1 f ( w , z i ) , the GD and SGD respectively start from w 0 follow the update rules of
[Equation]
8) and
[Equation]
9) where i t is randomly sampled from 1 to n .
Note the convergence rate of w t updated by GD and SGDare respectively O (1 /t )
[12] and ˜ O
(1 / √ t )
[64], we have the following two corollaries declare theconverged expected excess risks whose proofs appear in Appendix B.2.
Corollary 1.
Under Assumption 1-3, if w t is updated by GD in (8) with η t = 1 /L 1 , then
[Equation]
Corollary 2.
Under Assumption 1-3, if w t is updated by SGD in (9) with η t = D/ ( L 1 √ t + 1) , then
[Equation]
Excess Risk Under Non-Convex Problems
In this section, we present the upper bounds of the expected excess risk of iterates obtained by properalgorithms that approximate local minima under non-convex problems.
4.1 Generalization Error Under Non-Convex Problems
In this subsection, we study the generalization error under non-convex problems.
Unfortunately,the analysis in Section 3 can not be directly generalized here due to the following reason.
Thegeneralization error under convex problems relies on the fact that there exists the unique empiricallocal minima w ∗ S of R S ( · ) and w ∗ S (cid:48) of R S (cid:48) ( · ) that concentrate around the unique population localminimum w ∗ of R ( · ) .
Under non-convex problems, there can be many empirical and populationlocal minima.
The iterates obtained on S and S (cid:48) may converge to different empirical local minimaaway from each other, which invalidates our methods used in convex problems.
Fortunately, we can prove that for each population local minimum, there is an empirical localminimum concentrated around it with high probability.
If the generalization upper bound for theselocal minima is established, and there are no extra empirical local minima, the convergence resultsof the iterates obtained by proper algorithms imply their generalization ability.
Next, we prove ourresults following this road map.
First, we establish the generalization upper bound for the empirical local minima around the population local minima.
According to Proposition 1 in the Appendix C.1, there are only ﬁnite populationlocal minima, thus the non-convex problems with local minima consists of a manifold [48] is notconsidered in this paper.
Let M = { w ∗ 1 , · · · , w ∗ K } be the set of population local minima.
Thenumber of local minima K may depend on the problem of interest.
In many important non-convexproblems, K can be quite small, e.g., K = 2 for PCA [30] and K = 1 for robust regression [49].
Then, we notice that the population risk is strongly convex in B 2 ( w ∗ k , λ/ (4 L 2 )) .
Similar to thescenario under convex problems, we can verify that the empirical risk is locally strongly convex in
B 2 ( w ∗ k , ( λ/ 4 L 2 )) with high probability.
Next, we consider the following points
[Equation]
for k = 1 . . .
, K .
We show that w ∗ S ,k is a local minimum of R S ( · ) with high probability and presentthe generalization bound of it.
Note that in Theorem 1, A can be infeasible.
We construct an auxiliarysequence w t via an infeasible algorithm.
[Equation]
Then, as w t locates in B 2 ( w ∗ k , λ/ (4 L 2 )) in which R S ( · ) is strongly convex with high probability,we can establish the algorithmic stability bound of the w t .
Combining this with the convergenceresult of w t to w ∗ S ,k implies the generalization ability of w ∗ S ,k .
The following lemma states ourresult rigorously.
Lemma 1.
Under Assumption 1 and 4, for k = 1 , . . .
, K , with probability at least
[Equation]
w ∗ S ,k 3 is a local minimum of R S ( · ) .
Moreover, for such w ∗ S ,k , we have
[Equation]
The lemma is proved in Appendix C.1.1., and it guarantees the generalization ability of those empiricallocal minima located around population local minima.
The expected generalization error on theselocal minima is of order ˜ O
(1 /n )
as in convex problems.
In the sequel, we show that there are noextra empirical local minima expected for these w ∗ S ,k with high probability, under the following mildassumption, which also appears in [49, 30].
Assumption 4 (Strict saddle) .
There exists α, λ > 0 such that (cid:107)∇ R ( w ) (cid:107) > α on the boundary of
W , and
[Equation]
where σ min ( ∇ 2 R ( w )) is ∇ 2 R ( w ) ’s smallest eigenvalue.
The Assumption 4 is a generalized version of local strong convexity Assumption 2 (can be implied by
Assumption 4).
A vast vary of machine learning problems satisfy this assumption, e.g., generalizedlinear regression, robust regression, normal mixture model, tensor decomposition, matrix completion,
PCA, and ICA [30, 49, 81].
We refer readers to [30, 27, 28, 49] for more details of this assumption.
Let M S
= { w : w is a local minimum of R S ( · ) } be the set consists of all the local minima ofempirical risk R S ( · ) .
Then we establish the following non-asymptotic probability bound.
Lemma 2.
Under Assumption 1 and 4, for r = min (cid:110) λ 8 L 2 ,
[Equation]
1 (cid:111) , with probability at least
[Equation]
we havei:
M S = { w ∗ S , 1 , . . . , w ∗ S ,K }
ii: for any w ∈ W , if (cid:107)∇ R S ( w ) (cid:107) < α 2 / (2 L 0 ) and ∇ 2 R S ( w ) (cid:31) − λ/ 2 , then (cid:107) w −
P M S ( w ) (cid:107) ≤ λ (cid:107)∇ R S ( w ) (cid:107) / 4 , where ∇ 2 R S ( w ) (cid:31) − λ/ 2 means ∇ 2 R S ( w ) + λ/ 2 I d is a positive deﬁnite matrix.
The ﬁrst conclusion in this lemma states that there are no extra empirical local minima except forthose w ∗ S ,k concentrate around population local minima, which have guaranteed generalizationability (by Theorem 1).
The second result is that the empirical risk is “error bound” (see [42] for itsdeﬁnition) around its local minima, with high probability.
The “error bound” is a nice property inoptimization [42].
Proof of the lemma is in Appendix C.2.1.
The probability bound (17) will appearin the generalization bound of iterates obtained by proper algorithms accounting for the existence ofthose empirical local minima away from population local minima.
We defer the discussion to thebound after providing our generalization upper bound in Theorem 4.
We move forward to derive the generalization upper bound of those iterates obtained by the properalgorithm that approximates the local minima under non-convex problems.
Under strict saddle
Assumption 4, the proper algorithm A approximates the second-order stationary point (SOSP) 4 , thatsays with probability at least 1 − δ
( δ is a constant that can be arbitrary small),
[Equation]
where w t is updated by the algorithm A , and ζ ( t ) , ρ ( t ) → 0 (which may have poly-logarithmicdependence on δ [37]) as t → ∞ .
To instantiate such proper algorithms, we construct an algorithm that satisﬁes (18) in Appendix D.
The following theorem establishes a generalization upper bound of w t obtained by such A .
Theorem 4.
Under Assumption 1, 2 and 4, if w t satisﬁes (18) and r deﬁned in Lemma 2, by choosingt such that ζ ( t ) < α 2 / (2 L 0 ) and ρ ( t ) < λ/ 2 we have
[Equation]
where
[Equation]
and
[Equation]
If with probability at least 1 − δ (cid:48) ( δ (cid:48) can be arbitrary small) [Equation] R S ( · ) has no spurious local minimum,then
[Equation]
This theorem is proved in Appendix C.3, and it provides upper bounds of the expected generalizationerror of iterates obtained by any proper algorithm that approximates SOSP.
We present an explanationof each term in it as follows.
The 2 DL 0 δ is of order O (1 / √ n ) or O (1 /n ) as we take the correspondedδ = 1 / √ n or 1 /n , and 8 L 0 ζ ( t ) /λ can be arbitrary small if we take a sufﬁciently large t .
Since ξ n, 1is of order ˜ O (1 /n ) , we next explore ξ n, 2 .
The leading term in ξ n, 2 is
[Equation]
If d is large enough to make log 4 d ≤ d log(3 D/r ) , then ξ n, 2 ≤ exp( − c 2 n ( c 1 − dn )) , wherec 1 = λ
2 / (512 L 21 log(3 D/r )) and c 2 = 2 log(3 D/r ) .
Thus ξ n, 2 (cid:28) ˜ O
(1 /n ) provided by d/n < c 1 .
In this case, the 2 KM/ √ n appears in bound (19) implies it is of order ˜ O
(1 / √ n ) , even under highdimensional problems such that d is in the same order of n .
The K can be small here for manynon-convex problems, as previously discussed.
Moreover, the bound (22) improves the result in (19)to ˜ O (1 /n ) , under the condition of empirical risk has no spurious local minima with high probability(i.e. δ (cid:48) ≤ ˜ O
(1 /n ) ).
The condition can be satisﬁed by many important non-convex optimizationproblems e.g., PCA [30], matrix completion [28], and over-parameterized neural network [43, 1, 21]
Comparison.
Under the extra strictly saddle Assumption 4, our bounds (no matter whether imposing the no spurious local minima assumption) improve the classical results of order O ( (cid:112) d/n )based on the uniform convergence theory [63] or the one of order O
( t c /n ) for a positive c [32, 79]based on algorithmic stability.
[30] get the result of order ˜ O
( d/n ) under the same Assumptions
1and 4 imposed in this paper.
However, their bound has a linear dependence on d , thus can not benon-vacuous like ours when d is in the same order of n .
Speciﬁcally, if the parameter space satisﬁes some sparsity conditions [6, 80, 35, 36, 22, 72] , we canextrapolate Theorem 4 to ultrahigh-dimensional problem such that d (cid:29) n .
For example, suppose theparameter space W is contained in a (cid:96) 1 -ball, i.e., (cid:107) w 1 − w 2 (cid:107) 1 ≤ D (cid:48) for some positive D (cid:48) .
Note thatthe covering number (deﬁned in [72]) of polytopes (Corollary 0.0.4 in [71]) is much smaller thanthat of (cid:96) 2 -ball.
Then, applying the similar proof of Theorem 4 establishes the same upper bound ofgeneralization error w.r.t.
w t with ξ n, 2 in Theorem 4 replaced by
[Equation]
(24) where the much smaller relationship is valid as long as log( d ) /n
→ 0 .
4.2 Excess Risk Under Non-Convex Problems
In this subsection, we establish upper bounds for the expected excess risk of iterates obtainedby proper algorithms under non-convex problems.
In contrast to convex optimization, the properalgorithm under non-convex problems is not guaranteed to ﬁnd the global minimum, as it onlyapproximates SOSP.
Hence the optimization error may not vanish as in Theorem 3.
The followingtheorem proved in Appendix C.4 establishes an upper bound of the expected excess risk.
Theorem 5.
Under Assumption 1, 2 and 4, if w t satisﬁes (18) , by choosing t in (18) such thatζ ( t ) < α 2 / (2 L 0 ) and ρ ( t ) < λ/ 2 , we have
[Equation]
where w ∗ is the global minimum of the population risk.
If with probability at least 1 − δ (cid:48) ( δ [Equation] can bearbitrary small), R S ( · ) has no spurious local minimum, then
[Equation]
/n ≤ O (1)) , (26) where ξ n, 1 and ξ n, 2 are deﬁned in Theorem 4, and w ∗ S is the global minimum of R S ( · ) .
From the discussions in the last section, the bound (25) and (26) become O (1 / √ n ) and ˜ O
(1 /n ) ,respectively, when d is in the same order of n and t → ∞ .
Besides that, in (25), expected for the orderof convergence rate O ( ζ ( t )) and the generalization bound of order ˜ O
(1 / √ n +exp( − c 2 n ( c 1 − d/n )) 303, there is an extra E A , S [ R S ( P M S ( w t ))
− R S ( w ∗ S )] in the bound (25), compared with the resultof convex problems in Theorem 3.
This is the gap between the empirical global minimum and thealgorithmic approximated empirical local minimum.
The gap seems necessary as the proper algorithmis not guaranteed to ﬁnd the global minima, and if so, the gap becomes zero.
The bound (26) of order ˜ O
(1 /n ) is obtained under empirical risk without spurious local minima,which is proven to be hold on many important non-convex problems e.g., PCA [30], matrix completion[28], and over-parameterized neural network [43, 1, 21, 85].
Related Works
Generalization
The generalization error is the gap between the model’s performance on trainingand unseen test data.
One of the central tools to bound the generalization error in statistical learningis uniform convergence theory.
However, this method is unavoidably related to the capacity ofhypothesis space e.g., VC dimension [7, 17, 59, 31], Rademacher complexity [3, 51, 55], coveringnumber [73, 82, 65], or entropy integral [72].
Thus, these results are not well suited for highdimensional hypothesis spaces, which makes the mentioned measures to be large.
The generalization error of the iterates obtained by some algorithms, e.g., GD or SGD, is often ofmore interest.
There are plenty of papers working on this topic via the tool of algorithmic stability[9, 26, 10, 30, 63], differential privacy [18, 41], robustness of model [75, 66, 77], and informationtheory [74, 67, 11].
However, these tools either depend heavily on algorithm implementation(algorithmic stability and information theory) or require unveriﬁable conditions (robustness anddifferential privacy).
This paper combines the technique of characterizing empirical loss landscapeand algorithmic stability to explore the generalization under both convex and non-convex problems.
Our methods develop a new way to use algorithmic stability, which can be applied without restrictionson the algorithm, learning rate, and the number of iterations.
Optimization Results in this paper are related to both convex and non-convex problems.
For convex problems, [12] summarizes most of the classical algorithms in convex optimization.
Someother novel methods [40, 62, 56] with lower computational complexity have also been extensivelyexplored.
Recently, the non-convex optimization has attracted quite a lot attentions owing to thedevelopment of deep learning [33, 70].
But most of the existing algorithms [29, 2, 57, 15, 23, 78]approximate the ﬁrst-order stationary point instead of local minima.
Under non-convex problem, the algorithm that approximates SOSP is proper (approximate localminima) in this paper.
We refer readers for recent progress in the topic of developing algorithmsapproximating SOSP to [27, 24, 19, 37, 39, 76, 52, 84, 38].
The discussed proper algorithms in thispaper have constrained parameter space which is different from the ones in [5, 13, 52].
To resolvethis, we also develop an algorithm that approximates SOSP under our constraints in Appendix D.
Excess Risk
A straightforward way to characterize the excess risk is by controlling the generalization and optimization errors, respectively, as we did in this paper.
Thus, for this problem, theused tools are similar to the ones in analyzing generalization, e.g., uniform convergence theory[69, 81, 25], algorithmic stability [32, 14, 16, 79, 20], information theory [53, 54].
However, thediscussed drawbacks of these tools also appeared.
Our results are built upon the combination ofcharacterizing empirical risk’s landscape and algorithmic stability.
Moreover, they are dimensionalinsensitive, independent of algorithm’s implementation, and they improve the order of existing resultsunder both convex and non-convex problems.
Conclusion
This paper provides a uniﬁed analysis of the expected excess risk of models trained by properalgorithms under convex and non-convex problems.
Our primary techniques are algorithmic stabilityand the non-asymptotic characterization of the empirical risk’s landscape.
Under the conditions of local strong convexity around population local minima and some othermild regularity conditions, we establish the upper bounds of the expected excess risk in the orderof ˜ O
(1 /n ) and ˜ O (1 / √ n ) (can be improved to ˜ O
(1 /n ) when empirical risk has no spurious localminima with high probability) under convex and non-convex problems respectively.
The presented results improve the existing results in many aspects.
For convex problems, ourresults improve the standard excess risk bound of order O ( (cid:112) 1 /n )
[32] to ˜ O
(1 /n ) under locallyconvex assumption.
For non-convex problems, our results signiﬁcantly improve the standard uniformconvergence bound in the order of O ( (cid:112) d/n ) [63] when d/n is smaller than a universal constant.