Gradient Descent Is Optimal Under Lower Restricted Secant Inequality And Upper Error Bound
Abstract
The study of first-order optimization is sensitive to the assumptions made on the objective functions.
These assumptions induce complexity classes which play a key role in worst-case analysis, including the fundamental concept of algorithm optimality.
Recent work argues that strong convexity and smoothness—popular assumptions in literature—lead to a pathological definition of the condition number [12].
Motivated by this result, we focus on the class of functions satisfying a lower restricted secant inequality and an upper error bound.
On top of being robust to the aforementioned pathological behavior and including some non-convex functions, this pair of conditions displays interesting geometrical properties.
In particular, the necessary and sufficient conditions to interpolate a set of points and their gradients within the class can be separated into simple conditions on each sampled gradient.
This allows the performance estimation problem (PEP, [7, 33]) to be solved analytically, leading to a lower bound on the convergence rate that proves gradient descent to be exactly optimal on this class of functions among all first-order algorithms.
Introduction
The typical framework to study convergence properties of first-order algorithms in the context of machine learning is to first establish a class of objective functions to optimize through assumptions usually bound to a constant, such as L -smoothness and µ -strong convexity.
A tuning prescription of an algorithm is then made based on the constants, (e g. step size α = 2 µ + L in the case of the gradient descent method on smooth and strongly convex functions), and finally a worst-case convergence rate can be derived for this algorithm when using this tuning prescription.
In some cases, a lower bound on the achievable worst-case convergence rate can also be derived, leading to the theoretical optimality of an algorithm on the considered class of function, for instance the Nesterov accelerated gradient method [27] is known to be optimal up to a constant on strongly convex and smooth functions.
However in [12], the authors establish that such framework and its derived results are very sensitive to the choice of assumptions, and that strong convexity and smoothness can exhibit pathological behaviors leading to conservative tuning and arbitrarily sub-optimal convergence rates, even when the resulting algorithm achieves theoretical worst-case optimality on this class of functions.
Furthermore, they propose a set of more robust alternative conditions.
In this work, we focus on a specific pair of such alternative conditions : lower restricted secant inequality ( RSI − ) and upper error bounds ( EB + ).
Our main contribution is to show that the gradient descent (GD) method with a certain tuning is exactly optimal on the classes of objective functions induced by these conditions, confirming that optimality results are highly sensitive to the choice of conditions. Another consequence is that no algorithm can accelerate on this class of functions, implying that additional assumptions are required to explain the practical efficiency of accelerated methods.
Notation Let F the set of differentiable functions from R d to R that admit a convex set of global minima X ∗ f .
We focus on the problem of optimizing a function f ∈ F , i.e. finding x ∈ X ∗ f .
For any x ∈ R d , we denote x ∗ f
the orthogonal projection of x on X ∗ f .
By abuse of notation, when the context is not ambiguous, we will simply denote x ∗ f as x ∗ .
We call gradient descent (GD) the standard optimization algorithm based on the following update, where α is the step size :
[Equation]
We call first-order algorithm all A that consider past iterates, function values and gradients and output a next iterate.
Formally, A can be seen as a sequence of functions {A n | n ∈ N } such that for any n ∈ N , A n is a function defined on (cid:0) R d × R × R d (cid:1) n +1 with values in R d .
Under that formalism, applying A to optimize an objective function f starting in x 0 ∈ R d generates a sequence of iterates ( x i ) i such that ∀ i, x i +
=
A i (cid:16) ( x j , f ( x j ) , ∇ f ( x j ))
j ≤ i (cid:17) .
Note that we do not require the iterates of
A to lie within the span of observed gradients as it is often the case in the literature.
Outline Section 2 introduces RSI − and EB + and provides some basic properties and motivation.
Section 3 discusses related works in the literature.
Section 4 defines and establishes the necessary and sufficient interpolation conditions for RSI − and EB + , which is a key element of our analysis.
Section 5 proves the lower bound on the convergence rate of first-order algorithms under RSI − and
EB + , and finally Section 6 concludes our work.
Detailed proofs are provided in the appendix.
Lower restricted secant inequality and upper error bounds
We now define RSI − and EB + and discuss some basic properties.
Definition 2.1 (Lower restricted secant inequality) Let f ∈ F and µ >
[Equation]
Intuitively, RSI − ( µ ) enforces that the further x is from X ∗ f , the stronger the gradient of f in x must be in the opposite direction of X ∗ f .
Remark 2.2 RSI − ( µ ) includes non-convex functions.
However, it prevents flat landscapes outside of X ∗ f , and requires f to increase at least quadratically with the distance to X ∗ f , as established in [12] :
[Equation]
Definition 2.3 (Upper Error Bounds) Let f ∈ F and L >
[Equation]
EB + ( L ) thus enforces that the gradient of f is controlled by the distance to X ∗ f .
Remark 2.4 L -smoothness implies EB + ( L ) and µ -strong convexity implies RSI − ( µ ) .
However, one must be careful before claiming that RSI − and EB + are respectively weaker than strong convexity and smoothness : for µ 0 > µ 1 , RSI − ( µ 0 ) is neither stronger or weaker than µ 1 -strong convexity, and for L 0 < L 1 , EB + ( L 0 ) is neither stronger or weaker than L 1 -smoothness.
Therefore, even when the objective function is smooth and strongly convex, considering convergence results under RSI − and EB + is relevant, as we might obtain better constants µ and L .
Many convergence results depend of the condition number κ = Lµ .
Better constants leads to a better condition number, and thus a potentially better convergence rate (including when the dependence in the condition number κ is the same or worse).
As a consequence, machine learning problems with strongly convex and smooth objective functions are all potential applications of results under RSI − and EB + , provided we can obtain better constants under these conditions.
Prop 1 (convergence rate of GD under RSI − and EB + ) Let f ∈ RSI − ( µ ) ∩ EB + ( L ) .
Then gradient descent with learning rate α = µL 2 on f will guarantee the following convergence rate:
[Equation]
Moreover, when the learning rate is set to α 0 = 12 µ on the first step, and α = µL 2 on every other step, gradient descent guarantees the following convergence rate:
[Equation]
Proof.
When α = µ L 2 , we have
[Equation]
which proves (1).
To prove (2), we simply note that when using α = 12 µ , we have:
[Equation]
Interestingly, RSI − and EB + are direct bounds on the two additional terms obtained by developing ∥ x i − α ∇ f ( x i ) − x ∗ i ∥ 22 , leading to an extremely simple proof.
On the intuitive level, RSI − lower bounds the gain from stepping in the direction of X ∗ f , while EB + upper bounds the error coming from the component of the gradient orthogonal to that direction.
Remark 2.5 The literature gives a worst case convergence rate of gradient descent on µ -strongly convex and L -smooth functions of ∥ x i − x ∗ i ∥ 22 ≤ (cid:16) 1 − 2 µ µ + L (cid:17) 2 i ∥ x 0 − x ∗ 0 ∥ 22 , using the step size α = 2 µ + L [28, 30].
While this rate is better for a fixed µ and L , we again emphasize that the constants µ and L may be very different depending on the chosen conditions, and thus these two rates can not directly be compared.
Motivation: we empirically verify in Appendix A that the optimization path of a ResNet18 [16] trained for classification on CIFAR10 [20] verifies the interpolation conditions of RSI − ∩ EB + , which are introduced in Section 4.
This result guarantees the existence of a function in RSI − ∩ EB + which exactly interpolates the observed gradients, and thus all convergence guarantees of RSI − ∩ EB + naturally apply in this practical deep learning setting. RSI − ∩ EB + therefore provides linear convergence guarantees while being empirically applicable to neural networks (although with respects to a local minima), an impressive feat given the highly non-convex nature of neural networks loss functions.
Related Work
Throughout the literature, many choices of assumptions have been used to study first-order optimization.
Most assumptions fall into one of two categories : lower conditions and upper conditions that respectively take the form of a lower and an upper bound on properties of the objective function.
For instance, strong convexity lower bounds the curvature of the objective function and is thus a lower condition .
Similarly, smoothness is an upper condition .
Lower conditions have been the most extensively studied assumptions, such as Polyak-Łojasiewicz [29], local-quasi-convexity [15], weak quasi-convexity [14], quadratic growth [2, 4, 18], KurdykaŁojasiewicz [21, 3], optimal strong convexity [22, 25, 8], weak strong convexity [19, 26], error bounds [24].
Some recent works have explored the relations between these lower conditions [19, 38].
In this work we focus on the restricted secant inequality ( RSI − ) (which we denote as lower restricted secant inequality to differentiate it from its upper bound equivalent) which was introduced in [39], and has been used (along with its convex extension restricted strong convexity ) in many recent theoretical derivations of linear convergence rates [36, 31, 37].
On the contrary, because most machine learning objective functions are naturally smooth, fewer works have explored alternatives to smoothness.
However as discussed in Remark 2.4, it is still relevant to study these alternatives on smooth objectives due to potentially better conditioning.
The most notable ones in the literature are local smoothness [15], restricted smoothness [1], relative smoothness [23, 13, 40], weak-smoothness [14], expected smoothness [11].
In [12], the authors argue that lower conditions can be naturally translated into equivalent upper conditions by changing the lower bound into an upper bound, and vice-versa.
Subsequently, they introduce a set of upper equivalent to existing lower conditions , such as upper error bounds EB + , the natural upper equivalent to error bounds from [24].
Throughout this work, we focus on EB + as an upper condition .
Finally, a key to our analysis are the necessary and sufficient interpolation conditions of RSI − ( µ ) ∩
EB + ( L ) (Section 4).
The search of these conditions has been largely motivated by Performance
Estimation Problems (PEP), introduced in [33] and with many recent successful applications [17, 32, 6, 35, 34].
PEP is a framework for computer-assisted worst-case convergence analysis, that only requires necessary and sufficient interpolation conditions for the considered class of objective functions.
In the case of RSI − ∩ EB + however, we found the interpolation conditions to be independent (see Section 4), which made the worst-case analysis directly solvable analytically.
Interpolation conditions
In this section we provide and discuss the necessary and sufficient interpolation conditions for
RSI − ∩ EB + .
Their importance stems from the framework used in PEP and in this work to analyze worst-case convergence.
Given a first-order optimization algorithm A
we want to find the slowest convergence rate of A among all f within a class of objective functions C and starting point x 0 ∈ R d \ X ∗ f .
That is equivalent to solving the following optimization problem at any step number n :
[Equation]
Directly searching for f in the functional space is generally intractable, however if we can explicitly find the set G of all families ( x i , f i , g i )
i such that ∃ f ∈ C , ∀ i, ∇ f ( x i ) =
g i and f ( x i ) =
f i (we say that f interpolates ( x i , f i , g i ) i ), then problem (5) can be reduced to:
[Equation]
In many cases (see Section 3), problem (6) is tractable and becomes a very powerful analysis tool providing lower bounds, upper bounds, and optimal tuning for different types of algorithms and assumptions used.
A crucial and difficult component of this analysis is to formulate the interpolation conditions , that is the necessary and sufficient conditions for a family ( x i , f i , g i )
i to belong in G .
Driven by these motivations, we now establish the interpolation conditions for RSI − ∩ EB + .
In Theorem 1, we introduce the necessary and sufficient conditions to interpolate a family ( x i , g i ) , without considering the function values ( f i ) i .
This theorem could be used to find the worst case convergence rate over all first-order algorithms that ignore function values.
However, in Corollary 1, we deduce from Theorem 1 sufficient (but not necessary) conditions to interpolate a family ( x i , f i , g i ) .
These conditions allow us in Section 5 to find a lower bound on the worst-case convergence rate for all first-order algorithms (including algorithms that have access to function values information), which we know is tight thanks to Prop 1.
Theorem 1 (Interpolation conditions) Let ( x i , g i )
i ≤ n ∈ (cid:0)
R d × R d (cid:1) n +1 , such that the x i are separate points.
Then, ∀ µ, L > 0 :
[Equation] [Equation]
and
[Equation]
where x ∗ i is the orthogonal projection of x i onto X ∗ .
Proof.
In order to preserve concision and clarity, we will only present the broad outline of the proof here.
For a complete technical proof, see Appendix B.
The direct implication is trivial as it is a direct application of RSI − ( µ ) and EB + ( L ) definitions to f in ( x i ) i .
For the reverse implication, we will construct a function f ϵ,β that interpolates ( x i , g i ) i .
The function f ϵ,β is a quadratic everywhere except in the spheres of radius ϵ around each x
i , ϵ being small enough for these spheres to never intersect.
Inside the sphere of radius ϵ around a given x i , f ϵ,β will be perturbed by adding a term λ ( ∥ x − x i ∥ 2 ) h
( x ) where h is affine in x , and λ is a scaling term so that λ ( ϵ ) = 0 at the border of the sphere, and λ (0)
= 1 in its center x i .
The key is to find a function λ that preserves the properties of RSI − ( µ ) and EB + ( L ) .
We use
[Equation]
And our construction f ϵ,β is given by:
[Equation]
The rest of the proof is to use the Taylor expansions of f ϵ,β to show that for sufficiently small ϵ and β , f ϵ,β will belong in RSI − ( µ ) and EB + ( L ) (see Appendix B).
■
Corollary 1 Let ( x i , f i , g i ) i ≤ n ∈ (cid:0) R d × R × R d (cid:1) n +1 , such that the x i are separate points.
Then, ∀ µ, L > 0 :
[Equation]
where x ∗ i is the orthogonal projection of x i onto X ∗ .
Proof.
We simply use the function f
ϵ,β from the proof of Theorem 1 and note that ∀ i, f ϵ,β ( x i ) = µ + L 4 ∥ x i − x ∗ i ∥ 22 .
■
Theorem 1 and Corollary 1 are the key elements to prove the optimality of gradient descent on RSI − and EB + among all first-order algorithms (see Section 5).
Remark 4.1 The interpolation conditions in Theorem 1 are independent, in the sense that a family ( x i , g i ) i admits an interpolation in RSI − ( µ ) ∩ EB + ( L )
if and only if each { ( x i , g i ) } admits an interpolation in RSI − ( µ ) ∩ EB + ( L ) .
Similarly, the sufficient interpolation conditions in Corollaryare also independent.
This property of independent interpolation conditions drastically simplifies convergence analysis and is the main reason we are able to analytically derive a lower bound in Section 5.
Indeed, given an interpolable family ( x i , f i , g i ) i ≤ n for a given set X ∗ , it is sufficient to show that ( x n +1 , f n +1 , g n +1 ) is interpolable with X ∗ to prove that the entire family ( x i , f i , g i ) i ≤ n +1 can be interpolated.
It is thus simple to find the set of interpolable ( f n +1 , g n +1 ) given X ∗ , x n +1 , and an interpolable ( x i , f i , g i ) i ≤ n .
Lower bound on the convergence rate
In this Section we derive a lower bound on the convergence rate of first-order algorithms on RSI − and EB + .
This lower bounds applies under the assumption that the number of steps taken is smaller than the number of dimension d .
This assumption is frequent in the literature (e.g. [5]) and not constraining for high-dimensional optimization.
The observed gradients of the worst-case functions for optimal algorithms are typically orthogonal to one another (see [6]) which is not possible when the number of steps becomes larger than the dimension d .
We conjecture that when not bounding the number of steps, it is possible to achieve an asymptotic rate in O (cid:0) 2 − nd (cid:1) which would be better than the usual rates obtained for very ill-conditioned functions, while having little to no practical uses due to the bad convergence properties on a lower number of steps.
We now introduce Lemma 1, which is the cornerstone of the proof of Theorem 2:
Lemma 1 Let µ > 0 and L > µ .
Let α 0 ∈ (cid:104) µL 2 , max (cid:16) µL 2 , 12 µ (cid:17)(cid:105) .
For any first-order optimization algorithm A and starting point x 0 ∈ R d , there exists ( g i ) i ≤ d − 2 ∈ R d , ( f i ) i ≤ d − 2 ∈ R and S d − 2 ⊆
S d − 1 ⊆ · · · ⊆ S 0 ⊆
R d such that:
∀ i ≤ d − 2 , there exists a ( d − i − 1) -dimensional
affine space H
i containing S i and in which S i is a ( d − i − 2) -sphere of radius r i = (cid:113)
α 0 µ − α 20 ∥ g 0 ∥ 2 (cid:16) 1 − µ 2 L 2 (cid:17) i 2 and center c i ∈
H i .
Let ( x i )
i be the iterates generated by A starting from x 0 and reading gradients ( g i )
i and function values ( f i ) i , then for any i ≤ d − 2 and any x ∈ S i , there exists a function f in RSI − ( µ ) ∩ EB + ( L ) minimized by { x } that interpolates ( x j , f j , g j ) j ≤ i .
Proof.
In order to preserve concision and clarity, we will only present the broad outline of the proof here.
For a complete technical proof, see Appendix C.
We construct the sequence iteratively.
For initialisation, we take any non-zero g 0 , set f 0 = µ + L 4 µ α 0 ∥ g 0 ∥ 2 2 , c
= x 0 − α 0 g 0 , and finally
[Equation]
Then assuming we have a sequence ( f j , g j , S j ) j ≤ i< ( d − 2) respecting the conditions of the Lemma, noting H i the d − i − 1 dimensional affine space in which S
i is a ( d − i − 2) dimensional sphere, and x i +1 the ( i + 1) -th iterate returned by A .
Let h
i +1 the orthogonal projection of x i +1 into H i .
If h i +1 ̸ = c i , let v =
( h i +1 − c i )
∥ h
i +1 − c i ∥ 2 .
If h i +
=
c i , let s ∈ S
i and v = ( s − c i )
∥ s − c i ∥ 2 .
We then construct:
[Equation]
We verify in Appendix C that this construction respects the properties of Lemma 1.
■
We can now introduce Theorem 2 which gives us a lower bound on the worst-case convergence rate of any first-order algorithm on RSI − ( µ ) and EB + ( L ) .
Theorem 2 (Lower bound on RSI − ∩ EB + ) Let A be any first-order algorithm on R d , µ >and L ≥ µ .
For any x 0 ∈ R d , there exists x ∗ ∈ R d and a function f in RSI − ( µ ) ∩ EB + ( L ) minimized by X ∗ = { x ∗ } such that
[Equation]
Furthermore, if Lµ ≥ √ 2 , there exists h in RSI − ( µ ) ∩ EB + ( L ) minimized by X ∗ = { x ∗ } such that
[Equation]
where ( x i ) (resp.
( z i ) ) is the trajectory obtained by applying A to f (resp.
h ) starting in x 0 .
Note that since X ∗ is a singleton, ∀ i, x ∗ i =
z ∗ i = x ∗ .
Proof.
If L = µ , then the inequalities are trivial from the positivity of the norm.
If L > µ , let ( g i , f i , S i ) i ≤ ( d − 2) be the sequence introduced in Lemma 1 for α 0 ∈ (cid:104) µL 2 , max (cid:16) µL 2 , 12 µ (cid:17)(cid:105) .
Let us note that for any x ∈ S 0 , ∥ x 0 − x ∥ {+2 
+} = α
∥ g 0 ∥
22 µ (see initialisation in Appendix C).
Let i ∈ { 1 , . . .
, d − 1 } .
S i − 1 has radius r i − 1 , thus there exists x ∗ ∈ S
i − 1 such that ∥ x i − x ∗ ∥ 2 ≥ r i − 1 , and thus :
[Equation]
When setting α 0 = µL 2 and observing that x ∗ ∈ S i − 1 ⊆ S 0 and thus ∥ x 0 − x ∗ ∥ 22 = α
∥ g 0 ∥
22 µ in (13), we obtain (11).
If Lµ ≥ √ 2 , we set α 0 = 12 µ and (13) immediately yields (12).
■
Remark 5.1 Since the lower bounds established in Theorem 2 are exactly matched by the convergence guarantees of gradient descent (see Prop 1), these bounds are tight and gradient descent is exactly optimal on RSI − ( µ ) ∩ EB + ( L ) .
This is a concrete example of the sensitivity of theoretical optimality to the choice of complexity classes.
5.1 Discussion
The first bound presented in Theorem 2 gives the optimal solution when trying to solve
[Equation]
While the second bound gives the optimal solution when trying to solve
[Equation]
For general smooth and convex functions, (15) will not have a solution (for any A , the quantity will not have a worst case upper bound), which is why (14) has historically been the focus of optimization literature.
However in practice, when (15) admits a solution, as is the case for RSI − ( µ ) ∩ EB + ( L ) , it fits practical motivations better than (14) : when starting from x 0 and observing an initial gradient g 0 , the solution to (15) is the one that will minimize ∥ x n − x ∗ n ∥ 2 in the worst case.
In comparison, for a fixed ∥∇ f ( x 0 ) ∥ 2 , the solution to (14) will be faster when[-∥ x 0 − x ∗ 
-]
∥ {+x 0 − x ∗ 
 ∥+} 2 is small, and slower when ∥ x 0 − x ∗
∥ 2 is large, leading to a slower worst-case convergence.
When Lµ > √ 2 , using a tuning of α 0 = 12 µ on the first step instead of µL 2 leads to a worst-case convergence of ∥ x n − x ∗ n ∥ 2 better by a constant c =
L 2 2( L 2 − µ 2 ) .
While such small constant factor is often considered not impactful, the number of steps required to make up for this constant factor
[Figure]
is n =
[Figure]
[Equation]
[Figure]
− 1 , which yields n ≈ 68 for Lµ = 10 and n ≈ 6930 for Lµ = 100 , and can thus become substantial on ill-conditioned functions.
Finally, we propose a geometric interpretation of the threshold Lµ = √ 2 .
Given x 0 and g 0 , RSI − ( µ ) requires x ∗ to be within the circle of center x 0 − g 0 2 µ and radius ∥ g 0 ∥
2 2 µ , while EB + ( L ) requires x ∗ to not be within the circle of center x 0 and radius ∥ g [-
-]0 ∥
2 L .
In Figure 1 we show these circles for different values of Lµ .
When Lµ > √ 2 (Figure 1(a)), x 1 = x 0 − µL 2 g 0 minimizes ∥ x 1 − x ∗ ∥ 2 ∥ x 0 − x ∗ ∥ 2 over all possible x ∗ , while x 1 = x 0 − g 0 2 µ minimizes ∥ x 1 − x ∗ ∥ 2 .
As Lµ becomes smaller than √ 2 (Figure 1(b) and
the 5.2 PEP experiment
Since we have found necessary and sufficient interpolation conditions in Theorem 1, we can use the
PEP framework on RSI − ∩ EB + to confirm our results and derive the worst-case convergence rate of first-order algorithms.
In Figure 2 we show the worst-case linear rate of convergence of Heavy Ball (HB) [29] on RSI − (0 . 1) ∩ EB + (1 . 0) for regularly sampled learning rate α and momentum β (bright yellow means no linear convergence), generated with PEPit [9]. We remind the update rule of HB
[Equation]
Since gradient descent is a special case of HB where β = 0 , we observe as expected that the optimal rate of convergence is achieved for β = 0 and α = 0 .
1 = µL 2 .
Moreover, Figure 2 shows that momentum does not do well on RSI − ∩ EB + but gradient decent benefits from a relative robustness to the tuning of α : we get similar convergence rates for any α ∈ [0 .
Conclusion
Our main result is to prove that for any µ > 0 and L ≥ µ , gradient descent is exactly optimal on the class of functions RSI − ( µ ) ∩ EB + ( L ) (by exact optimality, we mean that the convergence guarantees of GD match the lower bound of worst-case performances exactly, without a constant factor of difference).
This result confirms the observation in [12] that optimality is overly sensitive to the choice of assumptions, and should thus be considered with a lot of caution.
Interestingly, our analysis also identifies two similar notions of optimality, one of which suggests using a larger step size on the first iteration when the function is not particularly well-conditioned to improve worst-case convergence speed (see section 5.1).
We verified empirically that RSI − ∩ EB + , with respect to the last iterate, are verified on the optimization paths of simple deep neural networks (c.f. Appendix A).
This suggests that unlike usual alternatives which are known to not be verified on the highly non-convex loss landscapes of neural networks, convergence guarantees on RSI − ∩ EB + realistically apply to deep learning. On the other hand, the impossibility to accelerate the convergence rate on RSI − ∩ EB + implies that these assumptions are insufficient to explain the empirical successes of accelerated methods.
For the scope of this work we have focused on worst-case convergence analysis.
While in practice average-case convergence rates are more insightful than their worst-case counterparts, such results are rare due to the necessity of defining a reasonable distribution on the considered class of functions, which is generally unfeasible.
While such distribution on RSI − ∩ EB + is equally difficult to define, it should be feasible to define instead a reasonable distribution of the observed gradient for a given sampling point x and nearest minima x ∗ , e.g. an uniform distribution over the (simple) set of possible gradients.
Such approach is conceivable with RSI − ∩ EB + only because the interpolation conditions are independent (see remark 4.1), and thus we can easily make sure that any set of gradients sampled from this distribution can be interpolated within the class.
While the distribution will necessarily be arbitrary, we believe such analysis could yield very useful insights and RSI − ∩ EB + is a rare opportunity to follow this approach.
Finally, many alternative conditions have been introduced in the literature (see Section 3), for which optimality results are still unknown.
The PEP framework is a powerful tool to study conditions for which we can determine sufficient and necessary interpolation conditions, which would improve our understanding of first-order algorithm properties and tuning on a wide variety of objective function classes. Acknowledgments and Disclosure of Funding The authors would like to thank Leonard Boussioux, for useful discussions and feedback. Ioannis
Mitliagkas acknowledges support by an NSERC Discovery grant (RGPIN-2019-06512), a Samsung grant and a Canada CIFAR AI chair.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] limitation discussions are less relevant to purely theoretical works such as ours.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work does not introduce any new method and focuses on theoretical results.
We do not believe a discussion on negative societal impacts is relevant to this situation.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] See detailed notations in Section 1, and full assumptions in Theorem 1 and Theorem 2. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B and Appendix C. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] The PEP experiment in Section 5 is extremely simple to reproduce with the PEPit library. Similarly, our experience in Appendix A is trivially reproduceable. We believe these experiments do not require a code release.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specify the details of the setting in Appendix A (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A] We do not report any tables.
[Yes] See Appendix A 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets? [No] We only use CIFAR10. (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] 5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] A Optimization paths of neural networks are in RSI − ∩ EB + In this section we present a short experiment motivating RSI − ∩ EB + as useful assumptions for the study of optimization of neural networks.
The goal is to estimate whether the gradients seen by training neural networks are interpolable by a function f ∈ RSI − ∩ EB + .
We propose the following process :
Set random seed S 2.
Train a ResNet18 on CIFAR10 until convergence, and save the last iterate x ∗ 3.
Reset random seed to S 4.
Train a ResNet18 on CIFAR10, and at each iteration i , sample batch B and measure
[Equation]
Where ∇ f B ( x ) is the gradient of the loss function on minibatch B , x i is the value of the weight at iteration i , and x ∗ is the value of the last iterate measured at step 2.
Due to resetting the seed to a same value, the two training runs will be identical.
We consider the last iterate x ∗ to approximate a local minima, and RSI i and EB i will indicate whether the gradients seen during optimization are compatible with RSI − ∩ EB + with respect to that minima.
Even if the full-batch objective function that we intend to optimize is in RSI − ∩ EB + , it is possible for RSI i to be negative due to the variance w.r.t the sampling of the minibatch B .
However, we observe empirically that despite this, RSI i is lower bounded by a strictly positive value for every single iteration, without exception.
This behavior is consistent across optimization algorithms (LARS,
SGD without momentum, SGD with momentum, and ADAM) and initializations.
For simplicity, we present here the results using SGD without momentum, with learning rate 0 .
1 and batch size | B | = 1000 , and train for 360 epochs.
Results: we report the log of training loss in Figure 3 and the measured RSI i and EB i in Figure 4.
Moreover, in order to better observe the behavior of RSI i and EB i outside of the initial peak, we report in Figure 5 the measured RSI i and EB i starting at epoch 30 .
We observe that EB i is upper bounded by L = 2 . 303 and RSI i is lower bounded by µ = 0 . 0010 , with a resulting condition number κ = Lµ
= 2196 . 0 .
Both have a significant peak at the beginning of training, justifying the popular use of learning rate warm-ups.
When measuring bounds after epoch 30 , we obtain L = 0 .
2308 resulting in a condition number κ = 220 . 1 .
Surprisingly, despite the variance induced by minibatch sampling, the observed RSI i are all lower bounded by µ = 0 . 0010 > 0 .
In particular, due to the necessary and sufficient conditions of
RSI − ∩ EB + (See Section 4), it is guaranteed that there exists a function f ∈ RSI − ∩ EB + which exactly interpolates the gradients seen by the optimizer.
And therefore, the convergence guarantees of RSI − ∩ EB + naturally apply to the optimization of neural networks in this setting.
Note that we do not claim that the objective function is in RSI − ∩ EB + , which seems unlikely, but that the iterates explored by first-order algorithms are interpolable by functions in RSI − ∩ EB + , including when sampling only part of the objective function through minibatches.
This result strongly motivates the study of RSI − ∩ EB + as its guarantees apply to the optimization of neural network under assumptions empirically verified (at least in this simple setting).
B Proof of Theorem
We start with two simple lemmas
Lemma 2 Let X ∗ be a closed convex set, and x ∗ ∈
X ∗ be the orthogonal projection of x onto
X ∗ .
Then for any y ∈ X ∗ ,
[Equation]
Proof.
Let y ∈
X ∗ .
For θ ∈ [0 , 1] , let h ( θ ) =
∥ x − ((1 − θ )
[Equation]
+
[Equation]
h is differentiable and
[Equation]
Since x ∗ is the orthogonal projection of x onto X ∗ and ∀ θ ∈ [0 , 1] , (1 − θ ) x ∗ + θy ∈ X ∗ , we have ∀ θ ∈ [0 , 1] , h ( θ ) ≥ h (0) , and thus h ′ (0) ≥ 0 .
This concludes the proof of the Lemma thanks to (17).
■
{+Lemma 
+}
If x and x i are two points with respective orthogonal projections x ∗ and x ∗ i on a closed convex set, then
[Equation]
Proof.
As the case x ∗ = x ∗ i is trivial, we may assume that x ∗ ̸
= x ∗ i .
Using lemma 2 twice, we get
[Equation]
[Equation]
Adding the two inequalities, we get that
[Equation]
[Equation]
Since x ∗ ̸ = x ∗ i , we obtain
[Equation]
And thus
[Equation]
We now move on to the proof of Theorem 1, that is :
Let ( x i , g i ) i ≤ n ∈ (cid:0)
R d × R d (cid:1) n +1 , such that the x i are separate points.
Then, ∀ µ, L > 0 :
[Equation]
Where x ∗ i is the orthogonal projection of x i onto X ∗ .
Proof.
The direct implication is trivial since the second property is simply the application of RSI − and EB + in each x i .
Let us now assume that (7) is verified.
First, let us note that if L = µ , then we have ∀ i, g i = µ ( x i − x ∗ i ) and thus we can easily interpolate the ( x i , g i ) using f ( x ) = µ 2 ∥ x − x ∗ ∥ 22 .
We now assume L > µ .
If there is only one pair ( x i , g i ) , then we can simply use f ( x ) =
⟨ g i | x − x i ⟩ + µ + L 4 ∥ x − x i ∥ 22 to interpolate ( x i , g i ) .
f is then µ -strongly convex and L -smooth
so it is also in RSI − ( µ ) and EB + ( L ) . Let us now assume there are at least two pairs ( x i , g i ) i . Let [Equation]
By construction,
[Equation]
Moreover, if ∀ i, x i ∈ X ∗ , we can simply take f ( x ) = µ 2 ∥ x − x ∗ ∥ 22 .
Otherwise, let
I = { i | x i ̸
= x ∗ i } , and let ϵ 1 = 12 min i ∈I ( ∥ x i − x ∗ i ∥ 2 ) >
Let ϵ < min( ϵ 0 , ϵ 1 ) and 0 < β < 12 .
We introduce the function λ ϵ,β from [0 , ϵ ] to [0 , 1] defined by :
[Equation]
We finally introduce our interpolation function :
[Equation]
First let us note that f ϵ,β is properly defined : as stated in (26), there may be at most one i such that ∥ x − x i ∥ 2 < ϵ .
Moreover, f ϵ,β is continuous because λ ϵ,β ( ϵ ) = 0 .
Since λ ϵ,β ( ϵ ) = 0 and λ ′ ϵ,β ( ϵ ) = 0 , we can easily verify that for x such that ∥ x − x i ∥ 2 = ϵ , f ϵ,β is differentiable in x with ∇ f ϵ,β ( x ) = µ + L 2 ( x − x ∗ ) .
Thus f ϵ,β is differentiable on R d .
For any x ∈ R d such that ∀ i, ∥ x − x i ∥ 2 ≥ ϵ , we have ∇ f ϵ,β ( x ) = µ + L 2 ( x − x ∗ ) and thus trivially
[Equation]
Let us now assume there is i such that ∥ x − x i ∥ 2 < ϵ .
If x i = x ∗ i , then g i = 0 and ∇ f ( x ) = µ + L 4 ( x − x ∗ ) and equations (29) and (30) are respected as well.
Otherwise, we have ∥ x − x ∗ ∥ 2 ≥ min i ∈I ( ∥ x i − x ∗ i ∥ 2 ) − ϵ
=
ϵ 1 − ϵ >
We then have, for x ̸
= x i :
[Equation]
Since X ∗ is a convex set (and closed by continuity of f ϵ,β ), we have from Lemma∥ ( x − x ∗ − ( x i − x ∗ i ))
∥ 2 ≤ 2 ∥ x − x i ∥ 2 .
To simplify notations, let us note u = ∥
x − x i ∥ 2 , λ = λ ϵ,β ( u ) , and
[Equation]
We first want to upper bound ∥∇ f ϵ,β ∥ 2 using (31):
[Equation]
Moreover, the 3 rd order remainder of the Taylor expansion of cos ( π u β ϵ β ) is cos ( c ) 4!
( π 4 u 4 β ϵ 4 β ) for some c in [0 , π uϵ ] and by upper bounding it we get cos ( π u β ϵ β ) ≤ 1 − π 2 u 2 β 2 ϵ 2 β
+ π 4 u 4 β 24 ϵ 4 β and thus, for uϵ ≤ 1 ,
[Equation]
[Equation]
with C 0 = (cid:16) π 2 4 − π 4 48 (cid:17) L − µ 2 ϵ 1 >
Furthermore, since 2 β < 1 and uϵ ≤ 1 , we can also bound (
[Equation]
34) with C 1 = µ + 3 L >
Finally, we can bound the last term using sin ( x ) ≤ | x |
[Equation]
35) with C 2 =
π
2 2 3 L + µmax i ( ∥ x i − x ∗ i ∥ 2 )
Finally, by choosing ϵ ≤ C 0 2 C 1 and β ≤ C 0 2 C 2 , and plugging (33), (34), (35) into (32), we get :
[Equation]
It only remains now to adequately lower bound ⟨∇ f ϵ,β ( x ) | x − x ∗ ⟩ .
We use the same method as before and keep the notations :
⟨∇
[Equation]
With M 0 = 4( µ + L ) max i ( ∥ x i − x ∗ i ∥ 2 ) + ( L + 3 µ ) ϵ 1 >and M 1 = C 2 (max i ( ∥ x i − x ∗ i ∥ 2 ) + ϵ 1 ) >
Therefore by taking ϵ ≤ C 0 2 M 0 and β ≤ C 0 2 M 1 , we guarantee
[Equation]
Finally, for any i , we have :
[Equation]
Which goes to 0 as x tends to x i .
Therefore using the definition of f ϵ,β in (28) and the fact that (cid:68) g i − µ + L 2 ( x i − x ∗ i ) | x − x i (cid:69) is linear in x − x i , we can conclude ∇ f ϵ,β ( x i ) = µ + L 2 ( x i − x ∗ i ) + λ ϵ,β (0)( g i − µ + L 2 ( x i − x ∗ i ))
=
g i .
We thus have proven that for sufficiently small ϵ and β , ∀ i, ∇ f ϵ,β ( x i ) =
g i , that for all x , ⟨∇ f ϵ,β ( x ) | x − x ∗ ⟩ ≥ µ ∥ x − x ∗ ∥ 22 and ∥∇ f ϵ,β ( x ) ∥ 2 ≤ L ∥ x − x ∗ ∥ 2 .
Therefore by definition, f ϵ,β is in RSI − ( µ ) ∩ EB + ( L ) and interpolates the x i , g i .
Which concludes the proof.
■
C Proof of Lemma
Let µ > 0 and L > µ .
Let α 0 ∈ (cid:104) µL 2 , max (cid:16) µL 2 , 12 µ (cid:17)(cid:105) .
For any first-order optimization algorithm
A and starting point x 0 ∈ R d , there exists ( g i ) i ≤ ( d − 2) ∈ R d , ( f i ) i ≤ ( d − 2) ∈ R and S d − 2 ⊆
S
d − 1 ⊆ · · · ⊆ S 0 ⊆
R d such that:
∀ i ≤ d − 2 , there exists a ( d − i − 1) -dimensional
affine space H
i containing S i and in which S i is a ( d − i − 2) -sphere of radius r i = (cid:113)
α 0 µ − α 20 ∥
g
∥ 2 (cid:16) 1 − µ 2 L 2 (cid:17) i 2 and center c i ∈ H i .
Let ( x i ) i the iterates generated by A starting from x 0 and reading gradients ( g i )
i and function values ( f i ) i , then for any i ≤ d − 2 and any x ∈ S i , there exists a function f in
RSI − ( µ ) ∩ EB + ( L ) minimized by { x } that interpolates ( x j , f j , g j ) j ≤ i .
Proof.
For any first-order optimization algorithm A and starting point x 0 , we are going to construct by induction the sequences ( g i ) i , ( f i ) i and ( S i ) i .
Initialisation: Let g 0 ∈ R d \ { 0 } .
We can take any non-zero gradient as initialisation.
{+Let c 
+}
= x 0 − α 0 g 0 , f 0 = µ + L 4 µ α 0 ∥ g 0 ∥ 22 , and
[Equation] = [Equation]
H 0 is an hyperplane with dimension d − 1 , and we finally introduce
[Equation]
{+By construction, c 0 ∈ H and S 0 is the ( d − 2) -sphere in H 0 of center c 0 and radius r 
+}
= (cid:113)
α 0 µ − α 20 ∥ g 0 ∥ 2 .
Moreover, let x ∗ ∈ S 0 .
We have
[Equation]
And thus f 0 = µ + L 4 ∥ x ∗ − x 0 ∥ 22 .
We also have :
[Equation]
Finally, since α 0 ≥ µL 2 ,
[Equation] Therefore all the sufficient conditions of Corollary 1 are verified, and there exists f ∈ RSI − ( µ ) ∩ EB + ( L ) which is minimized by { x ∗ } and interpolates ( x 0 , f 0 , g 0 ) . This concludes the initialization.
Induction: Let us assume the existence of such ( f j ) j , ( g j ) j and ( S j ) j up to step i ≤ d − 3 .
Let x i + be the iterate given by A after reading iterates ( x j ) j , function values ( f j ) j , and gradients ( g j ) j , let
H i the ( d − i − 1) -dimensional affine space in which S
i is a sphere, and let c i ∈ H
i the center of the sphere S i .
If there exists j ≤ i such that x i +1 =
{+x j , then we simply return g i +
+}
=
g j and f i +1 =
f j .
We can take as S
i +1 any ( d − i − 2)
-dimensional sphere of radius r
i +1 included in S
i , H
i +1 its supporting affine space and
c i +1 its center.
We now assume ∀ j ≤ i, x i +1 ̸ = x j .
Let h i +1 the orthogonal projection of x i +1 into H i .
If h i +1 ̸
= c i , let v =
( h i +1 − c i )
{+∥ h i +1 − c i ∥ 2 . If h i +
 =+}
c i , let s ∈ S
i and v = ( s − c i )
∥ s − c i ∥ 2 .
Let
[Equation]
v is the difference between two points of H i , therefore it is one of the direction of H i , and since c i ∈ H
i , H i +1 indeed defines an affine subspace of H i of dimension ( d − i − 2) .
Let C the sphere in
H i +1 of center c
i +1 ∈
H
i +1 and radius r
i +1 =
(cid:113) 1 − µ 2 L 2 r i .
We now want to prove that C = S
i +1 .
First, let x ∈ H
i +1 .
Then
[Equation]
38) i)
First, we show that C ⊆ S
i +1 .
Let x ∈ C ∥
[Equation]
using (38) since x ∈ C ⊆ H i +
[Equation]
39) since x ∈ H i +1 ⊆
H i and ∥ x − c i ∥ 2 = r
i , x ∈ S i and therefore x ∈ S i + ii) Conversely, we show that S i +1 ⊆ C .
Let x ∈ S
i +1 .
[Equation]
as x ∈ S
i
[Equation]
⊆ S
[Equation]
using (38) since x ∈ S i +1 ⊆
H i + (40) from which we obtain that
[Equation]
{+So x ∈ H i +1 and ∥ x − c i +1 ∥ 
+}
= r
i +1 , thus x ∈ C .
We have thus proved that S
i +1 is indeed a ( d − i − 3) -sphere in a ( d − i − 2) affine space with the desired radius and center which concludes the first item of the induction.
We now want to prove the second item .
For x ∗ ∈ S
{+i +1 , x i +
 − h i +1+} is orthogonal to H i +1 due to being orthogonal to H i by construction.
{+h i +1 − c i +1 is aligned with v and thus is orthogonal to H i +1 . Therefore, their sum x i +
+}
{+− c i +1 is orthogonal to H i +
 and we get+}
[Equation]
And thus [Equation] Let x ∗ ∈ S i +1 .
Since S i +1 ⊆
S i , then by recurrence hypothesis there exists an interpolation of the ( x j , f j , g j ) in RSI − ( µ ) ∩ EB + ( L ) minimized by x ∗ , hence from Theorem 1, [Equation] i, ∥ g j ∥ 2 ≤ L ∥ x j − x
[Equation]
∥and
[Equation]
−
[Equation]
[Equation]
Moreover, by construction of g i +1 ,
[Equation]
[Equation]
Since x i +1 − c i +1 is orthogonal to H i +1 and thus to c i +1 − x ∗ , we have [Equation] Besides, x i +1 − c
i +1 is orthogonal to c i +1 − x ∗ and thus
[Equation]
By construction,
[Equation]
48) and finally :
[Equation]
using
[Equation]
[Equation]
[Equation]
using ( [Equation]
[Equation]
And thus
[Equation]
[Equation]
Since ∀ j ≤ i, x ∗ ∈ S j , we also have
[Equation]
We finally apply Corollary 1 to all unique triples ( x j , f j , g j ) j ≤ i +1 (which ensures by construction that all x j are distincts) which allows us to conclude from (43), (45), (44), (50) and (51) that there exists an interpolation in RSI − ( µ ) ∩ EB + ( L ) that is minimized by { x ∗ } , proving the second item of the induction and thus concluding the proof.
■