Gradient Descent Is Optimal Under Lower Restricted Secant Inequality And Upper Error Bound
Abstract
The study of first-order optimization is sensitive to the assumptions made on the objective functions.
These assumptions induce complexity classes which play a key role in worst-case analysis, including the fundamental concept of algorithm optimality.
Recent work argues that strong convexity and smoothness—popular assumptions in literature—lead to a pathological definition of the condition number [12].
Motivated by this result, we focus on the class of functions satisfying a lower restricted secant inequality and an upper error bound.
On top of being robust to the aforementioned pathological behavior and including some non-convex functions, this pair of conditions displays interesting geometrical properties.
In particular, the necessary and sufficient conditions to interpolate a set of points and their gradients within the class can be separated into simple conditions on each sampled gradient.
This allows the performance estimation problem (PEP, [7, 33]) to be solved analytically, leading to a lower bound on the convergence rate that proves gradient descent to be exactly optimal on this class of functions among all first-order algorithms.
Introduction
The typical framework to study convergence properties of first-order algorithms in the context of machine learning is to first establish a class of objective functions to optimize through assumptions usually bound to a constant, such as L -smoothness and µ -strong convexity.
A tuning prescription of an algorithm is then made based on the constants, (e g. step size α = 2 µ + L in the case of the gradient descent method on smooth and strongly convex functions), and finally a worst-case convergence rate can be derived for this algorithm when using this tuning prescription.
In some cases, a lower bound on the achievable worst-case convergence rate can also be derived, leading to the theoretical optimality of an algorithm on the considered class of function, for instance the Nesterov accelerated gradient method [27] is known to be optimal up to a constant on strongly convex and smooth functions.
However in [12], the authors establish that such framework and its derived results are very sensitive to the choice of assumptions, and that strong convexity and smoothness can exhibit pathological behaviors leading to conservative tuning and arbitrarily sub-optimal convergence rates, even when the resulting algorithm achieves theoretical worst-case optimality on this class of functions.
Furthermore, they propose a set of more robust alternative conditions.
In this work, we focus on a specific pair of such alternative conditions : lower restricted secant inequality ( RSI − ) and upper error bounds ( EB + ).
Our main contribution is to show that the gradient descent (GD) method with a certain tuning is exactly optimal on the classes of objective functions induced by these conditions, confirming that optimality results are highly sensitive to the choice of conditions.
Another consequence is that no algorithm can accelerate on this class of functions, implying that additional assumptions are required to explain the practical efficiency of accelerated methods.
Notation Let F the set of differentiable functions from R d to R that admit a convex set of global minima X ∗ f .
We focus on the problem of optimizing a function f ∈ F , i.e. finding x ∈ X ∗ f .
For any x ∈ R d , we denote x ∗ f
the orthogonal projection of x on X ∗ f .
By abuse of notation, when the context is not ambiguous, we will simply denote x ∗ f as x ∗ .
We call gradient descent (GD) the standard optimization algorithm based on the following update, where α is the step size :
[Equation]
We call first-order algorithm all A that consider past iterates, function values and gradients and output a next iterate.
Formally, A can be seen as a sequence of functions {A n | n ∈ N } such that for any n ∈ N , A n is a function defined on (cid:0) R d × R × R d (cid:1) n +1 with values in R d .
Under that formalism, applying A to optimize an objective function f starting in x 0 ∈ R d generates a sequence of iterates ( x i ) i such that ∀ i, x i +
=
A i (cid:16) ( x j , f ( x j ) , ∇ f ( x j ))
j ≤ i (cid:17) .
Note that we do not require the iterates of
A to lie within the span of observed gradients as it is often the case in the literature.
Outline Section 2 introduces RSI − and EB + and provides some basic properties and motivation.
Section 3 discusses related works in the literature.
Section 4 defines and establishes the necessary and sufficient interpolation conditions for RSI − and EB + , which is a key element of our analysis.
Section 5 proves the lower bound on the convergence rate of first-order algorithms under RSI − and
EB + , and finally Section 6 concludes our work.
Detailed proofs are provided in the appendix.
Lower restricted secant inequality and upper error bounds
We now define RSI − and EB + and discuss some basic properties.
Definition 2.1 (Lower restricted secant inequality) Let f ∈ F and µ >
[Equation]
Intuitively, RSI − ( µ ) enforces that the further x is from X ∗ f , the stronger the gradient of f in x must be in the opposite direction of X ∗ f .
Remark 2.2 RSI − ( µ ) includes non-convex functions.
However, it prevents flat landscapes outside of X ∗ f , and requires f to increase at least quadratically with the distance to X ∗ f , as established in [12] :
[Equation]
Definition 2.3 (Upper Error Bounds) Let f ∈ F and L >
[Equation]
EB + ( L ) thus enforces that the gradient of f is controlled by the distance to X ∗ f .
Remark 2.4 L -smoothness implies EB + ( L ) and µ -strong convexity implies RSI − ( µ ) .
However, one must be careful before claiming that RSI − and EB + are respectively weaker than strong convexity and smoothness : for µ 0 > µ 1 , RSI − ( µ 0 ) is neither stronger or weaker than µ 1 -strong convexity, and for L 0 < L 1 , EB + ( L 0 ) is neither stronger or weaker than L 1 -smoothness.
Therefore, even when the objective function is smooth and strongly convex, considering convergence results under RSI − and EB + is relevant, as we might obtain better constants µ and L .
Many convergence results depend of the condition number κ = Lµ .
Better constants leads to a better condition number, and thus a potentially better convergence rate (including when the dependence in the condition number κ is the same or worse).
As a consequence, machine learning problems with strongly convex and smooth objective functions are all potential applications of results under RSI − and EB + , provided we can obtain better constants under these conditions.
Prop 1 (convergence rate of GD under RSI − and EB + ) Let f ∈ RSI − ( µ ) ∩ EB + ( L ) .
Then gradient descent with learning rate α = µL 2 on f will guarantee the following convergence rate:
[Equation]
Moreover, when the learning rate is set to α 0 = 12 µ on the first step, and α = µL 2 on every other step, gradient descent guarantees the following convergence rate:
[Equation]
Proof.
When α = µ L 2 , we have
[Equation]
which proves (1).
To prove (2), we simply note that when using α = 12 µ , we have:
[Equation]
Interestingly, RSI − and EB + are direct bounds on the two additional terms obtained by developing ∥ x i − α ∇ f ( x i ) − x ∗ i ∥ 22 , leading to an extremely simple proof.
On the intuitive level, RSI − lower bounds the gain from stepping in the direction of X ∗ f , while EB + upper bounds the error coming from the component of the gradient orthogonal to that direction.
Remark 2.5 The literature gives a worst case convergence rate of gradient descent on µ -strongly convex and L -smooth functions of ∥ x i − x ∗ i ∥ 22 ≤ (cid:16) 1 − 2 µ µ + L (cid:17) 2 i ∥ x 0 − x ∗ 0 ∥ 22 , using the step size α = 2 µ + L [28, 30].
While this rate is better for a fixed µ and L , we again emphasize that the constants µ and L may be very different depending on the chosen conditions, and thus these two rates can not directly be compared.
Motivation: we empirically verify in Appendix A that the optimization path of a ResNet18 [16] trained for classification on CIFAR10 [20] verifies the interpolation conditions of RSI − ∩ EB + , which are introduced in Section 4.
This result guarantees the existence of a function in RSI − ∩ EB + which exactly interpolates the observed gradients, and thus all convergence guarantees of RSI − ∩
EB + naturally apply in this practical deep learning setting.
RSI − ∩ EB + therefore provides linear convergence guarantees while being empirically applicable to neural networks (although with respects to a local minima), an impressive feat given the highly non-convex nature of neural networks loss functions.
Related Work
Throughout the literature, many choices of assumptions have been used to study first-order optimization.
Most assumptions fall into one of two categories : lower conditions and upper conditions that respectively take the form of a lower and an upper bound on properties of the objective function.
For instance, strong convexity lower bounds the curvature of the objective function and is thus a lower condition .
Similarly, smoothness is an upper condition .
Lower conditions have been the most extensively studied assumptions, such as Polyak-Łojasiewicz [29], local-quasi-convexity [15], weak quasi-convexity [14], quadratic growth [2, 4, 18], KurdykaŁojasiewicz [21, 3], optimal strong convexity [22, 25, 8], weak strong convexity [19, 26], error bounds [24].
Some recent works have explored the relations between these lower conditions [19, 38].
In this work we focus on the restricted secant inequality ( RSI − ) (which we denote as lower restricted secant inequality to differentiate it from its upper bound equivalent) which was introduced in [39], and has been used (along with its convex extension restricted strong convexity ) in many recent theoretical derivations of linear convergence rates [36, 31, 37].
On the contrary, because most machine learning objective functions are naturally smooth, fewer works have explored alternatives to smoothness.
However as discussed in Remark 2.4, it is still relevant to study these alternatives on smooth objectives due to potentially better conditioning.
The most notable ones in the literature are local smoothness [15], restricted smoothness [1], relative smoothness [23, 13, 40], weak-smoothness [14], expected smoothness [11].
In [12], the authors argue that lower conditions can be naturally translated into equivalent upper conditions by changing the lower bound into an upper bound, and vice-versa.
Subsequently, they introduce a set of upper equivalent to existing lower conditions , such as upper error bounds EB + , the natural upper equivalent to error bounds from [24].
Throughout this work, we focus on EB + as an upper condition .
Finally, a key to our analysis are the necessary and sufficient interpolation conditions of RSI − ( µ ) ∩
EB + ( L ) (Section 4).
The search of these conditions has been largely motivated by Performance
Estimation Problems (PEP), introduced in [33] and with many recent successful applications [17, 32, 6, 35, 34].
PEP is a framework for computer-assisted worst-case convergence analysis, that only requires necessary and sufficient interpolation conditions for the considered class of objective functions.
In the case of RSI − ∩ EB + however, we found the interpolation conditions to be independent (see Section 4), which made the worst-case analysis directly solvable analytically.
Interpolation conditions
In this section we provide and discuss the necessary and sufficient interpolation conditions for
RSI − ∩ EB + .
Their importance stems from the framework used in PEP and in this work to analyze worst-case convergence.
Given a first-order optimization algorithm A
we want to find the slowest convergence rate of A among all f within a class of objective functions C and starting point x 0 ∈ R d \ X ∗ f .
That is equivalent to solving the following optimization problem at any step number n :
[Equation]
Directly searching for f in the functional space is generally intractable, however if we can explicitly find the set G of all families ( x i , f i , g i )
i such that ∃ f ∈ C , ∀ i, ∇ f ( x i ) =
g i and f ( x i ) =
f i (we say that f interpolates ( x i , f i , g i ) i ), then problem (5) can be reduced to:
[Equation]
In many cases (see Section 3), problem (6) is tractable and becomes a very powerful analysis tool providing lower bounds, upper bounds, and optimal tuning for different types of algorithms and assumptions used.
A crucial and difficult component of this analysis is to formulate the interpolation conditions , that is the necessary and sufficient conditions for a family ( x i , f i , g i )
i to belong in G .
Driven by these motivations, we now establish the interpolation conditions for RSI − ∩ EB + .
In Theorem 1, we introduce the necessary and sufficient conditions to interpolate a family ( x i , g i ) , without considering the function values ( f i ) i .
This theorem could be used to find the worst case convergence rate over all first-order algorithms that ignore function values.
However, in Corollary 1, we deduce from Theorem 1 sufficient (but not necessary) conditions to interpolate a family ( x i , f i , g i ) .
These conditions allow us in Section 5 to find a lower bound on the worst-case convergence rate for all first-order algorithms (including algorithms that have access to function values information), which we know is tight thanks to Prop 1.
Theorem 1 (Interpolation conditions) Let ( x i , g i )
i ≤ n ∈ (cid:0)
R d × R d (cid:1) n +1 , such that the x i are separate points.
Then, ∀ µ, L > 0 :
[Equation] [Equation]
and
[Equation]
where x ∗ i is the orthogonal projection of x i onto X ∗ .
Proof.
In order to preserve concision and clarity, we will only present the broad outline of the proof here.
For a complete technical proof, see Appendix B.
The direct implication is trivial as it is a direct application of RSI − ( µ ) and EB + ( L ) definitions to f in ( x i ) i .
For the reverse implication, we will construct a function f ϵ,β that interpolates ( x i , g i ) i .
The function f ϵ,β is a quadratic everywhere except in the spheres of radius ϵ around each x
i , ϵ being small enough for these spheres to never intersect.
Inside the sphere of radius ϵ around a given x i , f ϵ,β will be perturbed by adding a term λ ( ∥ x − x i ∥ 2 ) h
( x ) where h is affine in x , and λ is a scaling term so that λ ( ϵ ) = 0 at the border of the sphere, and λ (0)
= 1 in its center x i .
The key is to find a function λ that preserves the properties of RSI − ( µ ) and EB + ( L ) .
We use
[Equation]
And our construction f ϵ,β is given by:
[Equation]
The rest of the proof is to use the Taylor expansions of f ϵ,β to show that for sufficiently small ϵ and β , f ϵ,β will belong in RSI − ( µ ) and EB + ( L ) (see Appendix B).
■
Corollary 1 Let ( x i , f i , g i ) i ≤ n ∈ (cid:0) R d × R × R d (cid:1) n +1 , such that the x i are separate points.
Then, ∀ µ, L > 0 :
[Equation]
where x ∗ i is the orthogonal projection of x i onto X ∗ .
Proof.
We simply use the function f
ϵ,β from the proof of Theorem 1 and note that ∀ i, f ϵ,β ( x i ) = µ + L 4 ∥ x i − x ∗ i ∥ 22 .
■
Theorem 1 and Corollary 1 are the key elements to prove the optimality of gradient descent on RSI − and EB + among all first-order algorithms (see Section 5).
Remark 4.1 The interpolation conditions in Theorem 1 are independent, in the sense that a family ( x i , g i ) i admits an interpolation in RSI − ( µ ) ∩ EB + ( L )
if and only if each { ( x i , g i ) } admits an interpolation in RSI − ( µ ) ∩ EB + ( L ) .
Similarly, the sufficient interpolation conditions in Corollaryare also independent.
This property of independent interpolation conditions drastically simplifies convergence analysis and is the main reason we are able to analytically derive a lower bound in Section 5.
Indeed, given an interpolable family ( x i , f i , g i ) i ≤ n for a given set X ∗ , it is sufficient to show that ( x n +1 , f n +1 , g n +1 ) is interpolable with X ∗ to prove that the entire family ( x i , f i , g i ) i ≤ n +1 can be interpolated.
It is thus simple to find the set of interpolable ( f n +1 , g n +1 ) given X ∗ , x n +1 , and an interpolable ( x i , f i , g i ) i ≤ n .
Lower bound on the convergence rate
In this Section we derive a lower bound on the convergence rate of first-order algorithms on RSI − and EB + .
This lower bounds applies under the assumption that the number of steps taken is smaller than the number of dimension d .
This assumption is frequent in the literature (e.g. [5]) and not constraining for high-dimensional optimization.
The observed gradients of the worst-case functions for optimal algorithms are typically orthogonal to one another (see [6]) which is not possible when the number of steps becomes larger than the dimension d .
We conjecture that when not bounding the number of steps, it is possible to achieve an asymptotic rate in O (cid:0) 2 − nd (cid:1) which would be better than the usual rates obtained for very ill-conditioned functions, while having little to no practical uses due to the bad convergence properties on a lower number of steps.
We now introduce Lemma 1, which is the cornerstone of the proof of Theorem 2:
Lemma 1 Let µ > 0 and L > µ .
Let α 0 ∈ (cid:104) µL 2 , max (cid:16) µL 2 , 12 µ (cid:17)(cid:105) .
For any first-order optimization algorithm A and starting point x 0 ∈ R d , there exists ( g i ) i ≤ d − 2 ∈ R d , ( f i ) i ≤ d − 2 ∈ R and S d − 2 ⊆
S d − 1 ⊆ · · · ⊆ S 0 ⊆
R d such that:
∀ i ≤ d − 2 , there exists a ( d − i − 1) -dimensional
affine space H
i containing S i and in which S i is a ( d − i − 2) -sphere of radius r i = (cid:113)
α 0 µ − α 20 ∥ g 0 ∥ 2 (cid:16) 1 − µ 2 L 2 (cid:17) i 2 and center c i ∈
H i .
Let ( x i )
i be the iterates generated by A starting from x 0 and reading gradients ( g i )
i and function values ( f i ) i , then for any i ≤ d − 2 and any x ∈ S i , there exists a function f in RSI − ( µ ) ∩ EB + ( L ) minimized by { x } that interpolates ( x j , f j , g j ) j ≤ i .
Proof.
In order to preserve concision and clarity, we will only present the broad outline of the proof here.
For a complete technical proof, see Appendix C.
We construct the sequence iteratively.
For initialisation, we take any non-zero g 0 , set f 0 = µ + L 4 µ α 0 ∥ g 0 ∥ 2 2 , c
= x 0 − α 0 g 0 , and finally
[Equation]
Then assuming we have a sequence ( f j , g j , S j ) j ≤ i< ( d − 2) respecting the conditions of the Lemma, noting H i the d − i − 1 dimensional affine space in which S
i is a ( d − i − 2) dimensional sphere, and x i +1 the ( i + 1) -th iterate returned by A .
Let h
i +1 the orthogonal projection of x i +1 into H i .
If h i +1 ̸ = c i , let v =
( h i +1 − c i )
∥ h
i +1 − c i ∥ 2 .
If h i +
=
c i , let s ∈ S
i and v = ( s − c i )
∥ s − c i ∥ 2 .
We then construct:
[Equation]
We verify in Appendix C that this construction respects the properties of Lemma 1.
■
We can now introduce Theorem 2 which gives us a lower bound on the worst-case convergence rate of any first-order algorithm on RSI − ( µ ) and EB + ( L ) .
Theorem 2 (Lower bound on RSI − ∩ EB + ) Let A be any first-order algorithm on R d , µ >and L ≥ µ .
For any x 0 ∈ R d , there exists x ∗ ∈ R d and a function f in RSI − ( µ ) ∩ EB + ( L ) minimized by X ∗ =
{ x ∗ } such that
[Equation]
Furthermore, if Lµ ≥ √ 2 , there exists h in RSI − ( µ ) ∩ EB + ( L ) minimized by X ∗ = { x ∗ } such that
[Equation]
where ( x i ) (resp.
( z i ) ) is the trajectory obtained by applying A to f (resp.
h ) starting in x 0 .
Note that since X ∗ is a singleton, ∀ i, x ∗ i =
z ∗ i = x ∗ .
Proof.
If L = µ , then the inequalities are trivial from the positivity of the norm.
If L > µ , let ( g i , f i , S i ) i ≤ ( d − 2) be the sequence introduced in Lemma 1 for α 0 ∈ (cid:104) µL 2 , max (cid:16) µL 2 , 12 µ (cid:17)(cid:105) .
Let us note that for any x ∈ S 0 ,
∥ x 0 − x ∥ 2
= α
∥ g 0 ∥
22 µ (see initialisation in Appendix C).
Let i ∈ { 1 , . . .
, d − 1 } .
S i − 1 has radius r i − 1 , thus there exists x ∗ ∈ S
i − 1 such that ∥ x i − x ∗ ∥ 2 ≥ r i − 1 , and thus :
[Equation]
When setting α 0 = µL 2 and observing that x ∗ ∈ S i − 1 ⊆ S 0 and thus ∥ x 0 − x ∗ ∥ 22 = α
∥ g 0 ∥
22 µ in (13), we obtain (11).
If Lµ ≥ √ 2 , we set α 0 = 12 µ and (13) immediately yields (12).
■
Remark 5.1 Since the lower bounds established in Theorem 2 are exactly matched by the convergence guarantees of gradient descent (see Prop 1), these bounds are tight and gradient descent is exactly optimal on RSI − ( µ ) ∩ EB + ( L ) .
This is a concrete example of the sensitivity of theoretical optimality to the choice of complexity classes.
5.1 Discussion
The first bound presented in Theorem 2 gives the optimal solution when trying to solve
[Equation]
While the second bound gives the optimal solution when trying to solve
[Equation]
For general smooth and convex functions, (15) will not have a solution (for any A , the quantity will not have a worst case upper bound), which is why (14) has historically been the focus of optimization literature.
However in practice, when (15) admits a solution, as is the case for RSI − ( µ ) ∩ EB + ( L ) , it fits practical motivations better than (14) : when starting from x 0 and observing an initial gradient g 0 , the solution to (15) is the one that will minimize ∥ x n − x ∗ n ∥ 2 in the worst case.
In comparison, for a fixed ∥∇ f ( x 0 ) ∥ 2 , the solution to (14) will be faster when
∥ x 0 − x ∗
∥ 2 is small, and slower when ∥ x 0 − x ∗
∥ 2 is large, leading to a slower worst-case convergence.
When Lµ > √ 2 , using a tuning of α 0 = 12 µ on the first step instead of µL 2 leads to a worst-case convergence of ∥ x n − x ∗ n ∥ 2 better by a constant c =
L 2 2( L 2 − µ 2 ) .
While such small constant factor is often considered not impactful, the number of steps required to make up for this constant factor
[Figure]
is n =
[Figure]
[Equation]
[Figure]
− 1 , which yields n ≈ 68 for Lµ = 10 and n ≈ 6930 for Lµ = 100 , and can thus become substantial on ill-conditioned functions.
Finally, we propose a geometric interpretation of the threshold Lµ = √ 2 .
Given x 0 and g 0 , RSI − ( µ ) requires x ∗ to be within the circle of center x 0 − g 0 2 µ and radius ∥ g 0 ∥
2 2 µ , while EB + ( L ) requires x ∗ to not be within the circle of center x 0 and radius ∥ g 0 ∥
2 L .
In Figure 1 we show these circles for different values of Lµ .
When Lµ > √ 2 (Figure 1(a)), x 1 = x 0 − µL 2 g 0 minimizes ∥ x 1 − x ∗ ∥ 2 ∥ x 0 − x ∗ ∥ 2 over all possible x ∗ , while x 1 = x 0 − g 0 2 µ minimizes ∥ x 1 − x ∗ ∥ 2 .
As Lµ becomes smaller than √ 2 (Figure 1(b) and
the
5.2 PEP experiment
Since we have found necessary and sufficient interpolation conditions in Theorem 1, we can use the
PEP framework on RSI − ∩ EB + to confirm our results and derive the worst-case convergence rate of first-order algorithms.
In Figure 2 we show the worst-case linear rate of convergence of Heavy Ball (HB) [29] on RSI − (0 . 1) ∩ EB + (1 . 0) for regularly sampled learning rate α and momentum β
(bright yellow means no linear convergence), generated with PEPit [9].
We remind the update rule of HB
[Equation]
Since gradient descent is a special case of HB where β = 0 , we observe as expected that the optimal rate of convergence is achieved for β = 0 and α = 0 .
1 = µL 2 .
Moreover, Figure 2 shows that momentum does not do well on RSI − ∩ EB + but gradient decent benefits from a relative robustness to the tuning of α : we get similar convergence rates for any α ∈ [0 .
Conclusion
Our main result is to prove that for any µ > 0 and L ≥ µ , gradient descent is exactly optimal on the class of functions RSI − ( µ ) ∩ EB + ( L ) (by exact optimality, we mean that the convergence guarantees of GD match the lower bound of worst-case performances exactly, without a constant factor of difference).
This result confirms the observation in [12] that optimality is overly sensitive to the choice of assumptions, and should thus be considered with a lot of caution.
Interestingly, our analysis also identifies two similar notions of optimality, one of which suggests using a larger step size on the first iteration when the function is not particularly well-conditioned to improve worst-case convergence speed (see section 5.1).
We verified empirically that RSI − ∩ EB + , with respect to the last iterate, are verified on the optimization paths of simple deep neural networks (c.f. Appendix A).
This suggests that unlike usual alternatives which are known to not be verified on the highly non-convex loss landscapes of neural networks, convergence guarantees on RSI − ∩ EB + realistically apply to deep learning.
On the other hand, the impossibility to accelerate the convergence rate on RSI − ∩ EB + implies that these assumptions are insufficient to explain the empirical successes of accelerated methods.
For the scope of this work we have focused on worst-case convergence analysis.
While in practice average-case convergence rates are more insightful than their worst-case counterparts, such results are rare due to the necessity of defining a reasonable distribution on the considered class of functions, which is generally unfeasible.
While such distribution on RSI − ∩ EB + is equally difficult to define, it should be feasible to define instead a reasonable distribution of the observed gradient for a given sampling point x and nearest minima x ∗ , e.g. an uniform distribution over the (simple) set of possible gradients.
Such approach is conceivable with RSI − ∩ EB + only because the interpolation conditions are independent (see remark 4.1), and thus we can easily make sure that any set of gradients sampled from this distribution can be interpolated within the class.
While the distribution will necessarily be arbitrary, we believe such analysis could yield very useful insights and RSI −
∩ EB + is a rare opportunity to follow this approach.
Finally, many alternative conditions have been introduced in the literature (see Section 3), for which optimality results are still unknown.
The PEP framework is a powerful tool to study conditions for which we can determine sufficient and necessary interpolation conditions, which would improve our understanding of first-order algorithm properties and tuning on a wide variety of objective function classes.
Acknowledgments and Disclosure of Funding
The authors would like to thank Leonard Boussioux, for useful discussions and feedback.
Ioannis
Mitliagkas acknowledges support by an NSERC Discovery grant (RGPIN-2019-06512), a Samsung grant and a Canada CIFAR AI chair.
[-Lemma 
-]
[-Let c 
-]
[-By construction, c 0 ∈ H and S 0 is the ( d − 2) -sphere in H 0 of center c 0 and radius r 
-]
[-x j , then we simply return g i +
-]
[-If h i +
-]
[-So x ∈ H i +1 and ∥ x − c i +1 ∥ 
-]
[-i +1 , x i +
-]
[-Therefore, their sum x i +
-]
[-i +
-]