On the Complexity of Finding Small Subgradients in Nonsmooth Optimization
Abstract
We study the oracle complexity of producing ( δ, ϵ ) -stationary points of Lipschitz functions, in the sense proposed by Zhang et al.
While there exist dimension-free randomized algorithms for producing such points within (cid:101) O (1 /δϵ 3 ) first-order oracle calls
, we show that no dimension-free rate can be achieved by a deterministic algorithm.
On the other hand, we point out that this rate can be derandomized for smooth functions with merely a logarithmic dependence on the smoothness parameter.
Moreover, we establish several lower bounds for this task which hold for any randomized algorithm, with or without convexity.
Finally, we show how the convergence rate of finding ( δ, ϵ )
-stationary
points can be improved in case the function is convex, a setting which we motivate by proving that in general no finite time algorithm can produce points with small subgradients even for convex functions.
1. Introduction
We consider the problem of optimizing Lipschitz continuous functions f : R d → R using a firstorder algorithm, which utilizes values and derivatives of the function at various points.
Though problems of this type are ubiquitous throughout modern machine learning, they are well known to be impossible to solve at a dimension-free rate without further assumptions such as convexity or smoothness.
For example, it is generally impossible to obtain local minima or approximatestationary points of f [14, 20], nor is it even possible to get close to such points within any finite time independent of d [11].
Seeking to design reachable optimization goals for this large function class, Zhang et al.
[20] proposed the relaxed notion of a ( δ, ϵ ) -stationary point.
Simply put, these are points for which there exists a convex combination of gradients in a δ -neighborhood whose norm is less than ϵ (see Section 2 for a formal definition).
Their main contribution is a randomized algorithm, INGD , which produces ( δ, ϵ ) -stationary points of any Lipschitz, bounded from below function within (cid:101) O
(1 /δϵ 3 ) iterations.
Although INGD initially required access to a slightly nonstandard oracle, follow-up works modified the algorithm such that it will rely on access to a standard first-order oracle at differentiable points [9, 17].
All of these algorithms are randomized and share the same oracle complexity of (cid:101) O
(1 /δϵ 3 ) .
In this work, we aim towards a better understanding of the oracle complexity of producing ( δ, ϵ ) -stationary points in various settings.
First, we examine whether the algorithms mentioned above can be derandomized.
Namely, what rate can be achieved by a deterministic algorithm that produces ( δ, ϵ ) -stationary points?
We solve this question by showing a strong lower bound, proving that deterministic algorithms cannot achieve any dimension-free rate.
On the other hand, we point out that if the objective function is even slightly smooth, it is possible to provide a deterministic algorithm whose rate is (cid:101) O
(1 /δϵ 3 ) with only a logarithmic dependence on the smoothness parameter.
Next, we take the first step towards providing lower bounds which hold for any (possibly randomized) first-order method.
In terms of ϵ dependence, we provide a lower bound of Ω(1 /ϵ 2 ) which holds even for convex functions.
In terms of δ dependence, we provide an Ω(log(1 /δ ))
lower bound for nonconvex functions.
Though these results are clearly not tight, there are currently no randomized lower bounds whatsoever for this setting in terms of neither ϵ nor δ .
Noticeably, Zhang et al.
[20] do prove an Ω(1 /δ ) lower bound, though their proof covers only deterministic algorithms which have access to the derivative alone (without function value).
As we conjecture the lower bounds are loose, especially with respect to δ , providing tighter and unified lower bounds - in terms of δ and ϵ - remains an intriguing open question.
Finally, we also examine the complexity of finding ( δ, ϵ ) -stationary of convex functions.
Although the vast majority of convex optimization literature deals with minimizing the function value [16], some applications require the design of algorithms that produce points with small subgradients (see [1, 15] and discussions therein).
Moreover, while for smooth convex function the picture is relatively well understood in terms of complexity upper and lower bounds [6], for nonsmooth convex optimization much less is known in terms of minimizing subgradient norm.
We start by proving that even for convex yet nonsmooth functions, it is generally impossible to produce ϵ -stationary pointsnamely, with some subgradient of norm at most ϵ (corresponding to δ → 0 ).
In fact, we prove an explicit δ -dependent lower bound for convex functions, hinting that it might be beneficial to consider the relaxation of ( δ, ϵ ) -stationarity in this setting as well.
Furthermore, we show that the complexity of producing a ( δ, ϵ ) -stationary point of a convex function improves (when compared to the nonconvex case) to (cid:101) O
(1 /δ 2 / 3 ϵ 2 ) iterations using a randomized algorithm, or (again) even a deterministic algorithm if the function is slightly smooth.
These algorithms have optimal ϵ -dependence as they match
our aforementioned lower
bound.
Overall, we hope this work will motivate further understanding of complexity guarantees for nonsmooth optimization as a whole, and of producing ( δ, ϵ ) -stationary points in particular.
This paper is structured as follows.
In Section 2 we formally introduce the terminology which we use throughout the paper.
In Section 3 we present our results for deterministic algorithms, while in Section 4 we provide our randomized lower bounds.
In Section 5 we turn to analyze the case of convex functions.
We conclude in Section 6 with a discussion of our results and future directions.
The appendices contain a brief account of the notation we use throughout this work, along with the full proofs of our results.
2. Preliminaries
Nonsmooth analysis.
We call a function f : R d → R L -Lipschitz if for any x, y ∈ R d : | f ( x ) − f
( y )
| ≤ L ∥ x − y ∥ , and H -smooth if it is differentiable and ∇ f : R d → R d is H -Lipschitz, namely for any x,
y ∈ R
d : ∥∇ f ( x ) − ∇ f ( y ) ∥
≤ H ∥ x − y ∥ .
By Rademacher’s theorem, Lipschitz functions are differentiable almost everywhere (in the sense of Lebesgue).
Hence, for any Lipschitz function f : R d → R and point x ∈ R d the Clarke sub-differential set [7] can be defined as ∂f ( x ) :
= conv { g : g = lim n →∞ ∇ f ( x n ) , x n → x } , namely, the convex hull of all limit points of ∇ f ( x n ) over all sequences of differentiable points which converge to x .
Note that if the function is differentiable at a point or convex, the Clarke sub-differential reduces to the gradient or subgradient in the convex analytic sense, respectively.
We say that a point x is an ϵ -stationary point of f ( · ) if there exists g ∈ ∂f ( x ) :
∥ g ∥ ≤ ϵ .
Furthermore, given δ > 0 the Goldstein δ -subdifferential [10] of f at x is the set ∂ δ f ( x ) := conv( (cid:83)
∥
z − x ∥ <δ ∂f ( z )) , namely all convex combinations of gradients at points in a δ -neighborhood of x .
We say that a point x is a ( δ, ϵ ) -stationary of f ( · ) if there exists g ∈ ∂
δ f ( x ) such that ∥ g ∥ ≤ ϵ .
Note that a point is ϵ -stationary if and only if it is ( δ, ϵ ) -stationary for all δ > 0 [20, Lemma 7].
Algorithms and complexity.
Throughout this work we consider (possibly randomized) iterative first-order algorithms, from an oracle complexity perspective [14].
Such an algorithm first produces x 1 possibly at random and receives ( f ( x 1 ) , ∂f ( x 1 )) .
Then, for any t > 1 produces x t possibly at random based on previously observed responses, and receives ( f ( x t ) , ∂f ( x t )) .
We are interested in the minimal number T for which it can guarantee to produce some ( δ, ϵ ) stationary point.
Namely, given some function class F we look at sup f ∈F min T ∈ N Pr A [ ∃ t ∈
[ T ] : x t is ( δ, ϵ ) − stationary] ≥ 23 , where the probability is with respect to the algorithm’s internal randomness when acting upon f ∈ F (or deterministically, if the algorithm is deterministic).
3. Deterministic algorithms
As discussed in the introduction, [9, 17, 20] all provide randomized algorithms that given any LLipschitz function f : R d → R and an initial point x 1 that satisfies f ( x 1 ) − inf x f ( x ) ≤ ∆ , produce a ( δ, ϵ ) -stationary point of f within (cid:101) O (∆ L 2 /δϵ 3 ) oracle calls to f .
We start by showing that this rate, let alone any dimension-free rate whatsoever, cannot be achieved by any deterministic algorithm.
Theorem 1 For any deterministic first-order algorithm and any iteration budget T ∈ N , there exists a 1 -Lipschitz function f : R d → R , d = T + 2 such that f ( x 1 ) − inf x f ( x ) ≤ 1 yet the
T iterates produced by the algorithm when applied to f are not ( δ, ϵ ) -stationary points for any δ ≤ 17 , ϵ ≤ 1252 .
We defer the full proof to Section B in the appendix, though the intuition can be described as follows.
For any deterministic first-order algorithm, if an oracle can always return the “uninformative” answer f ( x i ) = 0 , ∇ f ( x i ) =
e 1 this fixes the algorithm iterates x 1 , . . .
, x T .
It remains to construct a Lipschitz function that will be consistent with the oracle answers, yet all the queried points are not ( δ, ϵ ) -stationary.
To that end, we construct a function which in a very small neighborhood of each queried point x t looks like x (cid:55)→ e ⊤ 1 ( x − x t ) , yet in most of the space looks like x (cid:55)→ max { v ⊤ x, − 1 } which has ( δ, ϵ ) -stationary points only when x is correlated with − v .
By letting v be some vector which is orthogonal to all the queried points, we obtain the result.
The construction we just described crucially relies on the function being highly nonsmoothessentially interpolating between two orthogonal linear functions in an arbitrarily small neighborhood.
As it turns out, if the function to be optimized is even slightly smooth, then the theorem above can be bypassed, as manifested in following theorem.
Theorem 2 Suppose f : R d → R is L -Lipschitz, H -smooth, and x 1 ∈ R d is such that f ( x 1 ) − inf x f ( x ) ≤ ∆ .
Then there is a deterministic first-order algorithm that produces a ( δ, ϵ ) -stationary point of f within O ( ∆ L 2 log( Hδ/ϵ ) δϵ) oracle calls.
The simple idea which proves Theorem 2 is to replace a certain randomized line search in INGD with a deterministic binary search subroutine, which terminates within O (log( Hδ/ϵ )) steps if the function is H -smooth.
Such a procedure is derived by Davis et al.
[9] for any H -weakly convex function along differentiable directions, and since any H -smooth function is H -weakly convex and differentiable along any direction, this can be applied as is proving Theorem 2.
Although the algorithmic ingredient to this observation appears inside a proof of Davis et al.
[9], they use it in a different way in order to produce a randomized algorithm for weakly convex functions in low dimension, with different guarantees suitable for that setting.
For completeness we explain in
Section B in the appendix how this leads to Theorem 2.
4. Randomized lower bounds
We now turn to establish complexity lower bounds which will hold for any (possibly randomized) algorithm based on a first-order oracle, as common in the optimization literature stemming from the seminal work of Nemirovski and Yudin [14].
Our first lower bound will be for the complexity of producing ( δ, ϵ ) -stationary points in terms of ϵ -dependence.
Theorem 3 For any ϵ < 1 , δ ≤ 112 ϵ and any randomized first-order algorithm A , there exists a 1 -Lipschitz convex function f : R d → R , d = (cid:101)
O (1 /ϵ 6 ) such that f ( x 1 ) − inf x f ( x ) ≤ 1 yet
Pr A [ ∃ t ∈ [ T ] : x t is ( δ, ϵ ) − stationary] < 13 unless T = Ω(1 /ϵ 2 ) .
An observation we would like to make is that Theorem 3 actually provides an optimal Ω(1 /δϵ 3 ) lower bound in a certain parameter regime.
Indeed, since it holds even when δ = Ω(1 /ϵ ) , for such a choice of δ we have Ω(1 /ϵ 2 )
=
Ω(1 /δϵ 3 ) , resulting in the optimal bound.
In standard settings in optimization which involve optimizing with respect to a single complexity parameter (e.g. ϵ -suboptimality or ϵ -stationarity) providing a lower bound in a certain parameter regime typically extends to any parameter regime through rescaling tricks of the form f ( x ) (cid:55)→ C 1 f ( C 2 · x ) .
Unfortunately, this does not seem to be the case for ( δ, ϵ ) -stationarity as there aren’t enough degrees of freedom to control the scales of both δ and ϵ , while maintaining the Lipschitz and initial sub-optimality constant.
The proof of Theorem 3 appears in Section B in the appendix, and is based on the well established machinery of high-dimensional optimization lower bounds (see for example Carmon et al.
[4], Nemirovski and Yudin [14], Nesterov [16], Woodworth and Srebro [18]).
To sketch the proof idea, we examine a random orthogonal transformation on top of the so called “Nemirovski function”, which is approximately of the form x (cid:55)→ max {| x i − 1 |} .
Though the Nemirovski function is commonly used to prove lower bounds in terms of sub-optimality (i.e. f ( x t ) − inf x f ( x ) ), we develop an analysis of its ( δ, ϵ ) -stationary points from which a complexity lower bound can be derived.
On the one hand, any oracle response reveals information about a single coordinate which roughly leads to a lower bound of d queries in order to get information about d coordinates.
On the other hand, the subgradients at points which are not near (1 , 1 , . . .
, 1) ∈ R
d are convex combinations of ± e i for some standard basis vectors e i , thus have norm of at least Ω(1 / √ d ) which is smaller than ϵ only whenever d = Ω(1 /ϵ 2 ) .
As to δ -dependent lower bounds, we provide the following theorem.
Theorem 4 For any randomized first-order algorithm A and any δ, ϵ ≤ 14 , there exists a 1-Lipschitz function f : R → R such that f ( x 1 ) − inf x f ( x ) ≤ 1 , yet
Pr A [ ∃ t ∈ [ T ] :
Underlying the proof of Theorem 4 is a useful observation on the structure of ( δ, ϵ ) -stationary points in dimension one: Although in general it is not true that a ( δ, ϵ ) -stationary point is necessarily δ -close to an ϵ -stationary point [11] (yet the opposite implication is trivially true), for univariate functions these two notions are actually equivalent.
Claim 1 If f : R → R is Lipschitz, then x ∈ R is ( δ, ϵ ) -stationary if and only if it is δ -close to an ϵ -stationary point.
We provide a proof of Claim 1 in Section B in the appendix.
This claim is useful for our purposes since it allows us to convert any one-dimensional lower bound for getting δ -close to ϵstationary points, into a lower bound for finding ( δ, ϵ ) -stationary points.
The former can be done using a technique which appears in [11] which itself is based on a technique for the deterministic setting due to Nemirovski [13].
The basic idea is to construct a function which is parameterized by a binary vector, such that optimizing the function essentially reduces to guessing the binary vector.
If the function is constructed such that any oracle response reveals a constant number of bits from the vector, this implies that the length of the vector can serve as a complexity lower bound for the optimization problem.
Since the result in [11] is asymptotic - for δ → 0 , or equivalently for a binary vector of arbitrary length - providing a specific δ -dependent lower bound boils down to extracting and improving its δ -dependence through a more careful analysis which appears in Section B in the appendix.
5. ( δ, ϵ ) -stationarity for convex functions
Recalling that for convex functions the Clarke sub-differential coincides with the convex subgradient [7, Proposition 2.2.7], we are interested in the possibility of producing points with small subgradients of convex Lipschitz functions.
It turns out that the same discussed proof idea of Theoremalso extends to convex functions, albeit with substantially more technical work to ensure that global convexity is maintained.
This yields the impossibility of producing ϵ -stationary points within any finite time even for convex functions.
Theorem 5 For any randomized first-order algorithm A and any δ, ϵ ≤ 14 , there exists a convex 1-Lipschitz function f : R → R such that f ( x 1 ) − inf x f ( x ) ≤ 1 , yet
Pr A [ ∃ t ∈ [ T ] : x t is ( δ, ϵ ) − stationary] < 13 unless T = Ω( (cid:112) log(1 /δ )) .
In particular, no finite time algorithm can produce a 1 4 -stationary point of 1-Lipschitz convex functions with constant probability.
Following the theorem above, we aim towards achievable relaxations of stationarity for convex
Lipschitz functions.
One natural relaxation is getting δ -close to an ϵ -stationary point.
While it is known that without convexity no algorithm can guarantee getting to such points in a dimension-free rate [11], perhaps not surprisingly, this is not the case for convex functions.
Indeed, given a Lipschitz convex function and a diameter bound dist( x 1 , arg min x f ( x )) ≤ R , Davis and Drusvyatskiy [8] provide a first-order algorithm that produces a point which is δ -close to an ϵ -stationary point of f within T = (cid:101)
O ( R
δ 2 ϵ 2 ) oracle calls.
Furthermore, since any such point is also a ( δ, ϵ ) -stationary point, then by Theorem 3 this is the best achievable rate for this task in terms of ϵ -dependence.
As in the rest of this work, we consider the coarser relaxation of ( δ, ϵ ) -stationary points.
It is important to emphasize that even for convex functions, the two notions of ( δ, ϵ ) -stationarity and being δ -close to an ϵ -stationary point are inherently different.
Indeed, as we illustrate in Lemma in the appendix, a ( δ, ϵ ) -stationary point can be arbitrarily far away from any ϵ -stationary point.
It turns out that for the sake of producing ( δ, ϵ ) -stationary points a simple combination of gradient descent and the INGD algorithm of [20] can provide a better dependence in terms of δ than [8] as we claim next.
Theorem 6 Suppose f : R d → R is L -Lipschitz and convex, and x 1 ∈ R d , R > 0 are such that dist( x 1 , arg min x f ( x )) ≤ R .
Then there exists a randomized first-order algorithm that produces a ( δ, ϵ ) -stationary point of f with probability at least 23 within T =
(cid:101) O (cid:16) L 2 R 2 / 3 δ 2 / 3 ϵ 2 (cid:17) oracle calls.
Moreover, if f is also H -smooth, then there exists such a deterministic algorithm with
[Equation]
The proof is provided in Section B in the appendix.
We would like to point out is that the improved rates for convex functions provided by Theorem 6 require the stronger assumption of a domain bound R :=
dist( x 1 , arg min x f ( x )) , as opposed to a sub-optimality bound ∆ :
= f ( x 1 ) − inf x f ( x ) .
This is indeed a stronger assumption since for any L -Lipschitz function we have ∆ ≤ LR .
Furthermore, since the lower bound in Theorem 3 holds for convex functions, and as we explained why it provides an Ω(1 /δϵ 3 ) lower bound for a certain parameter regime, we cannot expect such an improvement in general without any further assumption.
We note that this is analogous to the smooth convex case for which a gap between ∆ -based rate and R -based rate exists as well [6].
6. Discussion
In this paper, we studied the oracle complexity of producing ( δ, ϵ ) -stationary points in various settings.
Following our results, there are several notes and open ends we would like to point out.
In our opinion, the main open question in this realm is narrowing the gap between the (cid:101) O
(1 /δϵ 3 ) complexity upper bound of the INGD algorithm, and our Ω(1 /ϵ 2 + log(1 /δ )) complexity lower bound (which follows by combining Theorem 3 and Theorem 4).
We observed that the Ω(1 /ϵ 2 ) lower bound in Theorem 3 actually matches the upper bound in a certain parameter regime, which we suspect hints that the lower bound can be further improved.
This would require in particular a polynomial lower bound in terms of δ − 1 which we do not currently know how to prove.
We also examined the complexity of producing small subgradients of convex Lipschitz functions, showing that no finite algorithm can obtain such points unless a positive δ -relaxation is introduced, similarly to the nonconvex case.
Given an additional assumption of a domain bound as opposed to a sub-optimality bound, we were able to provide improved rates for convex functions.
We would like to point out that although the lower bound in Theorem 3 is stated in terms of a suboptimality bound, the same proof can be easily adapted to the bounded domain case.
Indeed, by replacing | x i − 1 | in the proof by | x i − R/ √ d | , the same result holds for any δ
= O ( R ) .
Another interesting direction for future work is to establish dimension-dependent lower bounds for producing ( δ, ϵ ) -stationary points.
Although several recent works considered this setting [9, 12], no complementing lower bounds currently exist.
Appendix A. Notation
We denote by R d the d -dimensional Euclidean space, any by ⟨· , ·⟩ , ∥ · ∥ the standard inner product and its associated Euclidean norm, respectively.
Given a point x ∈ R d we denote by x i its i ’th coordinate, and for any subset A ⊂ R d we denote their distance dist( x, A ) :=
inf
y ∈
A ∥ x − y ∥ .
We let e i ∈
R
d be the i ’th standard basis vector, N = { 1 , 2 , . . .
} be the natural numbers starting from 1, and [ N ] = { 1 , 2 , . . .
, N } be the natural numbers up to N ∈ N .
We denote by B r ( x ) the open
Euclidean ball around x ∈ R d of radius r > 0 , where d is clear from context.
We use standard big-O asymptotic notation:
For functions f, g : R → [0 , ∞ ) we write f = O ( g ) if there exists c > 0 such that f ( x ) ≤ c · g ( x ) ; f = Ω( g ) if g = O ( f ) .
We occasionally hide logarithmic factors by writing f = (cid:101)
O ( g ) if f =
O ( g log( g + 1)) .
Appendix B. Proofs B.1. Proof of Theorem
Fix T ∈ N .
Suppose that for any i ∈ [ T − 1] the first-order oracle response is f ( x i ) = 0 , ∇ f ( x i ) =
e 1 .
Since the algorithm is deterministic this fixes the iterate sequence x 1 , . . .
, x T .
We will show this resisting strategy is indeed consistent with a function which satisfies the conditions in the theorem.
To that end, we denote r :=
min 1 ≤ i ̸ =
j ≤ T ∥ x i − x j ∥ / 4 and fix some v ∈ (span { e 1 , x 1 , . . .
, x T } ) ⊥ with ∥ v ∥ = 1 (which exists since d = T + 2 ).
For any z ∈ R d we define
[Equation]
and further define
[Equation]
Note that h is well defined since by definition of r there cannot be i ̸
=
j such that ∥ x − x i ∥ < r and ∥ x − x j ∥ < r .
Lemma 7 h : R d → R as defined above is 7 -Lipschitz, satisfies for any i ∈
[ T ] : h ( x i ) = 0 , ∇ h ( x i ) =
e 1 and has no ( δ, 136 ) -stationary points for any δ > 0 .
Proof We start by noting that h is continuous, since for any z and ( y n ) ∞ n =1 ⊂ B r ( z ) , y n n →∞
[Equation]
y such that ∥ y − z ∥ =
r we have
[Equation]
Having established continuity, since x (cid:55)→ v ⊤ x is clearly 1 -Lipschitz (in particular 7 -Lipschitz), in order to prove Lipschitzness of h it is enough to show that g x i ( x ) is 7 -Lipschitz in ∥ x − x i ∥ < r for any x i .
For any such x, x i we have
[Equation]
hence ∥∇ g x
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
which proves the desired Lipschitz bound
The fact that for any i ∈ [ T ] : h ( x i ) = 0 , ∇ h ( x i ) =
eis easily verified by construction and by Eq.
In order to finish the proof, we need to show that h has no ( δ, 136 ) stationary-points.
By construction we have
[Equation]
while for ∥ x − x i ∥ =
r we would get convex combinations of the two cases.
3 Inspecting the set {∇ g x i ( x ) : ∥ x − x i ∥ < r } through Eq.
(1), we see that it depends on x, x i only through x − x i and that actually
[Equation]
which is convenient since the latter set does not depend on x i .
Overall, we see that any convex combination of gradients of h is in the set
[Equation]
We aim to show that the set above does not contain any vectors of norm smaller than 136 .
Let u be an element in the set with corresponding λ 1 , λ 2 , x as above.
If ∥ u ∥ ≥ 1 then there is nothing to show, so we can assume ∥ u ∥ < 1 .
We have
[Equation]
which gives
[Equation]
Hence
[Equation]
Given the previous lemma we can easily finish the proof of the theorem by looking at
[Equation]
f is 1 -Lipschitz (since h is 7 -Lipschitz), and satisfies f ( x 1 ) − inf x f ( x ) =
− ( − 17 ) < 1 .
Furthermore, for any i ∈ [ T ] : h ( x i )
= 0 > − 1 =
⇒ f
( x i ) = 17 h ( x i ) = 0 , which by the fact that h is 7 -Lipschitz implies that for any x ∈ B 17 ( x i ) : h ( x ) > − 1 =
⇒ ∂f ( x ) = 1 7 ∂h ( x ) .
In particular, ∂ 17 f ( x i ) =
17 ∂ 17 h ( x i ) so we conclude using the lemma shows that no x i is a ( 17 , 17 · 136 ) =
( 17 , 1252 ) stationary point of f .
B.2. Proof of Theorem
The proof is based on a slight modification of the INGD algorithm of Zhang et al.
To give some background, this algorithm is based on a gradient-descent-like procedure, where in each iteration, given current point x t , it uses a certain element g t ∈ ∂ δ f ( x t ) , until it finds a g t such that ∥ g t ∥ ≤ ϵ .
If g t does not satisfy this condition, it is ensured that the function value decreases by at least δϵ/ 4 , which can happen at most 4∆ /δϵ times.
g t itself is produced by repeatedly searching in various directions away from x t , and maintaining a convex combination of elements in ∂ δ f ( x ) , until either a sufficient decrease is found or the norm of the convex combination becomes sufficiently small.
In the original INGD algorithm, a given direction is searched by picking a point at random in an interval starting from x t .
If there is no sufficient decrease in that direction, the fundamental theorem of calculus implies that the derivative along that direction is small on average, hence picking a point at random is sufficient.
In our case, we de-randomize this part of the algorithm, by replacing it with a deterministic binary search (following Davis et al.
More formally, we replace the INGD algorithm with Algorithms 1,2,3 below, where the change is in calling the binary search subroutine (Algorithm 3) instead of picking a point at random.
Inspecting Algorithm 2, we see that it maintains a convex combination of elements in ∂ δ f ( x ) , and calls the subroutine BinarySearch( x, g k ) only whenever ∥
g k ∥ >
ϵ and f ( x ) − f ( x − δ ∥
g k ∥ g k ) ≤ δ 4 ∥
g k ∥ .
That being the case, the fundamental theorem of calculus ensures
[Equation]
δ (cid:28) ∇ f (cid:18) x − t ∥
g k ∥
g k (cid:19) , g k (cid:29) dt .
Namely, on average along the segment [ x, x − δ ∥
g k ∥ g k ] , points have gradients whose dot product with g k is small.
Accordingly, using the fact the ∇ f is H -Lipschitz, BinarySearch
( x, g k ) performs a binary search along this segment and produces a point y k such that ⟨∇ f ( y k ) , g k ⟩ ≤ 12 ∥
g k ∥within O (log( δH/ ∥
g k ∥ ))
=
O (log( δH/ϵ )) first-order oracle calls [9, Lemma 3.9].
This is the only modification to INGD which is introduced to Algorithm 1, when compared to [9, 20].
Accordingly,
MinNorm finds an approximate minimal norm element of ∂ δ f ( x t ) within O (cid:16) L 2 log( δH/ϵ ) ϵ(cid:17) firstorder oracle calls [9, Corollary 2.5].
Since we claimed that this can happen at most O (∆ /δϵ ) times, overall this produces a ( δ, ϵ ) -stationary point of f within O (cid:16) ∆ L 2 log( δH/ϵ ) δϵ(cid:17) oracle calls.
Algorithm 1 INGD(
Input: Initialization x 1 ∈ R d , iteration budget T ∈ N .
for t = 1 , . . .
, T − 1 do g t ← MinNorm( x t )
▷ Complexity O ( L 2 log( δH/ϵ ) ϵ) if ∥ g t ∥
≤ ϵ then return x t else x t +1 ← x t − δ ∥
g t ∥
g t end if end for return x T
Input: x ∈ R d , δ, ϵ > 0 .
k ←g 0 ← ∇ f ( x ) while ∥ g k ∥ >
ϵ and f ( x ) − f ( x − δ ∥
g k ∥ g k ) ≤ δ
∥
g k ∥ do y k ← BinarySearch( x, g k )
▷ Finds y k ∈
[ x, x − δ ∥
g k ∥
g k ] : ⟨∇ f
( y k ) , g k ⟩ ≤ 12 ∥
g k ∥g k +1 ← arg min λ ∈ [0 , 1] ∥ λg k + (1 − λ ) ∇ f ( y k )
∥
k ← k +end while return g k
¯ g ← δ ∥ g
∥ g while b − a > ∥ g ∥ 8 δH do if f ( x − a + b 2 · ¯ g ) ≥ 12 ( f ( x − a ¯ g ) + f ( x − b ¯ g ))
then ( a, b ) ← ( a, a + b 2 )
▷ E t ∈ [ a, a + b 2 ] ⟨∇ f ( z − t ¯ g ) , g ⟩ ≤ E t ∈ [ a + b 2 ,b ] ⟨∇ f ( z − t ¯ g ) , g ⟩ else ( a, b ) ← ( a + b 2 , b )
▷ E t ∈ [ a, a + b 2 ] ⟨∇ f ( z − t ¯ g ) , g ⟩ > E t ∈ [ a + b 2 ,b ] ⟨∇ f
( z − t ¯ g ) , g ⟩ end if end while return x − a ¯ g
B.3. Proof of Theorem
We derive the lower bound using the so-called “zero chain” technique [4, 18].
While a full discussion of this technique can be found in [3, 4], we will only state the relevant definitions and propositions we will use in our proof.
Definition 8 For x ∈ R d , we define its α -progress as prog α ( x ) :
= max { i ∈
[ d ] : | x i | > α } .
Definition 9 A function f : R d → R is an α -robust zero-chain if for every x ∈ R d : prog α ( x ) <
i =
⇒ f
( y ) =
f ( y 1 , . . .
, y i , 0 , . . . , 0) for all y in a neighborhood of x.
Proposition 10 ([3], Proposition 2.3) Let R, α > 0 , T ∈ N and d ′ ≥ ⌈ T + 2 R 2 α 2 log(3 T 2 ) ⌉ .
Let f : R T
→ R be an α -robust zero-chain, let U ∈ R d ′ × T be a uniformly random orthogonal matrix and denote f U ( x ) =
f ( U T x ) .
If A is a (possibly randomized) algorithm interacting with f U such that its iterates satisfy (cid:13)(cid:13) x ( t ) (cid:13)(cid:13) ≤ R for all t with probability 1, then with probability at least 23 over the draw of U : prog α
( U T x ( t ) ) < t for all t ≤ T.
Given the proposition, we aim to construct a robust zero chain function which such that all its ( δ, ϵ ) -stationary points will have large progress.
To that end, for any T ∈ N , α > 0 we define f : R T → R as follows
[Equation]
Lemma 11 For any T ∈ N , α > 0 , f T,α is 1 -Lipschitz, convex and is an α -robust zero-chain.
The lemma and its proof are similar to [5, Lemma 23], though we provide a proof for completeness.
Proof Let T ∈ N , α > 0 and abbreviate f = f T,α .
The Lipschitz property is easily seen since is a composition of 1 -Lipschitz functions.
Convexity follows from the fact that the maximum of convex functions is itself convex.
To prove that f is an α -robust zero-chain, let x ∈ R T be such that prog α ( x ) <
i , and y ∈ B
α ( x ) .
Note that | x i | ≤ α and that ∀ j : | x j − y j | < α .
So for any i + 1 ≤ j ≤ T we get
[Equation]
By Eq.
(3) this shows f is an α -robust zero-chain by definition.
From now on we fix some ϵ < 1 , δ ≤ 112 ϵ and set T = 14 ϵ 2 , α = 19 T .
The following lemma provides a crucial property of the ( δ, ϵ ) -stationary point of f T,α .
Lemma 12 Let x be a ( δ, ϵ ) -stationary point of f T,α .
Then there exists I ⊂ [ T ] , | I | ≥ 3 T 4 such that ∀ i ∈ I :
x i ≥ 13 .
In particular prog α ( x ) ≥ 3 T 4 .
Proof Throughout the proof we abbreviate f = f T,α .
Denote f i ( x ) :=
| x i − 1 | + 3 α
( T − i ) , and note that f ( x ) =
max i ∈
[ T ] { f i ( x ) } .
Consequently,
[Equation]
= ⇒ ∂
δ f ( x ) = conv { g i ( y ) :
f i ( y ) =
f ( y ) , ∥ x − y ∥ < δ } .
From the representation above we can see that if x satisfies that
[Equation]
then any g ∈ ∂
δ f ( x ) is a certain convex combination of vectors of the form
±
e
i (since in that case the second condition in the definition of g i is never satisfied).
Furthermore, in this case note that either e i ∈
∂
δ f ( x ) or − e i ∈ ∂
δ f ( x ) but not both, since if both of them are in ∂ δ f ( x ) then by convexity of ∂ δ there exists some y
i − 1 = 0 contradicting Eq.
We will now show that if that is the case, then x is not a ( δ, ϵ ) -stationary point.
Indeed, if x satisfies Eq.
(4) then for any g ∈ ∂f δ ( x ) there exist S
= { i 1 , . .
, i | S | } ⊆
[ T ] , λ i 1 , . . .
, λ i | S | ≥ 0 : (cid:80) i ∈ S λ i = 1 and σ i 1 , . . .
, σ i | S | ∈ {± 1 } such that g = (cid:80) i ∈ S λ i · σ i e i .
Hence
[Equation]
This establishes that if x satisfies Eq.
(4) then it is indeed not a ( δ, ϵ ) -stationary point.
So if x is a ( δ, ϵ ) -stationary point, we deduce that there exist y ∈ B
δ ( x ) , i ∈ [ T ] such that f i ( y ) =
f
( y ) and also y i = 1 .
For this y it holds that ∀ j ∈ [ T ] : f j ( y ) ≤ f ( y ) =
f i ( y ) , hence
[Equation]
We see that for all j ∈ [ T ] : 23 ≤ y j ≤ 43 , and recall that ∥ x − y ∥ < δ .
Assuming towards contradiction that the lemma is not true, we get | J | ⊂ [ T ] , | J | ≥ T 4 such that ∀ j ∈ J : x j < 13 and in particular | x j − y j | > 13 .
But this means that ∥ x − y ∥ > 13 · (cid:113) T 4 = √
[Equation]
6 =
L 12 ϵ ≥ δ which is a contradiction.
Given the previous lemma, finishing up the proof is done by a standard application of the aforementioned lower bound techniques.
All we need to do is to apply a random orthogonal transformation on top of our constructed function, and to handle “large queries” by extending our function such that points of large norm do not depend on the orthogonal transformation (see [4, Section 5]).
We do this by defining
[Equation]
On the one hand, if ∥ x ∥ < 2 √ T then
[Equation]
On the other hand, if ∥ x ∥ > 6 √ T then
[Equation]
which does not depend on U .
Furthermore, repeating essentially the same calculation as in Lemmashows that the ( δ, ϵ ) -stationary points of (cid:101)
f U ( · ) are exactly the ( δ, ϵ )
-stationary points of f ( U T · ) , which in particular shows they satisfy prog α
( U T x ) ≥ 3 T 4 .
Applying Proposition 10 with R = 6 √ T finishes the proof.
B.4. Proof of Theorem
Let δ ≤ 14 , T < 16 (cid:112) log(1 /δ ) .
By Yao’s lemma [19], we may assume A is deterministic and provide a distribution over hard functions.
Namely, we will construct an index set Σ and a parameterized subset of convex Lipschitz functions { f σ : σ ∈ Σ } that satisfy for any σ ∈ Σ :
• f σ is convex and 1 -Lipschitz.
• f σ attains its minimum with arg min x f ( x ) ⊂ (0 , 1) .
In particular, by Lipschitzness f ( x 1 ) − inf x f ( x ) ≤ 1 .
• ∀ x / ∈ arg min x f ( x ) , ∀ f ′ ( x ) ∈ ∂f ( x ) : | f ′ ( x ) | ≥ 14 .
Yet, the first T iterates produced by A ( f σ ) :
x f σ 1 , . .
, x f σ T , satisfy
[Equation] [Equation]
for some distribution D over Σ .
In particular, by the third bullet above, with the same probability and Claim 1 they are not ( δ, 14 ) -stationary.
We start by defining a sequence of segments in [0 , 1] parameterized by binary vectors, which will form the basis for our upcoming function class construction.
For any i ∈ N we denote
I i
[Equation] [Equation]
(cid:19)
[Equation]
(cid:18)
[Equation]
(cid:19) ⊂ (
[Equation]
We interpret the subscript 0 as ”left” and 1 as ”right”, corresponding to the location of the segment with respect to the center 12 .
Denote by ϕ i 0 , ϕ
i 1 the unique affine functions with positive derivatives which map (0 , 1) to I i 0 , I i 1 respectively - which we think of as “rescaling” the unit segment to the left or to the right.
We can now define for any k ∈ N , ( σ 1 , . . .
, σ k ) ∈ { 0 , 1 } k :
[Equation]
The following lemma shows that the parameterization of the segments above forms a binary tree structure.
This correspondence is given by considering binary vectors as vertices, while its sub-tree consists of vectors for which it serves as a prefix.
Later on we will use this intuition to construct functions parameterized by binary vectors, in a way that essentially reduces optimization to traversing the tree.
In particular, optimizing the function will require getting to a randomly selected leave which is unknown in advance to the algorithm.
Lemma
For any l < k and any σ 1 , . . .
, σ l , . . .
σ k ∈ { 0 , 1 } : I σ 1 ,...,σ l ⊃ I σ 1 ,...,σ l ,...,σ k . 2.
If ( σ 1 , . . .
, σ k − 1 ) ̸
= ( σ ′ 1 , . . .
, σ ′ k − 1 ) then for all σ k , σ ′ k ∈ { 0 , 1 } : I σ 1 ,...,σ k ∩ I σ ′ 1 ,...,σ ′ k = ∅ .
Proof
I σ 1 ,...,σ l ,...,σ k = ϕ 1 σ 1 ◦ · · · ◦ ϕ lσ l ( ϕ l +
[Equation]
σ l +1 ◦
[Equation]
kσ
[Equation]
[Equation] [Equation]
=
I
[Equation]
1 ,...,σ l .
[Figure]
Let i ≤ k − 1 be the minimal index for which σ
i ̸
= σ ′ i , and assume without loss of generality that σ i = 0 , σ ′ i = 1 .
If x ∈ I σ 1 ,...,σ k , then
[Equation]
hence x / ∈ I σ ′ 1 ,...,σ ′ k .
With the parameterized family of segments in hand, we are ready to define a corresponding family of functions.
We define the function h i 1 : [0 , 1] \ I i
→ R as follows
[Equation]
which is the piecewise linear function which satisfies h (0)
=
h (1) = 1 , h ( 12 + 1 8 i +1 )
=
h ( 12 + 2 8 i +1 )
= 12 + 1
i +1 , and also define h i 0 :
[0 , 1] \ I i
→ R as h i 0 ( x ) :
=
h i 1 (1 − x ) - see Fig.
1 for an illustration.
We also denote Φ i ( x ) :
= ϕ
i 1 ( x ) − 1 8 i +1 which is the affine function which satisfies Φ i (0) = 12 , Φ i (1) = 12 + 1 8 i +1 , (Φ i ) ′ =
( ϕ i 0 )
′ =
( ϕ i 1 ) ′ .
Finally, given any N ≥ 2 and σ = ( σ 1 , . . .
, σ N ) ∈ { 0 , 1 } N we are ready to define f : R → R as
[Equation]
The following lemma show that the constructed function satisfies our desired requirements.
Lemma
For any N ≥ 2 , σ ∈ { 0 , 1 } N :
• f Nσ is 2 -Lipschitz and convex.
• f Nσ ( x ) attains its minimum, and
∀ x / ∈ arg min x f Nσ ( x ) ∀ g ∈
∂f Nσ ( x ) : | g | ≥ 12 .
• f Nσ (0) − min x f Nσ ( x ) ≤ 1 and ∀ x ∈ arg min x f Nσ ( x ) : | x | ≤ 1 .
Proof Let N ≥ 2 , σ ∈ { 0 , 1 } N .
We start by proving that f Nσ is continuous.
It is clear that f Nσ is piecewise linear since h iσ i ,
ϕ iσ
i , Φ i are, So in order to show continuity we need to show continuity at the endpoints of adjacent linear pieces.
Since by Lemma 13 it holds that [0 , 1] ⊃ I σ 1 ⊃ I σ 1 ,σ 2 ⊃ · · · ⊃ I σ 1 ,...,σ N , inspecting the definition of f Nσ we see that we need to verify continuity only at the endpoints of I σ 1 ,...,σ i for any i ∈ [ N ] .
For any i ∈ [ N ] , the left endpoint of I σ 1 ,...,σ i satisfies
[Equation] [Equation]
The right endpoint follows a similar calculation, resulting in continuity.
Having established that f Nσ is piecewise linear and continuous, Lipschitzness will follow from the fact the each linear segment has a bounded slope.
To see this holds, recall that
[Equation]
so we get
[Equation]
In order to prove convexity, it is enough to show that each linear segment has slope which is larger than the segment to its left (hence the derivative is increasing which implies convexity).
Recalling that by Lemma 13 we have [0 , 1] ⊃ I σ 1 ⊃
I σ 1 ,σ 2 ⊃ · · · ⊃ I σ 1 ,...,σ N , we see that it is enough to establish monotonicity between the left linear segments of I σ 1 ,...,σ i , I σ 1 ,...,σ i +1 (in that order) and between the right linear segments of I σ 1 ,...,σ i +1 , , I σ 1 ,...,σ i (in that order) for any i ∈ [ N ] .
In order to do that, first note that by definition of h i 1 , h i
we have
[Equation]
Denoting by L ( I σ 1 ,...,σ i ) the left linear segment of I σ 1 ,...,σ i , we see that for any x ∈ L ( I σ 1 ,...,σ i ) we have by Eq.
(6) and Eq.
[Equation]
while for any x ∈ L ( I σ 1 ,...,σ i +1 )
we have
[Equation]
It is easy to verify for all i ∈ N that − (cid:16) 8 i +3 − 2 8 i +3 − 4 (cid:17) ≤ − (cid:16)
[Equation]
+2 − 2 8 i +2 +2 (cid:17) , which established the desired monotonicity.
Checking the right linear segments results in the same calculation, overall proving the first bullet.
For the second bullet, note that f Nσ attains its minimum at I σ 1 ,...,σ N since till then the function is decreasing, and after that it’s increasing (and on that segment it is constant).
Elsewhere we already saw that the derivatives are ± (cid:16) 8 i +1 − 2
i +1 +2 (cid:17) , ± (cid:16) 8 i +1 − 2
i +1 +4 (cid:17) which are easily verified to have absolute value larger than 12 for any i ∈ N .
The third bullet easily follows since I σ 1 ,...,σ N ⊂ [0 , 1] and f Nσ (0)
= 1 , Φ 1 ◦ · · · ◦ Φ N − 1 (0) > 0 .
The following lemma follows immediately from the definition of f Nσ and from Lemma 13, though we state it for future reference.
Lemma
For any N ≥ 2 , σ ∈ { 0 , 1 } N , 1 ≤ k < N , it holds that f Nσ ( x ) , ( f Nσ ) ′
( x ) do not depend on σ k +1 , . . .
, σ N for x / ∈ I σ 1 ,...,σ k .
Given Lemma 13 and Lemma 15, we state the following lemma which whose proof appears in [11, Proof of Proposition 1].
Lemma 16 ([11])
For any t ∈ N , 1 ≤ l < k ≤ N :
[Equation] [Equation]
We now claim that points that are δ close to arg min x f Nσ ( x ) =
I σ 1 ,...,
σ N ⊂ I σ 1 ,...,σ N
[Equation]
⊂
I σ 1 ,...,σ N − 2 ⊂ . .
. are necessarily inside I σ 1 ,...,σ k with large enough k .
Lemma 17 Let k =
⌊ 14 (cid:112) log(1 /δ )
⌋ < N ∈ N .
Then dist( x, I σ 1 ,...,σ N ) <
δ implies x ∈ I σ 1 ,...,σ k .
Proof Recall that I σ 1 ,...,σ N ⊂ I σ 1 ,...,σ k , and note that for any i ∈ N :
[Equation]
where ( ∗ ) follows from the fact that for any affine mapping x (cid:55)→ ax + b it holds that ( ax 1 + b ) − ( ax 2 + b )
= a ( x 1 − x 2 ) .
Thus
[Equation]
where that last inequality is a straightforward computation for k
= ⌊ 14 (cid:112) log(1 /δ ) ⌋ .
We conclude that
[Equation]
while the same arguments also show that sup I σ 1 ,...,σ k > sup I σ 1 ,...,σ N + δ .
Overall, we see that for any x satisfying dist( x, I σ 1 ,...,σ N ) <
δ : inf I σ 1 ,...,σ k < I σ 1 ,...,σ N − δ <
x
< sup I σ 1 ,...,σ N + δ < sup I σ 1 ,...,σ k = ⇒ x ∈ I σ 1 ,...,σ k .
We are now ready to finish the proof.
We set k =
⌊ 14 (cid:112) log(1 /δ ) ⌋ , N = k +1 , σ ∼ Unif { 0 , 1 } N .
We consider the algorithm’s iterates x 1 , . . .
, x T as random variables which depend on the random choice of the function f σ fed to the algorithm.
Denote the stochastic process
[Equation]
[Equation]
and note that that Z t +1 − Z t ≥ 0 with probability 1, yet by Lemma 16:
Pr[ Z t +1 − Z t = m ] ≤
[Equation]
m − 1 .
Hence by Lemma
[Equation]
This proves Eq.
(5) for 2-Lipschitz function and 12 -stationary points.
By rescaling the Lipschitz constant and ϵ by a factor of 2, we have finished the proof.
B.5. Proof of Theorem
The proof is identical to the proof of Theorem 5 which appears above, by replacing the constructed family f σ by the family h σ constructed in [11, Proof of proposition 1].
The only modification throughout the whole proof is that Lemma 17 holds with k = Θ(log(1 /δ )) instead of k = 14 (cid:112) log(1 /δ ) , affecting the final bound on T appropriately.
B.6. Proof of Theorem
It is well known (see for example [2, Theorem 3.2]) that given a Lipschitz convex function, performing projected gradient descent for T 1 :
= L 2 R 2 / 3 ϵ 2 δ 2 / 3 iterations and setting x GD := 1 T 1 (cid:80) T 1 j =1 x j satisfies f ( x GD ) − inf x f ( x ) ≤ RL √ T 1 = R 2 / 3 ϵδ 1 / 3 .
Furthermore, according to [20, Theorem 8], if we initialize INGD at a point x such that f ( x ) − inf x f ( x ) ≤ ∆ then with probability 23 it produces a ( δ, ϵ ) stationary point within T
= (cid:101) O (cid:16) ∆
L 2 δϵ 3 (cid:17) oracle calls.
In particular, initializing at x GD for which we have ∆ =
R 2 / 3 ϵδ 1 / 3 yields T 2 =
L 2 R 2 / 3 ϵ 2 δ 2 / 3 and letting T
= T
+ T 2 proves the claim.
Replacing INGD with its deterministic counterpart from Theorem 2 proves the additional claim.
B.7. Proof of Claim
First, if x is δ -close to an ϵ -stationary point it is clear that it is also ( δ, ϵ ) -stationary.
For the nontrivial implication, suppose x is ( δ, ϵ ) -stationary.
Namely, there exists g ∈ conv( ∪ x − δ<
t<x + δ ∂f ( t )) with | g | ≤ ϵ .
By Caratheodory’s theorem (for d = 1 ), this means there exist t 1 , t 2 ∈ ( − δ, δ ) ,
g i ∈ ∂f ( t i ) and λ 1 , λ 2 ∈ [0 , 1] satisfying λ 1 + λ 2 = 1 such that g
= λ 1 g 1 + λ 2 g 2 .
If either g 1 or g 2 satisfy | g i | ≤ ϵ then we get that x is δ -close the ϵ -stationary point t i .
Otherwise, both g i satisfy |
g i | > ϵ , but their convex combination is of magnitude smaller than ϵ - thus they necessarily have different signs.
Without loss of generality suppose g 1 < 0 , g 2 > 0 .
We claim that in this case there exists t ∗ ∈
[ t 1 , t 2 ] such that 0 ∈ ∂f ( t ∗ ) , finishing the proof since x is δ -close to the stationary point t ∗ .
Indeed, let A :=
{ t ∈ [ t 1 , t 2 ] : ∀ g ∈ ∂f ( t ) , g < 0 } and note that by assumption t 1 ∈ A, t 2 ̸∈ A .
Denote t ∗ := sup
A ∈ [ t 1 , t 2 ] which exists since A is non-empty.
On the one hand, by definition of the supremum there exists a non-decreasing sequence { y n } ∞ n =1 ⊂
A with lim n →∞ y n = t ∗ .
Note that by definition of A all the sub-differentials at all of y n are negative, thus t ∗ has some nonpositive sub-differential (by Bolzano–Weierstrass one can construct a converging sub-sequence of derivatives).
On the other hand, by construction all differentiable points in ( t ∗ , t 2 ) have only positive differentials, thus x + t ∗ has some non-negative sub-differential.
Finally, recalling that ∂f ( t ∗ ) is convex we get 0 ∈ ∂f ( t ∗ ) as claimed.
Appendix C. Additional Lemma
Lemma
For any C > 1 , δ < 1 and any ϵ < 1 8 C , there exists a 1-Lipschitz convex function f : R d → R in dimension d = O (1 /ϵ 2 ) and a point x ∈ R d such that x is a ( δ, ϵ ) -stationary point of f , yet is at distance at least Cδ away from any ϵ -stationary point of f .
Proof Fix δ < 1 < C, ϵ < 18 C , and let d = ⌈ 3 ϵ 2 ⌉ .
We denote by A :=
diag(2 ϵ 2 , 1 , . .
. , 1) ∈ R
d × d the diagonal positive-definite matrix whose diagonal is (2 ϵ 2 , 1 , . .
. , 1) , and consider its corresponding Mahalanobis norm f ( x ) :=
√ x ⊤ Ax .
For any x ̸ =
we have
[Equation]
In particular, we see that for any x ̸ =
the gradient norm is at least √ 2 ϵ 2 = √ 2 · ϵ , thus it is not an ϵ -stationary point.
We deduce that f is indeed 1-Lipschitz, and that it has no ϵ -stationary points except for the origin (which is its minimum, hence stationary).
Denote x 0 :=
δ 8 ϵ e 1 and notice that its distance from the only ϵ -stationary point of f is ∥ x 0 −
∥ =
δ 8 ϵ > Cδ .
It remains to show that x 0 is a ( δ, ϵ ) -stationary point of f .
To that end, consider for any 2 ≤ j ≤ d : z j := x 0 + δ 2 · e j = δ
ϵ e
+ δ 2 · e j which are clearly inside a δ -ball around x 0 .
Furthermore,
[Equation]
Now, define g := 1 d − 1 (cid:80) dj
=2 ∇ f ( z j ) and notice that by definition g ∈ ∂f δ ( x 0 ) since ∥ x − z j ∥ < δ .
This establishes that x 0 is indeed a ( δ, ϵ ) -stationary point since
[Equation]
where the last inequality is easily verified for d = ⌈ 3 ϵ 2 ⌉ .