A Gradient Sampling Method With Complexity Guarantees for Lipschitz Functions in High and Low Dimensions ∗
Abstract
Zhang, Lin, Jegelka, Sra and Jadbabaie [1] introduce a method for minimizing
Lipschitz functions with an efficiency guarantee of O ( ε − 4 ) .
Their method is a novel modification of Goldstein’s classical subgradient method.
Their work, however, makes use of a nonstandard subgradient oracle model and requires the function to be directionally differentiable.
Our first contribution in this paper is to show that both of these assumptions can be dropped by simply adding a small random perturbation in each step of their algorithm.
The resulting method works on any Lipschitz function whose value and gradient can be evaluated at points of differentiability.
Our second contribution is a new cutting plane algorithm that achieves better efficiency in low dimensions:
O ( dε − 3 ) for Lipschitz functions and
O ( dε − 2 ) for those that are weakly convex.
Introduction
The subgradient method [2] is a classical procedure for minimizing a nonsmooth Lipschitz function f on R d .
Starting from an initial iterate x 0 , the method computes
[Equation]
where v
[Equation]
∂f ( x
[Equation] [Equation]
Here, the positive sequence { α t } t ≥ 0 is user-specified, and the set ∂f is the Clarke subdifferential [3, 4],
[Equation]
In classical circumstances, the subdifferential reduces to familiar objects: for example, when f is
C 1 -smooth at x , the subdifferential ∂f ( x ) comprises of only the gradient ∇ f ( x ) , while for convex functions, it reduces to the subdifferential in the sense of convex analysis.
For functions f that are weakly convex — a broad class of functions first introduced in English in [5] — the limit points ¯ x of the subgradient method are known to be first-order critical, meaning
0 ∈ ∂f (¯ x ) .
Recall that a function f is called ρ -weakly convex if the quadratically perturbed function x (cid:55)→ f ( x ) + ρ 2 ∥ x ∥ 2 is convex.
In particular, convex and smooth functions are weakly convex [6].
Going beyond asymptotic guarantees, finite-time complexity estimates are known for smooth, convex, or weakly convex problems [7–14].
Modern machine learning, however, has witnessed the emergence of problems far beyond the weakly convex problem class.
Indeed, tremendous empirical success has been recently powered by industry-backed solvers, such as Google’s TensorFlow and Facebook’s PyTorch, which routinely train nonsmooth nonconvex deep networks via (stochastic) subgradient methods.
Despite a vast body of work on the asymptotic convergence of subgradient methods for nonsmooth nonconvex problems [15–19], no finite-time nonasymptotic convergence rates were known outside the weakly convex setting until recently, with Zhang, Lin, Jegelka, Sra, and Jadbabaie [1] making a big leap forward towards this goal.
In particular, restricting themselves to the class of Lipschitz and directionally differentiable functions, [1] developed an efficient algorithm motivated by Goldstein’s conceptual subgradient method [20].
Moreover, this was recently complemented by [21] with lower bounds for finding near -approximatestationary points for nonconvex nonsmooth functions.
While a significant breakthrough in both result and technique, one crucial limitation of [1] is that their complexity guarantees and algorithm use a nonstandard first-order oracle whose validity is unclear in examples.
To elaborate, their algorithm requires the following oracle access: given x, u ∈ R d solve the auxiliary convex feasibility problem :
[Equation]
The first issue with this oracle is that no general recipe exists for representing the full subdifferential ∂f ( x ) analytically, and evaluating even an arbitrary element of the subdifferential can be highly non-trivial [22, 23].
Moreover, ∂f ( x ) could be a very complicated set, e.g., for a deep ReLU neural network, the subdifferential is a polyhedron with a potentially huge number of facets, making the complexity of (2) unclear.
Further, [1] claim that for a composition of directionally differentiable functions with a closed-form directional derivative for each function, we can find the desired g by the chain rule.
While the chain rule does compute the directional derivative f ′ ( x, u ) , to the best of our knowledge, this does not translate to solving (2).
This is owing to the crucial fact that the chain rule (and sum rule) can easily fail for the computation of the subdifferential 2 (although these are indeed valid for the directional derivative of a composition of directionally differentiable functions).
We believe that this could potentially render the oracle of [1] computationally intractable.
Finally, we are unaware of other optimization algorithms imposing this oracle model.
Therefore, at face value, the convergence guarantees of [1] are not comparable to those of others.
1.1 Our results
Weakly convex optimization via a standard oracle.
Our first contribution is to recover the complexity result of [1] under a much weaker assumption: specifically, we replace the non-standard assumption in (2) with a standard first-order oracle model .
We show (Section 2) that a simple, yet critical, modification of the algorithm of [1], wherein one simply adds a small random perturbation in each iteration, works for any Lipschitz function assuming only an oracle that can compute gradients and function values at almost every point of R d in the sense of Lebesgue measure.
In particular, such oracles arise from automatic differentiation schemes routinely used in deep learning [24, 19].
Our end result is a randomized algorithm for minimizing any L -Lipschitz function that outputs a ( δ, ϵ ) -stationary point (Definition 1) after using at most (cid:101) O (cid:16) ∆ L 2 ϵ 3 δ log(1 /γ )
(cid:17) 3 gradient and function evaluations.
Here ∆ is the initial function gap and γ is the failure probability.
, y ) =
| x | and f 2 ( x, y ) =
−| x | .
Choose the direction u
= (0 , 1) , and let z = (0 , 0) .
Then, f ′ 1 ( z, u ) = f ′ 2 ( z, u ) = 0 , and ∂f 1 ( z ) =
∂f 2 ( z ) =
[ − 1 , 1] × 0 .
Therefore, to satisfy the oracle (2), for f 1 , we may choose the subgradient v 1 = ( − 1 , 0) , and for f 2 , we may choose the subgradient v 2 = ( − 1 , 0) since ⟨ v 1 , u ⟩ =
= ⟨ v 2 , u ⟩ .
However, v 1 + v 2 = ( − 2 , 0) , which is not a subgradient of f = 0 at z .
Throughout the paper, we use (cid:101) O ( · ) to hide poly-logarithmic factors in L, δ, ∆ , and ϵ .
In light of the above modifications, our algorithm is implementable in the many important settings like deep neural networks that [1] is not.
Along the way, we also simplify their proof techniques by providing a geometric viewpoint of the algorithm.
We would like to highlight the concurrent work of Tian, Zhou, and Man-Cho
So [25], which obtains this part of our result with a very similar technique.
Improved complexity in low dimensions.
Having obtained the result of [1] within the standard first-order oracle model, we then proceed to investigate the following question.
Can we improve the efficiency of the algorithm in low dimensions ?
In addition to being natural from the viewpoint of complexity theory, this question is well-grounded in applications.
For instance, numerous problems in control theory involve minimization of highly irregular functions of a small number of variables.
We refer the reader to the survey [26, Section 6] for an extensive list of examples, including Chebyshev approximation by exponential sums, spectral and pseudospectral abscissa minimization, maximization of the “distance to instability”, and fixed-order controller design by static output feedback.
We note that for many of these problems, the gradient sampling method of [26] is often used.
Despite its ubiquity in applications, the gradient sampling method does not have finite-time efficiency guarantees.
The algorithms we present here offer an alternative approach with a complete complexity theory.
The second contribution of our paper is an affirmative answer to the highlighted question .
We present a novel algorithm that uses (cid:101) O (cid:0) ∆ Ldϵ 2 δ log(1 /γ ) (cid:1) calls to our (weaker) oracle.
Thus we are able to trade off the factor Lϵ − 1 with d .
Further, if the function is ρ -weakly convex, the complexity improves to (cid:101) O (cid:0) ∆ dϵδ log( ρ ) (cid:1) , which matches the complexity in δ =
ϵ of gradient descent for smooth minimization.
Strikingly, the dependence on the weak convexity constant ρ is only logarithmic.
To put this contribution in perspective, assume for now δ
= ϵ : then, our algorithm’s dependence on ϵ in the case of Lipschitz, weakly convex functions is likely optimal in low dimensions, following a conjecture by Bubeck and Mikulincer [27] on the optimality of gradient descent for smooth optimization in dimension d = log( 1 ϵ ) (thus matching the lower bound by Carmon, Duchi, Hinder, and Sidford [28]).
Aside from possible optimality, the logarithmic dependence on smoothness/weak convexity exhibited by our iteration complexity is a significant improvement over the prior result of either O (1 /ϵ 4 ) by [1] or Nemirovski and Yudin’s rate of O (1 /ϵ 2 ) with a polynomial dependence on smoothness.
In the process, we also show that the minimal-norm element of the Goldsteinsubdifferential in low dimensions can be found in time
O (log(1 /ϵ )) , thus settling a question open since the 70s.
Techniques.
The main idea underlying our improved dependence on ϵ in low dimensions is outlined next.
The algorithm of [1] comprises of an outer loop with O (cid:0) ∆ ϵδ (cid:1) iterations, each performing either a decrease in the function value or an ingenious random sampling step to update the descent direction.
Our observation, central to improving the ε dependence, is that the violation of the descent condition can be transformed into a gradient oracle for the problem of finding a minimal norm element of the Goldstein subdifferential.
This gradient oracle may then be used within a cutting plane method, which achieves better ε dependence at the price of a dimension factor (Section 3).
Limitations.
One limitation of our work is that our second contribution does not immediately extend to the stochastic setting.
We consider this to be an interesting open problem to resolve.
Notation.
Throughout, we let R d denote a d -dimensional Euclidean space equipped with a dot product ⟨· , ·⟩ and the Euclidean norm ∥ x ∥ 2 = (cid:112) ⟨ x, x ⟩ .
The symbol B r ( x ) denotes an open Euclidean ball of radius r > 0 around a point x .
Throughout, we fix a function f : R d → R that is L -Lipschitz, and let dom( ∇ f ) denote the set of points where f is differentiable—a full Lebesgue measure set by Rademacher’s theorem.
The symbol f ′ ( x, u ) def =
lim τ ↓ 0 τ − 1 ( f directional derivative whenever limit exists
Interpolated normalized gradient descent
In this section, we describe the results in [1] and our modified subgradient method that achieves finite-time guarantees in obtaining ( δ, ϵ ) -stationarity for an L -Lipschitz function f : R d → R .
The main construction we use is the Goldstein subdifferential [20].
Definition 1 (Goldstein subdifferential) .
Consider a locally Lipschitz function f : R d → R , a point x ∈ R d , and a parameter δ > 0 .
The Goldstein subdifferential of f at x is the set
[Equation]
A point x is called ( δ, ϵ ) -stationary if dist (0 , ∂ δ f ( x )) ≤ ϵ .
Thus, the Goldstein subdifferential of f at x is the convex hull of all Clarke subgradients at points in a δ -ball around x .
Famously, [20] showed that one can significantly decrease the value of f by taking a step in the direction of the minimal norm element of ∂ δ f ( x ) .
Throughout the rest of the section, we fix δ ∈ (0 , 1) and use the notation
[Equation]
Theorem 1 ([20]) .
Fix a point x , and let g be a minimal norm element of ∂ δ f ( x ) .
Then as long as g ̸ = 0 , we have f ( x − δ ˆ g ) ≤ f ( x ) − δ ∥
g ∥ 2 .
Theorem 1 immediately motivates the following conceptual descent algorithm:
[Equation]
In particular, Theorem 1 guarantees that, defining ∆ def = f
( x 0 ) − min f , the approximate stationarity condition min t
[Equation]
,...,T ∥ g t
[Equation]
≤ ϵ holds after T
= O (cid:18)
[Equation]
iterations of (4) .
Evaluating the minimal norm element of ∂ δ f ( x ) is impossible in general, and therefore the descent method described in (4) cannot be applied directly.
Nonetheless it serves as a guiding principle for implementable algorithms.
Notably, the gradient sampling algorithm [29] in each iteration forms polyhedral approximations K t of ∂ δ f
x t ) by sampling gradients in the ball B δ ( x ) and computes search directions g t ∈ argmin g ∈ K t ∥ g ∥ 2 .
These gradient sampling algorithms, however, have only asymptotic convergence guarantees [26].
The recent paper [1] remarkably shows that for any x ∈ R
d one can find an approximate minimal norm element of ∂ δ f ( x ) using a number of subgradient computations that is independent of the dimension.
The idea of their procedure is as follows.
Suppose that we have a trial vector g ∈ ∂
δ f ( x ) (not necessarily a minimal norm element) satisfying
[Equation]
That is, the decrease in function value is not as large as guaranteed by Theorem 1 for the true minimal norm subgradient.
One would like to now find a vector u ∈ ∂
δ f ( x ) so that the norm of some convex combination (1 − λ )
g + λu is smaller than that of g .
A short computation shows that this is sure to be the case for all small λ > 0 as long as ⟨ u, g ⟩ ≤ ∥ g ∥ 22 .
The task therefore reduces to:
[Equation]
The ingenious idea of [1] is a randomized procedure for establishing exactly that in expectation.
Namely, suppose for the moment that f happens to be differentiable along the segment [ x, x − δ ˆ g ] ; we will revisit this assumption shortly.
Then the fundamental theorem of calculus, in conjunction with (5), yields
[Equation]
Consequently, a point y chosen uniformly at random in the segment [ x, x − δ ˆ g ] satisfies
[Equation]
Therefore the vector u = ∇ f ( y ) can act as the subgradient we seek.
Indeed, the following lemma shows that, in expectation, the minimal norm element of [ g, u ] is significantly shorter than g .
The proof is extracted from that of [1, Theorem 8].
Lemma 2 ([1]) .
Fix a vector g ∈ R d , and let
u ∈ R
d be a random vector satisfying E ⟨ u, g ⟩ < 12 ∥ g ∥ 22 .
Suppose moreover that the inequality ∥ g ∥ 2 ,
∥ u ∥ 2 ≤ L holds for some L < ∞ .
Then the minimalnorm vector z in the segment [ g, u ] satisfies:
[Equation]
Proof.
Applying E ⟨ u, g ⟩ ≤ 12 ∥ g ∥ 22 and ∥ g ∥ 2 , ∥ u ∥ 2 ≤ L , we have, for any λ ∈ (0 , 1) ,
[Equation]
Plugging in the value λ = ∥ g ∥ 22 8 L 2 ∈ (0 , 1) minimizes the right hand side and completes the proof.
The last technical difficulty to overcome is the requirement that f be differentiable along the line segment [ g, u ] .
This assumption is crucially used to obtain (6) and (7).
To cope with this problem, [1] introduce extra assumptions on the function f to be minimized and assume a nonstandard oracle access to subgradients.
We show, using Lemma 3, that no extra assumptions are needed if one slightly perturbs g .
Lemma 3.
Let f : R d → R be a Lipschitz function, and fix a point x ∈ R d .
Then there exists a set
D ⊂ R d of full Lebesgue measure such that for every y ∈ D , the line spanned by x and y intersects dom( ∇ f ) in a full Lebesgue measure set in R .
Then, for every y ∈ D and all τ ∈ R , we have
[Equation]
Proof.
Without loss of generality, we may assume x = 0 and f ( x ) = 0 .
Rademacher’s theorem guarantees that dom( ∇ f ) has full Lebesgue measure in R d .
Fubini’s theorem then directly implies that there exists a set Q ⊂ S d − 1 of full Lebesgue measure within the sphere S d − 1 such that for every y ∈ Q , the intersection R + { y } ∩ (dom( ∇ f ))
c is Lebesgue null in R .
It follows immediately that the set D = { τy : τ > 0 , y ∈ Q } has full Lebesgue measure in R d .
Fix now a point y ∈ D and any τ ∈ R + .
Since f is Lipschitz, it is absolutely continuous on any line segment and therefore f ( x + τ ( y − x
[Equation]
[Equation]
The proof is complete.
We now have all the ingredients to present a modification of the algorithm from [1], which, under a standard first-order oracle model, either significantly decreases the objective value or finds an approximate minimal norm element of ∂ δ f .
The following theorem establishes the efficiency of Algorithm 1, and its proof is similar to that of [1,
Lemma 13].
For completeness, we include the full proof in Appendix A.
Theorem 4.
Let { g k } be generated by MinNorm ( x, δ, ϵ ) .
Fix an index k ≥ 0 , and define the stopping time τ def = inf { k : f
( x − δ ˆ g k ) < f ( x ) − δ ∥
g k ∥ 2 / 4 or ∥ g k ∥ 2 ≤ ϵ } .
Then, we have
[Equation]
An immediate consequence of Theorem 4 is that MinNorm ( x, δ, ϵ ) terminates with high-probability.
Corollary 5.
MinNorm ( x, δ, ϵ ) terminates in at most (cid:108) 64 L 2 ϵ 2 (cid:109) · ⌈ 2 log(1 /γ )
⌉ iterations with probability at least 1 − γ , where we define the stopping time τ def = inf { k : f
( x − δ ˆ g k ) < f ( x ) − δ ∥
g k ∥ 2 / 4 or ∥ g k ∥ 2 ≤ ϵ } .
Combining Algorithm 1 with (4) yields Algorithm 2, with convergence guarantees summarized in
Theorem 6, whose proof is identical to that of [1, Theorem 8].
Algorithm 2 Interpolated Normalized Gradient Descent ( INGD ( x 0 , T ))
Initial x 0 , counter T for t = 0 , . .
, T − 1 do g
=
MinNorm ( x t )
▷ Computational complexity (cid:101) O
( L 2 /ϵ 2 )
Set x t +1 = x t − δ ˆ g
▷ We define ˆ g in (3) end for
Return x T
Theorem 6.
Fix an initial point x 0 ∈ R d , and define ∆ = f ( x 0 ) − inf x f ( x ) .
Set the number of iterations T = 4∆ δϵ .
Then, with probability 1 − γ , the point x T = INGD ( x 0 , T ) satisfies dist (0 , ∂
δ f ( x T )) ≤ ϵ in a total of at most
[Equation]
function-value and gradient evaluations .
In summary, the complexity of finding a point x satisfying dist (0 , ∂ δ f ( x )) ≤ ϵ is at most
O (cid:16) ∆ L 2 δϵ 3 log (cid:16) 4∆ γδϵ (cid:17)(cid:17) with probability 1 − γ .
Using the identity ∂f ( x ) =
lim sup δ → 0 ∂ δ f ( x ) , this result also provides a strategy for finding a Clarke stationary point, albeit with no complexity guarantee.
It is thus natural to ask whether one may efficiently find some point x for which there exists y ∈ B δ ( x ) satisfying dist (0 , ∂f ( y ))
≤ ϵ .
This is exactly the guarantee of subgradient methods on weakly convex functions in [6].
[30] shows that for general Lipschitz functions, the number of subgradient computations required to achieve this goal by any algorithm scales with the dimension of the ambient space.
Finally, we mention that the perturbation technique similarly applies to the stochastic algorithm of [1, Algorithm 2], yielding a method that matches their complexity estimate.
Faster INGD in low dimensions
In this section, we describe our modification of Algorithm 1 (“ INGD ”) for obtaining improved runtimes in the low-dimensional setting.
Our modified algorithm hinges on computations similar to (5), (6), and (7) except for the constants involved, and hence we explicitly state this setup.
Given a vector g ∈ ∂
δ f ( x ) , we say it satisfies the descent condition at x if
[Equation] [Equation]
Recall that Lemma 3 shows that for almost all g , we have
[Equation]
Hence, when g does not satisfy the descent condition (8), we can output a random vector u ∈ ∂
δ f ( x ) such that
[Equation]
Then, an arbitrary vector g either satisfies (8) or can be used to output a random vector u satisfying (9).
As described in Corollary 5, Algorithm 1 achieves this goal in (cid:101)
O
( L 2 /ϵ 2 ) iterations.
In this section, we improve upon this oracle complexity by applying cutting plane methods (which we review shortly) to design Algorithm 3, which finds a better descent direction in (cid:101) O
( Ld/ϵ ) oracle calls for L -Lipschitz functions and O ( d log( L/ϵ ) log( δρ/ϵ )) oracle calls for ρ -weakly convex functions.
Brief overview of cutting-plane methods.
We first provide a brief relevant overview of cutting-plane methods here and refer the reader to standard textbooks in optimization for a more in-depth exposition.
Given a convex function f with its set S of minimizers, a cutting-plane method (CPM) minimizes f by maintaining a convex search set E ( k ) ⊇ S in the k th iteration and iteratively shrinking E ( k ) guided by the subgradients of f that act as “separation oracles” for the set S .
Specifically, this is achieved by noting that for any x ( k ) chosen from E ( k ) , if the gradient oracle indicates ∇ f ( x ( k ) )
̸ = 0 , (i.e. x ( k ) / ∈ S ), then the convexity of f guarantees S ⊆ H ( k ) : (cid:8)
y : ⟨∇ f
( x ( k ) ) , y − x ( k ) ⟩ ≤ 0 (cid:9) , and hence S ⊆ H ( k )
∩ E ( k ) .
The algorithm continues by choosing E ( k +1) ⊇ E ( k ) ∩ H ( k ) , and different choices of x ( k ) and E ( k ) yield different rates of shrinkage of E ( k ) until a point in S is found.
In light of this description, the minimization of a convex function over a constrained convex set via this cutting-plane method requires, at each iteration, merely a subgradient of the function.
Our novel insight is that a lack of function decrease implies we have roughly such a subgradient, which we may then use in a cutting-plane method for computing the minimum norm element of the subdifferential faster in low dimensions (with improved complexity for weakly convex functions).
Setting the stage for our algorithm.
In Appendix B, we demonstrate how to remove the expectation in (9) and turn the inequality into a high probability statement.
For now, we assume the existence of an oracle O as in Definition 2.
Definition 2 (Inner Product Oracle) .
Given a vector g ∈ ∂
δ f ( x ) that does not satisfy the descent condition (8) , the inner product oracle O ( g ) outputs a vector u ∈ ∂
δ f ( x ) such that
[Equation]
We defer the proof of the lemma below to Appendix B.
Lemma 7.
Fix x ∈ R d and a unit vector ˆ g ∈ R d such that f is differentiable almost everywhere on the line segment [ x, y ] , where y def = x − δ ˆ g .
Suppose that z ∈ R d sampled uniformly from [ x, y ] satisfies
E z ⟨∇ f
( z ) , ˆ g ⟩ ≤ ϵ 3 .
Then we can find ¯ z ∈ R d using at most O
( L ϵ log(1 /γ ))
gradient evaluations of f , such that with probability at least 1 − γ the estimate ⟨∇ f
(¯ z ) , ˆ g ⟩ ≤ ϵ 2 holds.
Moreover, if f is ρ -weakly convex, we can find ¯ z ∈ R d such that ⟨∇ f (¯ z ) , ˆ g ⟩ ≤ ϵ 2 using only O (log( δρ/ϵ )) function evaluations of f .
Our key insight is that this oracle is almost identical to the gradient oracle of the minimal norm element problem
[Equation]
Therefore, we can use it in the cutting plane method to find an approximate minimal norm element of ∂ δ f .
When there is no element of ∂ δ f with norm less than ϵ , our algorithm will instead find a vector that satisfies the descent condition.
The main result of this section is the following theorem.
Theorem 8.
Let f : R d → R be an L -Lipschitz function.
Fix an initial point x 0 ∈ R d , and let ∆ def = f ( x 0 ) − inf x f ( x ) .
Then, there exists an algorithm that outputs a point x ∈ R d satisfying dist(0 , ∂
δ f ( x )) ≤ ϵ and, with probability at least 1 − γ , uses at most
[Equation]
value/gradient evaluations.
If f is ρ -weakly convex, the analogous statement holds with probability one and with the improved efficiency estimate O (cid:0) ∆ dδϵ log( L/ϵ ) · log( δρ/ϵ ) (cid:1) of function value/gradient evaluations.
3.1 Finding a minimal norm element
In this section, we show, via Algorithm 3, how to find an approximate minimal norm element of ∂ δ f ( x ) .
Instead of directly working with the minimal norm problem, we note that, by CauchySchwarz inequality and the Minimax Theorem, for any closed convex set Q , we have
[Equation]
where ϕ Q ( v ) def = min g ∈ Q ⟨ g, v ⟩ , and Lemma 9 formally connects the problem of finding the minimal norm element with that of maximizing ϕ Q .
The key observation in this section (Lemma 10) is that the inner product oracle O is a separation oracle for the (dual) problem max ∥ v ∥ 2 ≤ 1 ϕ Q ( v ) with
Q = ∂
δ f ( x ) and hence can be used in cutting plane methods.
Lemma 9.
Let Q ⊂ R d be a closed convex set that does not contain the origin.
Let g ∗ Q be a minimizer of min g ∈ Q
∥ g ∥ 2 .
Then, the vector v ∗ Q = g ∗ Q / ∥ g ∗ Q ∥ 2 satisfies
[Equation]
∥for all g ∈ Q. and v ∗ Q = arg max ∥ v ∥ 2 ≤ 1 ϕ Q ( v ) .
Proof.
We omit the subscript Q to simplify notation.
Since, by definition, g ∗ minimizes ∥
g
∥ 2 over all g ∈ Q , we have
[Equation]
22 for all g ∈ Q, and the inequality is tight for g = g ∗ .
Using this fact and ϕ ( v ∗ ) =
min g ∈ Q ⟨ g, g ∗ ∥ g ∗ ∥
2 ⟩ gives
[Equation]
where we used Sion’s minimax theorem in the second to last step.
This completes the proof.
Using this lemma, we can show that O is a separation oracle.
Lemma 10.
Consider a vector g ∈
∂f δ ( x ) that does not satisfy the descent condition (8) , and let the output of querying the oracle at g be u ∈ O ( g ) .
Suppose that dist(0 , ∂
δ f ( x )) ≥
ϵ 2 .
Let g ∗ be the minimal-norm element of ∂ δ f ( x ) .
Then the normalized vector v ∗ def = g ∗ /
∥
g ∗ ∥ 2 satisfies the inclusion:
[Equation]
Proof.
Set Q = ∂
δ f ( x ) .
By using ⟨ u, ˆ g ⟩ ≤ ϵ 2 (the guarantee of O per Definition 2) and ⟨ u, v ∗ ⟩ ≥ ∥ g ∗ ∥ 2 (from Lemma 9), we have ⟨ u, ˆ g − v ∗ ⟩ = ⟨ u, ˆ g ⟩ − ⟨ u, v ∗ ⟩ ≤ ϵ 2 − ∥ g ∗ ∥ 2 ≤ 0 .
Thus Lemma 10 states that if x is not a ( δ, ϵ 2 ) -stationary point of f , then the oracle O produces a halfspace H v that separates ˆ g from v ∗ .
Since O is a separation oracle, we can combine it with any cutting plane method to find v ∗ .
For concreteness, we use the center of gravity method and display our algorithm in Algorithm 3.
We note that Ω k is an intersection of a ball and
some half spaces, hence we can compute its center of gravity in polynomial time by taking an average of the empirical samples from this convex set.
While we use a simple cutting-plane method, any algorithm in this class may be used; our focus is merely on minimizing the oracle complexity.
Further note that in our algorithm, we use a point ζ k close to the true center of gravity of Ω k , and therefore, we invoke a result about the perturbed center of gravity method.
Theorem 11 (Theorem 3 of [31]; see also [32]) .
Let K be a convex set with center of gravity µ and covariance matrix A .
For any halfspace H that contains some point x with ∥ x − µ ∥ A − 1 ≤ t , we have vol( K ∩ H ) ≤ (1 − 1 /e + t )vol( K ) .
1: Initialize center point x .
2: Set k = 0 , the search region Ω
= B 2 (0) , the set of gradients Q 0 = {∇ f ( x ) } , and r satisfying 0 < r < ϵ/ (32 dL ) 3: while min g ∈ Q k ∥ g ∥ 2 > ϵ do 4:
Let v k be the center of gravity of Ω k .
5: if v k satisfies the descent condition (8) at x then 6:
Return v k 7: end if 8:
Sample ζ k uniformly from B r ( v k ) 9: u k ← O
( ζ k ) 10: Ω k
+
=
Ω k ∩ { w : ⟨ u k , ζ k − w ⟩ ≤ 0 } .
Q k +1 = conv( Q k ∪ { u k } ) 12: k
= k +13: end while 14: Return arg min g ∈ Q k ∥ g ∥ 2 .
Theorem 12 (Theorem 4.1 of [33]) .
Let K be a convex set in R d with center of gravity µ and covariance matrix A .
Then,
[Equation]
We now have all the tools to show correctness and iteration complexity of Algorithm 3.
Theorem 13.
Let f : R d → R be an L -Lipschitz function.
Then Algorithm 3 returns a vector v ∈ ∂
δ f ( x ) that either satisfies the descent condition (8) at x or satisfies ∥ v ∥ 2 ≤ ϵ in ⌈ 8 d log(8 L/ϵ ))
⌉ calls to O .
Proof.
By the description of Algorithm 3, either it returns a vector v satisfying the descent condition or returns g ∈ ∂
δ f ( x ) with ∥ g ∥ 2 ≤ ϵ .
We now obtain the algorithm’s claimed iteration complexity.
Consider an iteration k such that Ω k does contain a ball of radius ϵ 4 L .
Let A k be the covariance matrix of convex set Ω k .
By Theorem 12, we have
[Equation]
Applying this result to the observation that in Algorithm 3 ζ k is sampled uniformly from B r ( v k ) gives
[Equation]
Recall from Algorithm 3 and the preceding notation that Ω k has center of gravity v k and covariance matrix
A k .
Further, the halfspace { w : ⟨ u k , ζ k − w ⟩ ≤ 0 } in Algorithm 3 contains the point ζ k satisfying (11).
Given these statements, since Algorithm 3 sets
Ω k +
= Ω k ∩ { w : ⟨ u k , ζ k − w ⟩} , we may invoke Theorem 11 to obtain
[Equation]
We claim that Algorithm 3 takes at most T + 1 steps where T =
d log (1 − 110 ) ( ϵ/ (8 L )) .
For the sake of contradiction, suppose that this statement is false.
Then, applying (12) with k = T + 1 gives
[Equation]
On the other hand, Algorithm 3 generates points u i
= O ( ζ i ) in the i -th call to O and the set
Q i = conv { u 1 , u 2 , · · · , u i } .
Since we assume that the algorithm takes more than T + 1 steps, we have min g ∈ Q T +1 ∥
g ∥ 2 ≥ ϵ .
Using this and u i ∈ Q T +1 , Lemma 10 lets us conclude that v ∗ Q T +1 ∈ (cid:8) w ∈ R d : ⟨ u i , ζ i − w ⟩ ≤ 0 (cid:9) for all i ∈
[ T +1] .
Since Ω T +1 is the intersection of the unit ball and these halfspaces, we have
[Equation]
Per (13), Ω T +1 does not contain a ball of radius ϵ 4 L , and therefore we may conclude that there exists a point (cid:101) v ∈ B ϵ 2 L ( v ∗ Q T +1 ) such that (cid:101) v / ∈ Ω T +1 .
Since (cid:101) v ∈ B 2 (0) , the fact (cid:101) v / ∈ Ω T +1 must be true due to one of the halfspaces generated in
Algorithm 3.
In other words, there must exist some i ∈
[ T + 1] with
[Equation]
By the guarantee of O , we have ⟨ u i , ζ i ⟩ ≤ ϵ 2 , and hence
[Equation]
By applying (cid:101) v ∈ B ϵ 2 L ( v ∗ Q T +1 ) , u i ∈
∂
δ f ( x ) , L -Lipschitzness of f , and Lemma 9, we have ⟨
[Equation] [Equation]
[Equation]
Combining (14) and (15) yields that min g ∈ Q T +1 ∥
g ∥ 2 = ∥
g ∗ Q T +1 ∥ 2 < ϵ .
This contradicts the assumption that the algorithm takes more than T + 1 steps and concludes the proof.
Now, we are ready to prove the main theorem.
Proof of Theorem 8.
We note that the outer loop in Algorithm 2 runs at most O ( ∆ δϵ ) times because we decrease the objective by Ω( δϵ ) every step.
Combining this with Theorem 13 and Lemma 7, we have that with probability 1 − γ , the oracle complexity for L -Lipschitz function is
[Equation]
· (cid:24) 2 log (cid:18) 4∆ γδϵ (cid:19)(cid:25) =
O (cid:18) ∆
Ld δϵ 2 · log( L/ϵ ) · log(1 /γ )
(cid:19) and for L -Lipschitz and ρ -weakly convex function is O (cid:0) ∆ dδϵ log( L/ϵ ) · log( δρ/ϵ ) (cid:1) .
Acknowledgements
Research of D. Davis was supported by an Alfred P. Sloan research fellowship and NSF DMS award 2047637 .
Research of D. Drusvyatskiy was supported by NSF DMS- 1651851 and CCF2023166 awards.
Research of YL was supported by NSF awards CCF- 1749609 , DMS- 1839116 ,
DMS- 2023166 , CCF- 2105772 , a Microsoft Research Faculty Fellowship, Sloan Research Fellowship, and Packard Fellowship.
Research of GY was supported by an MIT Presidential Fellowship.
Part of this work was done while GY was a student at the University of Washington.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 1.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Due to the theoretical nature of our work, we do not foresee any immediate potential negative societal impacts of it.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] All our complete proofs are in the appendix.
[-We thus conclude h (¯ t ) ≤ ϵ 3 + ϵ 
-]