Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimization
Abstract
Nonsmooth nonconvex optimization problems broadly emerge in machine learningand business decision making, whereas two core challenges impede the development of efﬁcient solution methods with ﬁnite-time convergence guarantee: the lackof computationally tractable optimality criterion and the lack of computationallypowerful oracles.
The contributions of this paper are two-fold.
First, we establishthe relationship between the celebrated Goldstein subdifferential [46] and uniformsmoothing, thereby providing the basis and intuition for the design of gradient-freemethods that guarantee the ﬁnite-time convergence to a set of Goldstein stationarypoints.
Second, we propose the gradient-free method (GFM) and stochastic GFMfor solving a class of nonsmooth nonconvex optimization problems and prove thatboth of them can return a ( δ, (cid:15) )
-Goldstein stationary point of a Lipschitz functionf at an expected convergence rate at O ( d 3 / 2 δ − 1 (cid:15) − 4 ) where d is the problem dimension.
Two-phase versions of GFM and SGFM are also proposed and proven toachieve improved large-deviation results.
Finally, we demonstrate the effectivenessof 2-SGFM on training ReLU neural networks with the M INST dataset.
Introduction
Many of the recent real-world success stories of machine learning have involved nonconvex optimization formulations, with the design of models and algorithms often being heuristic and intuitive.
Thusa gap has arisen between theory and practice.
Attempts have been made to ﬁll this gap for differentlearning methodologies, including the training of multi-layer neural networks [25], orthogonal tensordecomposition [41], M-estimators [63, 64], synchronization and MaxCut [6, 66], smooth semideﬁniteprogramming [15], matrix sensing and completion [10, 42], robust principal component analysis(RPCA)
[43] and phase retrieval [79, 64].
For an overview of nonconvex optimization formulationsand the relevant ML applications, we refer to a recent survey [51].
It is generally intractable to compute an approximate global minimum [69] or to verify whether a pointis a local minimum or a high-order saddle point [67].
Fortunately, the notion of approximate stationarypoint gives a reasonable optimality criterion when the objective function f is smooth; the goal here isto ﬁnd a point x ∈ R d such that (cid:107)∇ f ( x ) (cid:107) ≤ (cid:15) .
Recent years have seen rapid algorithmic developmentthrough the lens of nonasymptotic convergence rates to (cid:15) -stationary points [70, 44, 45, 20, 21, 53].
Another line of work establishes algorithm-independent lower bounds [22, 23, 3, 4].
Relative to its smooth counterpart, the investigation of nonsmooth optimization is relatively scarce,particularly in the nonconvex setting, both in terms of efﬁcient algorithms and ﬁnite-time convergenceguarantees.
Yet, over several decades, nonsmooth nonconvex optimization formulations have foundapplications in many ﬁelds.
A typical example is the training multi-layer neural networks with ReLUneurons, for which the piecewise linear activation functions induce nonsmoothness.
Another example arises in controlling ﬁnancial risk for asset portfolios or optimizing customer satisfaction in service systems or supply chain systems.
Here, the nonsmoothness arises from the payoffs of ﬁnancialderivatives and supply chain costs, e.g., options payoffs [38] and supply chain overage/underagecosts [78].
These applications make signiﬁcant demands with respect to computational feasibility,and the design of efﬁcient algorithms for solving nonsmooth nonconvex optimization problems hasmoved to the fore [65, 30, 28, 84, 12, 31, 80].
The key challenges lie in two aspects: (i) the lack of a computationally tractable optimality criterion,and (ii) the lack of computationally powerful oracles.
More speciﬁcally, in the classical setting wherethe function f is Lipschitz, we can deﬁne (cid:15) -stationary points based on the celebrated notion of Clarkestationarity [26].
However, the value of such a criterion has been called into question by Zhanget al.
[84], who show that no ﬁnite-time algorithm guarantees (cid:15) -stationarity when (cid:15) is less than aconstant.
Further, the computation of the gradient is impossible for many application problems andwe only have access to a noisy function value at each point.
This is a common issue in the context ofsimulation optimization [68, 48]; indeed, the objective function value is often achieved as the outputof a black-box or complex simulator, for which the simulator does not have the infrastructure neededto effectively evaluate gradients; see also Ghadimi and Lan [44] and Nesterov and Spokoiny [72] forcomments on the lack of gradient evaluation in practice.
Contribution.
In this paper, we propose and analyze a class of deterministic and stochastic gradientfree methods for nonsmooth nonconvex optimization problems in which we only assume that thefunction f is Lipschitz.
Our contributions can be summarized as follows.
We establish a relationship between the Goldstein subdifferential and uniform smoothing via appealto the hyperplane separation theorem.
This result provides the basis for algorithmic design andﬁnite-time convergence analysis of gradient-free methods to ( δ, (cid:15) ) -Goldstein stationary points.
We propose and analyze a gradient-free method (GFM) and stochastic GFM for solving a class ofnonsmooth nonconvex optimization problems.
Both of these methods are guaranteed to return a ( δ, (cid:15) ) -
Goldstein stationary point of a Lipschitz function f : R d (cid:55)→ R with an expected convergence rate of O ( d 3 / 2 δ − 1 (cid:15) − 4 ) where d ≥ 1 is the problem dimension.
Further, we propose the two-phase versionsof GFM and SGFM.
As our goal is to return a ( δ, (cid:15) ) -Goldstein stationary point with user-speciﬁedhigh probability 1 − Λ , we prove that the two-phase version of GFM and SGFM can improve thedependence from (1 / Λ) 4 to log(1 / Λ) in the large-deviation regime.
Related works.
Our work is related to a line of literature on gradient-based methods for nonsmoothand nonconvex optimization and gradient-free methods for smooth and nonconvex optimization.
Due to space limitations, we defer our comments on the former topic to Appendix A. In the contextof gradient-free methods, the basic idea is to approximate a full gradient using either a one-pointestimator [39] or a two-point estimator [1, 44, 37, 75, 72], where the latter approach achieves abetter ﬁnite-time convergence guarantee.
Despite the meteoric rise of two-point-based gradient-freemethods, most of the work is restricted to convex optimization [37, 75, 82] and smooth and nonconvexoptimization [72, 44, 61, 62, 24, 52, 49].
For nonsmooth and convex optimization, the best upperbound on the global rate of convergence is O ( d(cid:15) − 2 ) [75] and this matches the lower bound [37].
For smooth and nonconvex optimization, the best global rate of convergence is O ( d(cid:15) − 2 ) [72] and
O ( d(cid:15) − 4 ) if we only have access to noisy function value oracles [44].
Additional regularity conditions,e.g., a ﬁnite-sum structure, allow us to leverage variance-reduction techniques [62, 24, 52] and thebest known result is O ( d 3 / 4 (cid:15) − 3 )
However, none of these gradient-free methods have beendeveloped for nonsmooth nonconvex optimization and the only gradient-free method we are aware offor the nonsmooth is summarized in Nesterov and Spokoiny [72, Section 7].
Preliminaries and Technical Background
We provide the formal deﬁnitions for the class of Lipschitz functions considered in this paper, andthe deﬁnitions for generalized gradients and the Goldstein subdifferential that lead to optimalityconditions in nonsmooth nonconvex optimization.
2.1 Function classes
Imposing regularity on functions to be optimized is necessary for obtaining optimization algorithmswith ﬁnite-time convergence guarantees [71].
In the context of nonsmooth optimization there are twotypes of regularity conditions: Lipschitz properties of function values and bounds on function values.
[Equation]
We ﬁrst list several equivalent deﬁnitions of Lipschitz continuity.
A function f : R d (cid:55)→ R is said tobe L - Lipschitz if for every x ∈ R d and the direction v ∈ R d with (cid:107) v (cid:107) ≤ 1 , the directional projectionf x , v ( t ) :
= f ( x + t v ) deﬁned for t ∈ R satisﬁes [Equation] for
[Equation]
Equivalently, f is L -Lipschitz if for every x , x (cid:48) ∈ R d , we have
[Equation]
Further, the function value bound f ( x 0 ) − inf x ∈ R d f ( x ) appears in complexity guarantees for smoothand nonconvex optimization problems [71] and is often assumed to be bounded by a positive constant∆ > 0 .
Note that x 0 is a prespeciﬁed point (i.e., an initial point for an algorithm) and we simply ﬁx itfor the remainder of this paper.
We deﬁne the function class which will be considered in this paper.
Deﬁnition 2.1 Suppose that ∆ > 0 and L > 0 are both independent of the problem dimension d ≥ 1 .
Then, we denote F d (∆ , L ) as the set of L -Lipschitz functions f : R d (cid:55)→ R with the bounded functionvalue f ( x 0 ) − inf x ∈ R d f ( x ) ≤ ∆ .
The function class F d (∆ , L ) includes Lipschitz functions on R d and is thus different from thenonconvex function class considered in the literature [44, 72].
First, we do not impose a smoothnesscondition on the function f ∈ F d (∆ , L ) , in contrast to the nonconvex functions studied in Ghadimiand Lan [44] which are assumed to have Lipschitz gradients.
Second, Nesterov and Spokoiny [72,
Section 7] presented a complexity bound for a randomized optimization method for minimizing anonsmooth nonconvex function.
However, they did not clarify why the norm of the gradient of theapproximate function f ¯ µ of the order δ (we use their notation) serves as a reasonable optimalitycriterion in nonsmooth nonconvex optimization.
They also assume an exact function value oracle,ruling out many interesting application problems in simulation optimization and machine learning.
In contrast, our goal is to propose fast gradient-free methods for nonsmooth nonconvex optimizationin the absence of an exact function value oracle.
In general, the complexity bound of gradient-freemethods will depend on the problem dimension d ≥ 1 even when we assume that the function tobe optimized is convex and smooth [37, 75].
As such, we should consider a function class with agiven dimension d ≥ 1 .
In particular, we consider a optimality criterion based on the celebrated
Goldstein subdifferential [46] and prove that the number of function value oracles required by ourdeterministic and stochastic gradient-free methods to ﬁnd a ( δ, (cid:15) )
-Goldstein stationary point off ∈ F d (∆ , L ) is O ( poly ( d, L, ∆ , 1 /(cid:15), 1 /δ ))
when δ, (cid:15) ∈ (0 , 1) are constants (see the deﬁnition of
Goldstein stationarity in the next subsection).
It is worth mentioning that F d (∆ , L ) contains a rather broad class of functions used in real-worldapplication problems.
Typical examples with additional regularity properties include Hadamard semidifferentiable functions [76, 32, 84], Whitney-stratiﬁable functions [13, 30], o -minimally deﬁnablefunctions [27] and a class of semi-algebraic functions [5, 30].
Thus, our gradient-free methods canbe applied for solving these problems with ﬁnite-time convergence guarantees.
2.2 Generalized gradients and Goldstein subdifferential
We start with the deﬁnition of generalized gradients [26] for nondifferentiable functions.
This isperhaps the most standard extension of gradients to nonsmooth and nonconvex functions.
Deﬁnition 2.
Given a point x ∈ R d and a direction v ∈ R d , the generalized directional derivativeof a nondifferentiable function f is given by Df ( x ; v ) :=
lim sup y
→ x ,t ↓ 0 f
( y + t v ) − f ( y ) t .
Then, thegeneralized gradient of f is deﬁned as a set ∂f ( x ) :
= { g ∈ R d : g (cid:62) v ≤ Df ( x ; v ) , ∀ v ∈ R d } .
Rademacher’s theorem guarantees that any Lipschitz function is almost everywhere differentiable.
This implies that the generalized gradients of Lipschitz functions have additional properties and wecan deﬁne them in a relatively simple way.
The following proposition summarizes these results; werefer to Clarke [26] for the proof details.
Proposition 2.1 Suppose that f is L -Lipschitz for some L > 0 , we have that ∂f ( x ) is a nonempty,convex and compact set and (cid:107) g (cid:107) ≤ L for all g ∈ ∂f ( x ) .
Further, ∂f ( · ) is an upper-semicontinuousset-valued map.
Moreover, a generalization of mean-value theorem holds: for any x 1 , x 2 ∈ R d , there exist λ ∈ (0 , 1) and
g ∈ ∂f ( λ x 1 + (1 − λ ) x 2 ) such that f ( x 1 ) − f ( x 2 ) =
g (cid:62) ( x 1 − x 2 ) .
Finally,there is a simple way to represent the generalized gradient ∂f ( x ) :
[Equation]
which is the convex hull of all limit points of ∇ f ( x k ) over all sequences x 1 , x 2 , . .
of differentiablepoints of f ( · ) which converge to x .
Given this deﬁnition of generalized gradients, a Clarke stationary point of f is a point x satisfying0 ∈ ∂f ( x ) .
Then, it is natural to ask if an optimization algorithm can reach an (cid:15) -stationary point witha ﬁnite-time convergence guarantee.
Here a point x ∈ R d is an (cid:15) -Clarke stationary point ifmin {(cid:107) g
(cid:107) :
g ∈ ∂f ( x ) } ≤ (cid:15).
This question has been addressed by [84, Theorem 1], who showed that ﬁnding an (cid:15) -Clarke stationarypoints in nonsmooth nonconvex optimization can not be achieved by any ﬁnite-time algorithm givena ﬁxed tolerance (cid:15) ∈ [0 , 1) .
One possible response is to consider a relaxation called a near (cid:15) -Clarkestationary point.
Consider a point which is δ -close to an (cid:15) -stationary point for some δ > 0 .
A pointx ∈ R d is near (cid:15) -stationary
if the following statement holds true:
[Equation]
Unfortunately, however, [58, Theorem 1] demonstrated that it is impossible to obtain worst-caseguarantees for ﬁnding a near (cid:15)
-Clarke stationary point of f ∈ F d (∆ , L ) when (cid:15), δ > 0 are smallerthan some certain constants unless the number of oracle calls has an exponential dependence on theproblem dimension d ≥ 1 .
These negative results suggest a need for rethinking the deﬁnition oftargeted stationary points.
We propose to consider the reﬁned notion of Goldstein subdifferential.
Deﬁnition 2.
Given a point x ∈ R d and δ > 0 , the δ -Goldstein subdifferential of a Lipschitzfunction f at x is given by ∂
δ f ( x ) := conv ( ∪ y ∈ B
δ ( x ) ∂f ( y )) .
The Goldstein subdifferential of f at x is the convex hull of the union of all generalized gradients atpoints in a δ -ball around x .
Accordingly, we can deﬁne the ( δ, (cid:15) )
-Goldstein
stationary points; that is,a point x ∈ R d is a ( δ, (cid:15) )
-Goldstein
stationary point if the following statement holds:
[Equation]
It is worth mentioning that ( δ, (cid:15) ) -Goldstein stationarity is a weaker notion than (near) (cid:15) -Clarkestationarity since any (near) (cid:15) -stationary
point is a ( δ, (cid:15) )
-Goldstein stationary point but not viceversa.
However, the converse holds true under a smoothness condition [84, Proposition 6] andlim δ ↓
∂
δ f ( x ) =
∂f ( x ) holds as shown in Zhang et al.
[84, Lemma 7].
The latter result also enablesan intuitive framework for transforming nonasymptotic analysis of convergence to ( δ, (cid:15) )
-Goldsteinstationary points to classical asymptotic results for ﬁnding (cid:15) -Clarke stationary points.
Thus, weconclude that ﬁnding a ( δ, (cid:15) )
-Goldstein stationary point is a reasonable optimality condition forgeneral nonsmooth nonconvex optimization.
Remark 2.2 Finding a ( δ, (cid:15) )
-Goldstein stationary point in nonsmooth nonconvex optimization hasbeen formally shown to be computationally tractable in an oracle model [84, 31, 80].
Goldstein[46] discovered that one can decrease the function value of a Lipschitz f by using the minimal-normelement of ∂ δ f ( x ) and this leads to a deterministic normalized subgradient method which ﬁnds a( δ, (cid:15) )
-Goldstein stationary point within O ( ∆ δ(cid:15) ) iterations.
However, Goldstein’s algorithm is onlyconceptual since it is computationally intractable to return an exact minimal-norm element of ∂ δ f ( x ) .
Recently, the randomized variants of Goldstein’s algorithm have been proposed with a convergenceguarantee of O ( ∆ L 2 δ(cid:15) 3 ) [84, 31, 80].
However, it remains unknown if gradient-free methods ﬁnd a( δ, (cid:15) ) -Goldstein stationary point of a Lipschitz function f within O ( poly ( d, L, ∆ , 1 /(cid:15), 1 /δ )) iterationsin the absence of an exact function value oracle.
Note that the dependence on the problem dimensiond ≥ 1 is necessary for gradient-free methods as mentioned before.
2.3 Randomized smoothing
The randomized smoothing approaches are simple and work equally well for convex and nonconvexfunctions.
Formally, given the L -Lipschitz function f (possibly nonsmooth nonconvex) and a distribution P , we deﬁne f δ ( x ) =
E u ∼ P [ f ( x + δ u )] .
In particular, letting P be a standard Gaussiandistribution, the function f δ is a δL √ d -approximation of f ( · ) and the gradient ∇ f δ is L √ d
δ -Lipschitzwhere d ≥ 1 is the problem dimension; see Nesterov and Spokoiny [72, Theorem 1 and Lemma 2].
Letting P be an uniform distribution on an unit ball in (cid:96) 2 -norm, the resulting function f δ is a δLapproximation of f ( · ) and ∇ f δ is also cL √ d δ -Lipschitz where d ≥ 1 is the problem dimension;see
Youseﬁan et al.
[83, Lemma 8] and Duchi et al.
[36, Lemma E.2], rephrased as follows.
Proposition 2.3 Let f δ ( x ) =
E u ∼ P [ f ( x + δ u )] where P is an uniform distribution on an unit ball in(cid:96) 2 -norm.
Assuming that f is L -Lipschitz, we have (i) | f δ ( x ) − f ( x ) | ≤ δL , and (ii) f
δ is differentiableand L -Lipschitz with the cL √ d δ -Lipschitz gradient where c > 0 is a constant.
In addition, there existsa function f for which each of the above bounds are tight simultaneously.
The randomized smoothing approaches form the basis for developing gradient-free methods [39, 1,2, 44, 72].
Given an access to function values of f , we can compute an unbiased estimate of thegradient of f δ and plug them into stochastic gradient-based methods.
Note that the Lipschitz constantof f δ depends on the problem dimension d ≥ 1 with at least a factor of √ d for many randomizedsmoothing approaches [58, Theorem 2].
This is consistent with the lower bounds for all gradient-freemethods in convex and strongly convex optimization [37, 75].
Main Results
We establish a relationship between the Goldstein subdifferential and the uniform smoothing approach.
We propose a gradient-free method (GFM), its stochastic variant (SGFM), and a two-phase versionof GFM and SGFM.
We analyze these algorithms using the Goldstein subdifferential; we provide theglobal rate and large-deviation estimates in terms of ( δ, (cid:15) ) -Goldstein stationarity.
3.1 Linking Goldstein subdifferential to uniform smoothing
Recall that ∂
δ f and f δ are deﬁned by ∂
δ f ( x ) := conv ( ∪ y ∈ B
δ ( x ) ∂f ( y )) and f δ ( x ) =
E u ∼ P [ f ( x +δ u )] .
It is clear that f is almost everywhere differentiable since f is L -Lipschitz.
This implies that∇ f δ ( x ) =
E u ∼ P [ ∇ f ( x + δ u )] and demonstrates that ∇ f δ ( x ) can be viewed intuitively as a convexcombination of ∇ f ( z ) over an inﬁnite number of points z ∈ B δ ( x ) .
As such, it is reasonable toconjecture that ∇ f δ ( x ) ∈ ∂
δ f ( x ) for any x ∈ R d .
However, the above argument is not a rigorousproof; indeed, we need to justify why ∇ f δ ( x ) =
E u ∼ P [ ∇ f ( x + δ u )] if f is almost everywheredifferentiable and generalize the idea of a convex combination to include inﬁnite sums.
To resolvethese issues, we exploit a toolbox due to Rockafellar and Wets [74].
In the following theorem, we summarize our result and refer to Appendix C for the proof details.
Theorem 3.1 Suppose that f is L -Lipschitz and let f δ ( x ) =
E u ∼ P [ f ( x + δ u )] , where P is anuniform distribution on a unit ball in (cid:96) 2 -norm and let ∂ δ f be a δ -Goldstein subdifferential of f (cf.
Deﬁnition 2.3).
Then, we have ∇ f δ ( x ) ∈ ∂
δ f ( x ) for any x ∈ R d .
Theorem 3.1 resolves an important question and forms the basis for analyzing our gradient-freemethods.
Notably, our analysis can be extended to justify other randomized smoothing approachesin nonsmooth nonconvex optimization.
For example, Nesterov and Spokoiny [72] used Gaussiansmoothing and estimated the number of iterations required by their methods to output ˆ x ∈ R dsatisfying (cid:107)∇ f δ (ˆ x ) (cid:107) ≤ (cid:15) .
By modifying the proof of Theorem 3.1 and Zhang et al.
[84, Lemma 7], wecan prove that ∇ f δ belongs to Goldstein subdifferential with Gaussian weights and this subdifferentialconverges to the Clarke subdifferential as δ → 0 .
Compared to uniform smoothing and the original
Goldstein subdifferential, the proof for Gaussian smoothing is quite long and technical [72, Page554], and adding Gaussian weights seems unnatural in general.
3.2 Gradient-free methods
We analyze a gradient-free method (GFM) and its two-phase version (2-GFM) for optimizing a
Lipschitz function f .
Due to space limitations, we defer the proof details to Appendix D.
Global rate estimation.
Let f : R d (cid:55)→ R be a L -Lipschitz function and the smooth version of fis then the function f δ = E u ∼ P
[ f ( x + δ u )] where P is an uniform distribution on an unit ball in 2 -norm.
Equipped with Lemma 10 from Shamir [75], we can compute an unbiased estimator for thegradient ∇ f δ ( x t ) using function values.
This leads to the gradient-free method (GFM) in Algorithm 1 that simply performs a one-stepgradient descent to obtain x t .
It is worth mentioning that we use a random iteration count R toterminate the execution of Algorithm 1 and this will guarantee that GFM is valid.
Indeed, we onlyderive that min t =1 , 2 ,...,T (cid:107)∇ f δ ( x t ) (cid:107) ≤ (cid:15) in the theoretical analysis (see also Nesterov and Spokoiny[72, Section 7]) and ﬁnding the best solution from { x 1 , x 2 , . . .
, x T } is difﬁcult since the quantity(cid:107)∇ f δ ( x t ) (cid:107) is unknown.
To estimate them using Monte Carlo simulation would incur additionalapproximation errors and raise some reliability issues.
The idea of random output is not new but hasbeen used by Ghadimi and Lan [44] for smooth and nonconvex stochastic optimization.
Such schemealso gives us a computational gain with a factor of two in expectation.
Theorem 3.2 Suppose that f is L -Lipschitz and let δ > 0 and 0 < (cid:15) < 1 .
Then, there exists some
T > 0 such that the output of Algorithm 1 with η = 110 (cid:113) δ (∆+ δL ) cd 3 / 2 L 3 T satisﬁes that E [min {(cid:107) g (cid:107) : g ∈∂ δ f ( x R ) } ] ≤ (cid:15) and the total number of calls of the function value oracle is bounded by
[Equation]
, where d ≥ 1 is the problem dimension, L > 0 is the Lipschtiz parameter of f and ∆ > 0 is an upperbound for the initial objective function gap, f ( x 0 ) − inf x ∈ R d
f ( x ) > 0 .
Remark 3.3 Theorem 3.2 illustrates the difference between gradient-based and gradient-free methods in nonsmooth nonconvex optimization.
Indeed, Davis et al.
[31] has recently proved the rate of
O ( δ − 1 (cid:15) − 3 ) for a randomized gradient-based method in terms of ( δ, (cid:15) ) -Goldstein stationarity.
Further,
Theorem 3.2 demonstrates that nonsmooth nonconvex optimization is likely to be intrinsically harderthan all other standard settings.
More speciﬁcally, the state-of-the-art rate for gradient-free methodsis O ( d(cid:15) − 2 ) for nonsmooth convex optimization in terms of objective function value gap [37] andsmooth nonconvex optimization in terms of gradient norm [72].
Thus, the dependence on d ≥ 1 islinear in their bounds yet d 32 in our bound.
We believe it is promising to either improve the rate ofgradient-free methods or show the impossibility by establishing a lower bound.
Large-deviation estimation.
While Theorem 3.2 establishes the expected convergence rate overmany runs of Algorithm 1, we are also interested in the large-deviation properties for a single run.
Indeed, we hope to establish a complexity bound for computing a ( δ, (cid:15), Λ) -solution ; that is, a point x ∈ R d satisfying Prob
[Equation] min {
[Equation] x ) } ≤ [Equation] ) ≥ 1 − Λ for some δ > 0 and 0 < (cid:15), Λ < 1 .
By Theorem 3.
and Markov’s inequality,
[Equation]
we conclude that the total number of calls of the function value oracle is bounded by
[Equation]
This complexity bound is rather pessimistic in terms of its dependence on Λ which is often set tobe small in practice.
To improve the bound, we combine Algorithm 1 with a post-optimizationprocedure [44], leading to a two-phase gradient-free method (2-GFM), shown in Algorithm 2.
Theorem 3.4 Suppose that f is L -Lipschitz and let δ > 0 and 0 < (cid:15), Λ < 1 .
Then, there exists some
T, S, B > 0 such that the output of Algorithm 2 with η = 110 (cid:113) δ (∆+ δL ) cd 3 / 2 L 3 T satisﬁes that Prob (min {(cid:107) g (cid:107) :g ∈ ∂
δ f (¯ x s (cid:63) ) } ] ≥ (cid:15) ) ≤ Λ and the total number of calls of the function value oracle is bounded by
[Equation]
Λ (cid:15) 2 log(cid:18)Λ (cid:19)(cid:19) , where d ≥ 1 is the problem dimension, L > 0 is the Lipschtiz parameter of f and ∆ > 0 is an upperbound for the initial objective function gap, f ( x 0 ) − inf x ∈ R d
f ( x ) > 0 .
Clearly, the bound in Theorem 3.4 is signiﬁcantly smaller than the corresponding one in Eq.
(3.1) interms of the dependence on 1 / Λ , demonstrating the power of the post-optimization phase.
3.3 Stochastic gradient-free methods
We turn to the analysis of a stochastic gradient-free method (SGFM) and its two-phase version(2-SGFM) for optimizing a Lipschitz function f ( · ) =
E ξ ∈ P µ [ F ( · , ξ )] .
Global rate estimation.
In contrast to minimizing a deterministic function f , we only have accessto the noisy function value F ( x , ξ ) at any point x ∈ R d where a data sample ξ is drawn from adistribution P µ .
Intuitively, this is a more challenging setup.
It has been studied before in the settingof optimizing a nonsmooth convex function [37, 72] or a smooth nonconvex function [44].
As inthese papers, we assume that (i) F ( · , ξ ) is L ( ξ ) -Lipschitz with E ξ ∈ P µ [ L 2 ( ξ )] ≤ G 2 for some G > 0and (ii) E
[ F ( x , ξ t )]
= f
( x ) for all x ∈ R d where ξ t is simulated from P µ at the t th iteration.
Despite the noisy function value, we can compute an unbiased estimator of the gradient ∇ f δ ( x t ) ,where f δ = E u ∼ P [ f ( x + δ u )]
=
E u ∼ P ,ξ ∈ P µ [ F ( x + δ u , ξ )] .
In particular, we have ˆ g t =
d 2 δ ( F ( x t +δ w t , ξ t ) − F ( x t − δ w t , ξ t ))
w t .
Clearly, under our assumption, we have
[Equation]
where g t is deﬁned in Algorithm 1.
However, the variance of the estimator ˆ g t can be undesirablylarge since F ( · , ξ ) is L ( ξ ) -Lipschitz for a (possibly unbounded) random variable L ( ξ ) > 0 .
Toresolve this issue, we revisit Shamir [75, Lemma 10] and show that in deriving an upper bound for 283 E u ∼ P ,ξ ∈ P µ [ (cid:107) ˆ g t (cid:107) 2 ] it sufﬁces to assume that E ξ ∈ P µ [ L 2 ( ξ )] ≤ G 2 for some constant G > 0 .
Theresulting bound achieves a linear dependence in the problem dimension d > 0 which is the sameas in Shamir [75, Lemma 10].
Note that the setup with convex and L ( ξ ) -Lipschitz functions F ( · , ξ )has been considered in Duchi et al.
However, our estimator is different from their estimator ofˆ g t = dδ ( F ( x t + δ w t , ξ t ) − F ( x t , ξ t ))
w t which essentially suffers from the quadratic dependence ind > 0 .
It is also necessary to employ a random iteration count R to terminate Algorithm 3.
Theorem 3.5 Suppose that F ( · , ξ ) is L ( ξ ) -Lipschitz with E ξ ∈ P µ [ L 2 ( ξ )] ≤ G 2 for some G > 0 andlet δ > 0 and 0 < (cid:15) < 1 .
Then, there exists some T > 0 such that the output of Algorithm 3 withη = 110 (cid:113) δ (∆+ δG ) cd 3 / 2 G 3 T satisﬁes
that E [min {(cid:107) g (cid:107) : g ∈ ∂
δ f ( x R ) } ] ≤ (cid:15) and the total number of calls ofthe noisy function value oracle is bounded by
[Equation]
where d ≥ 1 is the problem dimension, L > 0 is the Lipschtiz parameter of f and ∆ > 0 is an upperbound for the initial objective function gap, f ( x 0 ) − inf x ∈ R d
f ( x ) > 0 .
− Output: ¯ x s (cid:63) . In the stochastic setting, the gradient-based method achieves the rate of O ( δ − 1 (cid:15) − 4 ) for a randomizedgradient-based method in terms of ( δ, (cid:15) ) -Goldstein stationarity [31].
As such, our bound in Theorem 3.5 is tight up to the problem dimension d ≥ 1 .
Further, the state-of-the-art rate for stochasticgradient-free methods is O ( d(cid:15) − 2 ) for nonsmooth convex optimization in terms of objective functionvalue gap [37] and O ( d(cid:15) − 4 ) for smooth nonconvex optimization in terms of gradient norm [44].
Thus, Theorem 3.5 demonstrates that nonsmooth nonconvex stochastic optimization is essentially themost difﬁcult one among than all these standard settings.
Large-deviation estimation.
As in the case of GFM, we hope to establish a complexity bound of
SGFM for computing a ( δ, (cid:15), Λ) -solution.
By Theorem 3.5 and Markov’s inequality, we obtain thatthe total number of calls of the noisy function value oracle is bounded by
[Equation]
We also propose a two-phase stochastic gradient-free method (2-SGFM) in Algorithm 4 by combining
Algorithm 3 with a post-optimization procedure.
Theorem 3.6 Suppose that F ( · , ξ ) is L ( ξ ) -Lipschitz with E ξ ∈ P µ [ L 2 ( ξ )] ≤ G 2 for some G > 0 andlet δ > 0 and 0 < (cid:15), Λ < 1 .
Then, there exists some T, S, B > 0 such that the output of Algorithm 4with η = 110 (cid:113) δ (∆+ δG ) cd 3 / 2 G 3 T satisﬁes that Prob (min {(cid:107) g (cid:107) : g ∈ ∂
δ f (¯ x s (cid:63) ) } ] ≥ (cid:15) ) ≤ Λ and the totalnumber of calls of the noisy function value oracle is bounded by
[Equation]
where d ≥ 1 is the problem dimension, L > 0 is the Lipschtiz parameter of f and ∆ > 0 is an upperbound for the initial objective function gap f ( x 0 ) − inf x ∈ R d
f ( x ) > 0 .
Further discussions. We remark that the choice of stepsize η in all of our zeroth-order methodsdepend on ∆ , whereas such dependence is not necessary in the ﬁrst-order setting; see e.g., Zhang et al.[84]. Setting the stepsize without any prior knowledge of ∆ , our methods can still achieve ﬁnite-timeconvergence guarantees but the order would become worse. This is possibly because the ﬁrst-orderinformation gives more characterization of the objective function than the zeroth-order information,so that for ﬁrst-order methods the stepsize can be independent of more problem parameters without
[Figure]
[Figure]
sacriﬁcing the bound.
A bit on the positive side is that, it sufﬁces for our zeroth-order methods toknow an estimate of the upper bound of Θ(∆) , which can be done in certain application problems.
Moreover, we highlight that δ > 0 is the desired tolerance in our setting.
In fact, ( δ, (cid:15) ) -Goldstein stationarity (see Deﬁnition 2.3) relaxes (cid:15) -Clarke stationarity and our methods pursue an ( δ, (cid:15) ) -stationarypoint since ﬁnding an (cid:15)
-Clarke point is intractable.
This is different from smooth optimization where(cid:15) -Clarke stationarity reduces to ∇ f ( x ) ≤ (cid:15) and becomes tractable.
In this context, the existing zerothorder methods are designed to pursue an (cid:15) -stationary point.
Notably, a ( δ, (cid:15) ) -Goldstein stationarypoint is provably an (cid:15) -stationary point in smooth optimization if we choose δ that relies on d and (cid:15) .
We conduct numerical experiments to validate the effectiveness of our proposed methods.
In particular,we evaluate the performance of our two-phase version of SGFM (Algorithm 4) on the task of imageclassiﬁcation using convolutional neural networks (CNNs) with ReLU activations.
All the experimentsare implemented using PyTorch [73] on a workstation with a 2.6 GHz Intel Core i7 and 16GB memory.
Figure 1 summarizes the numerical results on the performance of SGD, A DAGRAD , Adagrad, A DAM , INDG [84], and our method 2-SGFM with δ = 0 . 1 and B = 200 .
Notably, 2-SGFM is comparableto other gradient-based methods in terms of training/test accuracy/loss even though it only use thefunction values.
This demonstrates the potential value of our methods since the gradient-basedmethods are not applicable in many real-world application problems as mentioned before.
Figure 2aand 2b presents the effect of batch size B ≥ 1 in 2-SGFM; indeed, the larger value of B leads tobetter performance and this accords with Theorem 3.6.
We also compare the performance of SGDand 2-SGFM with different choices of η .
From Figure 2c and 2d, we see that SGD and 2-SGFMachieve similar performance in the early stage and converge to solutions with similar quality.
Conclusion
We proposed and analyzed a class of deterministic and stochastic gradient-free methods for optimizinga Lipschitz function.
Based on the relationship between the Goldstein subdifferential and uniformsmoothing that we have established, the proposed GFM and SGFM are proved to return a ( δ, (cid:15) ) - Goldstein stationary point at an expected rate of O ( d 3 / 2 δ − 1 (cid:15) − 4 ) .
We obtain a large-deviationguarantee and improve it by combining GFM and SGFM with a two-phase scheme.
Experiments ontraining neural networks with the MNIST and CIFAR10 datasets demonstrate the effectiveness ofour methods.
Future directions include the theory for non-Lipschitz and nonconvex optimization [11]and applications of our methods to deep residual neural network (ResNet) [47] and deep denseconvolutional network (DenseNet) [50].
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[No] Our workdoes not have any potential negative societal impacts
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] You can ﬁnd thecomplete proofs in the supplementary material.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] We run the experiment on the
workstation with any advanced computing sources.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]