Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimization
Abstract
Nonsmooth nonconvex optimization problems broadly emerge in machine learningand business decision making, whereas two core challenges impede the development of efﬁcient solution methods with ﬁnite-time convergence guarantee: the lackof computationally tractable optimality criterion and the lack of computationallypowerful oracles.
The contributions of this paper are two-fold.
First, we establishthe relationship between the celebrated Goldstein subdifferential [45] and uniformsmoothing, thereby providing the basis and intuition for the design of gradient-freemethods that guarantee the ﬁnite-time convergence to a set of Goldstein stationarypoints.
Second, we propose the gradient-free method (GFM) and stochastic GFMfor solving a class of nonsmooth nonconvex optimization problems and prove thatboth of them can return a ( δ, (cid:15) )
-Goldstein stationary point of a Lipschitz functionf at an expected convergence rate at O ( d 3 / 2 δ − 1 (cid:15) − 4 ) where d is the problem dimension.
Two-phase versions of GFM and SGFM are also proposed and proven toachieve improved large-deviation results.
Finally, we demonstrate the effectivenessof 2-SGFM on training ReLU neural networks with the M INST dataset.
Introduction
Many of the recent real-world success stories of machine learning have involved nonconvex optimization formulations, with the design of models and algorithms often being heuristic and intuitive.
Thusa gap has arisen between theory and practice.
Attempts have been made to ﬁll this gap for differentlearning methodologies, including the training of multi-layer neural networks [24], orthogonal tensordecomposition [40], M-estimators [59, 60], synchronization and MaxCut [6, 62], smooth semideﬁniteprogramming [14], matrix sensing and completion [10, 41], robust principal component analysis(RPCA)
[42] and phase retrieval [75, 60].
For an overview of nonconvex optimization formulationsand the relevant ML applications, we refer to a recent survey [48].
It is generally intractable to compute an approximate global minimum [65] or to verify whether a pointis a local minimum or a high-order saddle point [63].
Fortunately, the notion of approximate stationarypoint gives a reasonable optimality criterion when the objective function f is smooth; the goal here isto ﬁnd a point x ∈ R d such that (cid:107)∇ f ( x ) (cid:107) ≤ (cid:15) .
Recent years have seen rapid algorithmic developmentthrough the lens of nonasymptotic convergence rates to (cid:15) -stationary points [66, 43, 44, 19, 20, 50].
Another growing body of work establishes algorithm-independent lower bounds for ﬁnding stationarypoints [21, 22, 3, 4].
Relative to its smooth counterpart, the investigation of nonsmooth optimization is relatively scarce,particularly in the nonconvex setting, both in terms of efﬁcient algorithms and ﬁnite-time convergenceguarantees.
Yet, over several decades, nonsmooth nonconvex optimization formulations have foundapplications in many ﬁelds.
A typical example is the training multi-layer neural networks with ReLUneurons, for which the piecewise linear activation functions induce nonsmoothness.
Another example arises in controlling ﬁnancial risk for asset portfolios or optimizing customer satisfaction in servicesystems or supply chain systems.
Here, the nonsmoothness arises from the payoffs of ﬁnancialderivatives and supply chain costs, e.g., options payoffs [37] and supply chain overage/underagecosts [74].
These applications make signiﬁcant demands with respect to computational feasibility,and the design of efﬁcient algorithms for solving nonsmooth nonconvex optimization problems hasmoved to the fore [61, 29, 27, 79, 11, 30].
The key challenges lie in two aspects: (i) the lack of a computationally tractable optimality criterion,and (ii) the lack of computationally powerful oracles.
More speciﬁcally, in the classical setting wherethe function f is Lipschitz, we can deﬁne (cid:15) -stationary points based on the celebrated notion of Clarkstationarity [25].
However, the value of such a criterion has been called into question by Zhanget al.
[79], who show that no ﬁnite-time algorithm guarantees (cid:15) -stationarity when (cid:15) is less than aconstant.
Further, the computation of the gradient is impossible for many application problems andwe only have access to a noisy function value at each point.
This is a common issue in the context ofsimulation optimization [64, 46]; indeed, the objective function value is often achieved as the outputof a black-box or complex simulator, for which the simulator does not have the infrastructure neededto effectively evaluate gradients; see also Ghadimi and Lan [43] and Nesterov and Spokoiny [68] forcomments on the lack of gradient evaluation in practice.
Contribution.
In this paper, we propose and analyze a class of deterministic and stochastic gradientfree methods for nonsmooth nonconvex optimization problems in which we only assume that thefunction f is Lipschitz.
Our contributions can be summarized as follows.
• We establish a relationship between the Goldstein subdifferential and uniform smoothing via appeal to the hyperplane separation theorem.
This result provides the basis for algorithmic design and ﬁnite-time convergence analysis of gradient-free methods to ( δ, (cid:15) ) -Goldstein stationary points.
• We propose and analyze a gradient-free method (GFM) and stochastic GFM for solving aclass of nonsmooth nonconvex optimization problems.
Both of these methods are guaranteedto return a ( δ, (cid:15) )
-Goldstein stationary point of a Lipschitz function f : R d (cid:55)→ R with anexpected convergence rate of O ( d 3 / 2 δ − 1 (cid:15) − 4 ) where d ≥ 1 is the problem dimension.
Further, we propose the two-phase versions of GFM and SGFM.
As our goal is to return a( δ, (cid:15) ) -Goldstein stationary point with user-speciﬁed high probability 1 − Λ , we prove that the two-phase version of GFM and SGFM can improve the dependence from (1 / Λ) 4 tolog(1 / Λ) in the large-deviation regime.
Related works.
Our work is related to a line of literature on gradient-based methods for nonsmoothnonconvex optimization and gradient-free methods for smooth nonconvex optimization.
Due to spacelimitations, we defer our comments on the former topic to Appendix A. In the context of gradient-free methods, the basic idea is to approximate a full gradient using either aone-point estimator [38] or a two-point estimator [1, 43, 36, 71, 68], where the latter approach achievesa better ﬁnite-time convergence guarantee.
Despite the meteoric rise of two-point-based gradient-freemethods, most of the work is restricted to convex optimization [36, 71, 77] and smooth and nonconvexoptimization [68, 43, 57, 58, 23, 49, 47].
For nonsmooth and convex optimization, the best upperbound on the global rate of convergence is O ( d(cid:15) − 2 ) [71] and this matches the lower bound [36].
For smooth and nonconvex optimization, the best global rate of convergence is O ( d(cid:15) − 2 ) [68] and
O ( d(cid:15) − 4 ) if we only have access to noisy function value oracles [43].
Additional regularity conditions,e.g., a ﬁnite-sum structure, allow us to leverage variance-reduction techniques [58, 23, 49] and thebest known result is O ( d 3 / 4 (cid:15) − 3 )
However, none of these gradient-free methods have beendeveloped for nonsmooth nonconvex optimization and the only gradient-free method we are aware offor the nonsmooth is summarized in Nesterov and Spokoiny [68, Section 7].
Preliminaries and Technical Background
In this section, we provide formal deﬁnitions for the class of Lipschitz functions considered in thispaper, and the deﬁnitions for generalized gradients and the Goldstein subdifferential that lead tooptimality conditions in nonsmooth nonconvex optimization.
2.1 Function classes
Imposing regularity on the functions to be optimized is necessary for obtaining optimization algorithms with ﬁnite-time convergence guarantees [67].
In the context of nonsmooth optimization thereare two types of regularity conditions: Lipschitz properties of function values and bounds on functionvalues.
We ﬁrst list several equivalent deﬁnitions of Lipschitz continuity.
A function f : R d (cid:55)→ R is said tobe L - Lipschitz if for every x ∈ R d and the direction v ∈ R d with (cid:107) v (cid:107) ≤ 1 , the directional projectionf x , v ( t ) :
= f ( x + t v ) deﬁned for t ∈ R satisﬁes
[Equation]
Equivalently, f is L -Lipschitz if for every x , x (cid:48) ∈ R d , we have
[Equation]
Further, the function value bound f ( x 0 ) − inf x ∈ R d f ( x ) appears in complexity guarantees for smoothand nonconvex optimization problems [67] and is often assumed to be bounded by a positive constant∆ > 0 .
Note that x 0 is a prespeciﬁed point (i.e., an initial point for an algorithm) and we simply ﬁxit for the remainder of this paper.
We now deﬁne the following classes of functions which will beconsidered in this paper.
Deﬁnition 2.1 Suppose that ∆ > 0 and L > 0 are both independent of the problem dimension d ≥ 1 .
Then, we denote F d (∆ , L ) as the set of L -Lipschitz functions f : R d (cid:55)→ R with the bounded functionvalue f ( x 0 ) − inf x ∈ R d f ( x ) ≤ ∆ .
The function class F d (∆ , L ) includes Lipschitz functions on R d and is thus different from thenonconvex function class considered in the literature [43, 68].
First, we do not impose a smoothnesscondition on the function f ∈ F d (∆ , L ) , in contrast to the nonconvex functions studied in Ghadimiand Lan [43] which are assumed to have Lipschitz gradients.
Second, Nesterov and Spokoiny [68,
Section 7] presented a complexity bound for a randomized optimization method for minimizing anonsmooth nonconvex function.
However, they did not clarify why the norm of the gradient of theapproximate function f ¯ µ of the order δ (we use their notation) serves as a reasonable optimalitycriterion in nonsmooth nonconvex optimization.
They also assume an exact function value oracle,ruling out many interesting application problems in simulation optimization and machine learning.
In contrast, our goal is to propose fast gradient-free methods for nonsmooth nonconvex optimizationin the absence of an exact function value oracle.
In general, the complexity bound of gradient-freemethods will depend on the problem dimension d ≥ 1 even when we assume that the function tobe optimized is convex and smooth [36, 71].
As such, we should consider a function class with agiven dimension d ≥ 1 .
In particular, we consider a optimality criterion based on the celebrated
Goldstein subdifferential [45] and prove that the number of function value oracles required by ourdeterministic and stochastic gradient-free methods to ﬁnd a ( δ, (cid:15) )
-Goldstein stationary point off ∈ F d (∆ , L ) is O ( poly ( d, L, ∆ , 1 /(cid:15), 1 /δ ))
when δ, (cid:15) ∈ (0 , 1) are constants (see the deﬁnition of
Goldstein stationarity in the next subsection).
It is worth mentioning that F d (∆ , L ) contains a rather broad class of functions used in real-worldapplication problems.
Typical examples with additional regularity properties include Hadamard semidifferentiable functions [72, 31, 79], Whitney-stratiﬁable functions [12, 29], o -minimally deﬁnablefunctions [26] and a class of semi-algebraic functions [5, 29].
Thus, our gradient-free methods canbe applied for solving these problems with ﬁnite-time convergence guarantees.
2.2 Generalized gradients and Goldstein subdifferential
We start with the deﬁnition of generalized gradients [25] for nondifferentiable functions.
This isperhaps the most standard extension of gradients to nonsmooth and nonconvex functions.
Deﬁnition 2.
Given a point x ∈ R d and a direction v ∈ R d , the generalized directional derivativeof a nondifferentiable function f is given by Df ( x ; v ) :=
lim sup y
→ x ,t ↓ 0 f
( y + t v ) − f ( y ) t .
Then, thegeneralized gradient of f is deﬁned as a set ∂f ( x ) :
= { g ∈ R d : g (cid:62) v ≤ Df ( x ; v ) , ∀ v ∈ R d } .
Rademacher’s theorem guarantees that any Lipschitz function is almost everywhere differentiable.
This implies that the generalized gradients of Lipschitz functions have additional properties and we can deﬁne them in a relatively simple way.
The following proposition summarizes these results; werefer to [25] for the proof details.
Proposition 2.1 Suppose that f is L -Lipschitz for some L > 0 , we have that ∂f ( x ) is a nonempty,convex and compact set and (cid:107) g (cid:107) ≤ L for all g ∈ ∂f ( x ) .
Further, ∂f ( · ) is an upper-semicontinuousset-valued map.
Moreover, a generalization of mean-value theorem holds: for any x 1 , x 2 ∈ R d , thereexist λ ∈ (0 , 1) and
g ∈ ∂f ( λ x 1 + (1 − λ ) x 2 ) such that f ( x 1 ) − f ( x 2 ) =
g (cid:62) ( x 1 − x 2 ) .
Finally,there is a simple way to represent the generalized gradient ∂f ( x ) :
[Equation]
which is the convex hull of all limit points of ∇ f ( x k ) over all sequences x 1 , x 2 , . .
of differentiablepoints of f ( · ) which converge to x .
Given this deﬁnition of generalized gradients, a Clarke stationary point of f is a point x satisfying0 ∈ ∂f ( x ) .
Then, it is natural to ask if an optimization algorithm can reach an (cid:15) -stationary point witha ﬁnite-time convergence guarantee.
Here a point x ∈ R d is an (cid:15) -Clarke stationary point ifmin {(cid:107) g
(cid:107) :
g ∈ ∂f ( x ) } ≤ (cid:15).
This question has been addressed by [79, Theorem 1], who showed that ﬁnding an (cid:15) -Clarke stationarypoints in nonsmooth nonconvex optimization can not be achieved by any ﬁnite-time algorithm givena ﬁxed tolerance (cid:15) ∈ [0 , 1) .
One possible response is to consider a relaxation called a near (cid:15) -Clarkestationary point.
Consider a point which is δ -close to an (cid:15) -stationary point for some δ > 0 .
A pointx ∈ R d is near (cid:15) -stationary
if the following statement holds true:
[Equation]
Unfortunately, however, [55, Theorem 1] demonstrated that it is impossible to obtain worst-caseguarantees for ﬁnding a near (cid:15)
-Clarke stationary point of f ∈ F d (∆ , L ) when (cid:15), δ > 0 are smallerthan some certain constants unless the number of oracle calls has an exponential dependence onthe problem dimension d ≥ 1 .
These negative results suggest a need for rethinking the deﬁnitionof targeted stationary points.
We accordingly propose to consider the reﬁned notion of Goldsteinsubdifferential.
Deﬁnition 2.
Given a point x ∈ R d and δ > 0 , the δ -Goldstein subdifferential of a Lipschitzfunction f at x is given by ∂
δ f ( x ) := conv ( ∪ y ∈ B
δ ( x ) ∂f ( y )) .
The Goldstein subdifferential of f at x is the convex hull of the union of all generalized gradients atpoints in a δ -ball around x .
Accordingly, we can deﬁne the ( δ, (cid:15) )
-Goldstein
stationary points; that is,a point x ∈ R d is a ( δ, (cid:15) )
-Goldstein
stationary point if the following statement holds:
[Equation]
It is worth mentioning that ( δ, (cid:15) ) -Goldstein stationarity is a weaker notion than (near) (cid:15) -Clarkestationarity since any (near) (cid:15) -stationary
point is a ( δ, (cid:15) )
-Goldstein stationary point but not viceversa.
However, the converse holds true under a smoothness condition [79, Proposition 6] andlim δ ↓
∂
δ f ( x ) =
∂f ( x ) holds as shown in Zhang et al.
[79, Lemma 7].
The latter result also enablesan intuitive framework for transforming nonasymptotic analysis of convergence to ( δ, (cid:15) )
-Goldsteinstationary points to classical asymptotic results for ﬁnding (cid:15) -Clarke stationary points.
Thus, weconclude that ﬁnding a ( δ, (cid:15) )
-Goldstein stationary point is a reasonable optimality condition forgeneral nonsmooth nonconvex optimization.
Remark 2.2 Finding a ( δ, (cid:15) )
-Goldstein stationary point in nonsmooth nonconvex optimization hasbeen formally shown to be computationally tractable in an oracle models [79, 30].
More speciﬁcally, [45] discovered that one can decrease the function value of a Lipschitz f by using the minimalnorm element of ∂ δ f ( x ) and this leads to a deterministic normalized subgradient method which ﬁndsa ( δ, (cid:15) )
-Goldstein stationary point within O ( ∆ δ(cid:15) ) iterations.
However, Goldstein’s algorithm is onlyconceptual since it is computationally intractable to return an exact minimal-norm element of ∂ δ f ( x ) .
Very recently, [79] and [30] have proposed randomized variants of Goldstein’s algorithm and givena convergence guarantee of O ( ∆ L 2 δ(cid:15) 3 ) .
However, it remains unknown if gradient-free methods ﬁnd a( δ, (cid:15) ) -Goldstein stationary point of a Lipschitz function f within O ( poly ( d, L, ∆ , 1 /(cid:15), 1 /δ )) iterationsin the absence of an exact function value oracle.
Note that the dependence on the problem dimensiond ≥ 1 is necessary for gradient-free methods as mentioned before.
2.3 Randomized smoothing
The randomized smoothing approaches are simple and work equally well for convex and nonconvexfunctions.
Formally, given the L -Lipschitz function f (possibly nonsmooth nonconvex) and adistribution P , we deﬁne f δ ( x ) =
E u ∼ P [ f ( x + δ u )] .
In particular, letting P be a standard Gaussiandistribution, the function f δ is a δL √ d -approximation of f ( · ) and the gradient ∇ f δ is L √ d
δ -Lipschitzwhere d ≥ 1 is the problem dimension; see Nesterov and Spokoiny [68, Theorem 1 and Lemma 2].
Letting P be an uniform distribution on an unit ball in (cid:96) 2 -norm, the resulting function f δ is a δLapproximation of f ( · ) and ∇ f δ is also cL √ d δ -Lipschitz where d ≥ 1 is the problem dimension;see
Youseﬁan et al.
[78, Lemma 8] and Duchi et al.
[35, Lemma E.2].
Proposition 2.3 Let f δ ( x ) =
E u ∼ P [ f ( x + δ u )] where P is an uniform distribution on an unit ball in(cid:96) 2 -norm.
Assuming that f is L -Lipschitz, we have (i) | f δ ( x ) − f ( x ) | ≤ δL , and (ii) f
δ is differentiableand L -Lipschitz with the cL √ d δ -Lipschitz gradient where c > 0 is a constant.
In addition, there existsa function f for which each of the above bounds are tight—they cannot be improved by more than aconstant factor—simultaneously.
The randomized smoothing approaches form the basis for developing gradient-free methods [38, 1,2, 43, 68].
Given an access to function values of f , we can compute an unbiased estimate of thegradient of f δ and plug them into stochastic gradient-based methods.
Note that the Lipschitz constantof f δ depends on the problem dimension d ≥ 1 with at least a factor of √ d for many randomizedsmoothing approaches [55, Theorem 2].
This is consistent with the lower bounds for all gradient-freemethods in convex and strongly convex optimization [36, 71].
Main Results
In this section, we establish a relationship between the Goldstein subdifferential and the uniformsmoothing approach.
We propose a gradient-free method (GFM), its stochastic variant (SGFM),and a two-phase version of GFM and SGFM.
We analyze these algorithms using the Goldsteinsubdifferential; we provide the global rate and large-deviation estimates in terms of ( δ, (cid:15) ) -Goldsteinstationarity.
3.1 Linking Goldstein subdifferential to uniform smoothing
Recall that ∂
δ f and f δ are deﬁned by
[Equation]
and f δ ( x ) =
E u ∼ P [ f ( x + δ u )] .
It is clear that f is almost everywhere differentiable since f is L -Lipschitz.
This implies that ∇ f δ ( x ) =
E u ∼ P [ ∇ f ( x + δ u )] and demonstrates that ∇ f δ ( x ) can beviewed intuitively as a convex combination of ∇ f ( z ) over an inﬁnite number of points z ∈ B δ ( x ) .
As such, it is reasonable to conjecture that ∇ f δ ( x ) ∈ ∂
δ f ( x ) for any x ∈ R d .
However, the aboveargument is not a rigorous proof; indeed, we need to justify why ∇ f δ ( x ) =
E u ∼ P [ ∇ f ( x + δ u )] iff is almost everywhere differentiable and generalize the idea of a convex combination to includeinﬁnite sums.
To resolve these issues, we exploit a toolbox due to Rockafellar and Wets [70].
In the following theorem, we summarize our result and refer to Appendix C for the proof details.
Theorem 3.1 Suppose that f is L -Lipschitz and let f δ ( x ) =
E u ∼ P [ f ( x + δ u )] , where P is anuniform distribution on a unit ball in (cid:96) 2 -norm and let ∂ δ f be a δ -Goldstein subdifferential of f (cf.
Deﬁnition 2.3).
Then, we have ∇ f δ ( x ) ∈ ∂
δ f ( x ) for any x ∈ R d .
Theorem 3.1 resolves an important question and forms the basis for analyzing our gradient-freemethods.
Notably, our analysis can be extended to justify other randomized smoothing approachesin nonsmooth nonconvex optimization.
For example, Nesterov and Spokoiny [68] used Gaussiansmoothing and estimated the number of iterations required by their methods to output ˆ x ∈ R dsatisfying (cid:107)∇ f δ (ˆ x ) (cid:107) ≤ (cid:15) .
By modifying the proof of Theorem 3.1 and Zhang et al.
[79, Lemma 7], wecan prove that ∇ f δ belongs to Goldstein subdifferential with Gaussian weights and this subdifferentialconverges to the Clark subdifferential as δ → 0 .
Compared to uniform smoothing and the original
Goldstein subdifferential, the proof for Gaussian smoothing is quite long and technical [68, Page554], and adding Gaussian weights seems unnatural in general.
3.2 Gradient-free methods
We analyze a gradient-free method (GFM) and its two-phase version (2-GFM) for optimizing a
Lipschitz function f .
Due to space limitations, we defer the proof details to Appendix D.
Global rate estimation.
Let f : R d (cid:55)→ R be a L -Lipschitz function and the smooth version of fis then the function f δ = E u ∼ P
[ f ( x + δ u )] where P is an uniform distribution on an unit ball in(cid:96) 2 -norm.
Equipped with Lemma 10 from Shamir [71], we can compute an unbiased estimator forthe gradient ∇ f δ ( x t ) using function values.
This leads to the gradient-free method (GFM) in Algorithm 1 that simply performs a one-stepgradient descent to obtain x t .
It is worth mentioning that we use a random iteration count R toterminate the execution of Algorithm 1 and this will guarantee that GFM is valid.
Indeed, we onlyderive that min t =1 , 2 ,...,T (cid:107)∇ f δ ( x t ) (cid:107) ≤ (cid:15) in the theoretical analysis (see also Nesterov and Spokoiny[68, Section 7]) and ﬁnding the best solution from { x 1 , x 2 , . . .
, x T } is difﬁcult since the quantity(cid:107)∇ f δ ( x t ) (cid:107) is unknown.
To estimate them using Monte Carlo simulation would incur additionalapproximation errors and raise some reliability issues.
The idea of random output is not new but hasbeen used by Ghadimi and Lan [43] for smooth and nonconvex stochastic optimization.
Such schemealso gives us a computational gain with a factor of two in expectation.
Theorem 3.2 Suppose that f is L -Lipschitz and let δ > 0 and 0 < (cid:15) < 1 .
Then, there exists some
T > 0 such that the output of Algorithm 1 with η = 110 (cid:113) δ (∆+ δL ) cd 3 / 2 L 3 T satisﬁes that E [min {(cid:107) g (cid:107) : g ∈∂ δ f ( x R ) } ] ≤ (cid:15) and the total number of calls of the function value oracle is bounded by
[Equation]
where d ≥ 1 is the problem dimension, L > 0 is the Lipschtiz parameter of f and ∆ > 0 is an upperbound for the initial objective function gap, f ( x 0 ) − inf x ∈ R d
f ( x ) > 0 .
Remark 3.3 Theorem 3.2 illustrates the difference between gradient-based and gradient-free methods in nonsmooth nonconvex optimization.
Indeed, Davis et al.
[30] has recently proved the rate of
O ( δ − 1 (cid:15) − 3 ) for a randomized gradient-based method in terms of ( δ, (cid:15) ) -Goldstein stationarity.
Further,
Theorem 3.2 demonstrates that nonsmooth nonconvex optimization is likely to be intrinsically harderthan all other standard settings.
More speciﬁcally, the state-of-the-art rate for gradient-free methodsis O ( d(cid:15) − 2 ) for nonsmooth convex optimization in terms of objective function value gap [36] andsmooth nonconvex optimization in terms of gradient norm [68].
Thus, the dependence on d ≥ 1 islinear in their bounds yet d 32 in our bound.
We believe it is promising to either improve the rate ofgradient-free methods or show the impossibility by establishing a lower bound.
Large-deviation estimation.
While Theorem 3.2 establishes the expected convergence rate overmany runs of Algorithm 1, we are also interested in the large-deviation properties for a single run.
Indeed, we hope to establish a complexity bound for computing a ( δ, (cid:15), Λ) -solution ; that is, a pointx ∈ R d satisfying Prob (min {(cid:107) g
(cid:107) : g ∈ ∂
δ f ( x ) } ≤ (cid:15) ) ≥ 1 − Λ for some δ > 0 and 0 < (cid:15), Λ < 1 .
By Theorem 3.
and Markov’s inequality,
[Equation]
for all λ > 0 , we conclude that the total number of calls of the function value oracle is bounded by
[Equation]
ξ
− F x
− δ w ξ t
T − 1 } is uniformly sampled.
This complexity bound is rather pessimistic in terms of its dependence on Λ which is often set to besmall in practice.
To improve the bound, we combine Algorithm 1 with a post-optimization procedureproposed by Ghadimi and Lan [43], leading to a two-phase gradient-free method (2-GFM), shown in Algorithm 2.
Theorem 3.4 Suppose that f is L -Lipschitz and let δ > 0 and 0 < (cid:15), Λ < 1 .
Then, there exists some
T, S, B > 0 such that the output of Algorithm 2 with η = 110 (cid:113) δ (∆+ δL ) cd 3 / 2 L 3 T satisﬁes that Prob (min {(cid:107) g (cid:107) :g ∈ ∂
δ f (¯ x s (cid:63) ) } ] ≥ (cid:15) ) ≤ Λ and the total number of calls of the function value oracle is bounded by
[Equation]
, where d ≥ 1 is the problem dimension, L > 0 is the Lipschtiz parameter of f and ∆ > 0 is an upperbound for the initial objective function gap, f ( x 0 ) − inf x ∈ R d
f ( x ) > 0 .
Clearly, the bound in Theorem 3.4 is signiﬁcantly smaller than the corresponding one in Eq.
(3.1) interms of the dependence on 1 / Λ , demonstrating the power of the post-optimization phase.
3.3 Stochastic gradient-free methods
We turn to the analysis of a stochastic gradient-free method (SGFM) and its two-phase version(2-SGFM) for optimizing a Lipschitz function f ( · ) =
E ξ ∈ P µ [ F ( · , ξ )] .
Global rate estimation.
In contrast to minimizing a deterministic function f , we only have accessto the noisy function value F ( x , ξ ) at any point x ∈ R d where a data sample ξ is drawn from adistribution P µ .
Intuitively, this is a more challenging setup.
It has been studied before in the settingof optimizing a nonsmooth convex function [36, 68] or a smooth nonconvex function [43].
As in thesefoundational papers, we assume throughout that (i) F ( · , ξ ) is L ( ξ ) -Lipschitz with E ξ ∈ P µ [ L 2 ( ξ )] ≤ G 2for some G > 0 and (ii) E
[ F ( x , ξ t )]
= f
( x ) for all x ∈ R d where ξ t is simulated from P µ at the t thiteration.
Despite the noisy function value, we can compute an unbiased estimator of the gradient ∇ f δ ( x t ) ,where f δ = E u ∼ P [ f ( x + δ u )]
=
E u ∼ P ,ξ ∈ P µ [ F ( x + δ u , ξ )] .
In particular, we have ˆ g t =
d 2 δ ( F ( x t + δ w t ) − F ( x t − δ w t ξ t ))
w t .
Clearly, under our assumption, we have
[Equation]
where g t is deﬁned in Algorithm 1.
However, the variance of the estimator ˆ g t can be undesirablylarge since F ( · , ξ ) is L ( ξ ) -Lipschitz for a (possibly unbounded) random variable L ( ξ ) > 0 .
Toresolve this issue, we revisit Shamir [71, Lemma 10] and show that in deriving an upper bound for 293 E u ∼ P ,ξ ∈ P µ [ (cid:107) ˆ g t (cid:107) 2 ] it sufﬁces to assume that E ξ ∈ P µ [ L 2 ( ξ )] ≤ G 2 for some constant G > 0 .
Theresulting bound achieves a linear dependence in the problem dimension d > 0 which is the sameas in Shamir [71, Lemma 10].
Note that the setup with convex and L ( ξ ) -Lipschitz functions F ( · , ξ )has been considered in Duchi et al.
However, our estimator is different from their estimator ofˆ g t = dδ ( F ( x t + δ w t , ξ t ) − F ( x t , ξ t ))
w t which essentially suffers from the quadratic dependence ind > 0 .
Further, it is necessary to employ a random iteration count R to terminate the execution of Algorithm 3.
Theorem 3.5 Suppose that F ( · , ξ ) is L ( ξ ) -Lipschitz with E ξ ∈ P µ [ L 2 ( ξ )] ≤ G 2 for some G > 0 andlet δ > 0 and 0 < (cid:15) < 1 .
Then, there exists some T > 0 such that the output of Algorithm 3 withη = 110 (cid:113) δ (∆+ δG ) cd 3 / 2 G 3 T satisﬁes
that E [min {(cid:107) g (cid:107) : g ∈ ∂
δ f ( x R ) } ] ≤ (cid:15) and the total number of calls ofthe noisy function value oracle is bounded by
[Equation]
where d ≥ 1 is the problem dimension, L > 0 is the Lipschtiz parameter of f and ∆ > 0 is an upperbound for the initial objective function gap, f ( x 0 ) − inf x ∈ R d
f ( x ) > 0 .
In the stochastic setting, the gradient-based method achieves the rate of O ( δ − 1 (cid:15) − 4 ) for a randomizedgradient-based method in terms of ( δ, (cid:15) ) -Goldstein stationarity [30].
As such, our bound in Theorem 3.5 is tight up to the problem dimension d ≥ 1 .
Further, the state-of-the-art rate for stochasticgradient-free methods is O ( d(cid:15) − 2 ) for nonsmooth convex optimization in terms of objective functionvalue gap [36] and O ( d(cid:15) − 4 ) for smooth nonconvex optimization in terms of gradient norm [43].
Thus, Theorem 3.5 demonstrates that nonsmooth nonconvex stochastic optimization is essentially themost difﬁcult one among than all these standard settings.
Large-deviation estimation.
As in the case of GFM, we hope to establish a complexity bound of
SGFM for computing a ( δ, (cid:15), Λ) -solution.
By Theorem 3.5 and Markov’s inequality, we obtain thatthe total number of calls of the noisy function value oracle is bounded by
[Equation]
We also propose a two-phase stochastic gradient-free method (2-SGFM) in Algorithm 4 by combining
Algorithm 3 with a post-optimization procedure.
[Figure] [Figure] Theorem 3.6 Suppose that F ( · , ξ ) is L ( ξ ) -Lipschitz with E ξ ∈ P µ [ L 2 ( ξ )] ≤ G 2 for some G > 0 andlet δ > 0 and 0 < (cid:15), Λ < 1 .
Then, there exists some T, S, B > 0 such that the output of Algorithm 4with η = 110 (cid:113) δ (∆+ δG ) cd 3 / 2 G 3 T satisﬁes that Prob (min {(cid:107) g (cid:107) : g ∈ ∂
δ f (¯ x s (cid:63) ) } ] ≥ (cid:15) ) ≤ Λ and the totalnumber of calls of the noisy function value oracle is bounded by O
[Equation]
, where d ≥ 1 is the problem dimension, L > 0 is the Lipschtiz parameter of f and ∆ > 0 is an upperbound for the initial objective function gap f ( x 0 ) − inf x ∈ R d
f ( x ) > 0 .
Experiment
In this section, we conduct numerical experiments to validate the effectiveness of our proposedmethods.
In particular, we evaluate the performance of our two-phase version of SGFM (Algorithm 4)on the task of image classiﬁcation using convolutional neural networks (CNNs) with ReLU activations.
All the experiments are implemented using PyTorch [69] on a workstation with a 2.6 GHz Intel Corei7 and 16GB of memory.
Figure 1 summarizes the numerical results on the performance of SGD, A DAGRAD , Adagrad, A DAMand our 2-SGFM with δ = 0 . 1 and B = 200 .
Notably, we ﬁnd that 2-SGFM is comparable to othergradient-based methods in terms of training/test accuracy/loss even though it only use the functionvalues during the process of optimization.
This demonstrates the potential value of our methods sincethe gradient-based methods are not applicable in many real-world application problems as mentionedbefore.
Figure 2a and 2b presents the effect of batch size B ≥ 1 in 2-SGFM; indeed, the largervalue of B leads to better performance and this accords with Theorem 3.6.
We also compare theperformance of SGD and 2-SGFM with different choices of η .
From Figure 2c and 2d, we see that SGD and 2-SGFM achieve similar performance in the early stage and converge to solutions withsimilar quality.
Conclusion
We have proposed and analyzed a class of deterministic and stochastic gradient-free methods foroptimizing a Lipschitz function.
Based on the relationship between the Goldstein subdifferentialand uniform smoothing that we have established, we prove that the proposed GFM and SGFMreturn a ( δ, (cid:15) ) -Goldstein stationary point at an expected rate of O ( d 3 / 2 δ − 1 (cid:15) − 4 ) .
We also obtain alarge-deviation guarantee and improve it by combining GFM and SGFM with a two-phase scheme.
Experiments on training neural networks with the MNIST dataset and the numerical results demonstrate the effectiveness of our methods.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[No] Our workdoes not have any potential negative societal impacts
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] You can ﬁnd thecomplete proofs in the supplementary material.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No] We run the experiment on the
workstation with any advanced computing sources.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]