Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivity
Abstract ` 0 constrained optimization is prevalent in machine learning, particularly for highdimensional problems, because it is a fundamental approach to achieve sparselearning.
Hard-thresholding gradient descent is a dominant technique to solvethis problem.
However, ﬁrst-order gradients of the objective function may beeither unavailable or expensive to calculate in a lot of real-world problems, wherezeroth-order (ZO) gradients could be a good surrogate.
Unfortunately, whether ZOgradients can work with the hard-thresholding operator is still an unsolved problem.
To solve this puzzle, in this paper, we focus on the ` 0 constrained black-boxstochastic optimization problems, and propose a new stochastic zeroth-order gradient hard-thresholding (SZOHT) algorithm with a general ZO gradient estimatorpowered by a novel random support sampling.
We provide the convergence analysisof SZOHT under standard assumptions.
Importantly, we reveal a conﬂict betweenthe deviation of ZO estimators and the expansivity of the hard-thresholding operator, and provide a theoretical minimal value of the number of random directionsin ZO gradients.
In addition, we ﬁnd that the query complexity of SZOHT isindependent or weakly dependent on the dimensionality under different settings.
Finally, we illustrate the utility of our method on a portfolio optimization problemas well as black-box adversarial attacks.
Introduction
` 0 constrained optimization is prevalent in machine learning, particularly for high-dimensionalproblems, because it is a fundamental approach to achieve sparse learning.
In addition to improvingthe memory, computational and environmental footprint of the models, these sparse constraintshelp reduce overﬁtting and obtain consistent statistical estimation [42, 4, 29, 26].
We formulate theproblem as follows:
[Equation]
where f ( · , ⇠ ) : R d ! R is a differentiable function and ⇠ is a noise term, for instance related to an 25 underlying ﬁnite sum structure in f , of the form: E ⇠ f ( x , ⇠ ) = 1 n
P ni =
f i ( x ) .
Hard-thresholding 26 gradient algorithm [15, 28, 41] is a dominant technique to solve this problem.
It generally consists inalternating between a gradient step, and a hard-thresholding operation which only keeps the k -largestcomponents (in absolute value) of the current iterate.
The advantage of hard-thresholding over itsconvex relaxations ([35, 37]) is that it can often attain similar precision, but is more computationallyefﬁcient, since it allows to directly ensure a desired sparsity instead of going through several valuesof an ` 1 penalty or constraint hyperparameter.
The only expensive computation in hard-thresholdingis the hard-thresholding step itself, which requires ﬁnding the top k elements of the current iterate.
Hard-thresholding was originally developed in its full gradient form [15], but has been later onextended to the stochastic setting by Nguyen et al.
[28], which developed a stochastic gradient [Table] [Table]
[Table] [Table]
descent (SGD) version of hard thresholding (StoIHT), and further more with Zhou et al.
[43], Shenand Li [32] and Li et al.
[18], which used variance reduction technique to improve upon StoIHT.
However, the ﬁrst-order gradients used in the above methods may be either unavailable or expensiveto calculate in a lot of real-world problems.
For example, in certain graphical modeling tasks [38],obtaining the gradient of the objective function is computationally hard.
Even worse, in some settings,the gradient is inaccessible by nature, for instance in bandit problems [31], black-box adversarialattacks [36, 8, 9], or reinforcement learning [30, 24, 10].
To tackle those problems, ZO optimizationmethods have been developed [27].
Those methods usually replace the inaccessible gradient by itsﬁnite difference approximation which can be computed only from function evaluations, followingthe idea that for a differentiable function f : R ! R , we have: f 0 ( x ) =
lim h {+! 
+} f ( x + h )  f ( x ) h .
Lateron, ZO methods have been adapted to solve constrained problems. These could be used to solveproblem (1), by solving its convex relaxation, that is, with an ` 1 penalty/constraint instead of the ` 0one.
To that end, Ghadimi et al.
[12], and Cai et al.
[5] introduce proximal ZO algorithms, Liu et al.[21] introduce a ZO projected gradient algorithm and Balasubramanian and Ghadimi [2] introduce a ZO conditional gradient [17] algorithm.
We provide a review of those results in Table 1.
As can beseen from the table, their query complexity is high (linear in d ), except [5] that has a complexity of O ( s log( d ) log( 1 " )) , but assumes that gradients are sparse.
In addition, those methods must introducea hyperparameter  (the strength of the ` 1 penalty) or R (the radius of the ` 1 ball), for which severalvalues need to be tested to ﬁnd the one ensuring an output of sparsity k .
Therefore, it would beinteresting to use the hard-thresholding techniques described in the previous paragraph, instead ofthose convex relaxations.
Unfortunately, ZO hard-thresholding gradient algorithms have not been exploited formally.
Evenmore, whether ZO gradients can work with the hard-thresholding operator is still an unknownproblem.
Although there was one related algorithm proposed recently by Balasubramanian and
Ghadimi [2], they did not target ` 0 constrained optimization problem and importantly have strongassumptions in their convergence analysis.
Indeed, they assume that the gradients, as well as thesolution of the unconstrained problem, are s -sparse: kr f ( x ) k 0  s and k x {+⇤ k 
 +} s ⇤ ⇡ s , where
62 x ⇤ = arg min x f ( x ) .
In addition, it was recently shown by Cai et al.
[5] that they must in factassume that the support of the gradient is ﬁxed for all x 2 X , for their convergence result to hold, 64 which is a hard limitation, since that amounts to say that the function f depends on s ﬁxed variables.
To ﬁll this gap, in this paper, we focus on the ` 0 constrained black-box stochastic optimizationproblems, and propose a novel stochastic zeroth-order gradient hard-thresholding (SZOHT) algorithm.
Speciﬁcally, we propose a dimension friendly ZO gradient estimator powered by a novel randomsupport sampling technique, and then embed it into the standard hard-thresholding operator.
Combined with the classical convergence analysis from the hard-thresholding algorithm, this proposition allows us to provide the convergence and complexity analysis of SZOHT under the standardassumptions of sparse learning, which are restricted strong smoothness (RSS), and restricted strongconvexity (RSC) [28, 32], to retain generality, therefore providing a positive answer to the questionof whether ZO gradients can work with the hard-thresholding operator.
Crucial to our analysis is toprovide carefully tuned requirements on the parameters q and k .
Finally, we illustrate the utility ofour method on a portfolio optimization problem as well as black-box adversarial attacks, by showingthat our method can achieve competitive performance in comparison to state of the art methods forsparsity-enforcing zeroth-order algorithm described in Table 1, such as [12, 2, 5].
Importantly, we also show that in the smooth case, the query complexity of SZOHT is independent ofthe dimensionality, which is signiﬁcantly different to the dimensionality dependent result for mostexisting ZO algorithms.
Indeed, it is known from Jamieson et al.
[16] that the worst case querycomplexity of ZO optimization over the class F ⌫ ,L of ⌫ -strongly convex and L -smooth functions 82 deﬁned over a convex set X is linear in d .
Our work is thus in line with other works achieving 83 dimension-insensitive query complexity in zeroth-order optimization such as [13, 33, 40, 5, 6, 2, 5,20, 16], but contrary to those, instead of making further assumptions (i.e. restricting the class F ⌫ ,L 85 to a smaller class), we bypass the impossibility result by replacing the convex feasible set X by a 86 non-convex set (the ` 0 ball), which allows us to avoid making stringent assumptions on the class offunctions f .
Contributions.
We summarize the main contributions of our paper as follows:
We propose a new algorithm SZOHT that is, up to our knowledge, the ﬁrst zeroth-order sparsityconstrained algorithm that is dimension independent under the smoothness assumption, withoutassuming any gradient sparsity.2.
We reveal an interesting conﬂict between the error from zeroth-order estimates and the hardthresholding operator, which results in a minimal value for q that is necessary to ensure.3.
We also provide the convergence analysis of our algorithm in the more general RSS setting,providing, up to our knowledge, the ﬁrst zeroth-order algorithm that can work with the usualassumptions of RSS/RSC from the hard-thresholding literature.
Preliminaries
Throughout this paper, we denote by k x k the Euclidean norm for a vector x 2 R d , by k x k 1 the 99 maximum absolute component of that vector, and by k x k 0 the ` 0 norm (which is not a proper norm).
100 For simplicity, we denote f ⇠ ( · ) :
= f ( · , ⇠ ) .
We call u F (resp.
r F f ( x ) )
the vector which sets all 101 coordinates i 62 F of u (resp.
r f ( x ) ) to 0 .
We also denote by x ⇤ the solution of problem (1) deﬁned 102 in the introduction, for some target sparsity k ⇤ which could be smaller than k .
To derive our result,we will need the following assumptions on f .
Assumption 1 ( ( ⌫ s , s )
-RSC, [15, 25, 23, 41, 18, 32, 28]) .
f is said to be ⌫ s restricted strongly convex with sparsity parameter s if it is differentiable, and there exist a generic constant ⌫ s such that for all ( x , y ) 2 R d with k x  y k 0  s :
[Equation]
Assumption 2 ( ( L s , s ) -RSS, [32, 28]) .
For almost any ⇠ , f ⇠ is said to be L s restricted smooth with sparsity level s , if it is differentiable, and there exist a generic constant L s such that for all ( x , y ) 2 R d with k x  y [-∥ 
 ≤-]k 0  s :
[Equation]
Assumption 3 (  2 -FGN [14], Assumption 2.3 (Finite Gradient Noise)) .
f is said to have  -ﬁnite gradient noise if for almost any ⇠ , f ⇠ is differentiable and the gradient noise  =  ( f, ⇠ ) deﬁned below is ﬁnite:
[Equation]
Remark 1.
Even though the original version of [14] uses the ` 2 norm, we use the ` 1 norm here, 105 in order to give more insightful results in terms of k and d , as is done classically in ` 0 optimization,similarly to [43].
We also note that in [14], x ⇤ denotes an unconstrained minimum when in our caseit denotes the constrained minimum for some sparsity k ⇤ .
For Corollary 2, we will also need the more usual smoothness assumption:
Assumption 4 ( L -smooth) .
For almost any ⇠ , f ⇠ is said to be L smooth, if it is differentiable, and for all ( x , y ) 2 R d :
[Equation]
Algorithm3.1 Random support Zeroth-Order estimate
In this section, we describe our zeroth-order gradient estimator.
It is basically composed of a randomsupport sampling step, followed by a random direction with uniform smoothing on the spheresupported by this support.
We also use the technique of averaging our estimator over q dimensions,as described in [22].
More formally, our gradient estimator is described below:
[Equation]
where each random direction u i is a unit vector sampled uniformly from the set S ds 2 := { u {+
+}
R d : k u k 0  s 2 , k u k = 1 } .
We can obtain such vectors u by sampling ﬁrst a random support S(i.e.
a set of coordinates) of size s 2 from [ d ] , (denoted as S ⇠ U (  [ d ] s ) in Algorithm 1) and thenby sampling a random unit vector u supported on that support S , that is, uniformly sampled fromthe set S dS := { u 2 R d : u [ d ]  S = 0 , k u k = 1 } , (denoted as u ⇠ U ( S dS ) in algorithm 1).
The 121 original uniform smoothing technique on the sphere is described in more detail in [11].
However,in our case, the sphere along which we sample is restricted to a random support of size s 2 .
Ourgeneral estimator, through the setting of the variable s 2 , can take several forms, which are similar topre-existing gradient estimators from the literature described below:
• In the case s 2 = d , our estimator retrieves the usual vanilla estimator with uniform smoothing onthe sphere [11].
• In the case where 1  s 2  d , our estimator is similar to the Random Block-Coordinate gradientestimator from Lian et al.
[19], except that the blocks are not ﬁxed at initialization but chosenrandomly, and that we use a uniform smoothing with forward difference on the given block insteadof a coordinate-wise estimation with central difference.
This random support technique allows usto give a convergence analysis under the classical assumptions of the hard-thresholding literature(see Remark 3), and to deal with huge scale optimization, when even sampling uniformly from aunit d -sphere is costly [5, 6].
Error Bounds of the Zeroth-Order Estimator.
We now derive error bounds on the gradient 135 estimator, that will be useful in the convergence rate proof, except that we consider only the restrictionto some support F (that is, we consider a subset of components of the gradient/estimator).
Indeed,proofs in the hard-thresholding literature (see for instance [41]), are usually written only on thatsupport.
That is the key idea which explains how the dimensionality dependence is reduced whendoing SZOHT compared to vanilla ZO optimization.
We give more insight on the shape of theoriginal distribution of gradient estimators, and the distribution of their projection onto a hyperplane 141 F in Figure 5 in Appendix E.
We can observe that even if the original gradient estimator is poor, inthe projected space, the estimation error is reduced. In this section, we will quantify how exactlybetter that projected estimate is.
Proposition 1.
(Proof in Appendix C.3 ) Let us consider any support F ⇢ [ d ] of size s ( | F | = s ).
For the Z0 gradient estimator in (2) , with q random directions, and random supports of size s 2 , andassuming that each f ⇠ is ( L s 2 , s 2 ) -RSS, we have, with ˆ r F f ⇠ ( x ) denoting the hard thresholding of 147 the gradient r f ⇠ ( x ) on F (that is, we set all coordinates not in F to 0 ): (a) k E ˆ r F f ⇠ (
[Table]
where " µ = L 2 s 2 sd , " F = 2 d q ( s 2 +2) ⇣
( s  1)( s 2  1) d +⌘ + 2 , " F c = 2 d q ( s 2 +2) ⇣ s ( s 2  1)
d  1 ⌘ and" abs = [Equation]
[Equation] [Equation]
3.2 SZOHT Algorithm
We now present our full algorithm to optimize problem 1, which we name SZOHT (Stochastic
Zeroth-Order Hard Thresholding).
Each iteration of our algorithm is composed of two steps: (i) thegradient estimation step, and (ii) the hard thresholding step, where the gradient estimation step isthe one described in the section above, and the hard-thresholding is described in more detail in thefollowing paragraph.
We give the full formal description of our algorithm in Algorithm 1.
In the hard thresholding step, we only keep the k largest (in magnitude) components of the currentiterate x t .
This ensures that all our iterates (including the last one) are k -sparse.
This hard-thresholdingoperator has been studied for instance in [32], and possesses several interesting properties.
Firstly, itcan be seen as a projection on the ` 0 ball.
Second, importantly, it is not non-expansive, contrary toother operators like the soft-thresholding operator [32].
That expansivity plays an important role inthe analysis of our algorithm, as we will see later.
Compared to previous works, our algorithm can be seen as a variant of Stochastic Hard Thresholding(StoIHT from [28]) , where we replaced the true gradient of f ⇠ by the estimator ˆ r f ⇠ ( x ) .
It is alsovery close to Algorithm 5 from Balasubramanian and Ghadimi [2] (Truncated-ZSGD), with justa different zeroth-order gradient estimator: we use a uniform smoothing, random-block estimator,instead of their gaussian smoothing, full support vanilla estimator.
This allows us to deal with verylarge dimensionalities, in the order of millions, similarly to Cai et al.
Furthermore, as describedin the Introduction, contrary to Balasubramanian and Ghadimi [2], we provide the analysis of ouralgorithm without any gradient sparsity assumption.
The key challenge arising in our analysis is described in Figure 1: the hard-thresholding operator isknown to not be non-expansive [32], which means that each approximate gradient step must approachthe solution enough to stay close to it even after hard-thresholding.
Therefore, it is a priori unclearwhether the zeroth-order estimate can be accurate enough to guarantee the convergence of SZOHT.
Hopefully, as we will see in the next section, we can indeed ensure convergence, as long as wecarefully choose the value of q .  Convergence analysis
[Figure]
In this section, we provide the convergence analysis of SZOHT, using the assumptions from section2, and discuss an interesting property of the combination of the zeroth-order gradient estimate andthe hard-thresholding operator, providing a positive answer to the question from the previous section.
Theorem 1.
(Proof in Appendix D.1) Assume that that each f ⇠ is ( L s 0 , s 0 :=
max( s 2 , s ))
-RSS, and that f is ( ⌫ s , s ) -RSC and  -FGN, with s = 2 k + k ⇤  d , with k  ⇢ 2 k ⇤ / (1  ⇢ 2 ) 2 , with ⇢ deﬁned as below.
Suppose that we run SZOHT with random supports of size s 2 , with q random directions, a learning rate of ⌘ = ⌫ s (4 " F +1)
L 2 s 0 (with " F deﬁned above in Proposition 1), and with k coordinates kept at each iterations.
Then, we have a geometric convergence rate, of the following form, with x ( t ) denoting the t -iterate of SZOHT:
[Equation]
Initialization: Learning rate ⌘ , maximum number of iterations T , size of the random directions support s 2 , number of random directions q , number of coordinates to keep at each iteration k
= O (  4 k ⇤ ) , initial point x (0) with k x (0) k 0  k ⇤ (typically x (0) = 0 ), .
Output : x T .
for t = 1 , ..., T do Sample ⇠ (for instance sample a train sample i ) for i = 1 , ..., q do
Sample a random support S ⇠ U (  [ d ] s )
Sample a random direction u i from the unit sphere supported on S : u i ⇠ U
S dS  Compute ˆ r f ⇠ ( x t  1 ; u i ) =
dµ ( f ⇠ ( x + µ u i )  f ⇠ ( x ))
u i ; end
Compute ˆ r f ⇠ ( x t  1 ) = 1 q
P qi =1 ˆ r f ⇠ ( x t  1 ; u j )
Compute ˜ x t = x t  1  ⌘ ˆ r f ⇠ ( x t  1 ) ;
Compute x t = ˜ x tk as the truncation of ˜ x t with top k entries preserved; end with a = ⌘ ⇣p
[Equation] [Equation]  = r 1 + ⇣ k
⇤ /k
+ p (4 + k ⇤ /k )
k ⇤ /k
⌘ / 2 , and " F , " abs and " µ are deﬁned in Proposition 1above.
Remark 2 (System error) .
The format of our result is similar to the ones in [41] and [28], in that itcontains a linear convergence term, and a system error which depends on the expected norm of thegradient at x ⇤ (through the variable  ).
But, due to the error from the ZO estimate, it also containsanother system error term which depends on the smoothing radius µ .
Remark 3 (Generality) .
If we take s 2  s , the ﬁrst assumption of Theorem 1 becomes the requirement 191 that f ⇠ is ( L s , s ) -RSS.
Therefore, SZOHT as well as the theorem above provides, up to our knowledge,the ﬁrst algorithm that can work in the usual setting of hard-thresholding algorithms (that is, ( L s , s ) -
RSS and ( ⌫ s , s ) -RSC [28, 32]), as well as its convergence rate.
Interplay between hard-thresholding and zeroth-order error Importantly, contrary to previous 195 works in zeroth-order optimization, the number of random directions q must be chosen carefully here.
Indeed, there is an interesting phenomenon arising in our speciﬁc setting which combines zeroth-ordergradient estimates and the hard-thresholding operator.
As described in [32], the hard-thresholdingoperator is not non-expansive, contrary to the soft-thresholding operator or the projection onto the ` 1ball [32]. This means that projecting onto the ` 0 ball can drive the iterates away from the solution.
Therefore, enough descent must be made by the (approximate) gradient step to ensure to that weget close enough to the solution. Since any error introduced in the gradient estimate may worsenthe descent, it is crucial to limit those errors as much as possible.
This problem arises with anykind of gradient errors: for instance with SGD errors [28, 43], it is generally dealt with either byensuring some conditions on the function f [28], forming bigger batches of samples (to decreasethe error in the gradient) [43], and/or considering a larger number of components k kept in hardthresholding (to make the hard-thresholding less expansive). Since in our work we want to consideras much as possible assumptions-free settings, we don’t consider ensuring additional conditions on 208 f .
Rather, similarly to Zhou et al.
[43], we consider instead dealing with this problem by relaxing kand sampling more directions (that latter technique is the zeroth-order equivalent to taking biggerbatch-size in SGD).
However, there is an additional effect that happens in our case, speciﬁc to zerothorder estimation: indeed, as described in Proposition 1, the quality of our estimator also depends on 212 k !
Therefore, it may be hard to make the algorithm converge only by considering larger k : higherk means less expansivity (which helps convergence), but worse gradient estimate (which harmsconvergence) .
Therefore, it is even more crucial to tune precisely our degree of freedom at handwhich is q .
We further illustrate this conﬂict between the non-expansiveness of hard-thresholding(quantiﬁed by the parameter  [32]), and the error from the zeroth-order estimate in Figure 1. More precisely, we provide below the minimal value q , needed to ensure the descent of our algorithm,that is, to ensure ⇢ < 1 for some k ⇤  1 :
Lemma 1 (First condition on q , Proof in Appendix D.2) .
Let k ⇤ 2 { 1 , ..., d } . In the case s 2 > 1 , if q  q min , with q min deﬁned below, then there exist k 2 N such that k  k ⇤ ⇢ 2 (1  ⇢ 2 ) 2 .
If s 2 = 1 , no condition on q is needed to ensure such an existence. However, in the case s 2 = 1 , q = 1 does not ensure the second necessary condition in the following Remark 4, as detailed in the proof of the present Lemma. Therefore, a minimal value of q [-≥ 
-]is always necessary to ensure, for Theorem 1 to apply.
[Equation]
Remark 4 (Second condition on q ) . As mentioned in the Lemma 1 above, there is also a secondcondition on q necessary to ensure for Theorem 1 to be valid, which is simply to ensure that thesmallest valid k above (i.e. from the ﬁrst condition), is smaller or equal to d (since we cannot keepmore components than we have).
4.1 Weak/non dependence on dimensionality of the query complexity.
In this section, we provide Corollaries 1 and 2, following from Theorem 1, which give an exampleof q that is valid for both these conditions above, allowing to converge (that is, to obtain ⇢ < 1 in Theorem 1), and that achieves weak dimensionality dependence in the case of RSS, and completedimension independence in the case of smoothness.
Corollary 1 (RSS f ⇠ , proof in Appendix D.3) .
Assume that that almost all f ⇠ are ( L s 0 , s 0 :
=
max( s 2 , s ))
-RSS, and that f is ( ⌫ s , s ) -RSC and  -FGN, with s = 2 k + k ⇤  d , with k  (86  4  12  2 ) k ⇤ (with  :=
L s 0 ⌫ s ) .
Suppose that we run SZOHT with random support of size s 2 , a learning rate of ⌘ = ⌫ s 13 L 2 s 0 , with k coordinates kept at each iterations by the hard-thresholding, and with q  2 s + 6 ds 2 .
Then, we have a geometric convergence rate, of the following form, with x ( t ) denoting the t -iterate of SZOHT:
[Equation]
with a , b and  are deﬁned above in Theorem 1, and ⇢ = [Equation] 1  2 13  2 .
Therefore, the query complexityto ensure that E k [Equation]
We now turn to the case where the functions f ⇠ are smooth.
The key result in that case is that we canhave a query complexity independent of the dimension d , which is, up to our knowledge, the ﬁrstresult of such kind for sparse zeroth-order optimization without assuming any gradient sparsity.
Corollary 2 (Smooth f ⇠ , proof in Appendix D.4)) .
Assume that, in addition to the conditions from
Corollary 1 above, almost all f ⇠ are L -smooth, with k  (86  4  12  2 )
k ⇤ (with  := L ⌫ s ), and take q  2( s + 2) , and s {+
+} =
d (that is, no random support sampling).
Then, we have a geometric convergence rate, of the following form, with x ( t ) denoting the t -iterate of SZOHT:
[Equation]
Therefore, the query complexity to ensure that E k x ( t )  x ⇤ k  [Equation] ⌘ µ is O (  2 k log( 1 " )) .
Additionally, our convergence rate highlights an interesting connection between the geometry of f(deﬁned by the condition number  = L s 0 / ⌫ s ), and the number of random directions that we needto take at each iteration: if the problem is ill-conditioned, that is  is high, then we need a bigger k .
This result is standard in the ` 0 litterature (see for instance [41]).
But speciﬁcally, in our ZO case, italso impacts the query complexity: since the projected gradient is harder to approximate when the dimension k of the projection is larger, q needs to grow too, resulting in higher query complexity.
Webelieve this is an interesting result for the sparse zeroth-order optimization community: it revealsthat the query complexity may in fact depend on some notion of intrinsic dimension to the problem,related to both the sparsity of the iterates k , and the geometry of the function f for a given s 2 (throughthe restricted condition number  ), rather than the dimension of the original space d as in previousworks like [12].
Experiments5.1 Sensitivity analysis
We ﬁrst conduct a sensitivity parameter analysis on a toy example, to highlight the importance ofthe choice of q , as discussed in Section 4.
We ﬁx a target sparsity k ⇤ = 5 , choose k = {+
+} k ⇤ , andconsider a sparse quadric function f : R 5000 !
R , with: f ( x ) = 12 k a > ( x  b ) k 2 , with a i = 1 ifi  d  s and 0 otherwise (to ensure f is s -RSC and smooth, with ⌫ s =
L = 1 ), and b i =
i 100 d forall i 2 [ d ] .
We choose ⌘ as in Theorem 1: ⌘ = 1 (4 " F +1) with " F deﬁned in Proposition 1 in terms ofs and d (we take s 2 = d ), and present our results in Figure 2, for six values of q .
We can observe on Figure 2(b) that the smaller the q , the less f ( x ) can descend.
Interestingly, we can also see on Figure2(a) that for q = 1 and 20 , k x ( t )  x ⇤ k diverges: we can indeed compute that ⇢ > 1 for those q ,which explains the divergence, from Theorem 1.
5.2 Baselines
We compare our SZOHT algorithms with state of the art zeroth-order algorithms that can deal withsparsity constraints, that appear in Table 1:
1. ZSCG [12] is a Frank-Wolfe ZO algorithm, for which we consider an ` 1 ball constraint.2.
RSPGF [2] is a proximal ZO algorithm, for which we consider an ` 1 penalty.3.
ZORO [5] is a proximal ZO algorithm, that makes use of sparsity of gradients assumptions,using a sparse reconstruction algorithm at each iteration to reconstruct the gradient from a fewmeasurements.
Similarly, as for ZSCG, we consider an ` 1 penalty.
In all the applications below, we will tune the sparsity k of SZOHT, the penalty of RSPGF and ZORO,and the radius of the constraint of ZSCG, such that all algorithms attain a similar converged objectivevalue, for fair comparison.
5.3 Applications
We compare the algorithms above on two tasks: a sparse asset risk management task from [7], and anadversarial attack task [8] with a sparsity constraint.
Sparse asset risk management We consider the portfolio management task and dataset from [7],similarly to [5].
We have a given portfolio of d assets, with each asset i giving an expected returnm i , and with a global covariance matrix of the return of assets denoted as C .
The cost function weminimize is the portfolio risk:
[Equation]
, where x is a vector where each component x i denoteshow much is invested in each asset, and we require to minimize it under a constraint of minimalreturn r : P di
[-=
 m-][Equation] i
P di =1 x i .
We enforce that constraint using the Lagrangian form below.
Finally, we add asparsity constraint, to restrict the investments to only k assets.
Therefore, we obtain the cost functionbelow:
[Equation]
We use three datasets: port3, port4 and port5 from the OR-library [3], of respective dimensions 280 d = 89; 98; 225 .
We keep r and  the same for the 4 algorithms: r = 0 .
1 ,  = 10 (for obtain the optimal µ and ⌘ by grid search, both over the interval [10  3 , 10 3 ] .
For all other algorithms,we got the optimal hyper-parameters through grid search.
We present our results in Figure 3.
Few pixels adversarial attacks We consider the problem of adversarial attacks with a sparseconstraint.
Our goal is to minimize min  f ( x +  ) such that k  k 0  k , where f is the Carlini-Wagner 286 cost function [8], that is computed from the outputs of a pre-trained model on the correspondingdataset.
We consider three different datasets for the attacks: MNIST, CIFAR, and Imagenet, ofdimension respectively d = 784; 3072; 268203 .
All algorithms are initialized with  = 0 .
We setthe hyperparameters of SZOHT as follows:
MNIST: k = 20 , s 2 = 100 , q = 100 ; CIFAR: k = 60 ,s 2 = 100 , q = 1000 ; ImageNet: k = 100000 , s 2 = 1000 , q = 100 .
We present our results in Figure4.
All experiments are conducted in the workstation with four NVIDIA RTX A6000 GPUs, and takeabout one day to run.
5.4 Results and Discussion
We can observe from Figures 3 and 4 that the performance of SZOHT is comparable or better thanthe other algorithms.
This can be explained by the fact that SZOHT has a linear convergence, but thequery complexity of ZSCG and RSPGF is in O (1 /T ) .
We can also notice that RSPGF is faster than
ZSCG, which is natural since proximal algorithms are faster than Frank-Wolfe algorithms.
Finally, itappears that the convergence of ZORO is sometimes slower, particularly at the early stage of training,which may come from the fact that ZORO assumes sparse gradients, which is not necessarily veriﬁedin real-world use cases like the ones we consider; in those cases where the gradient is not sparse, it ispossible that the sparse gradient reconstruction step of ZORO does not work well.
This motivateseven further the need to consider algorithms able to work without those assumptions, such as SZOHT
[Figure] [Figure]
[Figure] [Figure]
Conclusion
In this paper, we proposed a new algorithm, SZOHT, for sparse zeroth-order optimization.
We gaveits convergence analysis and showed that it is dimension independent in the smooth case, and weakdimension-dependent in the RSS case.
We further veriﬁed experimentally the efﬁciency of SZOHTin several settings.
Moreover, throughout the paper, we showed how the condition number of f aswell as the gradient error have an important impact on the convergence of SZOHT.
As such, it wouldbe interesting to study whether we can improve the query complexity by regularizing f , by usingan adaptive learning rate or acceleration methods, or by using recent variance reduction techniques.
Finally, it would also be interesting to extend this work to a broader family of sparse structures, suchas low-rank approximations or graph sparsity.
We leave this for future work.