Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivity
Anonymous Author(s)
AffiliationAddress email Abstract ℓ 0 constrained optimization is prevalent in machine learning, particularly for highdimensional problems, because it is a fundamental approach to achieve sparselearning.
Hard-thresholding gradient descent is a dominant technique to solvethis problem.
However, first-order gradients of the objective function may beeither unavailable or expensive to calculate in a lot of real-world problems, wherezeroth-order (ZO) gradients could be a good surrogate.
Unfortunately, whether ZOgradients can work with the hard-thresholding operator is still an unsolved problem.
To solve this puzzle, in this paper, we focus on the ℓ 0 constrained black-boxstochastic optimization problems, and propose a new stochastic zeroth-order gradient hard-thresholding (SZOHT) algorithm with a general ZO gradient estimatorpowered by a novel random support sampling.
We provide the convergence analysisof SZOHT under standard assumptions.
Importantly, we reveal a conflict betweenthe deviation of ZO estimators and the expansivity of the hard-thresholding operator, and provide a theoretical minimal value of the number of random directionsin ZO gradients.
In addition, we find that the query complexity of SZOHT isindependent or weakly dependent on the dimensionality under different settings.
Finally, we illustrate the utility of our method on a portfolio optimization problemas well as black-box adversarial attacks.
Introduction
ℓ 0 constrained optimization is prevalent in machine learning, particularly for high-dimensionalproblems, because it is a fundamental approach to achieve sparse learning.
In addition to improvingthe memory, computational and environmental footprint of the models, these sparse constraintshelp reduce overfitting and obtain consistent statistical estimation [46, 5, 33, 29].
We formulate theproblem as follows:
[Equation]
where f ( · , ξ ) : R d → R is a differentiable function and ξ is a noise term, for instance related to anunderlying finite sum structure in f , of the form: E ξ f ( x , ξ ) = 1 n (cid:80) ni
=
f i ( x ) .
Hard-thresholdinggradient algorithm [17, 31, 45] is a dominant technique to solve this problem.
It generally consists inalternating between a gradient step, and a hard-thresholding operation which only keeps the k -largestcomponents (in absolute value) of the current iterate.
The advantage of hard-thresholding over itsconvex relaxations ([39, 41]) is that it can often attain similar precision, but is more computationallyefficient, since it can directly ensure a desired sparsity level instead of tuning an ℓ 1 penalty orconstraint.
The only expensive computation in hard-thresholding is the hard-thresholding step itself,which requires finding the top k elements of the current iterate.
Hard-thresholding was originallydeveloped in its full gradient form [17], but has been later on extended to the stochastic setting by Nguyen et al.
[31], which developed a stochastic gradient descent (SGD) version of hard thresholding
[Table] [Table]
(StoIHT), and further more with Zhou et al.
[47], Shen and Li [36] and Li et al.
[20], which usedvariance reduction technique to improve upon StoIHT.
However, the first-order gradients used in the above methods may be either unavailable or expensiveto calculate in a lot of real-world problems.
For example, in certain graphical modeling tasks [42],obtaining the gradient of the objective function is computationally hard.
Even worse, in some settings,the gradient is inaccessible by nature, for instance in bandit problems [35], black-box adversarialattacks [40, 9, 10], or reinforcement learning [34, 27, 11].
To tackle those problems, ZO optimizationmethods have been developed [30].
Those methods usually replace the inaccessible gradient by itsfinite difference approximation which can be computed only from function evaluations, followingthe idea that for a differentiable function f : R → R , we have: f ′ ( x ) =
lim h [-! 
-]→ 0 f ( x + h ) − f ( x ) h .
Lateron, ZO methods have been adapted to deal with a convex constraints set, and can therefore be usedto solve the ℓ 1 convex relaxation of problem (1).
To that end, Ghadimi et al.
[14], and Cai et al.
[6]introduce proximal ZO algorithms, Liu et al. [24] introduce a ZO projected gradient algorithm and Balasubramanian and Ghadimi [2] introduce a ZO conditional gradient [19] algorithm.
We providea review of those results in Table 1.
As can be seen from the table, their query complexity is high(linear in d ), except [6] that has a complexity of O ( s log( d ) log( 1 ε )) , but assumes that gradients aresparse.
In addition, those methods must introduce a hyperparameter λ (the strength of the ℓ 1 penalty)or R (the radius of the ℓ 1 ball), which need to be tuned to find which value ensures the right sparsitylevel.
Therefore, it would be interesting to use the hard-thresholding techniques described in theprevious paragraph, instead of those convex relaxations.
Unfortunately, ZO hard-thresholding gradient algorithms have not been exploited formally.
Evenmore, whether ZO gradients can work with the hard-thresholding operator is still an unknownproblem.
Although there was one related algorithm proposed recently by Balasubramanian and
Ghadimi [2], they did not target ℓ 0 constrained optimization problem and importantly have strongassumptions in their convergence analysis.
Indeed, they assume that the gradients, as well as thesolution of the unconstrained problem, are s -sparse: ∥∇ f ( x ) ∥ 0 ≤ s and ∥ x [-⇤ k 
 -]∗ ∥ 0 ≤ s ∗ ≈ s , wherex ∗ =
arg min x f ( x ) .
In addition, it was recently shown by Cai et al.
[6] that they must in factassume that the support of the gradient is fixed for all x ∈ X , for their convergence result to hold,which is a hard limitation, since that amounts to say that the function f depends on s fixed variables.
To fill this gap, in this paper, we focus on the ℓ 0 constrained black-box stochastic optimizationproblems, and propose a novel stochastic zeroth-order gradient hard-thresholding (SZOHT) algorithm.
Specifically, we propose a dimension friendly ZO gradient estimator powered by a novel randomsupport sampling technique, and then embed it into the standard hard-thresholding operator.
We then provide the convergence and complexity analysis of SZOHT under the standard assumptionsof sparse learning, which are restricted strong smoothness (RSS), and restricted strong convexity (RSC) [31, 36], to retain generality, therefore providing a positive answer to the question of whether ZO gradients can work with the hard-thresholding operator.
Crucial to our analysis is to providecarefully tuned requirements on the parameters q
(the number of random directions used to estimatethe gradient, further defined in Section 3.1) and k .
Finally, we illustrate the utility of our method on aportfolio optimization problem as well as black-box adversarial attacks, by showing that our methodcan achieve competitive performance in comparison to state of the art methods for sparsity-enforcingzeroth-order algorithm described in Table 1, such as [14, 2, 6].
Importantly, we also show that in the smooth case, the query complexity of SZOHT is independentof the dimensionality, which is significantly different to the dimensionality dependent results formost existing ZO algorithms.
Indeed, it is known from Jamieson et al.
[18] that the worst case querycomplexity of ZO optimization over the class F ν,L of ν -strongly convex and L -smooth functionsdefined over a convex set X is linear in d .
Our work is thus in line with other works achievingdimension-insensitive query complexity in zeroth-order optimization such as [15, 37, 44, 6, 7, 2, 6,22, 18], but contrary to those, instead of making further assumptions (i.e. restricting the class F ν,Lto a smaller class), we bypass the impossibility result by replacing the convex feasible set X by anon-convex set (the ℓ 0 ball), which is how we can avoid making stringent assumptions on the class offunctions f .
Contributions.
We summarize the main contributions of our paper as follows:
We propose a new algorithm SZOHT that is, up to our knowledge, the first zeroth-order sparsityconstrained algorithm that is dimension independent under the smoothness assumption, withoutassuming any gradient sparsity.2.
We reveal an interesting conflict between the error from zeroth-order estimates and the hardthresholding operator, which results in a minimal value for the number of random directions qthat is necessary to ensure at each iteration.
We also provide the convergence analysis of our algorithm in the more general RSS setting,providing, up to our knowledge, the first zeroth-order algorithm that can work with the usualassumptions of RSS/RSC from the hard-thresholding literature.
Preliminaries
Throughout this paper, we denote by ∥ x ∥ the Euclidean norm for a vector x ∈ R d , by ∥ x ∥ ∞ themaximum absolute component of that vector, and by ∥ x ∥ 0 the ℓ 0 norm (which is not a proper norm).
For simplicity, we denote f ξ ( · ) :
= f ( · , ξ ) .
We call u F (resp.
∇ F f ( x ) )
the vector which sets allcoordinates i ̸∈ F of u (resp.
∇ f ( x ) ) to 0 .
We also denote by x ∗ the solution of problem (1) definedin the introduction, for some target sparsity k ∗ which could be smaller than k .
To derive our result,we will need the following assumptions on f .
Assumption 1 ( ( ν s , s )
-RSC, [17, 28, 26, 45, 20, 36, 31]) .
f is said to be ν s restricted strongly convex with sparsity parameter s if it is differentiable, and there exist a generic constant ν s such that for all ( x , y ) ∈ R d with ∥ x − y ∥ 0 ≤ s :
[Equation]
Assumption 2 ( ( L s , s ) -RSS, [36, 31]) .
For almost any ξ , f ξ is said to be L s restricted smooth with sparsity level s , if it is differentiable, and there exist a generic constant L s such that for all ( x , y ) ∈ R d with ∥ x − y {+∥ 
 ≤+} s :
[Equation]
Assumption 3 ( σ 2 -FGN [16], Assumption 2.3 (Finite Gradient Noise)) .
f is said to have σ -finite gradient noise if for almost any ξ , f ξ is differentiable and the gradient noise σ = σ ( f, ξ ) defined below is finite:
[Equation]
Remark 1.
Even though the original version of [16] uses the ℓ 2 norm, we use the ℓ ∞ norm here,in order to give more insightful results in terms of k and d , as is done classically in ℓ 0 optimization,similarly to [47].
We also note that in [16], x ∗ denotes an unconstrained minimum when in our caseit denotes the constrained minimum for some sparsity k ∗ .
For Corollary 2, we will also need the more usual smoothness assumption:
Assumption 4 ( L -smooth) .
For almost any ξ , f ξ is said to be L smooth, if it is differentiable, and for all ( x , y ) ∈ R d :
[Equation]
Algorithm3.1 Random support Zeroth-Order estimate
In this section, we describe our zeroth-order gradient estimator.
It is basically composed of a randomsupport sampling step, followed by a random direction with uniform smoothing on the spheresupported by this support.
We also use the technique of averaging our estimator over q dimensions,as described in [25].
More formally, our gradient estimator is described below:
[Equation]
where each random direction u i is a unit vector sampled uniformly from the set S ds 2 := { u [-
-]∈
R d : ∥ u ∥ 0 ≤ s 2 , ∥ u ∥ = 1 } .
We can obtain such vectors u by sampling first a random support S(i.e.
a set of coordinates) of size s 2 from [ d ] , (denoted as S ∼ U ( (cid:0) [ d ] s 2 (cid:1) ) in Algorithm 1) and thenby sampling a random unit vector u supported on that support S , that is, uniformly sampled fromthe set S dS := { u ∈ R d : u [ d ] − S = 0 , ∥ u ∥ = 1 } , (denoted as u ∼ U ( S dS ) in algorithm 1).
Theoriginal uniform smoothing technique on the sphere is described in more detail in [12].
However,in our case, the sphere along which we sample is restricted to a random support of size s 2 .
Ourgeneral estimator, through the setting of the variable s 2 , can take several forms, which are similar topre-existing gradient estimators from the literature described below:
• If s 2 = d , ˆ ∇ f ξ ( x ) is the usual vanilla estimator with uniform smoothing on the sphere [12].• If 1 ≤
s 2 ≤ d , our estimator is similar to the Random Block-Coordinate gradient estimatorfrom Lian et al.
[21], except that the blocks are not fixed at initialization but chosen randomly,and that we use a uniform smoothing with forward difference on the given block instead of acoordinate-wise estimation with central difference.
This random support technique allows us togive a convergence analysis under the classical assumptions of the hard-thresholding literature(see Remark 3), and to deal with huge scale optimization, when sampling uniformly from a unitd -sphere is costly [6, 7]: in the distributed setting for instance, each worker would just need tosample an s 2 -sparse random vector, and only the centralized server would materialize the fullgradient approximation containing up to qs 2 non-zero entries.
Error Bounds of the Zeroth-Order Estimator.
We now derive error bounds on the gradientestimator, that will be useful in the convergence rate proof, except that we consider only the restrictionto some support F (that is, we consider a subset of components of the gradient/estimator).
Indeed,proofs in the hard-thresholding literature (see for instance [45]), are usually written only on thatsupport.
That is the key idea which explains how the dimensionality dependence is reduced whendoing SZOHT compared to vanilla ZO optimization.
We give more insight on the shape of theoriginal distribution of gradient estimators, and the distribution of their projection onto a hyperplane F in Figure 5 in Appendix E.
We can observe that even if the original gradient estimator is poor, inthe projected space, the estimation error is reduced, which we quantify in the proposition below.
Proposition 1.
(Proof in Appendix C.3 ) Let us consider any support F ⊂ [ d ] of size s ( | F | = s ).
For the Z0 gradient estimator in (2) , with q random directions, and random supports of size s 2 , andassuming that each f ξ is ( L s 2 , s 2 ) -RSS, we have, with ˆ ∇ F f ξ ( x ) denoting the hard thresholding ofthe gradient ∇ f ξ ( x ) on F (that is, we set all coordinates not in F to 0 ): [Equation] [Equation]
[Equation]
with ε µ = L 2 s 2 sd, ε F = 2 d q ( s 2 + 2) (cid:18) ( s − 1)( s 2 − 1)
d −+(cid:19) + 2 , ε F c = 2 d q ( s 2 + 2) (cid:18) s ( s 2 − 1)
d −(cid:19) and ε abs = 2 dL 2 s 2 ssq (cid:18) ( s − 1)( s 2 − 1)
d −+(cid:19) + L 2 s 2 sd (3)
3.2 SZOHT Algorithm
We now present our full algorithm to optimize problem 1, which we name SZOHT (Stochastic
Zeroth-Order Hard Thresholding).
Each iteration of our algorithm is composed of two steps: (i) thegradient estimation step, and (ii) the hard thresholding step, where the gradient estimation step isthe one described in the section above, and the hard-thresholding is described in more detail in thefollowing paragraph.
We give the full formal description of our algorithm in Algorithm 1.
In the hard thresholding step, we only keep the k largest (in magnitude) components of the currentiterate x t .
This ensures that all our iterates (including the last one) are k -sparse.
This hard-thresholdingoperator has been studied for instance in [36], and possesses several interesting properties.
Firstly, itcan be seen as a projection on the ℓ 0 ball.
Second, importantly, it is not non-expansive, contrary toother operators like the soft-thresholding operator [36].
That expansivity plays an important role inthe analysis of our algorithm, as we will see later.
Compared to previous works, our algorithm can be seen as a variant of Stochastic Hard Thresholding(StoIHT from [31]) , where we replaced the true gradient of f ξ by the estimator ˆ ∇ f ξ ( x ) .
It is alsovery close to Algorithm 5 from Balasubramanian and Ghadimi [2] (Truncated-ZSGD), with justa different zeroth-order gradient estimator: we use a uniform smoothing, random-block estimator,instead of their gaussian smoothing, full support vanilla estimator.
This allows us to deal with verylarge dimensionalities, in the order of millions, similarly to Cai et al.
Furthermore, as describedin the Introduction, contrary to Balasubramanian and Ghadimi [2], we provide the analysis of ouralgorithm without any gradient sparsity assumption.
The key challenge arising in our analysis is described in Figure 1: the hard-thresholding operatorbeing expansive [36], each approximate gradient step must approach the solution enough to stay closeto it even after hard-thresholding.
Therefore, it is a priori unclear whether the zeroth-order estimatecan be accurate enough to guarantee the convergence of SZOHT.
Hopefully, as we will see in thenext section, we can indeed ensure convergence, as long as we carefully choose the value of q .
[Figure]
In this section, we provide the convergence analysis of SZOHT, using the assumptions from section2, and discuss an interesting property of the combination of the zeroth-order gradient estimate andthe hard-thresholding operator, providing a positive answer to the question from the previous section.
Theorem 1.
(Proof in Appendix D.1) Assume that that each f ξ is ( L s ′ , s ′ : =
max( s 2 , s ))
-RSS, and that f is ( ν s , s ) -RSC and σ -FGN, with s = 2 k + k ∗ ≤ d , with d − k ∗≥ k ≥ ρ 2 k ∗ / (1 − ρ 2 ) 2 , with ρ defined as below.
Suppose that we run SZOHT with random supports of size s 2 , q random directions, a learning rate of η = ν s (4 ε F +1)
L 2 s ′ , and k coordinates kept at each iterations.
Then, we have a geometric convergence rate, of the following form, with x ( t ) denoting the t -iterate of SZOHT:
[Equation]
Initialization: Learning rate η , maximum number of iterations T , size of the random directions support s 2 , number of random directions q , number of coordinates to keep at each iteration k
= O ( κ 4 k ∗ ) , initial point x (0) with ∥ x (0) ∥ 0 ≤ k ∗ (typically x (0) = 0 ), .
Output : x T .
for t = 1 , ..., T do Sample ξ (for instance sample a train sample i ) for i = 1 , ..., q do
Sample a random support S ∼ U ( (cid:0) [ d ] s 2 (cid:1) )
Sample a random direction u i from the unit sphere supported on S : u i ∼ U (cid:0) S dS (cid:1)
Compute ˆ ∇ f ξ ( x t − 1 ; u i ) =
dµ ( f ξ ( x + µ u i ) − f ξ ( x ))
u i ; end
Compute ˆ ∇ f ξ ( x t − 1 ) = 1 q (cid:80) qi
=1 ˆ ∇ f ξ ( x t − 1 ; u j )
Compute ˜ x t = x t − 1 − η ˆ ∇ f ξ ( x t − 1 ) ;
Compute x t = ˜ x tk as the truncation of ˜ x t with top k entries preserved; end with a = η (cid:16)(cid:112) (4 ε F s + 2) + ε F c ( d − k ) + √ s (cid:17) , b = (cid:18) √ ε µ
L s ′ + η √ 2 ε abs (cid:19) , ρ 2 = 1 − ν 2 s (4 ε F + 1)
L 2 s ′ , and γ = (cid:114) 1 + (cid:16) k ∗ /k
+ (cid:112) (4 + k ∗ /k )
k ∗ /k
(cid:17) /and ε F , ε abs , and ε µ are defined in (3) .
Remark 2 (System error) .
The format of our result is similar to the ones in [45] and [31], in that itcontains a linear convergence term, and a system error which depends on the expected norm of thegradient at x ∗ (through the variable σ ).
We note that if f has a k ∗ -sparse unconstrained minimizer,which could happen in sparse reconstruction, or with overparameterized deep networks (see forinstance [32, Assumption (2)]), then we would have ∥∇ f ( x ∗ ) ∥ = 0 , and hence that part of thesystem error would vanish.
In addition to that usual system error, we also have here another systemerror, which depends on the smoothing radius µ , due to the error from the ZO estimate.
Remark 3 (Generality) .
If we take s 2 ≤ s , the first assumption of Theorem 1 becomes the requirementthat f ξ is ( L s , s ) -RSS.
Therefore, SZOHT as well as the theorem above provides, up to our knowledge,the first algorithm that can work in the usual setting of hard-thresholding algorithms (that is, ( L s , s ) -
RSS and ( ν s , s ) -RSC [31, 36]), as well as its convergence rate.
Interplay between hard-thresholding and zeroth-order error Importantly, contrary to previousworks in ZO optimization, q must be chosen carefully here, due to our specific setting combining
ZO and hard-thresholding.
Indeed, as described in [36], the hard-thresholding operator is not nonexpansive (contrary to projection onto the ℓ 1 ball) so it can drive the iterates away from the solution.
Therefore, enough descent must be made by the (approximate) gradient step to get close enough tothe solution, and it is therefore crucial to limit errors in gradient estimation.
This problem ariseswith any kind of gradient errors: for instance with SGD errors [31, 47], it is generally dealt witheither by ensuring some conditions on the function f [31], forming bigger batches of samples [47],and/or considering a larger number of components k kept in hard-thresholding (to make the hardthresholding less expansive).
In our work, similarly to Zhou et al.
[47], we deal with this problem byrelaxing k and sampling more directions u i (which is the ZO equivalent to taking bigger batch-sizein SGD).
However, there is an additional effect that happens in our case, specific to ZO estimation: asdescribed in Proposition 1, the quality of our estimator also depends on k .
Therefore, it may be hardto make the algorithm converge only by considering larger k : higher k means less expansivity (whichhelps convergence), but worse gradient estimate (which harms convergence) . We further illustratethis conflict between the non-expansiveness of hard-thresholding (quantified by the parameter γ
[36]),and the error from the zeroth-order estimate, in Figure 1.
Therefore, it is even more crucial to tuneprecisely our remaining degree of freedom at hand which is q .
More precisely, a minimal value ofq is always necessary to ensure convergence in our setting, contrary to most ZO setting (in which taking even q = 1 can work, as long as other constants like η are well chosen, see for instance [23, Corollary 3]). The remark below gives some necessary conditions on q to illustrate that fact.
Remark 4 (Some necessary condition on q , proof in D.2) .
Let k ∗ ∈ N ∗ and assume, that k is suchthat k > ρ 2 k ∗ / (1 − ρ 2 ) 2 (which ensures that ργ < 1 ), and that k ≤ d − k ∗ 2 .
These conditions implythe following necessary (but not sufficient) condition on q : • if s 2 > 1 : q {+≥ 
+}
d ( s 2 − 1) k ∗ κ 2 ( s 2 +2)( d − 1) (cid:104) 18 κ 2 − 1 + 2 (cid:113) 9 κ 2 (9 κ 2 − 1) + 12 − 12 k ∗ +d − 1 k ∗ ( s 2 − 1) (cid:105)• if s 2 = 1 : q ≥ 8 κ 2 d √ dk ∗ +
Remark 4 is just a warning that usual rules from ZO do not apply to SZOHT, but it does not sayhow to choose q to ensure convergence: for that we would need some sufficient conditions on q for Theorem 1 to apply. We give such conditions in the next section.
4.1 Weak/non dependence on dimensionality of the query complexity.
In this section, we provide Corollaries 1 and 2, following from Theorem 1, which give an exampleof q that is sufficient to converge (that is, to obtain γρ < 1 in Theorem 1), and that achieves weakdimensionality dependence in the case of RSS, and complete dimension independence in the case ofsmoothness.
Corollary 1 (RSS f ξ , proof in Appendix D.3) .
Assume that that almost all f ξ are ( L s ′ , s ′ :
=
max( s 2 , s ))
-RSS, and that f is ( ν s , s ) -RSC and σ -FGN, with s = 2 k + k ∗ ≤ d , with d − k ∗≥ k ≥ (86 κ 4 − 12 κ 2 ) k ∗ (with κ : =
L s ′ ν s ) .
Suppose that we run SZOHT with random support of size s 2 , a learning rate of η = ν s 13 L 2 s ′ , with k coordinates kept at each iterations by the hard-thresholding, and with q ≥ 2 s + 6 ds 2 .
Then, we have a geometric convergence rate, of the following form, with x ( t ) denoting the t -iterate of SZOHT:
[Equation]
with a , b and γ are defined in (4) , and ρ = (cid:113) 1 − 2 13 κ 2 .
Therefore, the query complexity (QC) toensure that E ∥ x ( t ) − x ∗ ∥ ≤ ε + (cid:16) γa 1 − γρ (cid:17) σ + (cid:16) γb 1 − γρ (cid:17) µ is O ( κ 2 ( k + ds 2 ) log( 1 ε )) .
We now turn to the case where the functions f ξ are smooth.
The key result in that case is that we canhave a query complexity independent of the dimension d , which is, up to our knowledge, the firstresult of such kind for sparse zeroth-order optimization without assuming any gradient sparsity.
Corollary 2 (Smooth f ξ , proof in Appendix D.4)) .
Assume that, in addition to the conditions from
Corollary 1 above, almost all f ξ are L -smooth, with d − k ∗≥ [Equation]
[Equation] ∗ (with κ := Lν s ), and take q ≥ 2( s + 2) , and s [-
-]2 =
d (that is, no random support sampling).
Then, we have a geometric convergence rate, of the following form, with x ( t ) denoting the t -iterate of SZOHT:
[Equation]
γρ (cid:19) σ + (cid:18) γb 1 − γρ (cid:19) µ
Therefore, the QC to ensure that E ∥ x ( t ) − x ∗ ∥ ≤ ε + (cid:16) γa 1 − γρ (cid:17) σ + (cid:16) γb 1 − γρ (cid:17) µ is O ( κ 2 k log( 1 ε )) .
Additionally, our convergence rate highlights an interesting connection between the geometry of f(defined by the condition number κ = L s ′ /ν s ), and the number of random directions that we needto take at each iteration: if the problem is ill-conditioned, that is κ is high, then we need a bigger k .
This result is standard in the ℓ 0 litterature (see for instance [45]).
But specifically, in our ZO case, italso impacts the query complexity: since the projected gradient is harder to approximate when thedimension k of the projection is larger, q needs to grow too, resulting in higher query complexity.
Webelieve this is an interesting result for the sparse zeroth-order optimization community: it revealsthat the query complexity may in fact depend on some notion of intrinsic dimension to the problem,related to both the sparsity of the iterates k , and the geometry of the function f for a given s 2 (throughthe restricted condition number κ ), rather than the dimension of the original space d as in previousworks like [14].
Experiments
5.1 Sensitivity analysis
We first conduct a sensitivity parameter analysis on a toy example, to highlight the importance ofthe choice of q , as discussed in Section 4.
We fix a target sparsity k ∗ = 5 , choose k = [-
-]74 k ∗ , andconsider a sparse quadric function f : R 5000 → R , with: f ( x ) = 12 ∥ a ⊙ ( x − b )
∥ 2 ( ⊙ denotes theelementwise product), with a i = 1 if i ≥ d − s and 0 otherwise (to ensure f is s -RSC and smooth,with ν s =
L = 1 ), and b i =
i 100 d for all i ∈
[ d − 70 k ∗ ] and 0 for all d − 70 k ∗ ≤ i ≤ d (we makesuch a choice in order to have ∥∇ f ( x ∗ ) ∥ small enough).
We choose η as in Theorem 1: η = 1 (4 ε F +1) 249 with ε F defined in Proposition 1 in terms of s and d (we take s 2 = d ), µ = 1 e − 4 , and present ourresults in Figure 2, for six values of q .
We can observe on Figure 2(b) that the smaller the q , the lessf ( x ) can descend.
Interestingly, we can also see on Figure 2(a) that for q = 1 and 20 , ∥ x ( t ) − x ∗ ∥diverges: we can indeed compute that ργ > 1 for those q , which explains the divergence, from Theorem 1.
5.2 Baselines
We compare our SZOHT algorithms with state of the art zeroth-order algorithms that can deal withsparsity constraints, that appear in Table 1:
ZSCG [2] is a Frank-Wolfe ZO algorithm, for which we consider an ℓ 1 ball constraint.2.
RSPGF [14] is a proximal ZO algorithm, for which we consider an ℓ 1 penalty.3.
ZORO [6] is a proximal ZO algorithm, that makes use of sparsity of gradients assumptions,using a sparse reconstruction algorithm at each iteration to reconstruct the gradient from a fewmeasurements.
Similarly, as for ZSCG , we consider an ℓ 1 penalty.
In all the applications below, we will tune the sparsity k of SZOHT, the penalty of RSPGF and ZORO,and the radius of the constraint of ZSCG, such that all algorithms attain a similar converged objectivevalue, for fair comparison.
5.3 Applications
We compare the algorithms above on two tasks: a sparse asset risk management task from [8], and anadversarial attack task [9] with a sparsity constraint.
Sparse asset risk management We consider the portfolio management task and dataset from [8],similarly to [6].
We have a given portfolio of d assets, with each asset i giving an expected returnm i , and with a global covariance matrix of the return of assets denoted as C .
The cost function weminimize is the portfolio risk:
[Equation]
, where x is a vector where each component x i denoteshow much is invested in each asset, and we require to minimize it under a constraint of minimalreturn r : (cid:80) di
{+=
 m+} i x i (cid:80) di
=1 x i .
We enforce that constraint using the Lagrangian form below.
Finally, we add asparsity constraint, to restrict the investments to only k assets.
Therefore, we obtain the cost functionbelow:
[Equation]
We use three datasets: port3, port4 and port5 from the OR-library [3], of respective dimensionsd = 89; 98; 225 .
We keep r and λ the same for the 4 algorithms: r = 0 .
1 , λ = 10 (for port3 andport4); and r = 1 e − 3 , λ = 1 e − 3 for port5. For SZOHT, we set k = 10 , s 2 = 10 , q = 10 , and( µ, η ) = (0 . 015 , 0 . 015) for port4, and ( µ, η ) = (0 . 1 , 1) for port5 ( µ and η are both obtained by gridsearch over the interval [10 − 3 , 10 3 ] ).
For all other algorithms, we got the optimal hyper-parametersthrough grid search.
We present our results in Figure 3.
Few pixels adversarial attacks We consider the problem of adversarial attacks with a sparseconstraint.
Our goal is to minimize min δ f ( x + δ ) such that ∥ δ ∥ 0 ≤ k , where f is the Carlini-Wagnercost function [9], that is computed from the outputs of a pre-trained model on the correspondingdataset.
We consider three different datasets for the attacks: MNIST, CIFAR, and Imagenet, ofdimension respectively d = 784; 3072; 268203 .
All algorithms are initialized with δ = 0 .
We setthe hyperparameters of SZOHT as follows:
MNIST: k = 20 , s 2 = 100 , q = 100 , µ = 0 . 3 , η = 1 ; CIFAR: k = 60 , s 2 = 100 , q = 1000 , µ = 1 e − 3 , η = 0 . ImageNet: k = 100000 , s 2 = 1000 ,q = 100 , µ = 0 .
01 , η = 0 .
We present our results in Figure 4.
All experiments are conducted inthe workstation with four NVIDIA RTX A6000 GPUs, and take about one day to run.
5.4 Results and Discussion
We can observe from Figures 3 and 4 that the performance of SZOHT is comparable or better thanthe other algorithms.
This can be explained by the fact that SZOHT has a linear convergence, but thequery complexity of ZSCG and RSPGF is in O (1 /T ) .
We can also notice that RSPGF is faster than
ZSCG, which is natural since proximal algorithms are faster than Frank-Wolfe algorithms (indeed, incase of possible strong-convexity, vanilla Frank-Wolfe algorithms maintain a O (1 /T ) rate [13], whenproximal algorithms get a linear rate [4, Theorem 10.29]).
Finally, it appears that the convergence of ZORO is sometimes slower, particularly at the early stage of training, which may come from the factthat ZORO assumes sparse gradients, which is not necessarily verified in real-world use cases likethe ones we consider; in those cases where the gradient is not sparse, it is possible that the sparsegradient reconstruction step of ZORO does not work well.
This motivates even further the need toconsider algorithms able to work without those assumptions, such as SZOHT.
[Figure] [Figure]
[Figure] [Figure]
Conclusion
In this paper, we proposed a new algorithm, SZOHT, for sparse zeroth-order optimization.
We gaveits convergence analysis and showed that it is dimension independent in the smooth case, and weakdimension-dependent in the RSS case.
We further verified experimentally the efficiency of SZOHTin several settings.
Moreover, throughout the paper, we showed how the condition number of f aswell as the gradient error have an important impact on the convergence of SZOHT.
As such, it wouldbe interesting to study whether we can improve the query complexity by regularizing f , by usingan adaptive learning rate or acceleration methods, or by using recent variance reduction techniques.
Finally, it would also be interesting to extend this work to a broader family of sparse structures, suchas low-rank approximations or graph sparsity.
We leave this for future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] In the conclusion we mentionsome possible ways to improve upon our work(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Our code will
be included in the supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
The code will be submitted in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]