Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivity
Abstract ℓ 0 constrained optimization is prevalent in machine learning, particularly for highdimensional problems, because it is a fundamental approach to achieve sparse learning.
Hard-thresholding gradient descent is a dominant technique to solve this problem.
However, first-order gradients of the objective function may be either unavailable or expensive to calculate in a lot of real-world problems, where zeroth-order (ZO) gradients could be a good surrogate.
Unfortunately, whether ZO gradients can work with the hard-thresholding operator is still an unsolved problem.
To solve this puzzle, in this paper, we focus on the ℓ 0 constrained black-box stochastic optimization problems, and propose a new stochastic zeroth-order gradient hard-thresholding (SZOHT) algorithm with a general ZO gradient estimator powered by a novel random support sampling.
We provide the convergence analysis of SZOHT under standard assumptions.
Importantly, we reveal a conflict between the deviation of ZO estimators and the expansivity of the hard-thresholding operator, and provide a theoretical minimal value of the number of random directions in ZO gradients.
In addition, we find that the query complexity of SZOHT is independent or weakly dependent on the dimensionality under different settings.
Finally, we illustrate the utility of our method on a portfolio optimization problem as well as black-box adversarial attacks.
Introduction
ℓ 0 constrained optimization is prevalent in machine learning, particularly for high-dimensional problems, because it is a fundamental approach to achieve sparse learning.
In addition to improving the memory, computational and environmental footprint of the models, these sparse constraints help reduce overfitting and obtain consistent statistical estimation [46, 5, 33, 29].
We formulate the problem as follows:
[Equation]
where f ( · , ξ ) : R d → R is a differentiable function and ξ is a noise term, for instance related to an underlying finite sum structure in f , of the form: E ξ f ( x , ξ ) = 1 n (cid:80) ni
=
f i ( x ) .
Hard-thresholding gradient algorithm [17, 31, 45] is a dominant technique to solve this problem.
It generally consists in alternating between a gradient step, and a hard-thresholding operation which only keeps the k -largest components (in absolute value) of the current iterate.
The advantage of hard-thresholding over its convex relaxations ([39, 41]) is that it can often attain similar precision, but is more computationally efficient, since it can directly ensure a desired sparsity level instead of tuning an ℓ 1 penalty or constraint.
The only expensive computation in hard-thresholding is the hard-thresholding step itself, which requires finding the top k elements of the current iterate.
Hard-thresholding was originally developed in its full gradient form [17], but has been later on extended to the stochastic setting by [Table] [Table]
Nguyen et al.
[31], which developed a stochastic gradient descent (SGD) version of hard thresholding (StoIHT)
[Table]
and further more with Zhou et al.
[47], Shen and Li [36] and Li et al.
[20], which used variance reduction technique to improve upon StoIHT.
However, the first-order gradients used in the above methods may be either unavailable or expensive to calculate in a lot of real-world problems.
For example, in certain graphical modeling tasks [42], obtaining the gradient of the objective function is computationally hard.
Even worse, in some settings, the gradient is inaccessible by nature, for instance in bandit problems [35], black-box adversarial attacks [40, 9, 10], or reinforcement learning [34, 27, 11].
To tackle those problems, ZO optimization methods have been developed [30].
Those methods usually replace the inaccessible gradient by its finite difference approximation which can be computed only from function evaluations, following the idea that for a differentiable function f : R → R , we have: f ′ ( x ) =
lim h → 0 f ( x + h ) − f ( x ) h .
Later on, ZO methods have been adapted to deal with a convex constraints set, and can therefore be used to solve the ℓ 1 convex relaxation of problem (1).
To that end, Ghadimi et al.
[14], and Cai et al.
[7] introduce proximal ZO algorithms, Liu et al.
[24] introduce a ZO projected gradient algorithm and
Balasubramanian and Ghadimi [2] introduce a ZO conditional gradient [19] algorithm.
We provide a review of those results in Table 1.
As can be seen from the table, their query complexity is high (linear in d ), except [7] that has a complexity of O ( s log( d ) log( 1 ε )) , but assumes that gradients are sparse.
In addition, those methods must introduce a hyperparameter λ (the strength of the ℓ 1 penalty) or R (the radius of the ℓ 1 ball), which need to be tuned to find which value ensures the right sparsity level.
Therefore, it would be interesting to use the hard-thresholding techniques described in the previous paragraph, instead of those convex relaxations.
Unfortunately, ZO hard-thresholding gradient algorithms have not been exploited formally.
Even more, whether ZO gradients can work with the hard-thresholding operator is still an unknown problem.
Although there was one related algorithm proposed recently by Balasubramanian and
Ghadimi [2], they did not target ℓ 0 constrained optimization problem and importantly have strong assumptions in their convergence analysis.
Indeed, they assume that the gradients, as well as the solution of the unconstrained problem, are s -sparse: ∥∇ f ( x ) ∥ 0 ≤ s and ∥ x ∗ ∥ 0 ≤ s ∗ ≈ s , where x ∗ = arg min x f ( x ) .
In addition, it was recently shown by Cai et al.
[7] that they must in fact assume that the support of the gradient is fixed for all x ∈ X , for their convergence result to hold, which is a hard limitation, since that amounts to say that the function f depends on s fixed variables.
To fill this gap, in this paper, we focus on the ℓ 0 constrained black-box stochastic optimization problems, and propose a novel stochastic zeroth-order gradient hard-thresholding (SZOHT) algorithm.
Specifically, we propose a dimension friendly ZO gradient estimator powered by a novel random support sampling technique, and then embed it into the standard hard-thresholding operator.
We then provide the convergence and complexity analysis of SZOHT under the standard assumptions of sparse learning, which are restricted strong smoothness (RSS), and restricted strong convexity (RSC) [31, 36], to retain generality, therefore providing a positive answer to the question of whether
ZO gradients can work with the hard-thresholding operator.
Crucial to our analysis is to provide carefully tuned requirements on the parameters q (the number of random directions used to estimate the gradient, further defined in Section 3.1) and k .
Finally, we illustrate the utility of our method on a portfolio optimization problem as well as black-box adversarial attacks, by showing that our method can achieve competitive performance in comparison to state of the art methods for sparsity-enforcing zeroth-order algorithm described in Table 1, such as [14, 2, 7].
Importantly, we also show that in the smooth case, the query complexity of SZOHT is independent of the dimensionality, which is significantly different to the dimensionality dependent results for most existing ZO algorithms.
Indeed, it is known from Jamieson et al.
[18] that the worst case query complexity of ZO optimization over the class F ν,L of ν -strongly convex and L -smooth functions defined over a convex set X is linear in d .
Our work is thus in line with other works achieving dimension-insensitive query complexity in zeroth-order optimization such as [15, 37, 44, 7, 6, 2, 7, 22, 18], but contrary to those, instead of making further assumptions (i.e. restricting the class F ν,L to a smaller class), we bypass the impossibility result by replacing the convex feasible set X by a non-convex set (the ℓ 0 ball), which is how we can avoid making stringent assumptions on the class of functions f .
Contributions.
We summarize the main contributions of our paper as follows:
We propose a new algorithm SZOHT that is, up to our knowledge, the first zeroth-order sparsity constrained algorithm that is dimension independent under the smoothness assumption, without assuming any gradient sparsity.
We reveal an interesting conflict between the error from zeroth-order estimates and the hardthresholding operator, which results in a minimal value for the number of random directions q that is necessary to ensure at each iteration.
We also provide the convergence analysis of our algorithm in the more general RSS setting, providing, up to our knowledge, the first zeroth-order algorithm that can work with the usual assumptions of RSS/RSC from the hard-thresholding literature.
Preliminaries
Throughout this paper, we denote by ∥ x ∥ the Euclidean norm for a vector x ∈ R d , by ∥ x ∥ ∞ the maximum absolute component of that vector, and by ∥ x ∥ {+
+} the ℓ 0 norm (which is not a proper norm).
For simplicity, we denote f ξ ( · ) :
= f ( · , ξ ) .
We call u F (resp.
∇ F f ( x ) )
the vector which sets all coordinates i ̸∈ F of u (resp.
∇ f ( x ) ) to 0 .
We also denote by x ∗ the solution of problem (1) defined in the introduction, for some target sparsity k ∗ which could be smaller than k .
To derive our result, we will need the following assumptions on f .
Assumption 1 ( ( ν s , s )
-RSC, [17, 28, 26, 45, 20, 36, 31]) .
f is said to be ν s restricted strongly convex with sparsity parameter s if it is differentiable, and there exist a generic constant ν s such that for all ( x , y ) ∈ R d with ∥ x − y ∥ 0 ≤ s :
[Equation]
Assumption 2 ( ( L s , s ) -RSS, [36, 31]) .
For almost any ξ , f ξ is said to be L s restricted smooth with sparsity level s , if it is differentiable, and there exist a generic constant L s such that for all ( x , y ) ∈ R d with ∥ x − y ∥
≤ s :
[Equation]
L s ∥
[Equation] − y ∥ Assumption 3 ( σ 2 -FGN [16], Assumption 2.3 (Finite Gradient Noise)) .
f is said to have σ -finite gradient noise if for almost any ξ , f ξ is differentiable and the gradient noise σ = σ ( f, ξ ) defined below is finite:
[Equation]
Remark 1.
Even though the original version of [16] uses the ℓ 2 norm, we use the ℓ ∞ norm here, in order to give more insightful results in terms of k and d , as is done classically in ℓ 0 optimization, similarly to [47].
We also note that in [16], x ∗ denotes an unconstrained minimum when in our case it denotes the constrained minimum for some sparsity k ∗ .
For Corollary 2, we will also need the more usual smoothness assumption:
Assumption 4 ( L -smooth) .
For almost any ξ , f ξ is said to be L smooth, if it is differentiable, and
for all ( x , y ) ∈ R
d :
[Equation]
Algorithm 3.1 Random support Zeroth-Order estimate
In this section, we describe our zeroth-order gradient estimator.
It is basically composed of a random support sampling step, followed by a random direction with uniform smoothing on the sphere supported by this support.
We also use the technique of averaging our estimator over q dimensions, as described in [25].
More formally, our gradient estimator is described below:
[Equation]
where each random direction u i is a unit vector sampled uniformly from the set S ds 2 := { u ∈
R d : ∥ u ∥ 0 ≤ s 2 , ∥ u ∥ = 1 } .
We can obtain such vectors u by sampling first a random support S (i.e. a set of coordinates) of size s 2 from [ d ] , (denoted as S ∼ U ( (cid:0)
[ d ] s 2 (cid:1) ) in Algorithm 1) and then by sampling a random unit vector u supported on that support S , that is, uniformly sampled from the set S dS := { u ∈ R d : u [ d ] − S = 0 , ∥ u ∥ = 1 } , (denoted as u ∼ U ( S dS ) in algorithm 1).
The original uniform smoothing technique on the sphere is described in more detail in [12].
However, in our case, the sphere along which we sample is restricted to a random support of size s 2 .
Our general estimator, through the setting of the variable s 2 , can take several forms, which are similar to pre-existing gradient estimators from the literature described below:
• If s 2 = d , ˆ ∇ f ξ ( x ) is the usual vanilla estimator with uniform smoothing on the sphere [12]. If
1 ≤ s 2 ≤ d , our estimator is similar to the Random Block-Coordinate gradient estimator from Lian et al.
[21], except that the blocks are not fixed at initialization but chosen randomly, and that we use a uniform smoothing with forward difference on the given block instead of a coordinate-wise estimation with central difference.
This random support technique allows us to give a convergence analysis under the classical assumptions of the hard-thresholding literature (see Remark 3), and to deal with huge scale optimization, when sampling uniformly from a unit d -sphere is costly [7, 6]: in the distributed setting for instance, each worker would just need to sample an s 2 -sparse random vector, and only the centralized server would materialize the full gradient approximation containing up to qs 2 non-zero entries.
Error Bounds of the Zeroth-Order Estimator.
We now derive error bounds on the gradient estimator, that will be useful in the convergence rate proof, except that we consider only the restriction to some support F (that is, we consider a subset of components of the gradient/estimator).
Indeed, proofs in the hard-thresholding literature (see for instance [45]), are usually written only on that support.
That is the key idea which explains how the dimensionality dependence is reduced when doing SZOHT compared to vanilla ZO optimization.
We give more insight on the shape of the original distribution of gradient estimators, and the distribution of their projection onto a hyperplane
F in Figure 5 in Appendix E.
We can observe that even if the original gradient estimator is poor, in the projected space, the estimation error is reduced, which we quantify in the proposition below.
Proposition 1.
(Proof in Appendix C.3 ) Let us consider any support F ⊂ [ d ] of size s ( | F | = s ).
For the Z0 gradient estimator in (2) , with q random directions, and random supports of size s 2 , and assuming that each f ξ is ( L s 2 , s 2 ) -RSS, we have, with ˆ ∇ F f ξ ( x ) denoting the hard thresholding of the gradient ∇ f ξ ( x ) on F (that is, we set all coordinates not in F to 0 ):
(a) ∥
E ˆ ∇ F f ξ ( x ) − ∇ F f ξ ( x ) ∥ 2 ≤ ε µ µ(b) E ∥ ˆ ∇ F f ξ ( x ) ∥ 2 ≤ ε F ∥∇ F f ξ ( x ) ∥ 2 + ε F c ∥∇ F c f ξ ( x ) ∥ 2 + ε abs µ(c) E ∥ ˆ ∇ F f ξ ( x ) − ∇ F f ξ ( x ) ∥ 2 ≤ 2( ε F + 1) ∥∇ F f ξ ( x ) ∥ 2 + 2 ε F c ∥∇ F c f ξ ( x ) ∥ 2 + 2 ε abs µ
with ε µ =
L 2 s 2 sd, ε F = 2 d q ( s 2 + 2) (cid:18) ( s − 1)( s 2 − 1)
d −+(cid:19) + 2 , ε F c = 2 d q ( s 2 + 2) (cid:18) s ( s 2 − 1)
d −(cid:19) and ε abs = 2 dL 2 s 2 ssq (cid:18) ( s − 1)( s 2 − 1)
d −+(cid:19) + L 2 s 2 sd (3)
3.2 SZOHT Algorithm
We now present our full algorithm to optimize problem 1, which we name SZOHT (Stochastic
Zeroth-Order Hard Thresholding).
Each iteration of our algorithm is composed of two steps: (i) the gradient estimation step, and (ii) the hard thresholding step, where the gradient estimation step is the one described in the section above, and the hard-thresholding is described in more detail in the following paragraph.
We give the full formal description of our algorithm in Algorithm 1.
In the hard thresholding step, we only keep the k largest (in magnitude) components of the current iterate x t .
This ensures that all our iterates (including the last one) are k -sparse.
This hard-thresholding operator has been studied for instance in [36], and possesses several interesting properties.
Firstly, it can be seen as a projection on the ℓ 0 ball.
Second, importantly, it is not non-expansive, contrary to other operators like the soft-thresholding operator [36].
That expansivity plays an important role in the analysis of our algorithm, as we will see later.
Compared to previous works, our algorithm can be seen as a variant of Stochastic Hard Thresholding (StoIHT from [31]) , where we replaced the true gradient of f ξ by the estimator ˆ ∇ f ξ ( x ) .
It is also very close to Algorithm 5 from Balasubramanian and Ghadimi [2] (Truncated-ZSGD), with just a different zeroth-order gradient estimator: we use a uniform smoothing, random-block estimator, instead of their gaussian smoothing, full support vanilla estimator.
This allows us to deal with very large dimensionalities, in the order of millions, similarly to Cai et al.
Furthermore, as described in the Introduction, contrary to Balasubramanian and Ghadimi [2], we provide the analysis of our algorithm without any gradient sparsity assumption.
The key challenge arising in our analysis is described in Figure 1: the hard-thresholding operator being expansive [36], each approximate gradient step must approach the solution enough to stay close to it even after hard-thresholding.
Therefore, it is a priori unclear whether the zeroth-order estimate can be accurate enough to guarantee the convergence of SZOHT.
Hopefully, as we will see in the next section, we can indeed ensure convergence, as long as we carefully choose the value of q .
Convergence analysis
In this section, we provide the convergence analysis of SZOHT, using the assumptions from section 2, and discuss an interesting property of the combination of the zeroth-order gradient estimate and the hard-thresholding operator, providing a positive answer to the question from the previous section.
Theorem 1.
(Proof in Appendix D.1) Assume that that each f ξ is ( L s ′ , s ′ :
=
max( s 2 , s ))
-RSS, and that f is ( ν s , s ) -RSC and σ -FGN, with s = 2 k
+ k ∗ ≤ d , with d − k ∗≥ k ≥ ρ 2 k ∗ / (1 − ρ 2 ) 2 , with ρ defined as below.
Suppose that we run SZOHT with random supports of size s 2 , q random directions, a learning rate of η = ν s (4 ε F +1)
L 2 s ′ , and k coordinates kept at each iterations.
Then, we have a geometric convergence rate, of the following form, with x ( t ) denoting the t -iterate of SZOHT: [Equation] γb
[Equation]
µ Initialization: Learning rate η , maximum number of iterations T , size of the random directions support s 2 , number of random directions q , number of coordinates to keep at each iteration k
= O
( κ 4 k ∗ ) , initial point x (0) with ∥ x (0) ∥ 0 ≤ k ∗ (typically x (0) = 0 ), .
Output : x T .
for t = 1 , ..., T do
Sample ξ (for instance sample a train sample i ) for i = 1 , ..., q do
Sample a random support S ∼ U ( (cid:0)
[ d ] s 2 (cid:1) )
Sample a random direction u i from the unit sphere supported on S : u i ∼ U (cid:0) S dS (cid:1)
Compute ˆ ∇ f ξ ( x t − 1 ; u i ) =
dµ ( f ξ ( x + µ u i ) − f ξ ( x ))
u
i ; end
Compute ˆ ∇ f ξ ( x t − 1 ) = 1 q (cid:80) qi
=1 ˆ ∇ f ξ ( x t − 1 ; u j )
Compute ˜ x t = x t − 1 − η ˆ ∇ f ξ ( x t − 1 ) ;
Compute x t = ˜ x tk as the truncation of ˜ x t with top k entries preserved; end with a = η (cid:16)(cid:112) (4 ε F s + 2) + ε F c ( d − k ) + √ s (cid:17) , b = (cid:18) √ ε µ
L s ′ + η √ 2 ε abs (cid:19) , ρ 2 = 1 − ν 2 s (4 ε F + 1)
L 2 s ′ , and γ = (cid:114) 1 + (cid:16) k ∗ /k
+ (cid:112) (4 + k ∗ /k )
k ∗ /k
(cid:17) /and
ε F , ε abs , and ε µ are defined in (3) .
Remark 2 (System error) .
The format of our result is similar to the ones in [45] and [31], in that it contains a linear convergence term, and a system error which depends on the expected norm of the gradient at x ∗ (through the variable σ ).
We note that if f has a k ∗ -sparse unconstrained minimizer, which could happen in sparse reconstruction, or with overparameterized deep networks (see for instance [32, Assumption (2)]), then we would have ∥∇ f ( x ∗ ) ∥ = 0 , and hence that part of the system error would vanish.
In addition to that usual system error, we also have here another system error, which depends on the smoothing radius µ , due to the error from the ZO estimate.
Remark 3 (Generality) .
If we take s 2 ≤ s , the first assumption of Theorem 1 becomes the requirement that f ξ is ( L s , s ) -RSS.
Therefore, SZOHT as well as the theorem above provides, up to our knowledge, the first algorithm that can work in the usual setting of hard-thresholding algorithms (that is, ( L s , s ) -
RSS and ( ν s , s ) -RSC [31, 36]), as well as its convergence rate.
Interplay between hard-thresholding and zeroth-order error Importantly, contrary to previous works in ZO optimization, q must be chosen carefully here, due to our specific setting combining
ZO and hard-thresholding.
Indeed, as described in [36], the hard-thresholding operator is not nonexpansive (contrary to projection onto the ℓ 1 ball) so it can drive the iterates away from the solution.
Therefore, enough descent must be made by the (approximate) gradient step to get close enough to the solution, and it is therefore crucial to limit errors in gradient estimation.
This problem arises with any kind of gradient errors: for instance with SGD errors [31, 47], it is generally dealt with either by ensuring some conditions on the function f [31], forming bigger batches of samples [47], and/or considering a larger number of components k kept in hard-thresholding (to make the hardthresholding less expansive).
In our work, similarly to Zhou et al.
[47], we deal with this problem by relaxing k and sampling more directions u
i (which is the ZO equivalent to taking bigger batch-size in SGD).
However, there is an additional effect that happens in our case, specific to ZO estimation: as described in Proposition 1, the quality of our estimator also depends on k .
Therefore, it may be hard to make the algorithm converge only by considering larger k : higher k means less expansivity (which helps convergence), but worse gradient estimate (which harms convergence) .
We further illustrate this conflict between the non-expansiveness of hard-thresholding (quantified by the parameter γ [36]), and the error from the zeroth-order estimate, in Figure 1.
Therefore, it is even more crucial to tune precisely our remaining degree of freedom at hand which is q .
More precisely, a minimal value of q is always necessary to ensure convergence in our setting, contrary to most ZO setting (in which taking even q = 1 can work, as long as other constants like η are well chosen, see for instance [23,
Corollary 3]).
The remark below gives some necessary conditions on q to illustrate that fact.
Remark 4 (Some necessary condition on q , proof in D.2) .
Let k ∗ ∈ N ∗ and assume, that k is such that k > ρ 2 k ∗ / (1 − ρ 2 ) 2 (which ensures that ργ < 1 ), and that k ≤ d − k ∗ 2 .
These conditions imply the following necessary (but not sufficient) condition on q :
• if s 2 > 1 : q ≥
d ( s 2 − 1) k ∗ κ 2 ( s 2 +2)( d − 1) (cid:104) 18 κ 2 − 1 + 2 (cid:113) 9 κ 2 (9 κ 2 − 1) + 12 − 12 k ∗ +d − 1 k ∗ ( s 2 − 1) (cid:105) • if s 2 = 1 : q ≥ 8 κ 2 d √ dk ∗ +
Remark 4 is just a warning that usual rules from ZO do not apply to SZOHT, but it does not say how to choose q to ensure convergence: for that we would need some sufficient conditions on q for
Theorem 1 to apply.
We give such conditions in the next section.
4.1 Weak/non dependence on dimensionality of the query complexity.
In this section, we provide Corollaries 1 and 2, following from Theorem 1, which give an example of q that is sufficient to converge (that is, to obtain γρ < 1 in Theorem 1), and that achieves weak dimensionality dependence in the case of RSS, and complete dimension independence in the case of smoothness.
Corollary 1 (RSS f ξ , proof in Appendix D.3) .
Assume that that almost all f ξ are ( L s ′ , s ′ :
=
max( s 2 , s ))
-RSS, and that f is ( ν s , s ) -RSC and σ -FGN, with s = 2 k
+ k ∗ ≤ d , with d − k ∗≥ k ≥ (86 κ 4 − 12 κ 2 ) k ∗ (with κ :
=
L s ′
ν s ) .
Suppose that we run SZOHT with random support of size s 2 , a learning rate of η = ν
s 13 L 2 s ′ , with k coordinates kept at each iterations by the hard-thresholding, and with q ≥ 2 s
+ 6 ds 2 .
Then, we have a geometric convergence rate, of the following form, with x ( t ) denoting the t -iterate of SZOHT:
[Equation]
with a , b and γ are defined in (4) , and ρ = (cid:113) 1 − 2 13 κ 2 .
Therefore, the query complexity (QC) to ensure that E ∥ x ( t [Equation]
We now turn to the case where the functions f ξ are smooth.
The key result in that case is that we can have a query complexity independent of the dimension d , which is, up to our knowledge, the first result of such kind for sparse zeroth-order optimization without assuming any gradient sparsity.
Corollary 2 (Smooth f ξ , proof in Appendix D.4)) .
Assume that, in addition to the conditions from
Corollary 1 above, almost all f ξ are L -smooth, with d − k ∗
[Equation]
with
[Equation]
[Equation] and take q ≥ 2( s + 2) , and s {+
+} =
d (that is, no random support sampling).
Then, we have a geometric convergence rate, of the following form, with x ( t ) denoting the t -iterate of SZOHT:
[Equation]
γρ (cid:19) µ
Therefore, the QC to ensure that E ∥ x ( t ) − x ∗ ∥ ≤ ε + (cid:16) γa 1 − γρ (cid:17) σ + (cid:16) γb 1 − γρ (cid:17) µ is O ( κ 2 k log( 1 ε )) .
Additionally, our convergence rate highlights an interesting connection between the geometry of f (defined by the condition number κ = L s ′ /ν s ), and the number of random directions that we need to take at each iteration: if the problem is ill-conditioned, that is κ is high, then we need a bigger k .
This result is standard in the ℓ 0 litterature (see for instance [45]).
But specifically, in our ZO case, it also impacts the query complexity: since the projected gradient is harder to approximate when the dimension k of the projection is larger, q needs to grow too, resulting in higher query complexity.
We believe this is an interesting result for the sparse zeroth-order optimization community: it reveals that the query complexity may in fact depend on some notion of intrinsic dimension to the problem, related to both the sparsity of the iterates k , and the geometry of the function f for a given s 2 (through the restricted condition number κ ), rather than the dimension of the original space d as in previous works like [14].
Experiments 5.1 Sensitivity analysis
We first conduct a sensitivity parameter analysis on a toy example, to highlight the importance of the choice of q , as discussed in Section 4.
We fix a target sparsity k ∗ = 5 , choose k = 74 k ∗ , and consider a sparse quadric function f : R 5000 → R , with: f ( x ) = 12 ∥ a ⊙ ( x − b ) ∥ 2 ( ⊙ denotes the elementwise product), with a i
= 1 if i ≥ d − s and 0 otherwise (to ensure f is s -RSC and smooth, with ν s =
L = 1 ), and b i =
i 100 d for all i ∈
[ d − 70 k ∗ ] and 0 for all d − 70 k ∗ ≤ i ≤ d (we make such a choice in order to have ∥∇ f ( x ∗ ) ∥ small enough).
We choose η as in Theorem 1: η = 1 (4 ε F +1) with ε F defined in Proposition 1 in terms of s and d (we take s 2 = d ), µ = 1 e − 4 , and present our results in Figure 2, for six values of q .
We can observe on Figure 2(b) that the smaller the q , the less f ( x ) can descend.
Interestingly, we can also see on Figure 2(a) that for q = 1 and 20 , ∥ x ( t ) − x ∗ ∥ diverges: we can indeed compute that ργ > 1 for those q , which explains the divergence, from
Theorem 1.
5.2 Baselines
We compare our SZOHT algorithms with state of the art zeroth-order algorithms that can deal with sparsity constraints, that appear in Table 1:
ZSCG [2] is a Frank-Wolfe ZO algorithm, for which we consider an ℓ 1 ball constraint.
RSPGF [14] is a proximal ZO algorithm, for which we consider an ℓ 1 penalty.
ZORO [7] is a proximal ZO algorithm, that makes use of sparsity of gradients assumptions, using a sparse reconstruction algorithm at each iteration to reconstruct the gradient from a few measurements.
Similarly, as for ZSCG, we consider an ℓ 1 penalty.
In all the applications below, we will tune the sparsity k of SZOHT, the penalty of RSPGF and ZORO, and the radius of the constraint of ZSCG, such that all algorithms attain a similar converged objective value, for fair comparison.
5.3 Applications
We compare the algorithms above on two tasks: a sparse asset risk management task from [8], and an adversarial attack task [9] with a sparsity constraint.
Sparse asset risk management We consider the portfolio management task and dataset from [8], similarly to [7].
We have a given portfolio of d assets, with each asset i giving an expected return m i , and with a global covariance matrix of the return of assets denoted as C .
The cost function we minimize is the portfolio risk:
[Equation]
, where x is a vector where each component x i denotes how much is invested in each asset, and we require to minimize it under a constraint of minimal return r : (cid:80) di
[-=
-][Equation]
1 m i x i (cid:80) di
=1 x i .
We enforce that constraint using the Lagrangian form below.
Finally, we add a sparsity constraint, to restrict the investments to only k assets.
Therefore, we obtain the cost function below:
[Equation]
We use three datasets: port3, port4 and port5 from the OR-library [3], of respective dimensions d = 89; 98; 225 .
We keep r and λ the same for the 4 algorithms: r = 0 .
1 , λ = 10 (for port3 and port4); and r = 1 e − 3 , λ = 1 e − 3 for port5.
For SZOHT, we set k = 10 , s 2 = 10 , q = 10 , and ( µ, η ) =
(0 . 015 , 0 . 015) for port4, and ( µ, η ) =
(0 . 1 , 1) for port5 ( µ and η are both obtained by grid search over the interval [10 − 3 , 10 3 ] ).
For all other algorithms, we got the optimal hyper-parameters through grid search.
We present our results in Figure 3.
Few pixels adversarial attacks We consider the problem of adversarial attacks with a sparse constraint.
Our goal is to minimize min δ f ( x + δ ) such that ∥ δ ∥ 0 ≤ k , where f is the Carlini-Wagner cost function [9], that is computed from the outputs of a pre-trained model on the corresponding dataset.
We consider three different datasets for the attacks: MNIST, CIFAR, and Imagenet, of dimension respectively d = 784; 3072; 268203 .
All algorithms are initialized with δ = 0 .
We set the hyperparameters of SZOHT as follows:
MNIST: k = 20 , s 2 = 100 , q = 100 , µ = 0 .
3 , η = 1 ;
CIFAR: k = 60 , s 2 = 100 , q = 1000 , µ = 1 e − 3 , η = 0 .
ImageNet: k = 100000 , s 2 = 1000 , q = 100 , µ = 0 .
01 , η = 0 .
We present our results in Figure 4.
All experiments are conducted in the workstation with four NVIDIA RTX A6000 GPUs, and take about one day to run.
5.4 Results and Discussion
We can observe from Figures 3 and 4 that the performance of SZOHT is comparable or better than the other algorithms.
This can be explained by the fact that SZOHT has a linear convergence, but the query complexity of ZSCG and RSPGF is in O (1 /T ) .
We can also notice that RSPGF is faster than
ZSCG, which is natural since proximal algorithms are faster than Frank-Wolfe algorithms (indeed, in case of possible strong-convexity, vanilla Frank-Wolfe algorithms maintain a O (1 /T ) rate [13], when proximal algorithms get a linear rate [4, Theorem 10.29]).
Finally, it appears that the convergence of
ZORO is sometimes slower, particularly at the early stage of training, which may come from the fact that ZORO assumes sparse gradients, which is not necessarily verified in real-world use cases like the ones we consider; in those cases where the gradient is not sparse, it is possible that the sparse gradient reconstruction step of ZORO does not work well.
This motivates even further the need to consider algorithms able to work without those assumptions, such as SZOHT.
[Figure]
[Figure] [Figure]
Conclusion
In this paper, we proposed a new algorithm, SZOHT, for sparse zeroth-order optimization.
We gave its convergence analysis and showed that it is dimension independent in the smooth case, and weak dimension-dependent in the RSS case.
We further verified experimentally the efficiency of SZOHT in several settings.
Moreover, throughout the paper, we showed how the condition number of f as well as the gradient error have an important impact on the convergence of SZOHT.
As such, it would be interesting to study whether we can improve the query complexity by regularizing f , by using an adaptive learning rate or acceleration methods, or by using recent variance reduction techniques.
Finally, it would also be interesting to extend this work to a broader family of sparse structures, such as low-rank approximations or graph sparsity.
We leave this for future work.
Acknowledgments and Disclosure of Funding
Xiao-Tong Yuan is funded in part by the National Key Research and Development Program of China under Grant No. 2018AAA0100400, and in part by the Natural Science Foundation of China (NSFC) under Grant No.
U21B2049, No.61876090 and No.61936005.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] In the conclusion we mention some possible ways to improve upon our work (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Our code will be included in the supplementary material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] The code will be submitted in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]