A gradient estimator via L1-randomization for online zero-order optimization with two point feedback
Abstract
This work studies online zero-order optimization of convex and Lipschitz functions.
We present a novel gradient estimator based on two function evaluations and randomization on the ℓ 1 -sphere.
Considering different geometries of feasible sets and Lipschitz assumptions we analyse online dual averaging algorithm with our estimator in place of the usual gradient.
We consider two types of assumptions on the noise of the zero-order oracle: canceling noise and adversarial noise.
We provide an anytime and completely data-driven algorithm, which is adaptive to all parameters of the problem.
In the case of canceling noise that was previously studied in the literature, our guarantees are either comparable or better than stateof-the-art bounds obtained by Duchi et al.
[14] and Shamir [33] for non-adaptive algorithms.
Our analysis is based on deriving a new weighted Poincaré type inequality for the uniform measure on the ℓ 1 -sphere with explicit constants, which may be of independent interest.
Introduction
In this work we study the problem of convex online zero-order optimization with two-point feedback, in which adversary fixes a sequence f 1 , f 2 , . . .
: R d → R of convex functions and the goal of the learner is to minimize the cumulative regret with respect to the best action in a prescribed convex set Θ ⊆ R d .
This problem has received significant attention in the context of continuous bandits and online optimization [see e.g., 1, 3, 12, 13, 16, 18, 21, 25, 29, 33, and references therein].
We consider the following protocol: at each round t = 1 , 2 , . . .
the algorithm chooses x ′ t , x ′′ t ∈ R d (that can be queried outside of Θ ) and the adversary reveals [Equation] where ξ ′ t , ξ ′′ t ∈ R are the noise variables (random or not) to be specified.
Based on the above information and the previous rounds, the learner outputs x t ∈ Θ and suffers loss f t ( x t ) .
The goal of the learner is to minimize the cumulative regret
[Equation]
At the core of our approach is a novel zero-order gradient estimator based on two function evaluations outlined in Algorithm 1.
A key novelty of our estimator is that it employs a randomization step over
Input: Convex function V ( · ) , step size η 1 > 0 , and parameters h t , for t = 1 , 2 , . . . ,
Initialization: Generate independently vectors ζ 1 , ζ 2 , . . .
uniformly distributed on ∂B d 1 , and set z 1 =for t = 1 , . .
, do x t = arg max x ∈ Θ { η t ⟨ z t , x ⟩ − V ( x ) } y ′ t
= f t ( x t + h t ζ t ) + ξ ′ t and y ′′ t = f t ( x t − h t ζ t ) + ξ ′′ t //
Query g t =
d 2 h t ( y ′ t − y ′′ t ) sign( ζ t ) //
ℓ 1 -gradient estimate z t +1 =
z t − g t // Update z t update the step-size η t + end the ℓ 1 sphere.
This is in contrast to most of the prior work [see e.g., 1, 3–5, 14, 16, 17, 22, 28, 32] that was employing ℓ 2 or ℓ ∞ type randomizations to define x ′ t , x ′′ t .
We use the proposed estimator within an online dual averaging procedure to tackle the zero-order online convex optimization problem, matching or improving the state-of-the-art results.
Duchi et al.
[14] and Shamir [33] have studied instances of the above problem under the assumption that ξ ′ t = ξ ′′ t , which we will further refer to as canceling noise assumption.
Specifically, [14] considered the stochastic optimization framework where f t = f , for every t , and obtained bounds on the optimization error rather than on cumulative regret, while [33] analyzed the case ξ ′ t = ξ ′′ t = 0 .
The results in [14, 33] are obtained for the objective functions that are Lipschitz with respect to the ℓ q -norm for q = 1 and q = 2 , although, with extra derivations it is possible to extend the above mentioned results beyond such cases.
The proposed method allows us to improve upon these results in several aspects.
Contributions.
The contributions of the present paper can be summarized as follows.
1) We present a new randomized zero-order gradient estimator and study its statistical properties, both under canceling noise and under adversarial noise (see Lemma 1 and Lemma 4); 2)
In the canceling noise case ( ξ ′ t = ξ ′′ t ) in Theorem 1 we show that dual averaging based on our gradient estimator either improves or matches the state-of-the-art bounds [14, 33].
We derive the results for Lipschitz functions with respect to all ℓ q -norms, q ∈ [1 , ∞ ] .
In particular, when q = 1 and Θ is the probability simplex, our bound is better by a (cid:112) log( d ) factor than that of [14, 33]; 3) We propose a completely data-driven and anytime version of the algorithm, which is adaptive to all parameters of the problem.
We show that it achieves analogous performance as the non-adaptive algorithm in the case of canceling noise and only slightly worse performance under adversarial noise.
To the best of our knowledge, no adaptive algorithms were developed for zero-order online problems in our setting so far; 4) As a key element of our analysis, we derive in Lemma 3 a weighted Poincaré type inequality [following the terminology of 10] with explicit constants for the uniform measure on ℓ 1 -sphere.
This result may be of independent interest.
Notation.
Throughout the paper we use the following notation.
We denote by ∥·∥ p the ℓ p -norm in
R d .
For any x ∈ R
d
we denote by x (cid:55)→ sign( x ) the component-wise sign function (defined at 0 as 1 ).
We let ⟨· , ·⟩ be the standard inner product in R d .
For p ∈ [1 , ∞ ] we introduce the open ℓ p -ball and ℓ p -sphere respectively as [Equation] and
[Equation]
For two a, b ∈ R , we denote by a ∧ b ( resp.
a ∨ b ) the minimum ( resp.
the maximum) between a and b .
We denote by Γ : (0 , ∞ )
→ R , the gamma function.
In what follows, log always stands for the natural logarithm and e is Euler’s number.
The algorithm
Let Θ be a closed convex subset of R d and let V :
Θ
→ R be a convex function.
The procedure that we propose in this paper is summarized in Algorithm 1.
Intuition behind the gradient estimate.
The form of gradient estimator g t in Algorithm 1 is explained by Stokes’ theorem (see Theorem 5 in the appendix and the discussion that follows).
Stokes’ theorem provides a connection between the gradient of a function f
(first order information) and f itself (zero order information).
Under some regularity conditions, it establishes that
[Equation]
where ∂D is the boundary of D , n is the outward normal vector to ∂D , and d S ( x ) denotes the surface measure.
Introducing U D and ζ ∂D distributed uniformly on D and ∂D respectively, we can rewrite the above identity as
[Equation]
where Vol d − 1 ( ∂D ) is the surface area of D and Vol d ( D ) is its volume.
In what follows we consider the special case D = B d 1 .
For this choice of D we have n ( x ) = 1 √ d · sign( x ) with
Vol d − 1 ( ∂D ) /
Vol d ( D ) =
d 3 / 2 leading to our gradient estimate for the two-point feedback setup.
Computational aspects.
Let us highlight two appealing practical features of the ℓ 1 -randomized gradient estimator g t in Algorithm 1.
First, we can easily evaluate any ℓ p -norm of g t .
Indeed, it holds that ∥ g t ∥ p
= ( d 1+1 /p / 2 h t ) |
y ′
t − y ′′ t | , i.e., computing ∥ g t ∥ p only requires O (1) elementary operations.
Second, this gradient estimator is very economic in terms of the required memory: in order to store g t we only need d bits and 1 float.
None of these properties is inherent to the popular alternatives based on the randomization over the ℓ 2 -sphere
[see e.g., 5, 16, 22] or on Gaussian randomization [see e.g., 19, 23, 24].
To compute g t one needs to generate ζ t distributed uniformly on ∂B d 1 .
The most straightforward way to do it consists in first generating a d -dimensional vector of i.i.d.
centered scaled Laplace random variables and then normalizing this vector by its ℓ 1 -norm.
The result is guaranteed to follow the uniform distribution on ∂B 1 d [see e.g., 30, Lemma 1].
Furthermore, to sample from the centered scaled Laplace distribution one can simply use inverse transform sampling.
Indeed, if U is distributed uniformly on (0 , 1) , then log(2 U ) 1 ( U > 1 / 2) − log(2 − 2 U ) 1 ( U ≥ 1 / 2) follows centered scaled
Laplace distribution.
Assumptions
We say that the convex function V ( · ) is 1 -strongly convex with respect to the ℓ p -norm on Θ if
[Equation]
, for all x , x ′ ∈ Θ and all w ∈ ∂V ( x ) , where ∂V ( x ) is the subdifferential of V at point x .
Throughout the paper, we assume that p, q ∈ [1 , ∞ ] , d ≥ 3 , and set p ∗ , q ∗ ∈ [1 , ∞ ] such thatp + 1 p ∗ = 1 and 1 q + 1 q ∗ = 1 , with the usual convention 1 / ∞ = 0 .
We will use the following assumptions. Assumption 1.
The following conditions hold:
The set Θ ⊂ R d is compact and convex.
There exists V : Θ → R , which is lower semi-continuous, 1 -strongly convex on Θ w.r.t.
the ℓ p -norm and such that [Equation] [Equation]  for some constant R > 0 .
Each function f t : R d → R is convex on R d for all t ≥ 1 . 4.
For all x , x ′ ∈ R d , and all t ≥ 1 we have | f t ( x ) − f t ( x ′ )
| ≤ L ∥ x − x ′ ∥ q for some constant L > 0 .
Assumption 1 is rather standard in the study of dual averaging-type algorithms and have been previously considered in the context of zero-order problems in [14, 33].
We assume that Θ is compact as we are interested in the worst-case regret, which ensures that R < + ∞ .
We discuss extensions of our results to the case of unbounded Θ in Section 8.
Note that the constant R > 0 is not necessarily dimension independent.
Below we provide two classical examples of V [see e.g., 31, Section 2].
Example 1.
Let Θ be any convex subset of R d and p ∈ (1 , 2] .
Then, V ( x ) = 1 2( p − 1) ∥ x ∥ 2 p is 1 -strongly convex on Θ w.r.t. the ℓ p -norm.
Example 2.
Let Θ = (cid:8) x ∈ R d : ∥ x ∥ 1 = 1 , x ≥ 0 (cid:9) .
Then 1 , V ( x ) =
(cid:80) dj
=1 x j log( x j ) is 1strongly convex on Θ w.r.t.
the ℓ 1 -norm and R 2 ≤ log( d ) .
Assumptions on the noise.
We consider two different assumptions on the noises ξ ′ t , ξ ′′ t .
The first noise assumption is common in the stochastic optimization context [see e.g., 14, 19, 23, 24, 33].
Assumption 2 (Canceling noise) .
For all t = 1 , 2 , . . .
, it holds that ξ ′ t = ξ ′′ t almost surely.
Formally, Assumption 2 permits noisy evaluations of function values.
However, due to the fact that we are allowed to query f t at two points, taking difference of y ′ t and y ′′ t in the estimator of the gradient effectively erases the noise.
It results in a smaller variance of our gradient estimator.
Importantly,
Assumption 2 covers the case of no noise, that is, the classical online optimization setting as defined, e.g., in [31].
Second, we consider an adversarial noise assumption, which is essentially equivalent to the assumptions used in [3, 4].
Assumption 3 (Adversarial noise) .
For all t = 1 , 2 , . . .
, it holds that: (i) E [( ξ ′ t ) 2 ] ≤ σ 2 and
E [( ξ ′′ t ) 2 ] ≤ σ 2 ; (ii) ( ξ ′ t ) t ≥ 1 and ( ξ ′′ t ) t ≥ 1 are independent of ( ζ t ) t ≥
Assumption 3 allows for stochastic ξ ′ t and ξ ′′ t that are not necessarily zero-mean or independent over the trajectory.
Furthermore, it permits bounded non-stochastic adversarial noises. Part (ii) of
Assumption 3 is always satisfied.
Indeed, ξ ′ t ’s and ξ ′′ t ’s are coming from the environment and are unknown to the learner while ζ t ’s are artificially generated by the learner.
We mention part (ii) only for formal mathematical rigor.
Note that, since the choice of function V belongs to the learner and Θ is given, it is always reasonable to assume that parameter R is known.
At the same time, parameters L and σ may be either known or unknown.
We will study both cases in the next sections.
Upper bounds on the regret
In this section, we present the main convergence results for Algorithm 1 when L, σ, T are known to the learner.
The case when they are unknown is analyzed in Section 5, where we develop fully adaptive versions of Algorithm 1.
To state our results in a unified way, we introduce the following sequence that depends on the dimension d and on the norm index q ≥ 1 :
[Equation]
The value b q ( d ) will explicitly influence the choice of the step size η > 0 and of the discretization parameter h > 0 .
The first result of this section establishes the convergence guarantees under the canceling noise assumption.
This case was previously considered by Duchi et al.
[14] and Shamir [33].
Theorem 1.
Let Assumptions 1 and 2 be satisfied.
Then, Algorithm 1 with the parameters
[Equation]
any
[Equation]
where A = ( √ 6 + √ 12) − 1 , satisfies, for any x ∈ Θ ,
[Equation]
Note that, as in other related works [14, 20, 23, 24, 33], under the canceling noise (or no noise) assumption the discretization parameter h > 0 can be chosen arbitrary small.
This is due to the fact that, under the canceling noise assumption, the variance of the gradient estimate g t is bounded by a constant independent of h .
It is no longer the case under the adversarial noise assumption as exhibited in the next theorem.
Theorem 2.
Let Assumptions 1 and 3 be satisfied.
Then Algorithm 1 with the parameters
[Equation]
where A = 6(1+ √ 2) 2 , satisfies, for any x ∈ Θ ,
[Equation]
Comparison to state-of-the-art bounds.
We provide two examples of p, q, Θ and compare results for our new method to those of [14, 33] where only the canceling noise Assumption 2 and q ∈ { 1 , 2 } were considered.
Corollary 1.
Let p = q = 2 and Θ
= B d 2 .
Then under Assumption 2, Algorithm 1 with V : Θ → R defined in Example 1, satisfies
[Equation]
In the setup of Corollary 1, Duchi et al.
[14] obtain O ( L (cid:112) dT log( d )) rate and Shamir [33] exhibits O ( L √ dT ) , which is the optimal rate.
Both results do not specify the leading absolute constants.
Corollary 2.
Let p = q = 1 and Θ = (cid:8) x ∈ R d : x ≥ 0 ,
∥ x ∥ 1 = 1 (cid:9) .
Then under Assumption 2,
Algorithm 1 with V : Θ → R defined in Example 2, satisfies
[Equation]
In the setup of Corollary 2, Shamir [33] proves the rate
O
( L √ dT log( d )) for the method with ℓ 2randomization.
On the other hand, Duchi et al.
[14] derived a lower bound Ω( (cid:112)
dT/ log( d )) .
Thus, our algorithm further reduces the gap between the upper and the lower bounds.
Finally, note that in the case p = 1 , q = 2 with V : Θ → R defined in Example 2 the bound of Theorem 1 is of the order O ( (cid:112) T log( d )) .
This case was handled by an algorithm with ℓ 1randomization slightly different from ours in [18] leading to the suboptimal rate O ( (cid:112) dT log( d )) .
Adaptive algorithms
Theorems 1 and 2 used the step size η and the discretization parameter h that depend on the potentially unknown quantities L , σ , and the optimization horizon T .
In this section, we show that, under the canceling noise Assumption 2, adaptation to unknown L comes with nearly no price.
On the other hand, under the adversarial noise Assumption 3, our adaptive rate has a slightly worse dependence on L and σ in the dominant term.
The proof is based on combining the adaptive scheme for online dual averaging [see Section 7.13 in 26, for an overview] with our bias and variance evaluations, cf.
Section 6 below.
Theorem 3.
Let Assumptions 1 and 2 be satisfied.
Then, Algorithm 1 with the parameters[-
-]
[Equation]
and any
[Equation]
satisfies for any x ∈ Θ
[Equation]
The above result gives, up to an absolute constant, the same convergence rate as that of the nonadaptive Theorem 1.
In other words, the price for adaptive algorithm does not depend on the parameters of the problem.
Finally, we derive an adaptive algorithm under Assumption 3.
Theorem 4.
Let Assumptions 1 and 3 be satisfied.
Then, Algorithm 1 with the parameters
[Equation]
and any h [Equation]
=
[Equation]
satisfies for any x ∈ Θ E (cid:34) T (cid:88)
[Equation]
Note that the bound of Theorem 4 has a less advantageous dependency on σ and L compared to
Theorem 2, where we had √ σL instead of σ + L .
We remark that if σ is known but L is unknown, one can recover the √ σL dependency by selecting h t depending on σ .
We do not state this result that can be derived in a similar way and favor here only the fully adaptive version.
Elements of proofs
In this section, we outline major ingredients for the proofs of Theorems 1 – 4.
The full proofs can be found in Appendix C. Here, we only focus on novel elements without reproducing the general scheme of online dual averaging analysis [see e.g., 26, 31].
Namely, we highlight two key facts, which are the smoothing lemma (Lemma 1) and the weighted Poincaré type inequality for the uniform measure on ∂B d 1 (Lemma 3) used to control the variance.
6.1 Bias and smoothing lemma
First, as in the prior work that was using smoothing ideas [see e.g., 16, 22, 33], we show that our gradient estimate g t is an unbiased estimator of a surrogate version of f t and establish its approximation properties.
Lemma 1 (Smoothing lemma) .
Fix h > 0 and q ∈ [1 , ∞ ] .
Let f : R d → R be an L -Lipschitz function w.r.t.
the ℓ q -norm.
Let U be distributed uniformly on B d 1 and ζ be distributed uniformly on ∂B 1 d .
Let f h ( x ) ≜ E [ f ( x + h U )] for x ∈ R d .
Then f h is differentiable and
[Equation]
Furthermore, we have for all d ≥ 3 and all x ∈ R d ,
[Equation]
Finally, if Θ ⊂ R d is convex, f is convex in Θ+ hB d 1 , then f h is convex in Θ and f h ( x ) ≥ f ( x ) for x ∈ Θ .
Proof.
There are three claims to prove.
For the first one, we notice that ζ has the same distribution as − ζ , hence,
[Equation]
and the first claim follows from Theorem 6 in the Appendix (a version of Stokes’, or divergence, theorem) applied to g ( · ) =
f ( x + h · ) with observation that ∇ g ( · ) =
h ∇ f
( x + h · ) where ∇ f is the gradient defined almost everywhere and whose existence is ensured by the Rademacher theorem.
We now prove the approximation property (1).
Assuming d ≥ 3 grants that log( d ) ≥ 1 .
Since f is
L -Lipschitz w.r.t.
the ℓ q -norm we get that, for any x ∈ R d ,
[Equation]
If q ∈ [1 , log( d )) then (1) follows from Lemma 2.
If q ≥ log( d ) then using again Lemma 2 we find
[Equation]
which together with (2) yields the desired bound.
Finally, if f is convex in Θ + hB d 1 , then for all x , x ′ ∈ Θ and α ∈ [0 , 1] we have
[Equation]
Thus f h is indeed convex on Θ .
Furthermore, again by convexity of f , we deduce that for any x ∈ Θ f h ( x ) =
E [ f ( x + h U )] ≥ E [ f ( x ) + ⟨ w , h U ⟩ ] = f ( x ) where w ∈ ∂f ( x ) .
The proof of Lemma 1 relies on the control of the ℓ q -norm of random vector U established in the next result.
Lemma 2.
Let q ∈
[1 , ∞ )
and let U be distributed uniformly on B d 1 .
Then E ∥ U ∥ q ≤ qd 1 q d +1 .
Proof.
Let W 1 , . .
, W d , W d
+1 be i.i.d.
random variables having the Laplace distribution with mean 0 and scale parameter 1 .
Set W =
( W 1 , . . .
, W d ) .
Then, following [6, Theorem 1] we have
[Equation]
where d = denotes equality in distribution.
Furthermore, [30, Lemma 1] states that the random variables [Equation] and ∥ W ∥
[Equation]
+ | W
[Equation]
+
[Equation]
| , are independent.
Hence, for any q ∈ [1 , ∞ ) , it holds that
[Equation]
where ( a ) follows from Jensen’s inequality and ( b ) uses the fact that Γ 1 /q ( q +1) ≤ q for q ≥ 1 .
6.2 Variance and weighted Poincaré type inequality
We additionally need to control the squared ℓ p ∗ -norm of each gradient estimator g t .
This is where we get the main improvement of our procedure compared to previously proposed methods.
To derive the result, we first establish the following lemma of independent interest, which allows us to control the variance of Lipschitz functions on ∂B d 1 .
The proof of this lemma is given in the Appendix.
Lemma 3.
Let d ≥ 3 .
Assume that G : R d → R is a continuously differentiable function, and ζ is distributed uniformly on ∂B d 1 .
Then
[Equation]
Furthermore, if G : R d → R is an L -Lipschitz function w.r.t.
the ℓ 2 -norm then
[Equation]
Remark 1.
Since d 2 / ( d ( d − 2)) ≤ 3 for all d ≥ 3 , the last inequality of Lemma 3 implies that
[Equation]
We can now deduce the following bound on the squared ℓ p ∗ -norm of g t .
Lemma 4.
Let p ∈
[1 , ∞ ] and p ∗ = pp − 1 .
Assume that f t is L -Lipschitz w.r.t.
the ℓ q -norm.
Then, for all d ≥ 3 ,
[Equation]
under canceling noise Assumption
[Equation]
under adversarial noise Assumption
[Equation]
Proof.
Using the definition of g t we get
[Equation]
Let G ( ζ ) ≜
f t ( x t + h ζ ) − f t ( x t − h ζ ) .
First, observe that E [ G ( ζ t ) | x t ] = 0 and under both
Assumption 2 and Assumption 3(ii) it holds that E [ G ( ζ t )( ξ ′ t − ξ ′′ t ) | x t ] = 0 .
Using these remarks and the fact that under adversarial noise Assumption 3, E [( ξ ′ t − ξ ′′ t ) 2 | x t ] ≤ 4 σ 2 , we find:
[Equation]
Var( G ( ζ t ) | x t ) + (cid:40)under cancelling noise Assumption4 σunder adversarial noise Assumption(cid:33) .
Furthermore, since f t is L -Lipschitz, w.r.t.
the ℓ q -norm, the map ζ (cid:55)→ G ( ζ ) is (cid:0) 2 Lhd 1 q ∧ 2 − 12 (cid:1) -
Lipschitz w.r.t.
the ℓ 2 -norm.
Applying (3) to bound Var( G ( ζ t ) | x t ) , yields the desired result.
Note that under adversarial noise Assumption 3, the bound on squared ℓ p ∗ -norm of g t gets an additional term d 4 − 2 p σ 2 h − 2 .
In contrast to the case of canceling noise Assumption 2, this does not allow us to take h arbitrary small hence inducing the bias-variance trade-off.
Numerical illustration
In this section, we provide a numerical comparison of our algorithm with the method based on ℓ 2 -randomization from Shamir [33] (see Appendix D for the definition).
We consider the no noise model and f t = f , ∀ t , with the function f : R d → R defined as
[Equation]
where c = ( c 1 , . . . , c d ) ⊤ ∈ R d such that c j = exp( j ) / (cid:80) di
=1 exp( i ) for j = 1 , . . .
, d .
We choose Θ = (cid:8) x ∈ R d : ∥ x ∥ 1 = 1 , x ≥ 0 (cid:9) and V ( x ) =
d (cid:88)
j = x j log( x j ) .
As stated in Example 2, V is 1 -strongly convex on Θ w.r.t.
the ℓ 1 -norm and R ≤ (cid:112) log( d ) .
Moreover, f is a Lipschitz function w.r.t.
the ℓ 1 -norm.
We deploy the adaptive parameterization that appears in
Theorem 3.
In Figure 1 we present the optimization error of the algorithms, which is defined as
[Equation]
The results are reported over 30 trials. We plot all the 30 runs alongside the average performance.
One can observe that the ℓ 1 -randomization method behaves significantly better than the ℓ 2 -randomization algorithm.
The theoretical bound for our method in this setup has a √ log d gain in the rate. [Figure] i [Figure]
n
[Figure]
Discussion and comparison to prior work
We introduced and analyzed a novel estimator for the gradient based on randomization over the ℓ 1 -sphere.
We established guarantees for the online dual averaging algorithm with the gradient replaced by the proposed estimator.
We provided an anytime and completely data-driven algorithm, which is adaptive to all parameters of the problem.
Our analysis is based on deriving a weighted Poincaré type inequality for the uniform measure on the ℓ 1 -sphere that may be of independent interest.
Under the canceling noise assumption and q ∈ { 1 , 2 } , our setting is analogous to [14, 33].
For the case q = p = 2 and canceling noise, we show that the performance of our method is the same as in [33, Corollary 2] up to absolute constants that were not made explicit in [33].
For the case of q = p = 1 and canceling noise, we improved the bound [33, Corollary 3] by a (cid:112) log( d ) factor.
For the case q = 2 , p ≥ 1 , comparing with the lower bound in [14, Proposition 1], shows that the result of Theorem 1 is minimax optimal.
For the case q = p = 1 , [14, Proposition 2] shows that our result in Theorem 1 is optimal up to a log( d ) factor.
Under the adversarial noise assumption , Theorem 2 provides the rate
O ( T 3 / 4 ) , that is, we get an additional T 1 / 4 factor compared to the canceling noise case.
It remains unclear whether it is optimal under adversarial noise – this question deserves further investigation.
Note that, under sub-Gaussian i.i.d.
noise assumption and q = p = 2 , one can achieve the rate ˜ O
( d a √ T ) with a relatively big a > 0 [2, 8, 13, 21].
In particular, with an ellipsoid type method [21] obtains the rate O ( d 4 . 5 √ T log( T ) 2 ) for the cumulative regret.
Finally, let us discuss the compactness of Θ . It is straightforward to extend the results of Theorems 1,to any closed convex Θ considering the regret against a fixed action x ∈ Θ . Indeed, using [26, Corollary 7.9], one only needs to replace R appearing in both Theorems 1, 2 by an upper bound on (cid:112)
V ( x ) − inf x ′ ∈ Θ V ( x ′ ) .
The adaptive case is more complicated. One way to tackle this case is to use [27, Theorem 1] requiring a control of E max t =1 ,...,T ∥ g t ∥ p ∗ . This term can be controlled under the canceling noise Assumption 2 using the Lipschitzness of f t ’s, so that Theorem 3 extends to unbounded Θ .
However, without the canceling noise assumption, following the approach outlined above, one needs to control E max t =1 ,...,T | ξ ′ t − ξ ′′ t | h t . The adversarial noise Assumption 3 is not sufficient to reasonably control this term, so that extending Theorem 4 to unbounded Θ is not possible without further assumptions. Acknowledgments and disclosure of funding The work of E. Chzhen was supported by ANR PIA funding: ANR-20-IDEES-0002.
The work of M.
Pontil is partially supported by European Union’s Horizon 2020 research and innovation programme under ELISE grant agreement No 951847.
The research of A. Akhavan and A. B. Tsybakov is supported by a grant of the French National Research Agency (ANR), “Investissements d’Avenir” (LabEx Ecodec/ANR-11-LABX-0047).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] at https: //github.com/evgchz/Akhavan_Chzhen_Pontil_Tsybakov (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplementary Material
This supplementary material contains the proofs and results omitted from the main body.
In Appendix A we recall the appropriate version of the Stokes’ theorem and discuss its applicability for
Lipschitz functions on B d 1 .
In Appendix B we provide the proof of Lemma 3.
Finally, in Appendix C we provide the proofs of Theorems 1, 2, 3, 4.
Additional notation For two functions g, η : R d → R , we denote by η ⋆ g their convolution defined point-wise for x ∈ R d as
[Equation]
The standard mollifier η ϵ : R d → R is defined as η ϵ ( x ) =
ϵ − d η 1 ( x /ϵ ) for ϵ > 0 and x ∈ R , where η 1 : R d → R is defined as
[Equation]
with C chosen so that (cid:82) R d η 1 ( x ) d x = 1 .
A Integration by parts
We first recall the following result that can be found in [34, Section 13.3.5, Exercise 14a].
Theorem 5 (Integration by parts in a multiple integral) .
Let D be an open connected subset of R d with a piecewise smooth boundary ∂D oriented by the outward unit normal n = ( n 1 , . . . , n d ) ⊤ .
Let g be a continuously differentiable function in D ∪ ∂D .
Then
[Equation]
Remark 2.
We refer to [34, Section 12.3.2, Definitions 4 and 5] for the definition of piecewise smooth surfaces and their orientations respectively.
The idea of using the instance of Theorem 5 (also called Stokes’ theorem) with D = B d 2 to obtain ℓ 2 -randomized estimators of the gradient belongs to Nemirovsky and Yudin [22].
It was further used in several papers [5, 16, 31, 33] to mention just a few.
Those papers were referring to [22] but [22] did not provide an exact statement of the result (nor a reference) and only tossed the idea in a discussion.
However, the classical analysis formulation as presented in Theorem 5 does not apply to
Lipschitz continuous functions that were considered in [5, 16, 31, 33].
We are not aware of whether its extension to Lipschitz continuous functions, though rather standard, is proved in the literature.
In this paper, we apply Theorem 5 with the ℓ 1 -ball D
= B d 1 .
Our aim in this section is to provide a variant of Theorem 5 applicable to a Lipschitz continuous function g : R d → R , which is not necessarily continuously differentiable on D ∪ ∂D =
B d 1 ∪ ∂B d 1 .
To this end, we will go through the argument of approximating g by C ∞ (Ω) functions, where Ω ⊂ R d is an open bounded connected subset of R d such that D ∪ ∂D ⊂ Ω .
Let g n = η 1 /n ⋆ g , where η 1 /n is the standard mollifier.
Let g : R d → R be a function satisfying the Lipschitz condition w.r.t.
the ℓ 1 -norm: | g ( u ) − g ( u ′ )
| ≤ L ∥
u − u ′ ∥ 1 .
Since g is continuous in Ω and, by construction D ∪ ∂D ⊂ Ω , then using basic properties of mollification [see e.g., 15, Theorem 4.1 (ii)] we have g n −→ g uniformly on D ∪ ∂D (in particular, uniformly on ∂D ).
Furthermore, let ∇ g be the gradient of g , which by Rademacher theorem [see e.g., 15, Theorem 3.2] is well defined almost everywhere w.r.t. the Lebesgue measure and
[Equation]
It follows that ∂g∂u j is absolutely integrable on Ω for any j ∈ [ d ] .
Furthermore, since
[Equation]
we can apply [15, Theorem 4.1 (iii)] that yields
[Equation]
Combining the above remarks we obtain that the result of Theorem 5 is valid for functions
g that are
Lipschitz continuous w.r.t.
the ℓ 1 -norm.
Thus, it is also valid when the Lipschitz condition is imposed w.r.t.
any ℓ q -norm with q ∈ [1 , ∞ ] .
Specifying this conclusion for the particular case D = B d 1 , we obtain the following theorem.
Theorem 6.
Let the function g : R d → R be Lipschitz continuous w.r.t.
the ℓ q -norm with q ∈ [1 , ∞ ] .
Then
[Equation]
where ∇ g ( · ) is defined up to a set of zero Lebesgue measure by the Rademacher theorem.
B Proof of Lemma
To prove Lemma 3, we first recall the weighted Poincaré inequality for the univariate exponential measure (mean 0 and scale parameter 1 Laplace distribution).
Lemma 5 (Lemma 2.1 in [9]) .
Let W be mean 0 and scale parameter 1 Laplace random variable.
Let g : R → R be continuous almost everywhere differentiable function such that
[Equation]
and
[Equation]
then,
[Equation]
We are now in a position to prove Lemma 3.
The proof is inspired by [7, Lemma 2].
Proof of Lemma 3.
Throughout the proof, we assume without loss of generality that E
[ G ( ζ )] = 0 .
Indeed, if it is not the case, we use the result for the centered function ˜ G ( ζ ) =
G ( ζ ) − E
[ G ( ζ )] , which has the same gradient.
First, consider the case of continuously differentiable G .
Let W =
( W 1 , . . .
, W d ) be a vector of i.i.d.
mean 0 and scale parameter 1 Laplace random variables and define T ( w ) = w / ∥ w ∥ 1 .
Introduce the notation
[Equation]
Lemma 1 in [30] asserts that, for ζ uniformly distributed on ∂B d 1 ,
[Equation]
and T ( W ) is independent of ∥ W ∥ 1 .
[Equation]
In particular,
[Equation]
Using the Efron-Stein inequality [see e.g., 11, Theorem 3.1] we obtain
[Equation]
where [Equation] with W − i ≜ ( W 1 , . .
, W i − 1 , W i +1 , . .
. , W d ) . Note that on the event { W − i ̸ = 0 } (whose complement has zero measure), the function w
[Equation]
W [Equation] , W i [Equation]
[Equation]
, W
[Equation]
. , W d ) , satisfies the assumptions of Lemma 5.
Thus,
[Equation]
In order to compute ∇ F ( W ) , we observe that for every i ̸ = j ∈ [ d ] we have for all w ̸ = 0 such that w i ,
[Equation]
and ∂T i[-̸ = j ∈ [ d ] we have for all w ̸ = 0 such that w i , w j ̸ = 
-]
[Equation]
Thus, the Jacobi matrix of T ( w ) has the form
[Equation]
It follows that almost surely
[Equation]
∇ G
[Equation]
W )) . Observe that since ⟨ sign( W ) , T ( W ) ⟩ = 1 almost surely, we have (cid:0) sign( W ) (cid:1) ⊤ (cid:18)
[Equation]
W
[Equation]
sign( W
[Equation]
⊤
[Equation]
( T ( W )) = 0 almost surely .
The above two equations imply that almost surely
[Equation]
where we used the fact that the operator norm of I − ab ⊤ is not greater than 1+ ∥ a ∥ 2 ∥ b ∥ 2 .
Combining the above bound with (5), and using the facts that E [ ∥ W ∥ − 1 1 ] = 1 d − 1 , E [ G ( T ( W ))]
= E [ G ( ζ )]
=and the independence of ∥ W ∥ 1 and T ( W ) (cf. (4)) yields
[Equation]
Rearranging, we deduce the first claim of the lemma since T ( W ) d = ζ .
To prove the second statement of the lemma regarding Lipschitz functions, it is sufficient to apply the first one to G n —the sequence of smoothed versions of G such that G n ∈ C ∞ ( R ) and
G n −→ G , uniformly on every compact subset, and sup n ≥ 1 ∥∇ G n ( x ) ∥ 2 ≤ L for almost all x ∈ R d . A sequence
G n satisfying these properties can be constructed by standard mollification due to the fact that G is
Lipschitz continuous [see e.g., 15, Theorem 4.2].
Finally, to obtain the value E ∥ T ( W ) ∥ 22 = E ∥
ζ ∥we use Lemma 6 below.
Lemma 6.
Let ζ be distributed uniformly on ∂B d 1 .
Then, E ∥ ζ ∥ 22 = 2 d +1 .
Proof.
We use the same tools as in the proof of Lemma 2. Let W =
( W 1 , . . .
, W d ) be a vector of i.i.d.
random variables following the Laplace distribution with mean 0 and scale parameter 1 .
By (4) we have that ζ d = W ∥ W ∥ 1 and ζ is independent of ∥ W ∥ 1 .
Therefore,
[Equation]
Here,
[Equation]
Furthermore, ∥ W ∥ 1 follows the Erlang distribution with parameters ( d, 1) , which implies
[Equation]
The lemma follows by combining (6) – (8).
C Upper bounds
The proofs of Theorems 1, 2, 3, 4 resemble each other. They only differ in the ways of handling the variance terms depending on ∥ g t ∥ 2 p ∗ and in the choice of parameters.
For this reason, we suggest the interested reader to follow the proofs in a linear manner starting from the next paragraph.
Common part of the proofs of Theorems 1, 2.
We start with the part of the proofs that is common for Theorems 1, 2.
Fix some x ∈ Θ .
Due to Assumption 1, we can use Lemma 1, which implies
E
[Equation]
T
[Equation]
where f t,h ( x ) =
E [ f t ( x + h U )] with U uniformly distributed on B d 1 .
Furthermore, by the approximation property derived in Lemma 1 and the standard bound on the cumulative regret of dual averaging algorithm [see e.g., 26, Corollary 7.9.]
we deduce that
[Equation]
where in the last inequality we used the identity η 1 = . . .
= η T = η . The results of Theorems 1,follow from the bound (9) as detailed below.
Proof of Theorem 1.
Here h 1 = . . .
=
h T = h , and we work under Assumption 2.
In this case, bounding E ∥ g t ∥
p ∗ in (9) via Lemma 4 yields
[Equation]
Minimizing the the right hand side of the above inequality over η > 0 and substituting η =
R L ( √
[Equation]
[Equation]
we deduce that
[Equation]
Taking h ≤ 7 R 100b q ( d ) √ T d 12 + 1 q ∧ 2 − 1 p makes negligible the second summand in the above bound.
This concludes the proof.
Proof of Theorem 2.
Here again {+h 
+} = . . .
=
h T =
h [Equation]
but we work under Assumption 3.
Then, bounding E ∥ g t ∥
p ∗ in (9) via Lemma 4 yields
[Equation]
Minimizing the right hand side of the above inequality over η > 0 and substituting the optimal value
[Equation]
results in the following upper bound on the regret
E
[Equation]
where for the last inequality we used the fact that √ a + b ≤ √ a + √ b for a, b ≥ 0 .
Minimizing over h > 0 the last expression and substituting the optimal value
h = (cid:16) √ 2 Rσ L b q ( d ) (cid:17) 12 T − 14 d 1 − 12 p we get E
[Equation]
Common part of the proofs of Theorems 3, 4. Here, we state the common parts of the proofs for Theorems 3, 4. Similar to the first inequality in (9), we have
[Equation]
Note that without loss of generality, we can assume that (cid:80) tk =1 ∥
g k ∥ 2 p ∗ ̸ = 0 , for all t ≥ 1 .
This is a consequence of the fact that if (cid:80) tk =1 ∥ g k ∥ 2 p ∗ = 0 , then the first term on the r.h.s.
of the above inequality will be zero up to round t .
Thus, we can erase these iterates from the cumulative regret, only paying the bias term for those rounds. In what follows we essentially use [27, Corollary 1], which we re-derive for the sake of clarity.
Assume that η t
[Equation]
for t ∈ { [Equation]
T } and λ > 0 . Then, applying [27, Theorem 1] we deduce that
[Equation]
where we introduced D = sup u , w ∈ Θ ∥ u − w ∥ p . By [27, Proposition 1], we have D ≤ √ 8 R .
Moreover, by Jensen’s inequality, using the rough bound E [max t ∈ [ T ] ∥ g t ∥ p ∗ ] ≤ (cid:114) (cid:80) Tt =1 E (cid:104) ∥ g t ∥ 2 p
[Equation]
(cid:105) , and substituting λ = R √ 2 . 75 , we deduce that
E
[Equation]
Proofs of Theorems 3, 4 provided below follow from the above inequality by properly selecting h t > 0 .
Proof of Theorem 3.
The bound of Lemma 4 under Assumption 2 applied to (10) yields
E (cid:34) T (
cid:88) t
[Equation]
[Equation] t [Equation] Td
[Equation] d [Equation] Taking h t ≤ 7 R 200b q ( d ) √ t d 12 + 1 q ∧ 2 − 1 p makes negligible the last summand in the above bound.
This concludes the proof.
Proof of Theorem 4.
Using (10), the bound of Lemma 4 under Assumption 3 and the fact that √ a + b ≤ √ a + √ b for a, b ≥ 0 , we deduce that
E
[Equation]
75 +
[Equation]
5 √
[Equation]
(cid:32) [Equation] h [Equation]
+
[Equation]
[Equation] − [Equation] p (cid:33)+
[Equation] ( d ) T (cid:88) t = h t ≤ 110 . 6 · RL (cid:113)
Td 1+ 2 q ∧ 2 − 2 p + 13 . 3 R · d 2 − 1 p σ (cid:32) T (cid:88) t =  h 2 t (cid:33)+ L b q ( d ) T (cid:88) t = h t . Since h t = (cid:16) 6 . 65 √ 6 · R b q ( d ) (cid:17) 12 t − 14 d 1 − 12 p and (cid:80) Tt =1 t 12 ≤ 23 T 32 and (cid:80) Tt =1 t − 14 ≤ 43 T 34 , we get
E (cid:34) T (cid:88) t = (cid:0) f t ( x t ) − f t ( x ) (cid:1)(cid:35) ≤ 110 . 6 · RL (cid:113) Td 1+ 2 q ∧ 2 − 2 p + 5 . 9 · √
[Equation]
( σ + L ) T 34 (cid:113) b q ( d ) d 12 − 12 p .
D Definition of ℓ 2 -randomized estimator
In this section we recall the algorithm of Shamir [33].
Let ζ ◦ ∈ R
d be distributed uniformly on ∂B d 2 .
Instead of the gradient estimator that we introduce in Algorithm 1, at a each step t ≥ 1 , Shamir [33] uses
[Equation]
where y ′ t = f t ( x t + h t ζ ◦ ) , y ′′ t = f t ( x t − h t ζ ◦ t ) , and ζ ◦ t ’s are independent random variables with the same distribution as ζ ◦ .