A gradient estimator via L1-randomization for online zero-order optimization with two point feedback
Abstract
This work studies online zero-order optimization of convex and Lipschitz functions.
We present a novel gradient estimator based on two function evaluations and randomization on the ` 1 -sphere.
Considering different geometries of feasible sets and Lipschitz assumptions we analyse online dual averaging algorithm with our estimator in place of the usual gradient.
We consider two types of assumptions on the noise of the zero-order oracle: canceling noise and adversarial noise.
We provide an anytime and completely data-driven algorithm, which is adaptive to all parameters of the problem.
In the case of canceling noise that was previously studied in the literature, our guarantees are either comparable or better than stateof-the-art bounds obtained by Duchi et al.
[14] and Shamir [33] for non-adaptive algorithms.
Our analysis is based on deriving a new weighted Poincaré type inequality for the uniform measure on the ` 1 -sphere with explicit constants, which may be of independent interest.
Introduction
In this work we study the problem of convex online zero-order optimization with two-point feedback, in which adversary ﬁxes a sequence f 1 , f 2 , . . .
: R d ! R of convex functions and the goal of the learner is to minimize the cumulative regret with respect to the best action in a prescribed convex set ⇥ ✓ R d .
This problem has received signiﬁcant attention in the context of continuous bandits and online optimization [see e.g., 1, 3, 12, 13, 16, 18, 21, 25, 29, 33, and references therein].
We consider the following protocol: at each round t = 1 , 2 , . . .
the algorithm chooses x 0 t , x 00 t 2 R d (that can be queried outside of ⇥ ) and the adversary reveals
[Equation]
where ⇠ 0 t , ⇠ 00 t 2 R are the noise variables (random or not) to be speciﬁed.
Based on the above information and the previous rounds, the learner outputs x t 2 ⇥ and suffers loss f t ( x t ) .
The goal of the learner is to minimize the cumulative regret
[Equation]
At the core of our approach is a novel zero-order gradient estimator based on two function evaluations outlined in Algorithm 1.
A key novelty of our estimator is that it employs a randomization step over 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Algorithm
Input: Convex function V ( · ) , step size ⌘ 1 > 0 , and parameters h t , for t = 1 , 2 , . . . ,
Initialization: Generate independently vectors ⇣ 1 , ⇣ 2 , . . .
uniformly distributed on @ B d 1 , and set z 1 =for t = 1 , . .
, do x t = arg max x 2 ⇥ { ⌘ t [Equation] z t , x [Equation]
[Equation]
x ) [Equation] y 0 t = f t ( x t + h t ⇣ t ) + ⇠ 0 t and y {+
+} t = f t ( x t  h t ⇣ t ) + ⇠ 00 t //
Query g t =
[Equation]
2 h t (
[Equation]
0 t
[Equation]
[Equation] ) sign( [Equation] t ) //
` 1 -gradient estimate z t +1 =
z t  g t // Update z t update the step-size ⌘ t + end the ` 1 sphere.
This is in contrast to most of the prior work [see e.g., 1, 3–5, 14, 16, 17, 22, 28, 32] that was employing ` 2 or ` 1 type randomizations to deﬁne x 0 t , x 00 t .
We use the proposed estimator within an online dual averaging procedure to tackle the zero-order online convex optimization problem, matching or improving the state-of-the-art results.
Duchi et al.
[14] and Shamir [33] have studied instances of the above problem under the assumption that ⇠ 0 t = ⇠ 00 t , which we will further refer to as canceling noise assumption.
Speciﬁcally, [14] considered the stochastic optimization framework where f t = f , for every t , and obtained bounds on the optimization error rather than on cumulative regret, while [33] analyzed the case ⇠ 0 t = ⇠ 00 t = 0 .
The results in [14, 33] are obtained for the objective functions that are Lipschitz with respect to the ` q -norm for q = 1 and q = 2 , although, with extra derivations it is possible to extend the above mentioned results beyond such cases.
The proposed method allows us to improve upon these results in several aspects.
Contributions.
The contributions of the present paper can be summarized as follows.
1) We present a new randomized zero-order gradient estimator and study its statistical properties, both under canceling noise and under adversarial noise (see Lemma 1 and Lemma 4); 2)
In the canceling noise case ( ⇠ 0 t = ⇠ 00 t ) in Theorem 1 we show that dual averaging based on our gradient estimator either improves or matches the state-of-the-art bounds [14, 33].
We derive the results for Lipschitz functions with respect to all ` q -norms, q 2 [1 , 1 ] .
In particular, when q = 1 and ⇥ is the probability simplex, our bound is better by a p log( d ) factor than that of [14, 33]; 3) We propose a completely data-driven and anytime version of the algorithm, which is adaptive to all parameters of the problem.
We show that it achieves analogous performance as the non-adaptive algorithm in the case of canceling noise and only slightly worse performance under adversarial noise.
To the best of our knowledge, no adaptive algorithms were developed for zero-order online problems in our setting so far; 4) As a key element of our analysis, we derive in Lemma 3 a weighted Poincaré type inequality [following the terminology of 10] with explicit constants for the uniform measure on ` 1 -sphere.
This result may be of independent interest.
Notation.
Throughout the paper we use the following notation.
We denote by k · k p the ` p -norm in
R d .
For any x 2 R d we denote by x 7! sign( x ) the component-wise sign function (deﬁned at 0 as 1 ).
We let h · , · i be the standard inner product in R d .
For p 2 [1 , 1 ] we introduce the open ` p -ball and ` p -sphere respectively as
[Equation]
For two a, b 2 R , we denote by a ^ b ( resp.
a _ b ) the minimum ( resp.
the maximum) between a and b .
We denote by  : (0 , 1 ) !
R , the gamma function.
In what follows, log always stands for the natural logarithm and e is Euler’s number.
The algorithm
Let ⇥ be a closed convex subset of R d and let V : ⇥ !
R be a convex function.
The procedure that we propose in this paper is summarized in Algorithm 1.
Intuition behind the gradient estimate.
The form of gradient estimator g t in Algorithm 1 is explained by Stokes’ theorem (see Theorem 5 in the appendix and the discussion that follows).
Stokes’ theorem provides a connection between the gradient of a function f
(ﬁrst order information) and f itself (zero order information).
Under some regularity conditions, it establishes that
[Equation]
where @ D is the boundary of D , n is the outward normal vector to @ D , and d S ( x ) denotes the surface measure.
Introducing U D and ⇣ @ D distributed uniformly on D and @ D respectively, we can rewrite the above identity as
[Equation]
where Vol d  1 ( @ D ) is the surface area of D and Vol d ( D ) is its volume.
In what follows we consider the special case D
= B d 1 .
For this choice of D we have n ( x ) = 1 p d · sign( x ) with
Vol d  1 ( @ D ) /
Vol d ( D ) =
d 3 / 2 leading to our gradient estimate for the two-point feedback setup.
Computational aspects.
Let us highlight two appealing practical features of the ` 1 -randomized gradient estimator g t in Algorithm 1.
First, we can easily evaluate any ` p -norm of g t .
Indeed, it holds that k g t k p =
( d 1+1 /p / 2 h t )
| y {+0 t  y 
+}
t | , i.e., computing k g t k p only requires O (1) elementary operations.
Second, this gradient estimator is very economic in terms of the required memory: in order to store g t we only need d bits and 1 ﬂoat.
None of these properties is inherent to the popular alternatives based on the randomization over the ` 2 -sphere [see e.g., 5, 16, 22] or on Gaussian randomization [see e.g., 19, 23, 24].
To compute g t one needs to generate ⇣ t distributed uniformly on @ B d 1 .
The most straightforward way to do it consists in ﬁrst generating a d -dimensional vector of i.i.d.
centered scaled Laplace random variables and then normalizing this vector by its ` 1 -norm.
The result is guaranteed to follow the uniform distribution on @ B 1 d [see e.g., 30, Lemma 1].
Furthermore, to sample from the centered scaled Laplace distribution one can simply use inverse transform sampling.
Indeed, if U is distributed uniformly on (0 , 1) , then log(2 U ) 1 ( U > 1 / 2)  log(2  2 U ) 1 ( U  1 / 2) follows centered scaled
Laplace distribution.
Assumptions
We say that the convex function V ( · ) is 1 -strongly convex with respect to the ` p -norm on ⇥ if [Equation]
[Equation]
, for all x , x 0 2 ⇥ and all w 2 @ V ( x ) , where @ V ( x ) is the subdifferential of V at point x .
Throughout the paper, we assume that p, q 2 [1 , 1 ] , d  3 , and set p ⇤ , q ⇤ 2 [1 , 1 ] such thatp + 1 p ⇤ = 1 and 1 q + 1 q ⇤ = 1 , with the usual convention 1 / 1 = 0 .
We will use the following assumptions.
Assumption 1.
The following conditions hold:
The set ⇥ ⇢ R d is compact and convex.
There exists V : ⇥ ! R , which is lower semi-continuous, 1 -strongly convex on ⇥ w.r.t.
the ` p -norm and such that
[Equation] [Equation]
for some constant R > 0 . 3.
Each function f t : R d ! R is convex on R d for all t  1 . 4.
For all x , x 0 2 R d , and all t  1 we have | f t ( x )  f t ( x 0 ) |  L k x  x 0 k q for some constant L > 0 .
Assumption 1 is rather standard in the study of dual averaging-type algorithms and have been previously considered in the context of zero-order problems in [14, 33].
We assume that ⇥ is compact as we are interested in the worst-case regret, which ensures that R < + 1 .
We discuss extensions of our results to the case of unbounded ⇥ in Section 8.
Note that the constant R > 0 is not necessarily dimension independent.
Below we provide two classical examples of V [see e.g., 31, Section 2].
Example 1.
Let ⇥ be any convex subset of R d and p 2 (1 , 2] .
Then, V ( x ) = 1 2( p  1) k x k 2 p is 1 -strongly convex on ⇥ w.r.t.
the ` p -norm.
Example 2.
Let ⇥ =  x 2 R d : k x k 1 = 1 , x  0 .
Then 1 , V ( x ) =
P dj =
1 x j log( x j ) is 1strongly convex on ⇥ w.r.t.
the ` 1 -norm and R 2  log( d ) .
Assumptions on the noise.
We consider two different assumptions on the noises ⇠ 0 t , ⇠ 00 t .
The ﬁrst noise assumption is common in the stochastic optimization context [see e.g., 14, 19, 23, 24, 33].
Assumption 2 (Canceling noise) .
For all t = 1 , 2 , . . .
, it holds that ⇠ 0 t = ⇠ 00 t almost surely.
Formally, Assumption 2 permits noisy evaluations of function values.
However, due to the fact that we are allowed to query f t at two points, taking difference of y 0 t and y 00 t in the estimator of the gradient effectively erases the noise.
It results in a smaller variance of our gradient estimator.
Importantly,
Assumption 2 covers the case of no noise, that is, the classical online optimization setting as deﬁned, e.g., in [31].
Second, we consider an adversarial noise assumption, which is essentially equivalent to the assumptions used in [3, 4].
Assumption 3 (Adversarial noise) .
For all t = 1 , 2 , . . .
, it holds that: (i) E [( ⇠ 0 t ) 2 ]   2 and
E [( ⇠ 00 t ) 2 ]   2 ; (ii) ( ⇠ 0 t ) t  1 and ( ⇠ 00 t ) t  1 are independent of ( ⇣ t ) t
Assumption 3 allows for stochastic ⇠ 0 t and ⇠ 00 t that are not necessarily zero-mean or independent over the trajectory.
Furthermore, it permits bounded non-stochastic adversarial noises.
Part (ii) of
Assumption 3 is always satisﬁed.
Indeed, ⇠ 0 t ’s and ⇠ 00 t ’s are coming from the environment and are unknown to the learner while ⇣ t ’s are artiﬁcially generated by the learner.
We mention part (ii) only for formal mathematical rigor.
Note that, since the choice of function V belongs to the learner and ⇥ is given, it is always reasonable to assume that parameter R is known.
At the same time, parameters L and  may be either known or unknown.
We will study both cases in the next sections.
Upper bounds on the regret
In this section, we present the main convergence results for Algorithm 1 when L,  , T are known to the learner.
The case when they are unknown is analyzed in Section 5, where we develop fully adaptive versions of Algorithm 1.
To state our results in a uniﬁed way, we introduce the following sequence that depends on the dimension d and on the norm index q  1 :
[Equation]
The value b q ( d ) will explicitly inﬂuence the choice of the step size ⌘ > 0 and of the discretization parameter h > 0 .
The ﬁrst result of this section establishes the convergence guarantees under the canceling noise assumption.
This case was previously considered by Duchi et al.
[14] and Shamir [33].
Theorem 1.
Let Assumptions 1 and 2 be satisﬁed.
Then, Algorithm 1 with the parameters
[Equation]
and any h 
[Equation]
where A = ( p 6 + p 12)  1 , satisﬁes, for any x 2 ⇥ ,
[Equation]
Note that, as in other related works [14, 20, 23, 24, 33], under the canceling noise (or no noise) assumption the discretization parameter h > 0 can be chosen arbitrary small.
This is due to the fact that, under the canceling noise assumption, the variance of the gradient estimate g t is bounded by a constant independent of h .
It is no longer the case under the adversarial noise assumption as exhibited in the next theorem.
Theorem 2.
Let Assumptions 1 and 3 be satisﬁed.
Then Algorithm 1 with the parameters
[Equation]
where A = 6(1+ p 2) 2 , satisﬁes, for any x 2 ⇥ ,
[Equation]
Comparison to state-of-the-art bounds.
We provide two examples of p, q, ⇥ and compare results for our new method to those of [14, 33] where only the canceling noise Assumption 2 and q 2 { 1 , 2 } were considered.
Corollary 1.
Let p = q = 2 and ⇥ =
B d 2 .
Then under Assumption 2, Algorithm 1 with V : ⇥ ! R deﬁned in Example 1, satisﬁes
[Equation]
In the setup of Corollary 1, Duchi et al.
[14] obtain O ( L p dT log( d )) rate and Shamir [33] exhibits O ( L p dT ) , which is the optimal rate.
Both results do not specify the leading absolute constants.
Corollary 2.
Let p = q = 1 and ⇥ =  x 2 R d : x  0 , k x k 1 = 1 .
Then under Assumption 2, Algorithm 1 with V : ⇥ ! R deﬁned in Example 2, satisﬁes
[Equation]
In the setup of Corollary 2, Shamir [33] proves the rate
O
( L p dT log( d )) for the method with ` 2randomization.
On the other hand, Duchi et al.
[14] derived a lower bound ⌦ ( p dT/ log( d )) .
Thus, our algorithm further reduces the gap between the upper and the lower bounds.
Finally, note that in the case p = 1 , q = 2 with V : ⇥ ! R deﬁned in Example 2 the bound of Theorem 1 is of the order O ( p T log( d )) .
This case was handled by an algorithm with ` 1randomization slightly different from ours in [18] leading to the suboptimal rate O ( p dT log( d )) .
Adaptive algorithms
Theorems 1 and 2 used the step size ⌘ and the discretization parameter h that depend on the potentially unknown quantities L ,  , and the optimization horizon T .
In this section, we show that, under the canceling noise Assumption 2, adaptation to unknown L comes with nearly no price.
On the other hand, under the adversarial noise Assumption 3, our adaptive rate has a slightly worse dependence on L and  in the dominant term.
The proof is based on combining the adaptive scheme for online dual averaging [see Section 7.13 in 26, for an overview] with our bias and variance evaluations, cf.
Section 6 below.
Theorem 3.
Let Assumptions 1 and 2 be satisﬁed.
Then, Algorithm 1 with the parameters⌘ [Equation] q
[Equation]
and any h t 
[Equation]
satisﬁes for any x 2 ⇥
[Equation]
The above result gives, up to an absolute constant, the same convergence rate as that of the nonadaptive Theorem 1.
In other words, the price for adaptive algorithm does not depend on the parameters of the problem.
Finally, we derive an adaptive algorithm under Assumption 3.
Theorem 4.
Let Assumptions 1 and 3 be satisﬁed.
Then, Algorithm 1 with the parameters
[Equation]
and any h t = ✓
[Equation]
satisﬁes for any x 2 ⇥
E "
[Equation]
Note that the bound of Theorem 4 has a less advantageous dependency on  and L compared to
Theorem 2, where we had p  L instead of  + L .
We remark that if  is known but L is unknown, one can recover the p  L dependency by selecting h t depending on  .
We do not state this result that can be derived in a similar way and favor here only the fully adaptive version.
Elements of proofs
In this section, we outline major ingredients for the proofs of Theorems 1 – 4.
The full proofs can be found in Appendix C.
Here, we only focus on novel elements without reproducing the general scheme of online dual averaging analysis [see e.g., 26, 31].
Namely, we highlight two key facts, which are the smoothing lemma (Lemma 1) and the weighted Poincaré type inequality for the uniform measure on @ B d 1 (Lemma 3) used to control the variance.
6.1 Bias and smoothing lemma
First, as in the prior work that was using smoothing ideas [see e.g., 16, 22, 33], we show that our gradient estimate g t is an unbiased estimator of a surrogate version of f t and establish its approximation properties.
Lemma 1 (Smoothing lemma) .
Fix h > 0 and q 2 [1 , 1 ] .
Let f : R d ! R be an L -Lipschitz function w.r.t.
the ` q -norm.
Let U be distributed uniformly on B d 1 and ⇣ be distributed uniformly on @ B 1 d .
Let f h ( x ) , E [ f ( x + h U )] for x 2 R d .
Then f h is differentiable and
[Equation]
Furthermore, we have for all d  3 and all x 2 R d ,
[Equation]
Finally, if ⇥ ⇢ R d is convex, f is convex in ⇥ + hB d 1 , then f h is convex in ⇥ and f h ( x )  f ( x ) for x 2 ⇥ .
Proof.
There are three claims to prove.
For the ﬁrst one, we notice that ⇣ has the same distribution as  ⇣ , hence,
[Equation]
and the ﬁrst claim follows from Theorem 6 in the Appendix (a version of Stokes’, or divergence, theorem) applied to g ( · ) =
f ( x + h · ) with observation that r g ( · ) =
h r f ( x + h · ) where r f is the gradient deﬁned almost everywhere and whose existence is ensured by the Rademacher theorem.
We now prove the approximation property (1).
Assuming d  3 grants that log( d )  1 .
Since f is
L -Lipschitz w.r.t.
the ` q -norm we get that, for any x 2 R d ,
[Equation]
If q 2 [1 , log( d )) then (1) follows from Lemma 2.
If q  log( d ) then using again Lemma 2 we ﬁnd
[Equation]
which together with (2) yields the desired bound.
Finally, if f is convex in ⇥ + hB d 1 , then for all x , x 0 2 ⇥ and ↵ 2 [0 , 1] we have
[Equation]
Thus f h is indeed convex on ⇥ .
Furthermore, again by convexity of f , we deduce that for any x 2 ⇥
[Equation] = f ( x ) where w 2 @ f ( x ) .
The proof of Lemma 1 relies on the control of the ` q -norm of random vector U established in the next result.
Lemma 2.
Let q 2 [1 , 1 ) and let U be distributed uniformly on B d 1 .
Then E k U k q  qd 1 q d +1 .
Proof.
Let W 1 , . .
, W d , W d
+1 be i.i.d.
random variables having the Laplace distribution with mean 0 and scale parameter 1 .
Set W =
( W 1 , . . .
, W d ) .
Then, following [6, Theorem 1] we have
[Equation]
where d = denotes equality in distribution.
Furthermore, [30, Lemma 1] states that the random variables
[Equation]
and
[Equation]
are independent.
Hence, for any q 2 [1 , 1 ) , it holds that
[Equation]
where ( a ) follows from Jensen’s inequality and ( b ) uses the fact that  1 /q ( q +1)  q for q  1 .
6.2 Variance and weighted Poincaré type inequality
We additionally need to control the squared ` p ⇤ -norm of each gradient estimator g t .
This is where we get the main improvement of our procedure compared to previously proposed methods.
To derive the result, we ﬁrst establish the following lemma of independent interest, which allows us to control the variance of Lipschitz functions on @ B d 1 .
The proof of this lemma is given in the Appendix.
Lemma 3.
Let d  3 .
Assume that G : R d ! R is a continuously differentiable function, and ⇣ is distributed uniformly on @ B d 1 .
Then
[Equation]
Furthermore, if G : R d ! R is an L -Lipschitz function w.r.t.
the ` 2 -norm then
[Equation]
Remark 1.
Since d 2 / ( d ( d  2))  3 for all d  3 , the last inequality of Lemma 3 implies that
[Equation]
We can now deduce the following bound on the squared ` p ⇤ -norm of g t .
Lemma 4.
Let p 2 [1 , 1 ] and p ⇤ = pp  1 .
Assume that f t is L -Lipschitz w.r.t.
the ` q -norm.
Then, for all d  3 ,
[Equation]
under canceling noise Assumption
[Equation]
under adversarial noise Assumption
[Equation]
Proof.
Using the deﬁnition of g t we get
[Equation]
Let G ( ⇣ ) , f t ( x t + h ⇣ )  f t ( x t  h ⇣ ) .
First, observe that E [ G ( ⇣ t ) | x t ] = 0 and under both
Assumption 2 and Assumption 3(ii) it holds that E [ G ( ⇣ t )( ⇠ 0 t  ⇠ 00 t ) | x t ] = 0 .
Using these remarks and the fact that under adversarial noise Assumption 3, E [( ⇠ 0 t  ⇠ 00 t ) 2 | x t ]  4  2 , we ﬁnd:
[Equation]
under cancelling noise Assumption4 under adversarial noise Assumption 3 ! .
Furthermore, since f t is L -Lipschitz, w.r.t.
the ` q -norm, the map ⇣ 7! G ( ⇣ ) is  2 Lhd 1 q ^ 2  -
Lipschitz w.r.t.
the ` 2 -norm.
Applying (3) to bound Var( G ( ⇣ t ) | x t ) , yields the desired result.
Note that under adversarial noise Assumption 3, the bound on squared ` p ⇤ -norm of g t gets an additional term d 4  2 p  2 h  2 .
In contrast to the case of canceling noise Assumption 2, this does not allow us to take h arbitrary small hence inducing the bias-variance trade-off.
Numerical illustration
In this section, we provide a numerical comparison of our algorithm with the method based on ` 2 -randomization from Shamir [33] (see Appendix D for the deﬁnition).
We consider the no noise model and f t = f , 8 t , with the function f : R d ! R deﬁned as
[Equation]
where c = ( c 1 , . . . , c d ) > 2 R d such that c j = exp( j ) /
P di =1 exp( i ) for j = 1 , . . .
, d .
We choose [Equation] and V ( x ) = [Equation]
X [Equation]
= x j log( x j ) .
As stated in Example 2, V is 1 -strongly convex on ⇥ w.r.t.
the ` 1 -norm and R  p log( d ) .
Moreover, f is a Lipschitz function w.r.t.
the ` 1 -norm.
We deploy the adaptive parameterization that appears in
Theorem 3.
In Figure 1 we present the optimization error of the algorithms, which is deﬁned as
[Equation]
The results are reported over 30 trials.
We plot all the 30 runs alongside the average performance.
One can observe that the ` 1 -randomization method behaves signiﬁcantly better than the ` 2 -randomization algorithm.
The theoretical bound for our method in this setup has a p log d gain in the rate.
[Figure]
i
[Figure]
n
[Figure]
Discussion and comparison to prior work
We introduced and analyzed a novel estimator for the gradient based on randomization over the ` 1 -sphere.
We established guarantees for the online dual averaging algorithm with the gradient replaced by the proposed estimator.
We provided an anytime and completely data-driven algorithm, which is adaptive to all parameters of the problem.
Our analysis is based on deriving a weighted
Poincaré type inequality for the uniform measure on the ` 1 -sphere that may be of independent interest.
Under the canceling noise assumption and q 2 { 1 , 2 } , our setting is analogous to [14, 33].
For the case q = p = 2 and canceling noise, we show that the performance of our method is the same as in [33, Corollary 2] up to absolute constants that were not made explicit in [33].
For the case of q = p = 1 and canceling noise, we improved the bound [33, Corollary 3] by a p log( d ) factor.
For the case q = 2 , p  1 , comparing with the lower bound in [14, Proposition 1], shows that the result of Theorem 1 is minimax optimal.
For the case q = p = 1 , [14, Proposition 2] shows that our result in Theorem 1 is optimal up to a log( d ) factor.
Under the adversarial noise assumption , Theorem 2 provides the rate
O ( T 3 / 4 ) , that is, we get an additional T 1 / 4 factor compared to the canceling noise case.
It remains unclear whether it is optimal under adversarial noise – this question deserves further investigation.
Note that, under sub-Gaussian i.i.d.
noise assumption and q = p = 2 , one can achieve the rate ˜ O
( d a p T ) with a relatively big a > 0 [2, 8, 13, 21].
In particular, with an ellipsoid type method [21] obtains the rate
O ( d 4 . 5 p T log( T ) 2 ) for the cumulative regret.
Finally, let us discuss the compactness of ⇥ .
It is straightforward to extend the results of Theorems 1,to any closed convex ⇥ considering the regret against a ﬁxed action x 2 ⇥ .
Indeed, using [26,
Corollary 7.9], one only needs to replace R appearing in both Theorems 1, 2 by an upper bound on p
V ( x )  inf x 0 2 ⇥ V ( x 0 ) .
The adaptive case is more complicated.
One way to tackle this case is to use [27, Theorem 1] requiring a control of E max t =1 ,...,T k g t k p
⇤ .
This term can be controlled under the canceling noise Assumption 2 using the Lipschitzness of f t ’s, so that Theorem 3 extends to unbounded ⇥ .
However, without the canceling noise assumption, following the approach outlined above, one needs to control E max t =1 ,...,T | ⇠ 0 t  ⇠ 00 t | h t .
The adversarial noise Assumption 3 is not sufﬁcient to reasonably control this term, so that extending Theorem 4 to unbounded ⇥ is not possible without further assumptions.
Acknowledgments and disclosure of funding
The work of E. Chzhen was supported by ANR PIA funding: ANR-20-IDEES-0002.
The work of M.
Pontil is partially supported by European Union’s Horizon 2020 research and innovation programme under ELISE grant agreement No 951847.
The research of A. Akhavan and A. B. Tsybakov is supported by a grant of the French National Research Agency (ANR), “Investissements d’Avenir” (LabEx Ecodec/ANR-11-LABX-0047).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] at https: //github.com/evgchz/ Akhavan
Chzhen Pontil Tsybakov (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[-h 
-]