B ANDIT L EARNING WITH G ENERAL F UNCTION C LASSES : H ETEROSCEDASTIC N OISE AND V ARIANCEDEPENDENT R EGRET B OUNDS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider learning a stochastic bandit model, where the reward function belongs to a general class of uniformly bounded functions, and the additive noise can be heteroscedastic.
Our model captures contextual linear bandits and generalized linear bandits as special cases.
While previous works (Kirschner & Krause, 2018; Zhou et al., 2021) based on weighted ridge regression can deal with linear bandits with heteroscedastic noise, they are not directly applicable to our general model due to the curse of nonlinearity.
In order to tackle this problem, we propose a multi-level learning framework for the general bandit model.
The core idea of our framework is to partition the observed data into different levels according to the variance of their respective reward and perform online learning at each level collaboratively.
Under our framework, we ﬁrst design an algorithm that constructs the variance-aware conﬁdence set based on empirical risk minimization and prove a variance-dependent regret bound.
For generalized linear bandits, we further propose an algorithm based on follow-the-regularized-leader (FTRL) subroutine and online-to-conﬁdence-set conversion, which can achieve a tighter variance-dependent regret under certain conditions.
I NTRODUCTION
Over the past decade, stochastic bandit algorithms have found a wide variety of applications in online advertising, website optimization, recommendation system and many other tasks (Li et al., 2010; McInerney et al., 2018).
In the model of stochastic bandits, at each round, an agent selects an action and observes a noisy evaluation of the reward function for the chosen action, aiming to maximize the sum of the received rewards.
A general reward function governs the reward of each action from the eligible action set.
A common assumption used in stochastic bandit problems is that the observation noise is conditionally independent and satisﬁes a uniform tail bound.
In real-world applications, however, the variance of observation noise is likely to be dependent on the evaluation point (chosen action) (Kirschner &
Krause, 2018).
Moreover, due to the dynamic environment in reality, the variance of each action may also be different at each round.
This motivates the studies of bandit problems with heteroscedastic noise.
For example, Kirschner & Krause (2018) introduced the heteroscedastic noise setting where the noise distribution is allowed to depend on the evaluation point.
They proposed weighted least squares to estimate the unknown reward function more accurately in the setting where the underlying reward function is linear or lies in a separable Hilbert space (Section 5, Kirschner & Krause 2018).
In this paper, we consider a general setting, where the unknown reward function belongs to a known general function class F with bounded eluder dimension (Russo & Van Roy, 2013).
This captures multi-armed bandits, linear contextual bandits (Abbasi-Yadkori et al., 2011) and generalized linear bandits (Filippi et al., 2010) simultaneously.
Since weighted least squares highly depends on the linearity of the function class, we propose a multi-level learning framework for our general setting.
The underlying idea of the framework is to partition the observed data into various levels according to the variance of the noise.
The agent then estimates the reward function at each level independently and then exploit all the levels when selecting an action at each round.
While previous work by Kirschner & Krause (2018) considered sub-Gaussian noise with nonuniform variance proxies, we only assume nonuniform variances of noise (Zhou et al., 2021; Zhang et al., 2021), which brings a new challenge of exploiting the variance information of the noise to obtain tighter variance-aware conﬁdence sets .
Under our multi-level learning framework, we ﬁrst design an algorithm based on empirical risk minimization and Optimism-in-the-Face-of-Uncertainty (OFU) principle, and prove a variancedependent regret bound.
For a special class of bandits namely generalized linear bandits with heteroscedastic noise, we further propose an algorithm using follow-the-regularized-leader (FTRL) as an online regression subroutine and adopting the technique of online-to-conﬁdence-set conversion (Abbasi-Yadkori et al., 2012;
Jun et al., 2017).
This algorithm achieves a provaly tighter regret bound when the range of the reward function is relatively wide compared to the magnitude of noise.
Our main contributions are summarized as follows:
• We develop a new framework called multi-level regression , which can be applied to heteroscedastic bandits, even when the reward function class does not lie in a separable Hilbert space.
• Under our framework, we design tighter variance-aware upper conﬁdence bounds for bandits with general reward functions, and propose an bandit learning algorithm based on empirical risk minimzation.
We show that our algorithm enjoys variance-dependent regret upper bounds which can be regarded as a strict extension of previous algorithms which obtain variance-dependent regret bounds on simpler bandit models (Zhou et al., 2021; Zhang et al., 2021).
• For generalized linear bandits (Filippi et al., 2010; Jun et al., 2017), which is a special case of our model class, we further propose an algorithm based on online-to-conﬁdence-set conversion.
We ﬁrst prove a variance-dependent regret bound for follow-the-regularized-leader (FTRL) for the online regression problem derived from generalized linear function class, and then convert the online learning regret bound to the bandit learning conﬁdence set.
We show that our algorithm can achieve a tighter regret bound for generalized linear bandits.
• As a by-product, our regret bound for FTRL improves the state-of-the-art regret result (cid:101)
O ( d 2 R 2 ) obtained by stochastic online linear regression (Ouhamma et al., 2021) to (cid:101) O
( dσ 2 max ) (omitting the terms without dependence on d ), where d is the dimension of contexts, R is the upper bound of the sub-Gaussian norm of the noises at each step, and σ max is the upper bound of the variances of the noises.
[Table]
Refer to Section 3 for the deﬁnitions of dim E , σ t , J and R , Section 6 for the deﬁnitions of κ, K, A, B .
We write general function class with eluder dimension dim E as ‘General’ and generalized linear function class as ‘G-Lin’ for short.
Oracle efﬁciency refers to the computational efﬁciency given a regression oracle (i
[Table]
e., empirical risk minimization) for the involved function class and an optimization oracle which maximizes the reward function f ( x ) for a ﬁxed x
Notation.
We use lower case letters to denote scalars, and use lower and upper case bold face letters to denote vectors and matrices respectively.
We denote by [ n ] the set { 1 , . . . , n } .
For a vector x ∈ R d and matrix Σ ∈ R d × d , a positive semi-deﬁnite matrix, we denote by (cid:107) x (cid:107) 2 the vector’s
Euclidean norm and deﬁne (cid:107) x (cid:107) Σ = √ x
(cid:62) Σx .
For two positive sequences { a n } and { b n } with n = 1 , 2 , . . .
, we write a n = O ( b n ) if there exists an absolute constant C > 0 such that a n ≤ Cb n holds for all n ≥ 1 and write a n = Ω( b n ) if there exists an absolute constant C > 0 such that a n ≥ Cb n holds for all n ≥ 1 .
Let N ( F , α, (cid:107) · (cid:107) ∞ ) denote the α -covering number of F in the sup-norm (cid:107) · (cid:107) ∞ .
If there is no ambiguity, we may write N ( F , α, (cid:107) · (cid:107) ∞ ) as N α for short.
We use (cid:101) O ( · ) to further hide the polylogarithmic factors other than log-covering numbers.
R ELATED WORK
Learning with heteroscedastic noise.
Heteroscedastic noise has been studied in many different settings such as active learning Antos et al.
, regression (Aitken, 1936; Goldberg et al., 1997;
Chaudhuri et al., 2017; Kersting et al., 2007), principle component analysis (Hong et al., 2016; 2018) and Bayesian optimization (Assael et al., 2014).
However, only a few works have considered heteroscedastic noise in bandit settings.
Cowan et al.
(2015) considered a variant of multi-armed bandits where the noise at each round is a Gaussian random variable with unknown variance.
Kirschner &
Krause (2018) is the ﬁrst to formally introduce the concept of stochastic bandits with heteroscedastic noise.
In their model, the variance of the noise at each round t is a function of the evaluation point x t , ρ t = ρ ( x t ) , and they further assume that the noise is ρ t -sub-Gaussian.
ρ t can either be observed at time t or either be estimated from the obsevations.
Zhou et al.
(2021) considered linear bandits with heteroscedastic noise and generalized the heteroscedastic noise setting in Kirschner &
Krause (2018) in the sense that they no longer assume the noise to be ρ t -sub-Gaussian, but only requires the variance of noise to be upper bounded by ρ 2 t and the variances are arbitrarily decided by the environment, which is not necessarily a function of the evaluation point.
In the same setting as in Zhou et al.
(2021), Zhang et al.
(2021) further considered a strictly harder setting where the noise has unknown variance.
They proposed an algorithm which can deal with unknown variance through a computationally inefﬁcient clip technique.
Our work basically considers the noise setting proposed by Zhou et al.
and further generalizes their setting to bandits with general function classes.
We will consider to extend it to the harder setting as Zhang et al.
(2021) as future work.
Bandits with known function classes.
Moving beyond multi-armed bandits, there have been signiﬁcant theoretical advances on stochastic bandits with function approximation.
Among them, there is a huge body of literature on linear bandit problems where the reward function is assumed to be a linear function of the feature vectors attached to the actions (Dani et al., 2008; Abbasi-Yadkori et al., 2011; Chu et al., 2011; Li et al., 2019; 2021b).
Generalizing the restrictive linear rewards, there has also been a ﬂurry of studies on generalized linear bandit problems (Filippi et al., 2010; Jun et al., 2017; Li et al., 2017; Kveton et al., 2020).
As for stochastic bandits with general function classes, the seminal work by Russo & Van Roy (2013) introduced the notion of eluder dimension to measure the complexity of the function class and provided a general UCB-like algorithm that works for any given class of reward functions with bounded eluder dimension.
They further proved a regret upper bound of order (cid:101) O ( √ dim E log N · T ) for their proposed algorithm where dim E is the eluder dimension and log N stands for the logcovering number of the funciton class.
Linear bandits and generalized linear bandit problems can be seen as special cases as their proposed general model.
Online-to-conﬁdence-set conversion.
Abbasi-Yadkori et al.
(2012) may be the ﬁrst one to introduce the technique that takes in an online learning subroutine and turns the output of it into a conﬁdence set at each round.
While Abbasi-Yadkori et al.
(2012) considered applying this technique in linear bandits, Jun et al.
(2017) generalized and introduced the previous approach to Generalized
Linear Online-to-conﬁdence-set Conversion (GLOC) and applied it to generalized linear bandits.
Online regression for linear functions.
Online linear regression has long been studied in the setting where the response variables (or labels) are bounded and chosen by an adversary (Bartlett et al., 2015;
Cesa-Bianchi et al., 1996; Kivinen & Warmuth, 1997; Littlestone et al., 1991; Malek &
Bartlett, 2018), to mention a few.
A recent work (Ouhamma et al., 2021) considers the stochastic setting where the response variables are unbounded and revealed by the environment with additional random noise on the true labels.
Ouhamma et al.
(2021) discussed the limitations of online learning algorithms in the adversarial setting and further advocates for the need of complementary analyses for existing algorithms under stochastic unbounded setting.
M ULTI - LEVEL LEARNING FRAMEWORK
We introduce the Multi-level learning framework in this section.
3. P RELIMINARIES
General function class Following (Russo & Van Roy, 2013), we introduce the (cid:15) -dependence and eluder dimension notion, which are used to measure the complexity of a general function class F .
Deﬁnition 3.1 ( (cid:15) -dependence, Russo & Van Roy 2013) .
An action a ∈ A is (cid:15) -dependent on actions { a 1 , a 2 , · · · , a n } ∈ A with respect to F if any pair of functions f, (cid:101) f ∈ F satisfying (cid:80) ni
=
( f ( a i ) − (cid:101) f ( a i )) 2 ≤ (cid:15) 2 also satisﬁes f ( a ) − (cid:101) f
( a ) ≤ (cid:15) .
Further, a is (cid:15) -independent of { a 1 , · · · , a n } with respect to F if a is not (cid:15) -dependent on { a 1 , · · · , a n } .
Deﬁnition 3.2 (eluder dimension, Russo & Van Roy 2013) .
The (cid:15) -eluder dimension dim E ( F , (cid:15) ) is the length d of the longest sequence of elements in A such that, for some (cid:15) (cid:48) ≥ (cid:15) , every element is (cid:15) (cid:48) -independent of its predecessors.
In this work we focus on general function class F with bounded eluder dimension, and we consider generalized linear function class as a special case.
We would like to point out that the function class with small eluder dimension is strictly larger than linear and generalized linear bandits (Li et al., 2021a), while neural networks with ReLU activation do not have a small eluder dimension (their eluder dimension has an exponential dependence on the input dimension)
(Dong et al., 2021).
We leave it as future work to consider even more general function classes.
Deﬁnition 3.3 (width) .
Let w (cid:101) F ( a ) =
sup f,f ∈ (cid:101) F (cid:0) f
( a ) − f ( a ) (cid:1) .
For an action set (cid:101)
A ⊆ A , we use w (cid:101) F ( (cid:101) A ) to denote sup a ∈ (cid:101)
A w (cid:101) F ( a ) .
To deal with inﬁnite or continuous action sets, we make the following assumption that the reward function class is known in advance by the agent.
Notice that for the ﬁnite multi-armed bandit case, we can choose F as the set that includes all the eligible functions.
Assumption 3.4 (Known Reward Function Class) .
The unknown reward function f ∗ belongs to an accessible function class F = { f θ : A → R | θ ∈ Θ } .
Bandit models We consider a heteroscedastic variant of the classic stochastic bandit problem with general function classes.
At each round t ∈ [ T ] ( T ∈ N ), the agent observes a decision set D t ⊆
A which is chosen by the environment.
The agent then selects an action a t ∈ D t and observes reward r t together with a corresponding variance upper bound σ 2 t .
We assume that r t = f ∗ ( a t ) + (cid:15) t where f ∗ :
A → R is an underlying real-valued reward function which is unknown to the learner and (cid:15) t is a random noise.
We make the following assumption on (cid:15) t .
Assumption 3.5.
For each t , (cid:15) t satisﬁes that (cid:15) t | a 1: t , (cid:15) 1: t − 1 is a R -sub-Gaussian random variable ( R > σ t ) and E [ (cid:15) t | a 1: t , (cid:15) 1: t − 1 ] = 0 , E [ (cid:15) 2 t | a 1: t , (cid:15) 1: t − 1 ] ≤ σ 2 t := σ 2 t ( a 1: t , r 1: t − 1 ) .
where σ t can be either a constant or a random variable dependent on a 1: t and r 1: t − 1 .
Remark 3.6.
σ t can be seen as either a given information from the environment, or an estimator of the noise variance at t -th round based on all past observations, as discussed in the information directed sampling bandit (Kirschner & Krause, 2018).
For instance, we consider a bandit problem with a two-point action distribution, where the variance of an action can be estimated through the estimation of the mean of the action.
Similar estimation procedure has been studied in Lattimore et al.
The details are in Appendix C. We can further consider the MDP setting.
With a conﬁdence set P that includes the true transition dynamic, the conditional variances of the value functions ¯ V at state s and action a can be estimated by sup p ∈P (cid:110)(cid:80) s (cid:48)
∈ S p ( s | s, a ) ¯ V 2 ( s (cid:48) ) − (cid:2)(cid:80) s (cid:48)
∈ S p ( s (cid:48) | s, a ) ¯ V ( s (cid:48) )
(cid:3) 2 (cid:111) .
So the variances of value functions can be efﬁciently estimated for the MDP setting.
For simplicity, let J = (cid:80) T t =1 σ 2 t .
This assumption on (cid:15) t is a slightly generalized version of that in
Zhou et al.
(2021) in the sense that the noise is not necessarily bounded by R .
The goal of the agent is to minimize the following cumulative regret:
[Equation]
3.1) where the optimal action a ∗ t at round t ∈
[ T ] is deﬁned as a ∗ t := argmax a ∈D t f ∗ ( a ) .
3. M ULTI - LEVEL L EARNING FRAMEWORK
Existing approach.
To tackle the heteroscedastic bandit problem, for the case where the F is the linear function class (i.e., f ( a ) = (cid:104) θ ∗ , a (cid:105) for some θ ∗ ∈ R d ), a weighted linear regression framework (Kirschner & Krause, 2018;
Zhou et al., 2021) has been proposed.
Generally speaking, at each round t ∈ [ T ] , weighted linear regression constructs a conﬁdence set C t based on the empirical risk minimizarion (ERM) for all previous observed actions a s and rewards r s as follows:
[Equation]
[Table]
where w s is the weight, and β t , λ are some parameters to be speciﬁed.
w s is selected in the order of the inverse of the variance σ 2 s at round s to let the variance of the rescaled reward √ w s r s upper bounded by 1.
Therefore, after the weighting step, one can regard the heteroscedastic bandits problem as a homoscedastic bandits problem and apply existing theoretical results to it.
To deal with the general function case, a direct attempt is to replace the (cid:104) θ , a (cid:105) appearing in above construction rules with f ( a ) .
However, such an approach requires that F is close under the linear mapping, which does not hold for general function class F .
Multi-level Learning framework (ML 2 ).
To deal with the nonlinearity issue, we propose a novel framework ML 2 in Algorithm 1.
At the core of our design is the idea of partitioning the observed data into several levels and ‘packing’ data with similar variance upper bounds into the same level as shown in line 7-8 of Algorithm 1.
Note that we use a small real number σ to ensure that the number of levels is bounded.
Speciﬁcally, for any two data belong to the same level with variaces larger than σ , their variance will be at most twice larger than the other.
Next in line 9, our framework calls a subroutine to estimate f ∗ according to the data points in Ψ t +1 ,l .
Since the variances of the data in the same level are nearly the same, we let algorithms that work for homoscedastic bandit problem run on the data in the same level.
Particularlly, we use the Empirical risk minimization (ERM) algorithm as described in Algorithm 2 for Sections 4 and 5.
In Section 6, we show the power of using Algorithm 4 as the regression subroutine.
Then in line 5, the agent makes use of L conﬁdence sets simultaneously to select an action based on the optimism-in-the-face-of-uncertainty (OFU) principle over all L levels.
More speciﬁcally, the algorithm chooses the action opmimisticly according to each conﬁdence set but inclines to select the conﬁdence set with the most pessimistic evaluation.
In the following sections, we will consider several different settings to show the power of ML 2 .
W ARMUP : NOISE WITH ADDITIONAL SUB -G AUSSIAN ASSUMPTION
We ﬁrst consider a simpliﬁed variant of our problem, where each noise is also sub-Gaussian.
Assumption 4.1 (Sub-Gaussianity of noise) .
(cid:15) t is conditionally σ t -sub-Gaussian on a 1: t , (cid:15) 1: t − 1 .
Such a sub-Gaussian assumption on noise has been considered by Kirschner & Krause (2018).
Next we show the regret upper bound for ML 2 with ERM.
For simplicity, in the following results, let dim E denote dim E ( F , 1 /T 2 ) .
Theorem 4.2 (Gap-independent regret bound for bandits with heteroscedastic sub-Gaussian noise) .
Suppose Assumption 3.4 and 4.1 hold and | f ∗ ( a ) | ≤ C for all a ∈ A .
For all t ∈ [ T ] , l ∈ [ L ] and δ ∈ (0 , 1) , α > 0 , σ > 0 , if we apply Algorithm 2 as a subroutine of Algorithm 1 (in line 9) and set [Equation] [Equation] β t,l as the square root of
[Equation]
where N α = N ( F , α, (cid:107) · (cid:107) ∞ ) and L = (cid:100) log 2 R/σ (cid:101) (recall the deﬁnition of L in Algorithm 1), then with probability at least 1 − δ , the regret for the ﬁrst T rounds is bounded as follows:
[Equation]
Corollary 4.3.
Let the same conditions as in Theorem 4.2 hold.
Set α = T − 2 and σ = dim − 1 E (log(2 N α L/δ ) √ T ) − 1 .
Then with probability at least 1 − δ , when T is large enough, the regret for the ﬁrst T rounds is bounded as Regret( T ) =
(cid:101) O (cid:16)(cid:112) dim E log( N ( F , T − 2 , (cid:107) · (cid:107) ∞ ))
J (cid:17) .
Remark 4.4.
Our result is strictly tighter than the (cid:101) O (cid:16) R (cid:112) dim E log( N ( F , T − 2 , (cid:107) · (cid:107) ∞ ))
T (cid:17) regret achieved by Russo & Van Roy (2013) since J = (cid:80) Tt =1 σ 2 t ≤ R 2 T .
In the worst case, when σ 1 = · · · = σ T = R , our result degrades to their result.
Our improvement in regret is due to the utilization of variance information.
When the variance information is provided or can be estimated, our algorithm can achieve better regrets for bandits with general function classes studied in this paper, while existing algorithms cannot.
Remark 4.5.
When restricted to linear contextual bandits with dimension d , since log N ( F , T − 2 , (cid:107)· (cid:107) ∞ ) = (cid:101) O ( d ) , dim E = (cid:101) O
( d ) (Russo & Van Roy, 2013), our result can be written as (cid:101) O
( d √ J ) , which matches the result of using weighted linear ridge regression for heteroscedastic linear bandit under our assumptions on noise (Kirschner & Krause, 2018;
Zhou et al., 2021).
We also provide a gap-dependent regret bound for general function class setting in section D, generalizing the previous gap-dependent regret bound in linear bandits (Abbasi-Yadkori et al., 2011).
G ENERAL RESULTS FOR BANDITS WITH HETEROSCEDASTIC NOISE
In this section, we consider the original setting introduced in Section 3 without Assumption 4.1.
In the following subsections, we will show that Algorithm 2 still works with reﬁned value of β .
5. V ARIANCE - AWARE CONFIDENCE SET
In this genral setting, directly applying the conﬁdence set used in the previous work (Russo &
Van Roy, 2013) gives no improvement since our conﬁdence sets C t,l do not adopt the variance information.
We show in the following theorem that our new designed C t,l with a new conﬁdence radius β still ensures that the conﬁdence set is large enough to contain f ∗ with high probability, and exploits the variance information at the same time.
Theorem 5.1 (Variance-dependent conﬁdence sets) .
Suppose that | f ∗ ( a ) | ≤ C for all a ∈ A .
For any α > 0 and δ ∈ (0 , 1 / 2) , if we set β t,l as the square root of
[Equation]
where R = R (cid:112) 2 log(4 t 2 /δ ) , N α = N ( F , α, (cid:107)·(cid:107) ∞ ) , then f ∗ ∈ C t,l with probability at least 1 − 2 δ for any ﬁxed t, l .
Remark 5.2.
With a small α , we have β t,l = (cid:101) O
(2 2 l σ 2 log N α + CR log N α ) .
Compared with the corresponding previous result (cid:101) O
( R 2 log N α ) (Russo & Van Roy, 2013; Ayoub et al., 2020), our conﬁdence set is tighter when C is relatively small compared to R .
5. R EGRET UPPER BOUNDS FOR ML 2 WITH ERM
We derive our general results with the variance-aware conﬁdences sets described in the last subsection.
In this part, we write dim E ( F , T − 1 ) as dim E for short.
Theorem 5.3 (Gap-independent regret bound for bandits with heteroscedastic noise) .
Suppose As- sumption 3.4 holds and | f ∗ ( a ) | ≤ 1 for all a ∈ A .
For all t ∈ [ T ] , l ∈ [ L ] and δ ∈ (0 , 1) , α > 0 , σ > 0 , if we apply Algorithm 2 as a subroutine of Algorithm 1 (in line 9) and set β t,l as the square root of 12 αt +4 αRt +8 / 3 · R log(2 N α t 2 L/δ )+16 · (2 l +1 σ ) 2 log(2 N α t 2 L/δ ) , where L = (cid:100) log 2 R/σ (cid:101) ,
N α = N ( F , α, (cid:107) · (cid:107) ∞ ) and R = R (cid:112) 2 log(4 t 2 L/δ ) (with a slight abuse of notation), then with probability at least 1 − 2 δ , the regret for the ﬁrst T rounds is bounded as follows:
[Equation]
Corollary 5.4.
Assume R = Ω(1) .
Let the same conditions as in Theorem 5.3 hold.
Set α =
T − 2 , σ = 1 .
Then with probability at least 1 − δ , when T is large enough, the regret for the ﬁrst T rounds is bounded as Regret( T ) =
(cid:101) O (cid:0) √ dim E log N α J + √ R dim E log
N α T (cid:1) .
Remark 5.5.
Compared with the result shown in Corollary 4.3, the additional term of order (cid:101) O ( √ dim E log N α RT ) is due to a larger conﬁdence set by the absence of Assumption 4.1.
Remark 5.6.
When restricted to heteroscedastic linear contextual bandits of dimension d , our regret bound can be written as (cid:101) O
( d √ J + √ Rd √ T ) .
With a slightly more restricted assumption on noise,
Zhou et al.
(2021) achieved a result of order (cid:101) O ( d √ J + R √ dT ) .
Our result is appealing when the sub-Gaussian parameter of noise R is much larger than 1 (or the range of the reward function, equivalently).
When R is small, our result becomes sub-optimal due to the property of our varianceaware conﬁdence set.
We also provide a gap-dependent regret bound in Appendix D.
T IGHTER BOUNDS FOR GENERALIZED LINEAR BANDITS
Our general result shown in Theorem 5.3 has an additional term of order (cid:101) O ( √ dim E log N α RT ) which makes the result sub-optimal when R is close to the range of the reward function.
In this section, we consider a special case, generalized linear bandits with heteroscedastic noise.
We show how to get rid of the (cid:101) O ( √ dim E log N α RT ) term in the upper bound of the regret, and achieve a better result when R is relatively small or close to the bound of the reward function.
6. G ENERALIZED LINEAR BANDITS
Following Filippi et al.
; Jun et al.
(2017), we consider the generalized linear function class deﬁned as follows.
Assumption 6.1 (Generalized linear function class) .
Action set A and Θ in Assumption 3.4 are subsets of R d .
There exists a known link function h , such that ∀ a ∈ A and f θ ∈ F , f θ ( a )
=
h ( θ (cid:62) a ) .
Let f ∗ =
f θ ∗ .
Assume that (cid:107) θ ∗ (cid:107) 2 ≤ B, sup a ∈A (cid:107) a (cid:107) 2 ≤ A .
To make the problem tractable, we need the following assumption on h .
Assumption 6.2 (Assumption 1, Jun et al. 2017) .
h is K -Lipschitz on [ − A · B, A · B ] and continuously differentiable on ( − A · B, A · B ) .
Furthermore, inf z ∈ ( − A · B,A · B ) h (cid:48) ( z ) =
κ for some κ > 0 .
Next we propose the follow-the-regularized-leader (FTRL) framework (Shalev-Shwartz & Singer, 2007; Xiao, 2010; Hazan, 2019) in Algorithm 3, which is the key component of our ﬁnal algorithm.
Note that when dealing with bandit setting, we maintain an independent process executing
Algorithm 3 for each variance level instead of feeding all the data points into a single FTRL online learner.
Here we number the data points with t = 1 , 2 , · · · for simplicity with a slight abuse of notation.
In Algorithm 3 we will use a loss function (cid:96) and a regularized function φ .
φ is deﬁned as Algorithm 3 Follow The Regularized Leader (FTRL) 1: Input: F , R . 2: for t ≥ 1 do 3: Output θ t ← argmin
∈
d ( θ ) + (cid:80) t − 1 s =1 (cid:96) ( θ (cid:62) a s , r s ) . Observe a t , r t . 5: end for φ ( θ )
=
c · (cid:107) θ (cid:107) 22 where c is a constant which will be speciﬁed later in Theorem 6.5.
To align with our Assumption 6.1, we select the loss function as follows, following Jun et al.
Assumption 6.3 (Loss function, Jun et al. 2017) .
The loss function (cid:96) in Algorithm 3 is selected as follows: (cid:96) ( z, r )
=
− rz
+ m ( z ) , (cid:96) t
( θ ) =
(cid:96) ( θ (cid:62) a t , r t ) , where m ( z ) satisﬁes m (cid:48)
( z ) =
h ( z ) .
V ARIANCE - DEPENDENT REGRET FOR FTRL
Before proposing our ﬁnal algorithm for generalized linear bandits, we ﬁrst propose a variancedependent complexity result for FTRL, since it is already nontrivial and reveals some interesting properties about our setting.
We deﬁne a notion of regret of online regression , named by reg t , as follows.
The concept of regret of online regression has been introduced in the previous work (Abbasi-Yadkori et al., 2012;
Jun et al., 2017).
In detail, it is used to characterize the complexity for
FTRL to learn the generalized linear function.
Deﬁnition 6.4.
Let reg
[Equation]
Our deﬁnition of regret for online regression is slightly different that in prior works (Abbasi-Yadkori et al., 2012;
Jun et al., 2017).
Here θ ∗ is choosen to be the true parameter for the bandit model, while θ ∗ is often chosen as arg inf θ ∈ Θ (cid:96) s ( θ ) in Abbasi-Yadkori et al.
Jun et al.
From the perspective of online learning, the algorithms and the corresponding analyses are usually introduced for either the realizable setting where there exists an underlying θ ∗ that incurs zero loss, or the adversarial setting where the bounded label r s in each round s can be arbitrarily chosen by the adversary.
As a result, the previous approaches by Abbasi-Yadkori et al.
Jun et al.
(2017) do not exploit the ‘stochastic’ property of the labels.
Provided that the labels are sequentially generated with additional stochastic noise, our deﬁnition is more reasonable and natural.
A recent work focusing on stochastic online linear regression also discussed the limitation of adversarial setting (Section 2.2, Ouhamma et al. 2021).
Next we propose a bound for reg t which adopts the variance information.
Theorem 6.5 (Regret of FTRL) .
Set φ ( θ ) =
A 2 K 2 (cid:107) θ (cid:107) 22 /κ and assume that all the data points fed into the algorithm are of noise variance bounded by σ 2max , then with probability at least 1 − 3 δ , ∀ t ≥ 1 , the regret of Algrithm 3 for the ﬁrst t rounds is bounded as follows:
[Equation]
Remark 6.6.
Jun et al.
(2017) analyzed the online learning regret for the same function class and loss function with our setting.
Their result yields a reg t in the order of (cid:101)
O
( K 2 A 2 B 2 + R 2 κ d ) .
Our result improves their result in two aspects.
First, R is strictly larger than σ max since a R -sub-Gaussian random variable is deﬁnitely of variance lower than R 2 .
Second, when we consider cases where the bound of reward functions (i.e., KAB ) is extremely large compared to R , their result becomes (cid:101) O
( K 2 A 2 B 2 d/κ ) , which has an additional linear dependence on d .
Remark 6.7.
Consider a special case where κ = K = 1 .
Our result degrades to (cid:101) O
( A 2 B 2 +
R + σ 2max d ) .
This is essentially a regret upper bound for stochastic online linear regression with square loss.
Recently Ouhamma et al.
(2021) studied this stochastic setting and managed to get rid of the (cid:101) O
( A 2 B 2 d ) term in classic result for online linear regression considering adversarial setting.
Ouhamma et al.
(2021) derived a high probability regret bound of (cid:101)
O
( R 2 d 2 ) after omitting the o (log( T ) 2 ) terms (Theorem 3.3, Ouhamma et al. 2021).
Unlike their result, our result does not suffer from the quadratic dependence on d , and our result depends on σ 2max d rather than R 2 d 2 .
Therefore, our result is better than that in Ouhamma et al.
(2021) when d is large.
We also notice that the discussion in Sec.
3.3 in Ouhamma et al.
(2021) yields an improved expected regret of order
O ( R 2 d log 2 T + R 2 d 2 log T log log T ) , but the ﬁrst term dominates the regret in the asymptotic sense, i.e., only when T is very large ( T ≥ (log T ) d ).
1: Initialize: V 0 ,l ← λ I for all l ∈
[ L ] .
2: while input t, l t , a t , r t do 3:
Set θ t,l t following Algorithm 3, where θ t,l t ← argmin θ ∈ R d φ ( θ ) + (cid:80) s ∈ Ψ t
+1 ,lt (cid:96) ( θ (cid:62) a s , r s ) .
Find t (cid:48)
=
max Ψ t,l t .
Update V t,l t ← V t (cid:48) ,l t + a t a (cid:62) t , z t,l t ← a (cid:62) t θ t,l t .
Compute (cid:98) θ t,l t ← V − 1 t,l t (cid:16)(cid:80)
s
∈ Ψ t +1 ,lt z s,l t · a s (cid:17) .
Deﬁne C t,l t ← { θ ∈ R d : (cid:107) θ − (cid:98) θ t,l t (cid:107) 2 V − 1 t,lt ≤ β t,l } .
Deﬁne C t,l ← C t − 1 ,l for all l ∈ [ L ] \{ l t } .
Return C t,l ← { f θ ∈ F θ ∈ C t, for all l
10: end while
6. R EGRET BOUND OF A LGORITHM 1 WITH GLOC
With our new technical tool presented in the last subsection, we now show our ﬁnal algorithm for the generalized linear bandit setting.
We propose our algorithm in Algorithm 4.
Generally speaking,
Algorithm 4 is a multi-level version of the generalized linear online-to-conﬁdence-set conversion (GLOC) algorithm proposed by Jun et al.
(2017), equipped with FTRL.
As shown in Algorithm 4, we maintain L FTRL online learners in parallel.
Under our framework, a single learner only receives data with similar variances of noise.
As a result, we can make use of the variance-dependent result shown in Theorem 6.5 to derive a tighter regret bound for generalized linear bandits with heteroscedastic noises.
Theorem 6.8 (Regret bound for generalized linear bandits, informal) .
Suppose that Assumption 6.1 and 6.2 hold for the known reward function class F .
If we apply Algorithm 4 as a subroutine of
Algorithm 1 (in line 9) and set β t,l to
[Equation]
for all t ∈ [ T ] , l ∈ [ L ] , where L = (cid:100) log 2 R/σ (cid:101) , σ = R/ √ d , then with probability 1 − 4 δ , the regret of Algorithm 1 for the ﬁrst T rounds is bounded as follows:
[Equation]
Remark 6.9.
In the worst case, i.e. σ 1 = · · · = σ T =
R , our result degraded to (cid:101)
O ( KRd √ T/κ +
K 2 AB √ dT/κ ) , which still improves the (cid:101) O (cid:16) K ( KAB + R ) d √ T/κ (cid:17) result provided by Jun et al.
Remark 6.10.
Applying the regret bound in Corollary 5.4 in generalized linear bandits, we obtain a regret bound of (cid:101) O (cid:16) K ( d √ J + d √ RT ) /κ
(cid:17) for the case where K · A · B = 1 and R = Ω(1) .
Our bound here improved the general result when R = o ( d ) .
Remark 6.11.
When restricted to the heteroscedastic linear bandits by setting κ = K = 1 , our result becomes (cid:101) O (cid:16) ( R + A · B ) √ dT + d √ J (cid:17) , which is the same as the (cid:101) O (cid:16) R √ dT + d √ J (cid:17) regret in Zhou et al.
(2021) when R = Ω( AB ) .
C ONCLUSION AND FUTURE WORK
In this work we study heteroscedastic stochastic bandits problem for a general reward function class.
We propose a multi-level regression framework ML 2 to deal with the heteroscedastic noises.
Under three different settings with additional assumptions on the noise and the function class, we study the performance of ML 2 and propose corresponding variance-dependent regret bounds, which strictly improves previous algorithms for homoscedastic bandit setting.
We leave to study the optimal regret bound of heteroscedastic stochastic bandits for a general reward function class for future work.
A E XPERIMENTS
In this section we conduct some experiments of the proposed ML 2 +FTRL algorithm for generalized linear bandits. We compare our results with GLOC proposed by Jun et al. For each trial the dimension of θ ∗ is set to d =
and θ ∗ is sampled uniformly from [ − (cid:112) 1 /d, (cid:112) 1 /d ]
d .
At each round, the action set is of cardinality 100, where each action is uniformly sampled from [ − 1 , 1] d and normalized to have a unit (cid:96) 2 norm.
The link function is set to be
the sigmoid function h ( z ) =1+ e − z .
The variance upper bound at each round is uniformly sampled from [0 , 0 .
To simulate the variance bounded condition considered in this paper, our reward is not sampled from Bernoulli distribution.
Instead, at each round t , we sample a random variable (cid:15) (cid:48) t from Poisson distribution, i e., (cid:101) (cid:15) t ∼ Pois ( σ 2 t ) and (cid:15) (cid:48) t = min { (cid:101) (cid:15) t , R } where R is set to 10.
Our zero-mean noise (cid:15) t is computed by (cid:15) t = (cid:15) (cid:48)
t − E [ (cid:15) (cid:48) t ] .
With this construction, it is not hard to see that (cid:15) t satisﬁes Assumption 3.5 with sub-Gaussian parameter R .
We plot the cumulative regret of ML 2 and GLOC in Figure 1.
We can see that by adapting the multi-level and variance-aware scheme, our ML 2 algorithm outperforms the previous GLOC algorithm by a large margin.
B P ROOF S KECTCH OF T HEOREM 5.
Proof Sketch.
We prove the result by showing the validity of the conﬁdence sets and bounding the sum of single-step regret incurred by each variance level.
Step 1: Construction of conﬁdence sets We ﬁrst show that the construction of C t,l is big enough to contain f ∗ with high probability.
Recall the deﬁnition of C t,l as follows
[Equation]
Thus, it only sufﬁces to show that f ∗ satisﬁes the inequality in (B.1).
According to our ERM subroutine, we have (cid:88)
[Equation]
where the inequality holds since (cid:98) f t,l is the function which minimizes the cumulative squared loss at level l , i.e., (cid:80) s ∈ Ψ t,l ( f ( a s ) − r s ) 2 .
Therefore, to bound (cid:80) s ∈ Ψ t +1 ,l ( (cid:98)
f t +1 ,l ( a s ) − f ∗ ( a s )) 2 , it sufﬁces to bound the absolute value of (cid:80) s ∈ Ψ t,l (cid:15) s
[ f ∗ ( a s ) − (cid:98) f t , l ( a s )] .
Since (cid:98) f suffers from the measurability issue, we use the concentration of self-normalized process with an α -cover discretization argument to bound this term, which can ﬁnally show that f ∗ satisﬁes (B.1).
Step 2: Regret decomposition With the conﬁdence sets corresponding to different variance levels constructed in Step 1, we decompose the ﬁnal regret in the following way:
[Equation]
where the second inequality holds due to the optimism principle for arm selection, the last one holds due to the deﬁnition of width w C t,l ( a ) :
= max f ∈C t,l f ( a ) − min f ∈C t,l f ( a ) and the fact that f ∗ ∈ C t,l from Step 1.
Step 3:
Bounding the sum of widths From Step 2, the last step to bound the regret is to bound the summation of the width w C t,l ( a t ) .
According to the deﬁnition of eluder dimension, we can further bound w C t,l ( a t ) in terms of the eluder dimension of F and the value of β (See Lemma E.3 for more details).
According to the decomposition, we can ﬁrst bound the regret at each level l by a (cid:101) O
(2 l (cid:112) | Ψ T +1 ,l | ) term (omitting the dependencies on d, R and ¯ σ ) and then naturally bound the total regret by Lemma E.3 and the Cauchy Schwartz inequality.
The full proof is given in Section F.
C E XAMPLE : V ARIANCE - DEPENDENT R EGRET B OUNDS FOR B ERNOULLI B ANDITS
In this section, we consider the following speciﬁc Bernoulli multi-armed bandits with general function approximation, where the observed reward r t incurred by action a t is subject to the following
Bernoulli distribution:
[Equation]
It is easy to see that the noise (cid:15) t satisﬁes R = 1 /f ∗ ( a t ) ≤ 1 /ξ and σ 2 t = 1 /f ∗ ( a t )+1 /
(1 − f ∗ ( a t )) .
As discussed in Remark 3.6, we can use estimators of variance in Algorithm 1 instead of the true variance at each round.
The following corollary is an illustration on how to apply Algorithm 1 when variance information is not accessible but can be estimated through past observation.
Corollary C.1.
Let the same conditions as in Theorem 5.3 hold.
Set
α
= T − 2 , ¯ σ = 1 .
At each round, set σ 2 t ( a t ) to min { 1 , (cid:98) σ 2 ( a t ) + 2 /ξ 2 · w C t,lt ( a t ) } where (cid:98) σ 2 ( a t ) :
= 1 / (cid:98) f t,l t ( a t ) − 1 / (1 − (cid:98) f t,l t ( a t )) .
Then with probability at least 1 − δ , when T is large enough, the regret of Algorithm 1 for the ﬁrst T rounds is bounded as Regret( T ) =
(cid:101) O
( (cid:113) dim E log N α (cid:80) Tt =1 Var [ r t ] + ξ − 3 dim E log N α + ξ − 1 / 2 √ dim E log N α T ) .
Proof.
We suppose the event described in Theorem 5 . 1 holds and the conditions of Theorem 5.
hold without the given variance at each round.
At each round t , we can upper bound the variance of (cid:15) t :=
r t − f ∗ ( a t ) by making use of C t,l and (cid:98) f t,l returned by Algorithm 2.
Speciﬁcally, we estimate Var [ r t ] = σ 2 t = 1 /f ∗ ( a t )
+ 1 / (1 − f ∗ ( a t )) as 1 / (cid:98) f t,l t ( a t ) − 1 / (1 − (cid:98) f t,l t ( a t )) and bound the gap between (cid:98) σ 2 t and Var [ r t ] : | (cid:98) σ 2 t ( a t ) − Var [ r t ] | ≤ | 1 / (cid:98) f t,l t ( a t ) − 1 /f ∗ ( a t ) | + | 1 / (1 − (cid:98) f t,l ( a t ))
− 1 / (1 − f ∗ ( a t ))
|
[Equation]
where the ﬁrst inequality follows from the distribution of reward, and the last inequality holds according to Theorem 5.1.
Therefore, it is valid to replace σ 2 t by min { 1 , (cid:98) σ 2 t + 2 /ξ 2 · w C t,lt ( a t ) } in Algorithm 1.
Similar to the proof of Theorem 5.3, we have
[Equation]
where the ﬁrst equality follows from the deﬁnition of J , the ﬁrst inequality follows from (C.2), the last inequality holds due to Lemma E.2.
Hence J is of order (cid:101) O
( (cid:80) Tt =1 Var [ r t ] + 1 /ξ 3 · √ d log N α T ) .
According to Theorem 5.3, the regret bound of Algorithm 1 for this Bernoulli bandit problem is of order
[Equation]
ξ − 1 dim E log N α T ) ≤ (cid:101) O
( (cid:113) dim E log N α (cid:80)
Tt =1 Var [ r t ] + ξ − 3 dim E log
N α + ξ − 1 / 2 (cid:112) dim E log N α T ) .
This completes the proof.
D G AP - DEPENDENT R EGRET B OUNDS
In this section, we provide gap-dependent regret bounds for Algorithm 1 with various subroutines.
Denote ∆ t as the smallest gap between the reward of an optimal action and the reward of a suboptimal action:
[Equation]
where D ∗ t := argmax a ∈D t f ∗ ( a ) .
Let ∆ be the smallest gap in all the rounds: ∆ :=
min t ∈ [ T ] ∆ t .
Theorem [-C.
-]D.1 (Gap-dependent regret bound for bandits with heteroscedastic sub-Gaussian noise) .
Suppose Assumption 3.4 and 4.1 hold and | f ∗ ( a ) | ≤ C for all a ∈ A .
Let σ max = max t ∈
[ T ] σ t and suppose σ max > σ .
If we apply Algorithm 2 as a subroutine of Algorithm 1 (in line 9) and set β t,l as the same value in Theorem 4.2, then with probability at least 1 − δ , the regret of Algorithm 1 for the ﬁrst T rounds is bounded as follows:
[Equation]
Corollary D.2.
Let the same conditions as in Theorem D.1 hold.
Set
α = T − 2 .
Then with probability at least 1 − δ , when T is large enough, the regret for the ﬁrst T rounds is bounded as follows:
[Equation]
Remark D.3.
Corollary D.2 immediately suggests an (cid:101) O
( R 2 dim E log( N ( F , T − 2 , (cid:107)·(cid:107) ∞ )) / ∆) gapdependent regret by the fact σ max
= O ( R ) , which provides a novel instance-dependent bound for the original problem considered by Russo & Van Roy (2013).
To our knowledge, this is the ﬁrst result of its kind for the general bandit model.
Remark D.4.
When restricted to linear contextual bandits with dimension d , our result reduces to (cid:101) O ( σ 2max d 2 / ∆) , which matches the previous result derived in Abbasi-Yadkori et al.
Theorem {+D.
+} (Gap-dependent regret bound for bandits with heteroscedastic noise, informal) .
Suppose Assumption 3.4 holds and | f ∗ ( a ) | ≤ 1 for all a ∈ A .
Let σ max = max t ∈
[ T ] σ t (suppose σ max > σ ) and d = dim E ( F , 1 /T ) .
If we apply Algorithm 2 as a subroutine of Algorithm 1 (in line 9) and set β t,l as the same value in Theorem 5.3, then with probability at least 1 − 2 δ , the regret of Algorithm 1 for the ﬁrst T rounds is bounded as follows:
[Equation]
if T is large enough and we set α = T − 2 .
Remark D.6.
Similar to Remark 5.5, compared with the regret in Corollary D.2, the regret in Theorem D.5 has an additional term that depends on R .
E P ROOFS FROM S ECTION
Lemma E.1 (Proposition 3, Russo & Van Roy 2013) .
If ( β t ≥ 0 | t ∈ N ) is a nondecreasing sequence and F t :=
(cid:26) f ∈ F : (cid:80) t − 1 s =1 (cid:16) (cid:98) f t ( a s ) − f ( a s ) (cid:17) 2 ≤ β 2 t (cid:27) , then
T (cid:88) t = 1 ( w F t ( a t ) > (cid:15) ) ≤ (cid:18) 4 β 2 T (cid:15) 2 +(cid:19) dim E ( F , (cid:15) ) for all T ∈ N and (cid:15) > 0 .
Lemma E.2 (Lemma 2, Russo & Van Roy 2013) .
If ( β t ≥ 0 | t ∈ N ) is a nondecreasing sequence and F t :=
(cid:26) f ∈ F : (cid:80) t − 1 s =1 (cid:16) (cid:98) f t ( a s ) − f ( a s ) (cid:17) 2 ≤ β 2 t (cid:27) , then
T (cid:88) t = [Equation] for all T ∈ N .
Instead of using the previous approach that bounds the sum of widths, we take another approach to bound the sum of squared widths, which can further provide a novel gap-dependent result later.
Lemma E.3 (Bounding the sum of the square of widths) .
If ( β t ≥ 0 | t ∈ N ) is a nondecreasing sequence and F t :=
(cid:26) f ∈ F : (cid:80) t − 1 s =1 (cid:16) (cid:98) f
[Equation]
( a s ) − f ( a s ) (cid:17) 2 ≤ β 2 t (cid:27) , then T (cid:88) t =
[Equation]
for all T ∈ N .
Proof.
Following a similar approach as Russo & Van Roy (2013), we reorder the set { w F t ( a t ) } t ∈
[ T ] to { w t } t ∈ [ T ] , such that w 1 ≥ w 2 ≥ · · · ≥ w T .
Let T (cid:48)
=
max { t ∈
[ T ] , w t ≥ 1 T } .
By Lemma E.1,
[Equation]
for any δ ∈ (0 , (cid:15) ) .
Taking δ → 0 , we have
[Equation]
Hence,
T
[Equation]
where the ﬁrst inequality holds due to (cid:80)
Tt = T (cid:48) +1 w 2 t ≤ 1 /T under our deﬁnition of T (cid:48) , the second inequality follows from (E.2), the third inequality is derived by taking the integral.
With Assumption 4.1, we can directly apply the previous result on conﬁdence set by replacing the sub-Gaussianity η parameter by 2 (cid:96) +1 σ .
Previous result by Russo & Van Roy (2013) achieved a conﬁdence set of radius (cid:101) O
( (cid:112) η 2 log N α
+ αt ( C + η )) .
Ayoub et al.
(2020) later provides a result of the same order with improvement in terms of smaller constants.
Lemma E.4 (Theorem 5, Ayoub et al. 2020) .
Suppose that | f ∗ ( a ) | ≤ C for all a ∈ A .
For any α > 0 , if we set
[Equation]
then with probability at least 1 − δ , for all t ≥ 1 , l ∈
[ L ] , f ∗ ∈ C t,l .
Theorem E.5 (Restatement of Theorem 4.2) .
Suppose Assumptions 3.4 and 4.1 hold and | f ∗ ( a ) | ≤
C for all a ∈ A .
For all t ∈ [ T ] , l ∈ [ L ] and δ ∈ (0 , 1) , α > 0 , σ > 0 , if we apply Algorithm 2 as a subroutine of Algorithm 1 (in line 9) and set β t,l as the square root of [Equation] where N α = N ( F , α, (cid:107) · (cid:107) ∞ ) and L = (cid:100) log 2 R/σ (cid:101) (recall the deﬁnition of L in Algorithm 1), then with probability at least 1 − δ , the regret for the ﬁrst T rounds is bounded as follows:
[Equation]
Proof.
For simplicity, let d = dim E ( F , 1 /T 2 ) .
With probability at least 1 − δ , we have
[Equation]
≤ (cid:88) l ∈
{+[ L ] (cid:18) 
+}
+ 2 C · d + 4 β T,l (cid:113) d | Ψ T +1 ,l | (cid:19)
≤ L + 2 CdL + 2 (cid:115) L (cid:88)
l ∈
[ L ] β 2 T,l d | Ψ T
+1 ,l | (cid:124) (cid:123)(cid:122) (cid:125)
I, (E.6) where the ﬁrst equality holds by the deﬁnition in (3.1) [Equation] the second equality holds since Ψ T +1 , · forms a partition of [ T ] , the ﬁrst inequality holds due to Lemma E.4, the second inequality follows from
Lemma E.3, the third inequality is obtained by applying Cauchy-Schwarz inequality.
Then we continue to bound I 0 ,
[Equation]
where the ﬁrst inequality follows from the deﬁnition of β T [Equation] l
[Equation]
the third inequality holds due to the fact that ∀ [Equation] ∈ [ L ] , t ∈ Ψ T +
[Equation]
,l
[Equation]
the fourth inequality follows from the deﬁnition of J .
Substituting (E.7) into (E.6), we obtain
[Equation]
which completes the proof
[Equation]
Theorem E.6 (Restatement of Theorem D.1) .
Suppose Assumptions 3.4 and 4.1 hold and | f ∗ ( a ) | ≤
C for all a ∈ A .
Let σ max = max t ∈
[ T ] σ t .
If we apply Algorithm 2 as a subroutine of Algorithm 1 (in line 9) and set β t,l as the same value in Theorem 4.2, then with probability at least 1 − δ , the regret of Algorithm 1 for the ﬁrst T rounds is bounded as follows:
[Equation]
Proof.
For simplicity, let d = dim E ( F , 1 /T 2 ) .
Suppose the event described in Lemma E.4 holds.
With probability at least 1 − δ ,
Regret( T )
[Equation]
where the ﬁrst equality follows from the deﬁnition in (3.1), the second equality holds by the fact that Ψ T +1 ,l ( l ∈ [ L ]) forms a partition of [ T ] , the ﬁrst inequality holds due to the deﬁnition of ∆ in
Subsection 3.1, the second inequality follows from Lemma E.4, the fourth inequality holds due to
Lemma E.3 and the last inequality is derived by directly substituting the value of β T,l .
F P ROOFS FROM S ECTION
Lemma F.1 (Freedman 1975) .
Let M, v > 0 be ﬁxed constants.
Let { x i } ni =1 be a stochastic process, {G i } i be a ﬁltration so that for all i ∈ [ n ] , x i is G i -measurable, while most surely E [ x i |G
i − 1 ]
= 0 , | x i | ≤ M and n (cid:88) i =
E ( x 2 i |G i ) ≤ v.
Then, for any δ > 0 , with probability 1 − δ , for all t ∈ [ n ] ,
[Equation]
Lemma F.2.
Suppose a, b ≥ 0 .
If x 2 ≤ a + b · x , then x 2 ≤ 2 b 2 + 2 a .
Proof.
By solving the root of quadratic polynomial q ( x ) := x 2 − b · x − a , we obtain max { x 1 , x 2 } =
( b + √ b 2 + 4 a ) / 2 .
Hence, we have x ≤ ( b + √ b 2 + 4 a ) / 2 provided that q ( x ) ≤ 0 .
Then we further have
[Equation]
F.
[Equation]
Theorem F.3 (Restatement of Theorem 5.1) .
Suppose that | f ∗ ( a ) | ≤ C for all a ∈ A .
For any α > 0 and δ ∈ (0 , 1 / 2) , if we set β t,l as the square root of
[Equation]
where
[Equation]
then f ∗ ∈ C t,l for all t with probability at least 1 − 2 δ for any ﬁxed l .
Proof.
By simple calculation, for all f ∈ F we have
[Equation]
By sub-Gaussianity of (cid:15) t we have
[Equation] For simplicity, let event E subG :=
[Equation]
∀ t ≥
[Equation]
max 1 ≤ s ≤ t |
[Equation]
≤ R
[Equation]
2 log
[Equation]
4 t
[Equation]
δ
[Equation]
(cid:111)
[Equation]
Let G ( α ) ⊂
F be an α -cover of F in (cid:107) · (cid:107) ∞ .
From the deﬁnition of (cid:98) f t,l , we have
[Equation]
Let g = argmin G ( α ) (cid:107) (cid:98) f t,l − g (cid:107) ∞ .
We then bound the gap I ( g ) − I ( (cid:98) f t,l ) under event E subG ,
I ( g ) − I ( (cid:98) f t,l ) =
[Equation]
F.
[Equation]
Fix an f ∈ F .
Applying Freedman’s inequality (Lemma F.1), with probability at least 1 − δ , we have
[Equation]
for all t ≥ 1 .
Using a union bound on all the f ∈ G ( α ) and E subG , we further obtain that
[Equation]
for all f ∈ G ( α ) with probability at least 1 − 2 δ .
Substituting (F.7) into the deﬁnition of I ( f ) , we have that for g , it holds for probability at least 1 − 2 δ that
[Equation]
where the ﬁrst inequality is obtained by substituting (F.5) and (F.4) into the inequality below
[Equation]
the second inequality follows from the deﬁnition of I ( f ) and (F.7).
Using Lemma F. [Equation] we can deduce that
[Equation]
Then we can complete the proof by bounding the gap between (cid:80) s ∈ Ψ t,l ( g ( a s ) − f ∗ ( a s ))
and (cid:80) s
[Equation]
Theorem F.4 (Restatement of Theorem 5.3) .
Suppose Assumption 3.4 holds and | f ∗ ( a ) | ≤ 1 for all a ∈ A .
For all t ∈ [ T ] , l ∈ [ L ] and δ ∈ (0 , 1) , α > 0 , σ > 0 , if we apply Algorithm 2 as a subroutine of Algorithm 1 (in line 9) and set β t,l as the square root of
[Equation]
where N α = N ( F , α, (cid:107) · (cid:107) ∞ ) and R = R (cid:112) 2 log(4 t 2 L/δ )
(with a slight abuse of notation), then with probability at least 1 − 2 δ , the regret for the ﬁrst T rounds is bounded as follows:
[Equation]
Proof.
For simplicity, let d = dim E ( F , 1 /T ) .
Based on Theorem 5.1, for any ﬁxed l , we have f ∗ ∈ C t,l with probability 1 − 2 δ/L .
Applying a union bound on all l ∈ [ L ] , we have f ∗ ∈ C t,l for all t, l with probability at least 1 − 2 δ .
Then we further obtain, with probability at least 1 − 2 δ ,
Regret
[Equation]
[Equation] where the ﬁrst equality follows from the deﬁnition in (3.1), the second equality holds by the fact that Ψ T +1 ,l ( l ∈ [ L ]) forms a partition of [ T ] , the second inequality follows from Cauchy-Schwarz inequality and Deﬁnition in (3.3), the third inequality follows from Lemma E.3.
Following the deﬁnition of β t,l , we further calculate
Regret
[Equation]
where the ﬁrst inequality holds by the deﬁnition of β t,l and the fact that √ a + b ≤ √ a + √ b for a, b > 0 , the second inequality follows from the deﬁnition of l t , the third inequality follows from the deﬁnition of J .
Theorem F.5 (Formal version of Theorem D.5) .
Suppose Assumption 3.4 holds and | f ∗ ( a ) | ≤for all a ∈ A .
Let σ max = max t ∈
[ T ] σ t and d = dim E ( F , 1 /T ) .
If we apply Algorithm 2 as a subroutine of Algorithm 1 (in line 9) and set β t,l as the same value in Theorem 5.3, then with probability at least 1 − 2 δ , the regret of Algorithm 1 for the ﬁrst T rounds is bounded as follows: [Equation] (log T + 1)(12 + 4 R ) + 32 3 L ∆ Rd (log T + 1) log(2 N α T 2 L [Equation]
δ ) + 256 L ∆ σ
[Equation]
(log
[Equation]
log
[Equation]
Proof.
For simplicity, let d = dim E ( F , 1 /T ) .
Basically following the previous approach in Theorem D.1, with probability 1 − 2 δ , we have
Regret( T ) =
T (cid:88) t = ( f ∗ ( a ∗ t ) − f ∗ ( a t ))
[Equation]
where the ﬁrst equality holds by the deﬁnition in (3.1), the second equality holds by the fact that Ψ T +1 ,l ( l ∈ [ L ]) forms a partition of [ T ] , the ﬁrst inequality follows from the deﬁnition of ∆ in (D.1), the third inequality follows from Deﬁnition (3.3), the fourth inequality holds by Lemma E.3, the ﬁfth inequality follows from the deﬁnition of β T,l .
G P ROOFS FROM S ECTION
Lemma G.1.
Let reg t :=
(cid:80) ts =1 (cid:96) ( a (cid:62) s θ , r s ) − (cid:80) ts =1 (cid:96) ( a (cid:62)
s θ ∗ , r s ) .
Following Algorithm 3, with probability at least 1 − δ ,
[Equation]
for all t ≥ 1 .
We denote the corresponding event by E 1 .
Proof.
[Equation]
where the ﬁrst equality follows from Deﬁnition 6.4, the second equality holds by Taylor series expansion and ξ s is a point between a (cid:62) s θ and a (cid:62) s θ ∗ , the ﬁrst inequality follows from Assumption 6.2 and Assumption 6.3.
Further we obtain t (cid:88)
[Equation]
[Equation]
with probability 1 − δ , where the second inequality follows from the sub-Gaussianity of (cid:15) s .
Applying Lemma F.2, we obtain that with probability at least 1 − δ ,
[Equation]
Applying union bound on all t ≥ 1 , we have with probability at least 1 − δ ,
[Equation]
for all t ≥ 1 .
Lemma G.2 (Lemma 11, Abbasi-Yadkori et al. 2011) .
For any λ > 0 and sequence { x t } Tt =1 ⊂ R d for t ∈ { 0 , 1 , · · · , T } , deﬁne Z t
= λ
I + (cid:80) ti =
1 x i x (cid:62) i .
Then, provided that (cid:107) x t (cid:107) 2 ≤ M for all t ∈
[ T ] , we have
[Equation]
Lemma G.3.
For any λ > 0 and sequence { x t } Tt =1 ⊂ R d for t ∈ { 0 , 1 , · · · , T } , deﬁne Z t
= λ
I + (cid:80) ti =
1 x i x (cid:62) i .
Then, provided that (cid:107) x t (cid:107) 2 ≤ M for all t ∈
[ T ] , we have
[Equation]
Proof.
Applying matrix inversion lemma,
[Equation]
where the second equality follows from matrix inversion lemma
[Equation]
the second inequality holds by
Lemma G.
[Equation]
Lemma G.4.
Let
[Equation]
Suppose R t is an upper bound of max 1 ≤ s ≤ t | (cid:15) s | .
Then with probability at least 1 − δ , it holds simultaneously for all t ≥ 1 that
[Equation]
Proof.
To bound the sum of variance of each term, we caulculate
[Equation]
where the second inequality follows from the deﬁnition of Σ t , the third inequality holds by Lemma
G.2.
Also note that
[Equation]
since Σ t (cid:23) 4 A 2 K 2 κ
I + κ a s · a (cid:62) s .
Then we apply Freedman’s inequality, which gives for arbitrary t ≥ 1 , t (cid:88)
[Equation]
with probability at least 1 − δ .
Applying a union bound on all t ≥ 1 , we have with probability at least 1 − δ , t (cid:88)
[Equation]
since (cid:80) t ≥ 1 δ 2 t 2 ≤ δ .
Theorem G.5 (Restatement of Theorem 6.5) .
If we set φ ( θ ) = 2 A 2 K 2 κ (cid:107) θ (cid:107) 22 and assume that all the data points fed into the algorithm are of noise variance bounded by σ 2max , then with probability at least 1 − 3 δ , ∀ t ≥ 1 , the regret of Algrithm 3 for the ﬁrst t rounds is bounded as follows:
[Equation]
Proof.
For simplicity, let L t ( θ ) =
(cid:80) t − 1 s =1 (cid:96) ( θ (cid:62) a s , r s )
+ φ ( θ ) and loss t ( θ ) =
L t ( θ ) − φ ( θ ) .
Suppose event E 1 , event E subG :=
(cid:110) ∀ t ≥ 1 , max 1 ≤ s ≤ t | (cid:15) s | ≤ R (cid:112) 2 log(4 t 2 /δ )
(cid:111) and the event described in Lemma G.4 (denoted by E 2 ) simultaneously hold in the following proof.
By sub-Gaussianity of (cid:15) t we have
P
[Equation]
∃ t [Equation] G.
[Equation]
Hence, P ( E subG ) ≥ 1 − δ .
Applying Lemma G.1 and Lemma G.4, we have that P ( E subG ∩ E 1 ∩ E 2 ) ≥ 1 − 3 δ by union bound.
From the update rule of Algorithm 3, we calculate t
[Equation]
[Equation]
where the ﬁrst equality follows from the deﬁnition of loss , the second equality holds by the deﬁnition of L , the ﬁrst inequality holds since L t +1 ( θ t +1 )
=
min θ
∈ R
d
L t +1 ( θ ) .
Then we continue to bound I 1 .
[Equation]
G
[Equation]
where the second equality holds due to Taylor Expansion, (Let H be the Hessian matrix and θ (cid:48) s ∈ R d ), the third equality follows from the fact that ∂ L s ∂
θ ( θ s )
= 0 and ∂ L s
+1 ∂ θ ( θ s ) =
∂
L s ∂ θ ( θ s ) + ∂(cid:96) ( θ (cid:62) a s ,r s ) ∂
θ ( θ s ) the ﬁrst inequality is obtained by solving the quadratic function with respect to θ s +1 − θ s , the second inequality follows from ( a + b ) 2 ≤
a
+ 2 b 2 .
From the deﬁnition of H , we calculate
[Equation]
From Lemma G.1,
[Equation]
We bound I 3 by decomposing it into its expected value and a zero-mean term.
[Equation]
[Equation]
[Equation]
where the second inequality follows from (G.9), the third inequality holds by Lemma G.3.
Substituting (G.11) and (G.10) into (G.6), we have
[Equation]
Applying Lemma G.4 to bound I 4 , we calculate
[Equation]
where the second inequality holds due to event E subG , the third inequality follows from σ max ≤ R
[Equation]
Substituting (G.13) into (G.12) , we have
[Equation]
[Equation]
Substituting the upper bound of I 1 above to (G.5),
[Equation]
from which we can further complete the proof by the arbitrariness of t .
In the following lemma, we formally introduce the conversion from online learning regret to conﬁdence set in our setting.
For simplicity in analysis, we omit the level subscript and suppose all the data is fed into the same level.
Lemma G.6.
Suppose we feed loss function { (cid:96) s ( θ ) } ts =1 into a single online learner B .
Assume that B has an online learning (OL) regret bound reg t : ∀ t ≥ 1 , t
[Equation]
s [Equation] (G.16) Deﬁne X t as the design matrix consisting of a 1 , · · · , a t , z t =
[ z 1 , · · · , z t ] .
Then, with probability at least 1 − 4 δ ,
[Equation]
Proof.
With Lemma G.1, we can prove this lemma by following nearly the same proof for Theorem 1 in Jun et al.
(We can set β (cid:48) t in their proof to be
+ 4 κ reg t + 8 R 2 κ 2 log (cid:0) 4 t 2 /δ (cid:1) according to
Lemma G.1. )
Lemma G.7.
For all t , with z t and X t deﬁned as in Lemma G.8, we have
[Equation]
Proof.
After ridge regression, (cid:98) θ
[Equation]
= V
[Equation]
X
[Equation]
z t where V t :=
λ I + X (cid:62) t X t .
Then we have
[Equation]
We consider
[Equation]
From Schur complement theorem, we have
[Equation]
Then we can complete the proof by substituting (G.20) into (G.18) [Equation]
Lemma G.8 (Variance-dependent conﬁdence set for generalized linear bandits) .
Suppose that Assumption [Equation] 1, 6.2, 6.3 hold.
For any δ ∈ (0 , 1 / 4) , if we set
[Equation]
21) then with probability at least 1 − 4 δ
[Equation]
we have θ ∗ ∈ C t,l for all t ≥ 1 , l ∈
[ L ] .
Proof.
For any l ∈ [ L ] [Equation] with probability at least 1 − δL ,
[Equation]
making use of Lemma G.8 and Lemma G.7.
From Theorem 6.5, with probability at least 1 − 3 δL , we can set
[Equation]
1 + tAκ
[Equation]
[Equation]
(G.22)
By Union Bound, with probability 1 − 4 δ , for all l ∈ [ L ] ,
[Equation]
Theorem G.9.
Suppose that Assumptions 6.1 and 6.2 hold for the known reward function class F .
If we apply Algorithm 4 as a subroutine of Algorithm 1 (in line 9) and set β t,l to
[Equation]
for all t ∈ [ T ]
[Equation]
l ∈ [ L ]
[Equation] σ = R/ √ d [Equation] then with probability 1 − 4 δ , the regret of Algorithm 1 for the ﬁrst T rounds is bounded as follows:
[Equation]
Proof.
[Equation]
where the ﬁrst equality holds by the deﬁnition in (3.1), the ﬁrst inequality follows from Assumption 6.2, the second inequality holds by Lemma G.8.
For an arbitrary l ∈ [ L ] , let I 1 ( l ) :
= (cid:110) t ∈ Ψ T +1 ,l
[Equation]
x t
[Equation]
V − 1 t − 1 ,l ≤ 1 (cid:111) and I 2 ( l )
[Equation]
= Ψ T +1 ,l \I 1 ( l ) .
[Equation]
where the second inequality holds by Cauchy-Schwartz inequality, the third inequality follows from
Lemma G.2.
Similarly, we calculate
[Equation]
Substituting (G.24) and (G.25) into (G.23), we obtain [Table]
≤
[Table]
K
[Table]
[Equation]
where the second inequality holds due to Cauchy-Schwarz inequality, the fourth inequality follows from the deﬁnition of β t,l , the sixth inequality holds since l t in Algorithm 1 satisﬁes 2 l t σ ≤ max { σ t , σ } , the last inequality follows from the deﬁnition of J .