Improved Regret Analysis for Variance-Adaptive Linear Bandits and Horizon-Free Linear Mixture MDPs
Abstract
In online learning problems, exploiting low variance plays an important role in obtaining tight performance guarantees yet is challenging because variances are often not known a priori.
Recently, considerable progress has been made by
Zhang et al.
(2021) where they obtain a variance-adaptive regret bound for linear bandits without knowledge of the variances and a horizon-free regret bound for linear mixture Markov decision processes (MDPs).
In this paper, we present novel analyses that improve their regret bounds signiﬁcantly.
For linear bandits, we achieve ˜ O ( min
[Equation]
where d is the dimension of the features, K is the time horizon, and  2 k is the noise variance at time step k , and ˜ O ignores polylogarithmic dependence, which is a factor of d 3 improvement.
For linear mixture MDPs with the assumption of maximum cumulative reward in an episode being in , we achieve a horizon-free regret bound of ˜ O
[Equation]
where d is the number of base models and K is the number of episodes.
This is a factor of d 3 .
5 improvement in the leading term and d 7 in the lower order term.
Our analysis critically relies on a novel peeling-based regret analysis that leverages the elliptical potential ‘count’ lemma.
Introduction
In online learning, variance often plays an important role in achieving low regret bounds.
For example, for the prediction with expert advice problem, Hazan and Kale [11] proposed an algorithm that achieves a regret bound of O ( √
VAR K ) where VAR K is a suitably-deﬁned variance of the loss function up to time step K , without knowing VAR K ahead of time.
The implication is that when the given sequence of loss functions has a small variance, one can perform much better than the previously known regret bound O ( √
K ) .
For multi-armed bandits, Audibert et al.
[2] proposed an algorithm that achieves regret bounds that depends on the variances of the arms, which means that, again, the regret bound becomes smaller as the variances become smaller.
It is thus natural to obtain similar variance-adaptive bounds for other problems.
For example, in ddimensional stochastic contextual bandit problems, the optimal worst-case regret bound is ˜ O
(  d √
K ) where ˜ O hides polylogarithmic dependencies and  2 is a uniform upper bound on the noise variance.
Following the developments in other online learning problems, it is natural to ask if we can develop a similar variance-adaptive regret bound.
The recent work by Zhang et al.
[31] has provided an afﬁrmative answer.
Their algorithm called VOFUL achieves a regret bound of ˜ O (
[Equation]
) where  2 k is the (unknown) noise variance at time step k .
This implies that, indeed, it is possible to adapt to the variance and suffer a much lower regret.
Furthermore, they show that a similar variance-adaptive analysis can be used to solve linear mixture Markov decision processes (MDPs) with the unit cumulative rewards assumption :
[Equation] where r kh is the reward received at episode k and horizon h .
They show a regret bound of ˜ O
( d [Equation] √
K + d) , which does not depend on the planning horizon length H up to polylogarithmic factors.
We elaborate more on the linear bandit and linear mixture MDP problems in Section 2.
However, the regret rates of these problems have a large gap between the known lower and the upper bounds.
For example, in linear bandits, it is well-known that the regret bound has to be ⌦ ( d √
K ) [6], which rejects the possibility of obtaining o ( d ￿ ∑ [Equation] =
[Equation]  [Equation] ) , yet the best upper bound obtained so far is
O ( d 4 .
[Equation]
Thus, the gap is a factor of d 3 . 5 , which is quite large.
In this paper, we reduce such gaps signiﬁcantly by obtaining much tighter regret upper bounds.
Speciﬁcally, we show that a slight variation of VOFUL [31] for linear bandits has a regret bound of ˜ O
( min {
[Equation] } without knowledge of the variances.
This reduces the gap between the upper and lower bounds to only √ d for the leading term in the regret.
Furthermore, we employ a similar technique to show that the algorithm VARLin [31] for linear mixture MDPs with unit cumulative rewards has a regret bound of ˜ O ( d √
K + d) .
At the heart of our analysis is a direct peeling of the instantaneous regret terms using an elliptical potential ‘count’ lemma (EPC).
EPC bounds, given q
[Equation]
, how many times
[Equation]
≥ q happens from time k = 1 to ∞ where V k − =
∑ k − 1 s = 1 x s x ￿ s .
Our lemma is an improved and generalized version of [16, Exercise 19.3], which was originally used for improving the regret bound of linear bandit algorithms.
We provide the proofs of our main results for linear bandits and linear mixture MDPs in Section 3 and Section 4 respectively.
Finally, we conclude the paper with exciting future directions.
Related work.
There are numerous works on linear bandit problems such as [6, 4, 1, 17] where the information of variance is not used.
On the other hand, variance can be exploited to obtain better regret [2].
Recently, works by [31, 33] proposed ways to infuse the variance information in the regret analysis which improves the standard regret bound.
Reinforcement learning with linear function approximation has been widely studied to develop efﬁcient learning methods that work for large state-action space [27, 26, 13, 8, 14, 23, 24, 25, 29, 18, 15, 7, 21, 10, 9, 28].
To our knowledge, all aforementioned works derived a regret bound that depends on the planning horizon H polynomially.
It was Zhang et al.
[31] who ﬁrst remove the polynomial dependence of H in the linear mixture
MDP problem, achieving a bound of ˜ O
( d 4 .√
K + d) .
In contrast, our analysis shows that their algorithm in fact achieves signiﬁcantly better bound of ˜ O
( d √
K + d) .
Note that these results assume the unit cumulative rewards assumption (1) and time-homogeneous transition models.
In a similar setup where r h,k ∈ [ 0 ,] with time-inhomogeneous transition models, Zhou et al.
[33] achieve the regret ˜ O ( √ d 2 H + dH√
HK + d 2 H+ d 3 H) and show a lower bound of ˜ ⌦ ( dH￿ √
K ) .
These problem setups are incompatible to the setup of [31] and ours.
Problem Deﬁnition
Notations.
We denote d -dimensional ` 2 ball by B d(
R ) ∶ = { x ∈
R d ∶ ￿ x ￿ ≤
R } and deﬁne B d(
R ) similarly for the ` 1 ball.
Let [
N ] ∶ = { 1 , 2 ,....,N } for N ∈
N .
Given ` ∈
R and x ∈
R d , we deﬁne the clipping operator as follows (take
[Equation]
Linear bandits.
The linear bandit problem has the following protocol.
At time step k , the learner observes an arm set
[Equation]
d
[Equation]
) , chooses an arm x k ∈ X k , pulls it.
The learner then receives a stochastic reward r k = x ￿ k ✓ ∗ + ✏ k where ✓ ∗ ∈
B d( ) is an unknown parameter and ✏ k is a zero-mean stochastic noise.
Following [31], we assume that ( i ) ∀ k ∈ [
K ] , ￿ r k ￿ ≤ [ − 12 ,] almost surely, ( ii )
E [ ✏ k ￿ F k ] = 0 where
F k =  ( x 1 , ✏ 1 ,...,x k − 1 , ✏ k − 1 ,x k ) , and ( iii )
E [ ✏ 2 k ￿ F k ] =
2 k .
Note that the bound on ￿ r k ￿ implies that ￿ ✏ k ￿ ≤ 1 almost surely.
Our goal is to minimize the regret
R
[Equation]
Linear mixture MDPs.
We consider an episodic Markov Decision Process (MDP) with a tuple ( S ,
A ,r ( s,a ) ,P ( s ′ ￿ s,a ) ,K,H ) where
S is the state space,
A is the action space, r ∶ S × A → [ 0 ,] is the reward function, P ( s ′ ￿ s,a ) is the transition probability, K is the number of episodes, and H is the planning horizon.
A policy is deﬁned as ⇡ = { ⇡ h ∶ S → D ( A )}
Hh = 1 where
D ( A ) is a set of all distributions over
A .
For each episode k ∈ [
K ] , the learner chooses a policy ⇡ k , and then the environment executes ⇡ k on the MDP by successively following a kh ∼ ⇡ kh
( s kh ) and s kh + ∼
P ( ⋅ ￿ s kh ,a kh ) .
Then, the learner observes the rewards { r kh ∈ [ 0 ,]} k,h and moves onto the next episode.
The key modeling assumption of linear mixture MDPs is that the transition probability P is a linear combination of a known set of models {
P i } , namely, P =
∑ di = 1 ✓ ∗ i P i where ✓ ∗ ∈
B d( ) is an unknown parameter.
We follow [31] and make the following assumptions:
• The reward at each time step h and episode k is r kh =
r ( s
kh , a kh
) for some known function r ∶ S × A →
• Unit cumulative rewards
: ∑ Hh = 1 r kh ∈ [ 0 ,] for any policy ⇡ k .
For a policy ⇡ , V ⇡ h ( s ) ∶ = max a ∈ A
Q ⇡ h ( s,a ) where Q ⇡ h ( s,a ) =
r ( s,a ) +
E s ′ ∼
P ( ⋅ ￿ s,a )
V ⇡ h + ( s ′ ) and
V ⇡ H + [Equation]
Denoting V ⇡
[Equation]
and V
[Equation]
) , our goal is to minimize the regret
[Equation]
Variance-Adaptive Linear Bandits
In this section, we show that VOFUL of Zhang et al.
[31] has a tighter regret bound than what was reported in their work.
Our version of VOFUL, which we call VOFUL2, has slightly different conﬁdence set for ease of exposition.
Speciﬁcally, we use a conﬁdence set that works for every µ ∈
B d( ) rather than over an ✏ -net of B d( ) (but we do use an ✏ -net for the proof of the conﬁdence set).
The full pseudocode can be found in Algorithm 1.
VOFUL2 follows the standard optimism-based arm selection [3, 6, 1].
Let ✏ s ( ✓ ) ∶ = r s − x ￿ s ✓ and ✏ 2 s ( ✓ )
∶ = ( ✏ s ( ✓ )) 2 .
With L and ◆ deﬁned in Algorithm 1, we deﬁne our conﬁdence set after k time steps as
[Equation]
where ⇥
[Equation]
￿￿￿￿￿￿￿ and the clipping operator ( z ) ` is deﬁned in (2).
The role of clipping is two-fold: (i) it allows us to factor out ∑
[Equation] by
[Equation] and (ii) the lower order term is reduced to the order of− ` .
Both properties are critical in obtaining variance-adaptive regret bounds as discussed in [31].
The true parameter is contained in our conﬁdence set with high probability as follows.
Lemma 1.
(Conﬁdence set)
Let L , ◆ , and  be given as those in Algorithm 1.
Then,
P
[Equation]
In fact, in our algorithm, we use the conﬁdence set of ∩ k
[Equation]
= 1 ⇥ s at time step k for a technical reason.
VOFUL2 has the following regret bound.
Theorem 1.
VOFUL2 satisﬁes, with probability at least− 2  ,
R
[Equation]
￿￿￿
[Equation]
￿￿￿￿
[Equation]
( ￿  ) ￿￿￿
Algorithm 1 VOFUL 1: Initialize: L = ∨ ￿ log(
K ) ￿ where ◆ = 128ln (( 12 K 2 L )
d + ￿  ) and  ≤ e − 1 .
2: for k = 1 , 2 ,...,K do 3:
Observe a decision set
X k ⊆
B d( ) .
Compute the optimistic arm as following: x k = arg max x ∈ X k max ✓ ∈ ∩ k − 1 s = 1 ⇥ s x ￿ ✓ where ⇥ s is deﬁned in (3).
Receive a reward r k .
6: end for where ˜ O hides poly-logarithmic dependence on { d,K, ∑
[Equation]
[Equation]
Note that one can also show that VOFUL2 can be slightly modiﬁed to achieve the regret bound of ˜ O ￿ min
[Equation]
, thus being no worse than OFUL.
We postpone the proof of this to Section A.4 to avoid clutter.
Properties of the conﬁdence sets and implications on the regret.
Before presenting the proof of
Theorem 1, we provide some key properties of our conﬁdence set (Lemma 3) and the intuition behind our regret bound.
First, let us describe a few preliminaries.
Deﬁne
[Equation]
Let ✓ k be the maximizer of the optimization problem at line 4 of Algorithm 1 and deﬁne µ k = ✓ k − ✓ ∗ .
For brevity, we use a shorthand of
[Equation]
Finally, we need to deﬁne the following event regarding the concentration of the empirical variance around the true variance:
[Equation]
which is true with high probability as follows.
Lemma 2.
We have P ( E ) ≥ −
Proof.
The proof is a direct consequence of Lemma 13 in our appendix.
Let ` k be the integer ` such that x ￿ k
[Equation]
k
[Equation]
] and deﬁne A k ∶
=
∑ ks = 1  2 s .
Lemma 3 below states the properties of our conﬁdence set.
Lemma 3.
Suppose the events
E 1 and
E 2 are true.
Then, for any k with ` k = ` , (
i)
For some absolute constant c 1 ,
[Equation]
(ii) There exists an absolute constant c 2 such that x k µ k
[Equation]
The key difference between Lemma 3 and the results of Zhang et al.
[31] is that we use the norm notations, although the norm involves a rather complicated matrix W ` ,k − 1 .
This opens up possibilities of analyzing the regret of VOFUL2 with existing tools such as applying Cauchy-Schwarz inequality and the elliptical potential lemma [1, 5, 16].
In particular, Lemma 3(ii) seems useful because if we had such a result with W ` ,k
[Equation]
1 replaced by V k
[Equation]
￿ s , then we would have, ignoring the additive term ◆ ,
[Equation]
Together with the optimism and the standard elliptical potential lemma (see Section 3.1 for details), this leads to
R K ≤
K ￿ k = x ￿ k µ k ≤ c
K ￿ k = ￿ x k ￿ 2 V − 1 k − 1 ￿￿￿￿ k − ￿ s =  2 s ◆ ≤ c⋅
O ( d log (
T ￿ d ))
⋅ ￿￿￿￿
K ￿ s =
2 s ◆ .
Since ◆ is linear in d , we would get the regret bounded by the order of d 1 .￿ ∑
Kk = 1  2 k , roughly speaking.
However, the discrepancy between W ` ,k − 1 and V k − 1 is not trivial to resolve, especially due to the fact that Lemma 3(ii) has µ k on both left and the right hand side.
That is, µ k is the key quantity that we need to understand, but we are bounding x k µ k as a function of µ k .
The novelty of our analysis of regret is exactly at relating W ` ,k − 1 to V k − 1 via a novel peeling-based analysis, which we present below.
3.1 Proof of Theorem
Throughout the proof, we condition
E 1 and
E 2 where each one is true with probability at least−  , as shown in Lemma 1 and 2 respectively.
For our regret analysis, it is critical to use Lemmabelow, which we call the elliptical ‘count’ lemma.
This lemma is a generalization of Lattimore and
Szepesvári [16, Exercise 19.3], which was originally used therein to improve the dependence of the range of the expected rewards in the regret bound.
Similar lemmas have been used in parallel studies [12, 22].
In particular, He et al.
[12] employ a lemma similar to elliptical potential count and peeling technique for the regret analysis for the linear MDP as well, which we compare in detail in Section C due to space constraint.
We remark that a similar strategy appears in disguise in Russo and Van Roy [20, Proposition 3] as well.
Lemma 4.
(Elliptical potential count) Let x 1 ,...,x k ∈
R d be a sequence of vectors with ￿ x s ￿ ≤
X for all s ∈ [ k ] .
Let V k = ⌧
I + ∑ ks = 1 x s x ￿ s for some ⌧ > 0 .
Let J ⊆
[ k ] be the set of indices where ￿ x s ￿ 2 V − 1 s − ≥ q .
Then, ￿
[Equation]
As the name explains, the lemma above bounds how many times ￿ x s ￿ 2 V − 1 s − 1 can go above a given value q > 0 , which is different from existing elliptical potential lemmas that bound the sum of ￿ x s ￿ 2 V − 1 s − 1 .
Let ✓ k be the ✓ that maximizes the optimization problem at line 4 of Algorithm 1.
We start by the usual optimism-based bounds: due to
E 1 , we have
R
[Equation]
We now take a peeling-based regret analysis that is quite different from existing analysis techniques:
[Equation]
− ✓ ∗ ) ≤ − L K +
L ￿ ` =  − ` + 2 K ￿ k =
￿ x ￿ k µ k ∈ ( ⋅ − ` ,⋅ − ` + ] ￿ , where L is deﬁned in Algorithm 1.
Given ` and k , let n k, ` be n such that max v ∶ k ≤ v ≤
K, ` v = ` ￿ x ￿ k µ v ￿ ∈ ( − ` + n ,− ` + n + ] if such n satisﬁes n ≥ 1 .
Otherwise, set n k, ` = 0 , which means max v ∶ k ≤ v ≤
K, ` v = ` ￿ x ￿ k µ v ￿ ≤ − ` + n + 1 with n = 0 .
We then deﬁne G ` ,n ∶ = { s ∈ [
K − ] ∶ ` s = ` ,n s, ` = n } and let G ` ,n [ k ] ∶ =
G ` ,n ∩ [ k ] .
Then,
K ￿ k = ￿ x ￿ k µ k ∈ ( ⋅ − ` ,⋅ − ` + ] ￿ =
K ￿ k = { ` k = ` } ≤ + ￿ n ￿ s ∈
G k,n 1 .
Letting V ` ,n,k − ∶ = − ` I + ∑ s ∈
G ` ,n [ k − ] x s x ￿ s , a comparison between two matrices W and V is given as follows for every v ∈ { k,...,K } :
[Equation]
For k ∈
G ` ,n [
K − ] and u = arg max v ∶ k ≤ v ≤
K, ` v = ` ￿ x ￿ k µ v ￿ , we have − ` + n < ￿ x k µ u ￿ ≤ ￿ x k ￿
W − 1 ` ,k − ( µ u ) ￿ µ u ￿
W ` ,k − ( µ u ) ≤ ￿ x k ￿
W − 1 ` ,k − ( µ u ) ￿ µ u ￿
W ` ,u − ( µ u ) ( u ≥ k ) ≤ c √ 2 n ￿ x k ￿
V − 1 ` ,n,k − 1 ￿ 2 − ` ( ￿
A K ◆ + ◆ ) .
((4) and Lemma 3(i))
Consequently, ￿ x k ￿ 2 V − 1 ` ,n,k − ≥ c 2 − ` + n √
A K ◆ + ◆ .
Thus, using Lemma 4 with ⌧ = − ` , + ` ￿ n = ￿ k ∈
G k,n ≤ + ￿ n ￿ k ∈
G k,n ￿￿￿￿￿￿￿￿ x k ￿ 2 V − 1 ` ,n,k − ≥ c − ` + n √
A K ◆ + ◆ ￿￿￿￿￿￿￿ ≤ + c ￿ n 2 ` − n ( ￿
A K ◆ + ◆ )
d ln ￿ + c 4 ` ( ￿
A K ◆ + ◆ ) ￿ ≤ c 2 ` ( ￿
A K ◆ + ◆ )
d ln ￿ + c 4 ` ( ￿
A K ◆ + ◆ ) ￿ .
where we use the fact that￿ ln ( + q ) ≤ c ￿ q for an absolute constant c if q is bounded by an absolute constant.
Finally,
[Equation]
We choose L = ∨ ￿ log(
K ) ￿ , which leads to
[Equation]
This concludes the proof.
Linear Mixture MDP
As linear bandits and linear mixture MDPs have quite a similar nature, we bring the techniques in our analysis of VOFUL2 to improve the regret bound of VARLin of Zhang et al.
A key feature of linear mixture MDP setting is that one can estimate the upper bound of the variance as it is a quadratic function of ✓ ∗ while linear bandits do not have a structural assumption on the variance.
Thanks to such a structural property, we obtain a slightly better dependence on the dimension d .
The conﬁdence set derived for our proposed algorithm is slightly different from that of VARLin as ours is deﬁned with
[Equation]
rather than an ✏ -net.
Our version of VARLin, which we call VARLin2, is described in Algorithm 2.
Given s kh and a kh , let us deﬁne P s kh ,a kh (
[Equation] =
E s ′ ∼
P skh,akh [
V kh + ( s ′ )] and x mk,h ∶ =
P 1 s kh ,a kh (
V kh + ) 2 m ,...,P ds kh ,a kh (
V kh + ) 2 m ] ￿ and let L , ◆ , and  be given as deﬁne Algorithm 2.
Let ✏ mv,u ( ✓ ) ∶ = ✓ ￿ x mv,u − (
V vu + ( s vu + )) 2 m for ( v,u ) ∈ [
K ] ×
H ] , m ∈ { 0 , 1 ,...,L } where L is deﬁned in Algorithm 2.
We construct our conﬁdence set as
[Equation]
(5) where we deﬁne ⇥ m,i, ` k below, based on the data collected up to episode k − 1 .
First, let ⌘ mk,h ∶ = max ✓ ∈ ⇥ k
{ ✓ ￿ x m + 1 k,h − ( ✓ ￿ x mk,h ) } and
T m,ik,h ∶ = {( v,u ) ∈ ([ k ] × [
H ]) ∪ ({ k } ×
[ h ]) ∶ ⌘ mv,u ∈ ( − i , 2− i ])} .
We naturally deﬁne
T m,L + 1 k,h ∶ = {( v,u ) ∈ T m,ik,h ∶ ⌘ mv,u ≤ −
L } .
With ◆ deﬁned in Algorithm 2, deﬁne ⇥ m,i, ` k
[Equation]
￿￿￿￿￿￿￿
[Equation]
￿￿￿￿￿￿￿￿￿￿￿￿￿￿￿￿
[Equation]
Algorithm
1: Initialize: L = ￿ log 2 HK ￿ + 1 , ◆ = 3ln (( 2 HK )
( d + ) ￿  ) ,  ≤ e − 1 .
2: for k = 1 , 2 ,...,K do 3: for h =
H,..., 1 do 4:
For each ( s,a ) ∈ S × A , deﬁne Q kh
( s,a ) =
min { 1 ,r ( s,a ) + max ✓ ∈ ⇥ k − ∑ di = 1 ✓ i P is,a V kh + } where ⇥ k − 1 is deﬁned in Lemma5:
For each state s , V kh ( s ) = max a ∈ A
Q kh ( s,a ) .
6: end for 7: for h = 1 ,...,H do 8:
Choose a kh = arg max a ∈
A
Q kh ( s kh ,a ) .
Observe a reward r kh and the next state s kh + 1 . 10: end for 11: end for ￿￿￿￿￿ ￿ ( v,u ) ∈ T m,ik − 1 ,H ￿ ( x mv,u ) ￿ µ ￿ ` ⌘ mv,u ◆ + ⋅ − ` ◆ , ∀ µ ∈ B d( ) ￿￿￿￿￿ ￿￿ (6) We show that the conﬁdence set is correct w.h.p. in the following lemma.
Lemma 5. (Conﬁdence set for MDP) P [Equation] The consequence is that the Q values computed in VarLin2 is optimistic with high probability due to the following property:
Lemma 6.
For every k
[Equation]
✓
[Equation]
Now with the conﬁdence set deﬁned above we state our main result.
Theorem 2.
With probability at least 1-  ,
[Equation]
where ˜ O hides poly-logarithmic dependence on
[Equation]
4.1 Proof of Theorem
The main idea of the proof is to infuse a peeling-based argument together with elliptical potential count lemma to both the planning horizon and episode.
Noting that the regret of the predicted variance is controlled by the variance of variance, one can expect to reduce the total regret using this information, as done in [31].
We begin by introducing relevant quantities that are parallel with those in linear bandits.
Let us ﬁrst introduce the following lemma.
Lemma 7.
Let x 1 ,...,x T ∈
R d be a sequence of vectors with ￿ x t ￿ ≤
X for all t ∈ [
T ] .
Let
V t =  I + ∑ ts = 1 x s x ￿ s .
Let= ⌧< ⌧< ⌧< ...
< ⌧ z =
T where ⌧ i marks the last time step of the i -th block formed by { ⌧ i − + 1 ,..., ⌧ i } for all i ∈ { 0 ,...,z } .
Let anc ( t ) be the ‘anchor’ of t , the last time step of the i -th block such that the ( i + ) -th block contains t : anc ( t )
=
max { ⌧ i ∶ i ∈ { 0 ,...,z } , ⌧ i < t } .
Let r > 0 .
Deﬁne J ⊆
T ] to be the set of indices t such that ￿ x t ￿ 2 V − 1 anc ( t ) >
r ￿ x t ￿ 2 V − 1 t − 1 is true for the ﬁrst time in the block containing t .
Then, ￿
J ￿ ≤ ln ( r ) d
ln ￿ + ￿ e ln ( r )
X ￿ .
To keep track of episode-horizon index pairs concisely, we use a ﬂat index t ∈ [
T ] where T ∶ =
HK .
Speciﬁcally, an episode k and a horizon h corresponds to the ﬂat index t =
( k − )
H + h .
Let t ( k,h ) ∶ =
( k − )
H + h .
Let k ( t ) and h ( t ) be the mapping from t to its corresponding episode and horizon index respectively
so that k = k ( t ( k,h )) and h = h ( t ( k,h )) .
By taking ⌧ k in Lemma 7 as t ( k,H ) , we have that anc ( t ) ∶ = t ( t − 1 ,H ) .
We deﬁne
T m,it ∶ = { t ( k ′ ,h ′ )
∶ ( k ′ ,h ′ ) ∈
T m,ik,h } and µ mt ∶ = µ m k ( t ) , h ( t ) .
Similarly, we deﬁne x mt , etc., by replacing the subscript k,h with t .
Hereafter, any appearance of subscript k,h can be replaced with t such that t
[Equation] without changing the meaning.
Given m , k and h , we deﬁne ` mk,h as the integer ` such that ( x mk,h ) ￿ µ mk,h ∈ ( ⋅ − ` ,⋅ − ` + ] where µ mk,h ∶ = ✓ mk,h − ✓ ∗ and ✓ mt = arg max ✓ ∈ ⇥ k − ￿ { ✓ ￿ x m + 1 t − ( ✓ ￿ x mt ) } ￿ .
For simplicity, we abbreviate ` mk,h by ` .
Deﬁne W m,i, ` t [Equation] ) ∶ [Equation]
) ￿ and introduce a shorthand
W m,i, ` anc ( [Equation] [Equation] =
W m, i [Equation] ) as before.
With the deﬁnition above we have the following:
[Equation]
We now show the key result of the conﬁdence set of VarLin2 that parallels Lemma 3 for bandits.
Lemma 8.
Fix m ∈ { 0 ,...,L m } and i ∈ [
L ] .
Let t ∈ T m,iT .
Then, with ` =
[Equation]
mt ,
[Equation]
} where c M > 0 is an absolute constant.
What is different from the linear bandit problem is that we do not update ✓ until the planning horizon is over and an additional layer for peeling is imposed on variance.
In [31], the authors introduce an indicator I k h to characterize episode-horizon pairs for which growth of the norm of µ with respect to
W t is controlled by the norm with respect to W anc ( t ) with d 2 growth rate, i.e.,
I kh ∶ = { ￿ µ t ￿
W m,i, ` t ≤ ( d + ) ￿ µ t ￿
W m,i, ` anc ( t ) } .
where t = t ( k,h ) .
Our novelty novelty lies in being able to replace( d + ) 2 above by a constant rate r that is set to 2 later (modulo some differences due to technical reasons).
To distinguish we denote such a set by I k,h .
See B.5 for the deﬁnition of I k,h and the proof of the following lemma.
Lemma 9.
[Equation]
Note here that once we ﬁx r such as r = 2 , the bound can be replaced by O ( d log
[Equation]
dHK )) .
We now use the following regret decomposition due to [31] which just come from replacing I kh by I k,h .
Lemma 10.
(Zhang et al.
R
[Equation]
Reg+ Reg+ Reg+ ∑
Kk =
∑ H − 1 h (
I k,h −
I k,h + ) where Reg= ∑ k,h ( P s kh ,a kh V kh + − V kh + ( s kh + ))
I k,h , Reg= ∑ k,h (
V kh ( s kh )
− r kh −
P s kh ,a kh V kh + )
I k,h , and Reg= ∑
Kk =
( ∑
Hh = 1 r kh −
V ⇡ k( s k)) .
Let ˘ x mk,h ∶ = x mk,h
I k,h and deﬁne R m , M m as
R m ∶ = ￿ k,h ( ˘ x mk,h ) ￿ µ mk,h , and M m ∶ =
￿ k,h (
P s kh ,a kh (
V kh + ) 2 m − (
V kh + ( s kh + )) 2 m )
I k,h .
We have that Reg=
M 0 and Reg≤
R 0 since
Q k h ( s,a ) − r ( s,a ) − P s,a V k h + ≤ max ✓ ∈ ⇥ k x 0 k,h ( ✓ − ✓ ∗ ) .
To proceed, we ﬁrst note that ∑ k,h (
I k,h −
I k,h + ) and Reg 3 are bounded by O ( d log( dHK )) and
O ( ￿
K log ( ￿  )) respectively from Lemma 9 and Lemma 16.
Since Reg+
Reg≤
R+
M 0 , it remains to ﬁnd a bound on R+
M 0 .
This, however, involves solving a series of recursive inequalities.
We leave the details in the appendix and provide a high-level description below.
Let us begin with Lemma 15 in the appendix that shows
[Equation]
where the RHS is a function of √
M m + 1 and √
R 0 .
Taking Proposition 1 below for granted and combining it with the relation from Zhang et al.
[31, Eq.
(57)] showing ￿
[Equation]
one arrives at R m ≤ ˜ O
￿ d
[Equation]
￿ (
M m + [Equation] +
[Equation]
K +
[Equation]
+
R
[Equation]
+ + [Equation] R m + d ) .
This bound is the key improvement we obtain via our peeling-based regret analysis.
Speciﬁcally, the bound on R m obtained by [31] has d 4 and d 6 in place of d￿ 2 and d above.
We ﬁrst show how our regret bound helps in obtaining the stated regret bound and then present
Proposition 1.
Noting that both R L and M L are trivially bounded by HK , one can solve the series of inequalities on R m and ￿
M m ￿ to obtain a bound on R 0 :
[Equation]
Solving it for R 0 , we obtain R
[Equation]
One can now plug in R 0 to the bound (7) and obtain a bound on ￿
M￿ in a similar way as follows, which concludes the proof:
[Equation]
We now show the key proposition that allows us to improve the bound on R m .
In the paper by Zhang et al.
[31], d 4 was derived while we propose the following.
Proposition 1.
Let ˘ ⌘ mk,h
[Equation]
mk,h I k,h .
Then, we have
[Equation]
Proof.
Deﬁne
T m,i, ` ∶ = { t ∈ T m
[Equation]
∶ (
[Equation]
) ￿ µ mt ∈ ( ⋅ −
[Equation]
,⋅ 2− ` ]} and split the time steps
T m,i, ` by
T m,i, ` ,
￿ ￿ ∶ =
￿￿￿￿￿￿￿ t ∈ T m,i, ` ∶ ￿ µ mk,h ￿
W m,i, ` anc ( a ) ≤ c M √ − i ◆ ￿￿￿￿￿￿￿ and
T m,i, ` , ￿ ￿ ∶ = T m,i, ` ￿ T m,i, ` , ￿ ￿ .
Having deﬁned I k,h with r = 2 , we also denote ˘
T m,i, ` , ￿ z ￿ ∶ = T m,i, ` , ￿ z ￿ ∩ ￿ t ∈ [
T ] ∶
I t =
￿ .
Now we decompose R m as
[Equation]
i [Equation] ` ￿
[Equation] ˘ T m [Equation]
i [Equation] ` [Equation]
￿ ￿
[Equation] ) ￿ µ mt
[Equation] i, ` ￿ t ∈ ˘
T m,i, ` , ￿ ￿ ( ˘ x mt ) ￿ µ mt .
Fix m , i and ` and focus on ∑ t ∈ ˘
T m,i, ` , ￿ z ￿ ( ˘ x mt )
￿ µ mt for z = 1 , 2 .
Hereafter, we omit the superscripts and subscripts of ( m,i, ` ) to avoid clutter, unless there is a need.
Note that for t ∈ ˘
T ￿ ￿ ,n and b such that t
[Equation] − ` I + ￿ t ′ ∈ T ￿ ￿ ,n anc ( t ) ( ∧ − ` − ` + n + ) x t ′ x ￿ t ′ ￿ − ` I + − n − ￿ t ′ ∈ T ￿ ￿ ,n anc ( t ) x t ′ x ￿ t ′ ￿ c− n V ￿ ￿ ,n anc ( t ) .
For the same t , letting b = arg max t ≤ b ′ ∈ ˘
T ￿ ￿ ￿ x ￿ t µ b ′ ￿ , − ` + n ≤ ￿ x ￿ t µ b ￿ ≤ ￿ x t ￿
W − 1 anc ( t ) ( µ b ) ￿ µ b ￿
W anc ( t ) ( µ b ) ≤ √ 2 n ￿ x t ￿ (
V ￿ 1 ￿ ,n anc ( t ) )
− ￿ µ b ￿
W anc ( b ) ( µ b ) ≤ c √ r √ 2 n ￿ x t ￿ (
V ￿ 1 ￿ ,n t − ) − 1 √ 2 − i ◆ (by b ∈ ˘
T ￿ ￿ )
This implies that ￿ x t ￿ (
V ￿ 1 ￿ ,n t − ) − ≥ c 2 − 2 ` + n r− i ◆ .
Thus, ￿ t ∈ ˘
T ￿ ￿ x ￿ t µ t ≤ c− ` ￿ t ∈ ˘
T ￿ ￿ ≤ c− ` ￿￿ ˘
T ￿ ￿ ￿ ￿ t ∈ ˘
T ￿ ￿ ≤ c− ` ￿￿￿￿￿￿ ˘
T ￿ ￿ ￿ ` ￿ n = ￿ t ∈ ˘
T ￿ ￿ ,n ≤ c− ` ￿￿￿￿￿￿ ˘
T ￿ ￿ ￿ ` ￿ n = ￿ t ∈ ˘
T ￿ ￿ ,n ￿￿ x t ￿ (
V ￿ 1 ￿ ,n t − ) − ≥ c 2 − 2 ` + n r− i ◆ ￿ ≤ c− ` ￿￿￿￿￿ ˘
T ￿ ￿ ￿ ` ￿ n = ￿ t ∈ T ￿ ￿ ,n ￿￿ x t ￿ (
V ￿ 1 ￿ ,n t − ) − ≥ c 2 − 2 ` + n r− i ◆ ￿ ( ˘
T ￿ ￿ ,n ⊆ T ￿ ￿ ,n ) ≤ c− ` ￿￿￿￿￿ ˘
T ￿ ￿ ￿ ` ￿ n = r− i ◆ − 2 ` + n d ln ￿ + c r− i ◆ − 2 ` + n− ` ￿ ≤ c ￿ dr ￿ ˘
T ￿ ￿ ￿ − i ◆ ln ￿ + cr ◆ 8 ` ￿ ≤ c ￿ dr ( + ￿ t ∈ ˘
T ￿ ￿ ⌘ mt ) ◆ ln ￿ + cr ◆ 8 ` ￿ where we use the fact that ￿ T ￿ ￿ ￿ ⋅ − i ≤
O ( + ∑ t ∈ ˘
T ￿ ￿ ⌘ mt ) , which is straightforward by the deﬁnition.
The summation over ˘
T ( ) can be handled in a similar way and the details of the proof is provided in
Section B.7 in our appendix.
Conclusion
In this work, we have made signiﬁcant improvements in the regret upper bounds for linear bandits and linear mixture MDPs by employing a novel peeling-based regret analysis based on the elliptical potential count lemma.
Our study opens up numerous future research directions.
First, the optimal regret rates are still not identiﬁed for these problems.
It would be interesting to close the gap between the upper and lower bound.
Second, our algorithms are not computationally tractable.
We believe computationally tractable algorithms, even at the price of increased regret, may lead to practical algorithms.
Finally, characterizing variance-dependent uncertainty in the linear regression setting without prior knowledge of variances is an interesting statistical problem on its own.
Identifying novel estimators for it and proving their optimal coverage would be interesting.
Acknowledgments and Disclosure of Funding
The authors thank Liyu Chen for ﬁnding an error in our earlier version. Insoon Yang is supported in part by the National Research Foundation of Korea (MSIT2020R1C1C1009766), the Information and Communications Technology Planning and Evaluation (IITP) grants (MSIT2022-0-00124, MSIT20220-00480), and Samsung Electronics. Kwang-Sung Jun is supported by Data Science Academy and
Research Innovation & Impact at University of Arizona.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] There is no negative impacts, to our knowledge.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]