Improved Regret Analysis for Variance-Adaptive Linear Bandits and Horizon-Free Linear Mixture MDPs
AfﬁliationAddress email
Abstract
In online learning problems, exploiting low variance plays an important role inobtaining tight performance guarantees yet is challenging because variances areoften not known a priori.
Recently, considerable progress has been made by
Zhang et al.
(2021) where they obtain a variance-adaptive regret bound for linearbandits without knowledge of the variances and a horizon-free regret bound forlinear mixture Markov decision processes (MDPs).
In this paper, we presentnovel analyses that improve their regret bounds signiﬁcantly.
For linear bandits,we achieve ˜ O
( d 1 .￿ ∑
Kk = 1  2 k + d) where d is the dimension of the features, Kis the time horizon, and  2 k is the noise variance at time step k , and ˜ O ignorespolylogarithmic dependence, which is a factor of d 3 improvement.
For linearmixture MDPs with the assumption of maximum cumulative reward in an episodebeing in [ 0 ,] , we achieve a horizon-free regret bound of ˜ O
( d √
K + d) where dis the number of base models and K is the number of episodes.
This is a factor ofd 3 .
5 improvement in the leading term and d 7 in the lower order term.
Our analysiscritically relies on a novel peeling-based regret analysis that leverages the ellipticalpotential ‘count’ lemma.
Introduction
In online learning, variance often plays an important role in achieving low regret bounds.
Forexample, for the prediction with expert advice problem, Hazan and Kale [11] proposed an algorithmthat achieves a regret bound of O ( √
VAR K ) where VAR K is a suitably-deﬁned variance of the lossfunction up to time step K , without knowing VAR K ahead of time.
The implication is that whenthe given sequence of loss functions has a small variance, one can perform much better than thepreviously known regret bound O
[Equation]
For multi-armed bandits, Audibert et al.
[2] proposed analgorithm that achieves regret bounds that depends on the variances of the arms, which means that,again, the regret bound becomes smaller as the variances become smaller.
It is thus natural to obtain similar variance-adaptive bounds for other problems.
For example, in ddimensional stochastic contextual bandit problems, the optimal worst-case regret bound is ˜ O
(  d √
K ) where ˜ O hides polylogarithmic dependencies and  2 is a uniform upper bound on the noise variance.
Following the developments in other online learning problems, it is natural to ask if we can developa similar variance-adaptive regret bound.
The recent work by Zhang et al.
[27] has provided anafﬁrmative answer.
Their algorithm called VOFUL achieves a regret bound of ˜ O
( d 4 .￿ ∑
Kk =
[Equation]
k + d) where  2 k is the (unknown) noise variance at time step k .
This implies that, indeed, it is possibleto adapt to the variance and suffer a much lower regret.
Furthermore, they show that a similarvariance-adaptive analysis can be used to solve linear mixture Markov decision processes (MDPs) with the unit cumulative rewards assumption :
￿ h r kh ∈ [ 0 ,] , ∀ k (1) where r kh is the reward received at episode k and horizon h .
They show a regret bound of ˜ O
( d 4 .√
K + d) , which does not depend on the planning horizon length H up to polylogarithmic factors.
Weelaborate more on the linear bandit and linear mixture MDP problems in Section 2.
However, the regret rates of these problems have a large gap between the known lower and the upperbounds.
For example, in linear bandits, it is well-known that the regret bound has to be ⌦ ( d √
K )
[6],which rejects the possibility of obtaining o ( d ￿ ∑
Kk = 1  2 k ) , yet the best upper bound obtained so far is
O ( d 4 .￿ ∑
Kk = 1  2 k ) .
Thus, the gap is a factor of d 3 . 5 , which is quite large.
In this paper, we reduce such gaps signiﬁcantly by obtaining much tighter regret upper bounds.
Speciﬁcally, we show that VOFUL [27] for linear bandits with unknown variance has a regretbound of ˜ O
( d 1 .￿ ∑
Kk = 1  2 k ) .
This reduces the gap between the upper and lower bounds to only√ d for the leading term in the regret.
Furthermore, we employ a similar technique to show that thealgorithm VARLin [27] for linear mixture MDPs with unit cumulative rewards has a regret bound of˜ O ( d √
K + d) .
At the heart of our analysis is a direct peeling of the instantaneous regret terms usingan elliptical potential ‘count’ lemma (EPC).
EPC bounds, given q > 0 , how many times ￿ x k ￿ 2 V − 1 k − ≥ qhappens from time k
= 1 to ∞ where V k − =
∑ k
[Equation]
=
[Equation]
s
[Equation]
s .
Our lemma is an improved and generalized 50 version of [15, Exercise 19.3], which was originally used for improving the regret bound of linearbandit algorithms.
We provide the proofs of our main results for linear bandits and linear mixture
MDPs in Section 3 and Section 4 respectively.
Finally, we conclude the paper with exciting futuredirections.
Related work.
There are numerous works on linear bandit problems such as [6, 4, 1, 16] where theinformation of variance is not used.
On the other hand, variance can be exploited to obtain betterregret [2].
Recently, works by [27, 28] proposed ways to infuse the variance information in the regretanalysis which improves the standard regret bound.
Reinforcement learning with linear functionapproximation has been widely studied to develop efﬁcient learning methods that work for largestate-action space [24, 23, 12, 8, 13, 20, 21, 22, 26, 17, 14, 7, 19, 10, 9, 25].
To our knowledge, allaforementioned works derived a regret bound that depends on the planning horizon H polynomially.
It was Zhang et al.
[27] who ﬁrst remove the polynomial dependence of H in the linear mixture
MDP problem, achieving a bound of ˜ O
( d 4 .√
K + d) .
In contrast, our analysis shows that theiralgorithm in fact achieves signiﬁcantly better bound of ˜ O
( d √
K + d) .
Note that these results assumethe unit cumulative rewards assumption (1) and time-homogeneous transition models.
In a similarsetup where r h,k ∈ [ 0 ,] with time-inhomogeneous transition models, Zhou et al.
[28] achieve theregret ˜ O ( √ d 2 H + dH√
HK + d 2 H+ d 3 H) and show a lower bound of ˜ ⌦ ( dH￿ √
K ) .
Theseproblem setups are incompatible to the setup of [27] and ours.
Problem Deﬁnition
Notations.
We denote d -dimensional ` 2 ball by B d(
R ) ∶ = { x ∈
R d ∶ ￿ x ￿ ≤
R } and deﬁne B d(
R ) similarly for the ` 1 ball.
Let [
N ] ∶ = { 1 , 2 ,....,N } for N ∈
N .
Given ` ∈
R and x ∈
R d , we deﬁne theclipping operator as follows (take￿ = 0 ):
[Equation]
Linear bandits.
The linear bandit problem has the following protocol.
At time step k , the learnerobserves an arm set
[Equation]
, chooses an arm x k ∈ X k , pulls it.
The learner then receives astochastic reward r k = x
￿ k ✓ ∗ + ✏ k where ✓ ∗ is an unknown parameter and ✏ k is a zero-mean stochasticnoise.
Following [27], we assume that ( i ) ∀ k ∈ [
K ] , ￿ r k ￿ ≤ 1 almost surely, ( ii )
E [ ✏ k ￿ F k ] = 0 where
F k =  ( x 1 , ✏ 1 ,...,x k − 1 , ✏ k − 1 ,x k ) , and ( iii )
E [ ✏ 2 k ￿ F k ] =
2 k .
Note that ￿ r k ￿ implies that ￿ ✏ k ￿ ≤ 1almost surely.
Our goal is to minimize the regret
R
K =
∑
Kk [Equation] max
[Equation]
✓ [Equation] x ￿ k ✓ ∗ .
Linear mixture MDPs.
We consider an episodic Markov Decision Process (MDP) with a tuple( S ,
A ,r ( s,a ) ,P ( s ′ ￿ s,a ) ,K,H ) where
S is the state space,
A is the action space, r ∶ S × A → [ 0 ,] is the reward function, P ( s ′ ￿ s,a ) is the transition probability, K is the number of episodes, and H is the planning horizon.
A policy is deﬁned as ⇡ = { ⇡ h ∶ S → D ( A )}
Hh = 1 where
D ( A ) is a set ofall distributions over
A .
For each episode k ∈ [
K ] , the learner chooses a policy ⇡ k , and then theenvironment executes ⇡ k on the MDP by successively following a kh ∼ ⇡ kh
( s kh ) and s kh + ∼
P ( ⋅ ￿ s kh ,a kh ) .
Then, the learner observes the rewards { r kh ∈ [ 0 ,]} k,h and moves onto the next episode.
Thekey modeling assumption of linear mixture MDPs is that the transition probability P is a linearcombination of a known set of models {
P i } , namely, P =
∑ di = 1 ✓ ∗ i P i where ✓ ∗ ∈
B d( ) is anunknown parameter.
We follow [27] and make the following assumptions:
• The reward at each time step h and episode k is r kh
kh
kh
for some known functionr
S × A → [ 0 ,
Unit cumulative rewards:
Hh =
kh
for any policy ⇡ k .
For a policy ⇡
, V ⇡ h ( s ) ∶ = max a ∈ A
Q ⇡ h ( s,a ) where Q ⇡ h ( s,a ) =
r ( s,a ) +
E s ′ ∼
P ( ⋅ ￿ s,a )
V ⇡ h + ( s ′ ) and
V ⇡ H + ( s ) ∶ = 0 .
Denoting V ⇡ ( s) =
V ⇡( s) and V ∗ ( s) =
V ⇡
[Equation]
( s) , our goal is to minimize theregret
R
[Equation]
Variance-Adaptive Linear Bandits
In this section, we show that VOFUL of Zhang et al.
[27] has a tighter regret bound than whatwas reported in their work.
Our version of VOFUL, which we call VOFUL2, has slightly differentconﬁdence set for ease of exposition.
Speciﬁcally, we use a conﬁdence set that works for everyµ ∈
B d( ) rather than over an ✏ -net of B d( ) (but we do use an ✏ -net for the proof of the conﬁdenceset).
The full pseudocode can be found in Algorithm 1.
VOFUL2 follows the standard optimism-based armselection [3, 6, 1].
Let ✏ s ( ✓
[Equation]
∶ =
[Equation]
and ✏
[Equation]
( ✓ ) ∶ = ( ✏ s
[Equation]
✓ )) 2 .
With L and ◆ deﬁned in Algorithm 1,we deﬁne our conﬁdence set after k time steps as
[Equation]
(3) where
[Equation]
￿￿￿￿￿￿￿ and the clipping operator ( z ) ` is deﬁned in (2).
The role of clipping is two-fold: (i) it allows usto factor out
∑ s = 1 ✏ 2 s ( ✓ ) by ∑ s ￿ x ￿ s µ ￿ ` ✏ 2 s ( ✓ ) ≤ ( − ` )
∑ s = 1 ✏ 2 s ( ✓ ) and (ii) the lower order term isreduced to the order of− ` .
Both properties are critical in obtaining variance-adaptive regret boundsas discussed in [27].
The true parameter is contained in our conﬁdence set with high probability asfollows.
Lemma 1.
(Conﬁdence set)
Let L , ◆ , and  be given as those in Algorithm 1.
Then,
[Equation]
In fact, in our algorithm, we use the conﬁdence set of
[Equation]
⇥ s at time step k for a technical reason.
VOFUL2 has the following regret bound.
Theorem 1.
VOFUL2 satisﬁes, with probability at least
[Equation] [Equation]
￿￿￿ where ˜ O hides poly-logarithmic dependence on {
[Equation]
Properties of the conﬁdence sets and implications on the regret.
Before presenting the proof of
Theorem 1, we provide some key properties of our conﬁdence set (Lemma 3) and the intuition behindour regret bound.
First, let us describe a few preliminaries.
Deﬁne
[Equation]
∧ − ` ￿ x ￿ s µ ￿ ￿ ￿ x s x ￿ s .
Algorithm 1 VOFUL
= ∨
K ) where ◆ = 128ln (( 12 K 2 L ) d + ￿  ) and  ≤ e − 1 .
Observe a decision set
X k ⊆
B d( ) .
Compute the optimistic arm as following: x k = arg max x ∈ X k max ✓ ∈ ∩ k − 1 s = 1 ⇥ s x ￿ ✓ where ⇥ s is deﬁned in
Let ✓ k be the maximizer of the optimization problem at line 4 of Algorithm 1 and deﬁne µ k = ✓ k − ✓ ∗ .
For brevity, we use a shorthand of
W ` ,k − ∶ =
W ` ,k − ( µ k ) =
− ` I + k − ￿ s = 1 ￿ ￿ ∧ − ` ￿ x ￿ s µ k ￿ ￿ ￿ x s x ￿ s .
Finally, we need to deﬁne the following event regarding the concentration of the empirical variancearound the true variance:
[Equation]
( 4 K ( log(
K ) + )  ) ￿￿￿￿￿￿￿ , which is true with high probability as follows.
Lemma 2.
We have P ( E ) ≥ −
Proof.
The proof is a direct consequence of Lemma 12 in our appendix.
Let ` k be the integer ` such that x ￿
[Equation] [Equation]
] and deﬁne A k ∶
=
∑ ks = 1  2 s .
Lemma 3 belowstates the properties of our conﬁdence set.
Lemma 3.
Suppose the events
E 1 and
E 2 are true.
Then, for any k with ` k = ` ,
(i)
For some absolute constant c 1 ,
[Equation]
(ii) There exists an absolute constant c 2 such that x k µ k
[Equation]
The key difference between Lemma 3 and the results of Zhang et al.
[27] is that we use the normnotations, although the norm involves a rather complicated matrix W ` ,k − 1 .
This opens up possibilitiesof analyzing the regret of VOFUL2 with existing tools such as applying Cauchy-Schwarz inequalityand the elliptical potential lemma [1, 5, 15].
In particular, Lemma 3(ii) seems useful because if wehad such a result with W ` ,k − 1 replaced by V k −
[Equation]
[Equation]
1 x s x ￿ s , then we would have, ignoring the 134 additive term ◆ ,
[Equation]
Together with the optimism and the standard elliptical potential lemma (see Section 3.1 for details),this leads to
[Equation]
k = x ￿ k µ k ≤ c
K ￿ k = ￿ x k ￿ 2 V − 1 k − 1 ￿￿￿￿ k − ￿ s =  2 s ◆ ≤ c⋅
O ( d log (
T ￿ d ))
⋅ ￿￿￿￿
K ￿ s =
2 s ◆ .
Since ◆ is linear in d , we would get the regret bounded by the order of d 1 .￿ ∑
K k = 1  2 k , roughlyspeaking.
However, the discrepancy between W ` ,k − 1 and V k − 1 is not trivial to resolve, especiallydue to the fact that Lemma 3(ii) has µ k on both left and the right hand side.
That is, µ k is the keyquantity that we need to understand, but we are bounding x k µ k as a function of µ k .
The novelty ofour analysis of regret is exactly at relating W ` ,k − 1 to V k − 1 via a novel peeling-based analysis, whichwe present below.
3.1 Proof of Theorem
Throughout the proof, we condition
E 1 and
E 2 where each one is true with probability at least−  ,as shown in Lemma 1 and 2 respectively.
For our regret analysis, it is critical to use Lemma 4below, which we call the elliptical ‘count’ lemma.
This lemma is a generalization of Lattimore and
Szepesvári [15, Exercise 19.3], which was originally used therein to improve the dependence of therange of the expected rewards in the regret bound.
We remark that a similar result appears in Russoand Van Roy [18, Proposition 3] as well.
Lemma 4.
(Elliptical potential count) Let x 1 ,...,x k ∈
R d be a sequence of vectors with ￿ x s ￿ ≤
Xfor all s ∈ [ k ] .
Let V k = ⌧
I + ∑ ks = 1 x s x ￿ s for some ⌧ > 0 .
Let J ⊆
[ k ] be the set of indices where￿ x s ￿ 2 V − 1 s − ≥ q .
Then,￿
J ￿ ≤ ln ( + q ) d ln ￿ + ￿ e ln ( + q )
X⌧ ￿ .
As the name explains, the lemma above bounds how many times ￿ x s ￿ 2 V − 1 s − 1 can go above a given valueq > 0 , which is different from existing elliptical potential lemmas that bound the sum of ￿ x s ￿ 2 V − 1 s − 1 .
Let ✓ k be the ✓ that maximizes the optimization problem at line 4 of Algorithm 1.
We start by theusual optimism-based bounds: due to
E 1 , we have
R
[Equation] − x [Equation]
[Equation] ≤ [Equation] ￿
[Equation]
( max
[Equation] [Equation]
= x
[Equation]
µ k .
We now take a peeling-based regret analysis that is quite different from existing analysis techniques:
R
K ≤
[Equation]
￿
[Equation]
[Equation]
− ✓ [Equation] + [Equation]
￿
[Equation]
￿
[Equation]
=
[Equation]
( ⋅ [Equation] ,⋅ − ` + ] ￿ , where L is deﬁned in Algorithm 1.
Given ` and k , let n k, ` be n such that max v ∶ k ≤ v ≤
K, ` v = ` ￿ x ￿ k µ v ￿ ∈ ( − ` + n ,− ` + n + ] if such n satisﬁes n ≥ 1 .
Otherwise, set n k, ` = 0 , which meansmax v ∶ k ≤ v ≤
K, ` v = ` ￿ x ￿ k µ v ￿ ≤ − ` + n + 1 with n = 0 .
We then deﬁne G ` ,n ∶ = { s ∈ [
K − ] ∶ ` s = ` ,n s, ` = n } and let G ` ,n [ k ] ∶ =
G ` ,n ∩ [ k ] .
Then,
K ￿ k = ￿ x ￿ k µ k ∈ ( ⋅ − ` ,⋅ − ` + ] ￿ =
K ￿ k = { ` k = ` } ≤ + ￿ n ￿ s ∈
G k,n 1 .
Letting V ` ,n,k − ∶ = − ` I + ∑ s ∈
G ` ,n [ k − ] x s x ￿ s , a comparison between two matrices W and V is givenas follows for every v ∈ { k,...,K } :
W ` ,k − ( µ v ) =
− ` I + k − ￿ s = 1 ￿ ￿ ∧ − ` ￿ x ￿ s µ v ￿ ￿ ￿ x s x ￿ s ￿ − ` I + ￿ s ∈
G ` ,n [ k − ] ￿ ￿ ∧ − ` − ` + n + 1 ￿ ￿ x s x ￿ s ￿ c ⋅ − n V ` ,n,k − 1 .
For k ∈
G ` ,n [
K − ] and u = arg max v ∶ k ≤ v ≤
K, ` v = ` ￿ x ￿ k µ v ￿ , we have − ` + n < ￿ x k µ u ￿ ≤ ￿ x k ￿
W − 1 ` ,k − ( µ u ) ￿ µ u ￿
W ` ,k − ( µ u ) ≤ ￿ x k ￿
W − 1 ` ,k − ( µ u ) ￿ µ u ￿
W ` ,u − ( µ u ) ( u ≥ k ) ≤ c √ 2 n ￿ x k ￿
V − 1 ` ,n,k − 1 ￿ 2 − ` ( ￿
A K ◆ + ◆ ) .
((4) and Lemma 3(i))
Consequently, ￿ x k ￿ 2 V − 1 ` ,n,k − ≥ c 2 − ` + n √
A K ◆ + ◆ .
Thus, using Lemma 4 with ⌧ = − ` , + ` ￿ n = ￿ k ∈
G k,n ≤ + ￿ n ￿ k ∈
G k,n ￿￿￿￿￿￿￿￿ x k ￿ 2 V − 1 ` ,n,k − ≥ c − ` + n √
A K ◆ + ◆ ￿￿￿￿￿￿￿ ≤ + c ￿ n 2 ` − n ( ￿
A K ◆ + ◆ )
d ln ￿ + c 4 ` ( ￿
A K ◆ + ◆ ) ￿ ≤ c 2 ` ( ￿
A K ◆ + ◆ )
d ln ￿ + c 4 ` ( ￿
A K ◆ + ◆ ) ￿ .
where we use the fact that￿ ln ( + q ) ≤ c ￿ q for an absolute constant c if q is bounded by an absoluteconstant.
Finally,
[Equation]
￿
A K ◆
[Equation]
◆
[Equation]
d ln ￿ + c 4 ` ( ￿
A K ◆ + ◆ ) ￿ =
[Equation]
◆
[Equation]
[Equation]
c
[Equation] A [Equation] ◆
[Equation]
◆
[Equation]
We choose L = ∨ ￿ log(
K ) ￿ , which leads to
R
K ≤ c ￿√
A K − 1 ◆ + ◆ ￿ d ln￿ + cK( √
A K − 1 ◆ + ◆ ) ￿ .
This concludes the proof.
Linear Mixture MDP
As linear bandits and linear mixture MDPs have quite a similar nature, we bring the techniques inour analysis of VOFUL2 to improve the regret bound of VARLin of Zhang et al.
A key featureof linear mixture MDP setting is that one can estimate the upper bound of the variance as it is aquadratic function of ✓ ∗ while linear bandits do not have a structural assumption on the variance.
Thanks to such a structural property, we obtain a slightly better dependence on the dimension d .
Theconﬁdence set derived for our proposed algorithm is slightly different from that of VARLin as ours isdeﬁned with
[Equation]
) rather than an ✏ -net.
Our version of VARLin, which we call VARLin2, isdescribed in Algorithm 2.
Given s kh and a kh , let us deﬁne P s kh ,a kh (
V kh + ) ∶ =
E s ′ ∼
P skh,akh [
V kh + ( s ′ )] andx mk,h ∶ =
P 1 s kh ,a kh (
V kh + ) 2 m ,...,P ds kh ,a kh (
V kh + ) 2 m ] ￿ and let L , ◆ , and  be given as deﬁne Algorithm 2.
Let ✏ mv,u ( ✓ ) ∶ = ✓ ￿ x mv,u − (
V vu + ( s vu + )) 2 m for ( v,u ) ∈ [
K ] ×
H ] , m ∈ { 0 , 1 ,...,L } where L is deﬁnedin Algorithm 2.
We construct our conﬁdence set as
[Equation]
(5) where we deﬁne ⇥ m,i, ` k below, based on the data collected up to episode k − 1 .
First, let⌘ mk,h ∶ = max ✓ ∈ ⇥ k
{ ✓ ￿ x m + 1 k,h − ( ✓ ￿ x mk,h ) } and
T m,ik,h ∶ = {( v,u ) ∈ ([ k ] × [
H ]) ∪ ({ k } ×
[ h ]) ∶ ⌘ mv,u ∈ ( − i , 2− i ])} .
We naturally deﬁne
T m,L + 1 k,h ∶ = {( v,u ) ∈ T m,ik,h ∶ ⌘ mv,u ≤ −
L } .
With ◆ deﬁned in Algorithm 2, deﬁne⇥ m [Equation] i, ` k −
[Equation]
We show that the conﬁdence set is correct w.h.p.
in the following lemma.
Lemma 5.
(Conﬁdence set for MDP) P
[Equation]
The consequence is that the Q values computed in VarLin2 is optimistic with high probability due tothe following property:
Lemma 6.
For every k
[Equation]
Now with the conﬁdence set deﬁned above we state our main result.
Theorem 2.
With probability at least 1-  ,
R
K
[Equation]
￿
[Equation]
where ˜ O hides poly-logarithmic dependence on { d,K,H, ln ( ￿  )} .
Algorithm
1: Initialize: L = ￿ log 2 HK ￿ + 1 , ◆ = 3ln (( 2 HK )
( d + ) ￿  ) ,  ≤ e − 1 .
2: for k = 1 , 2 ,...,K do 3: for h =
H,..., 1 do 4:
For each ( s,a ) ∈ S × A , deﬁne Q kh
( s,a ) =
min { 1 ,r ( s,a ) + max ✓ ∈ ⇥ k − ∑ di = 1 ✓ i P is,a V kh + } where ⇥ k − 1 is deﬁned in Lemma5:
For each state s , V kh ( s ) = max a ∈ A
Q kh ( s,a ) .
6: end for 7: for h = 1 ,...,H do 8:
Choose a kh = arg max a ∈
A
Q kh ( s kh ,a ) .
Observe a reward r kh and the next state s kh + 1 . 10: end for 11: end for
4.1 Proof of Theorem
The main idea of the proof is to infuse a peeling-based argument together with elliptical potentialcount lemma to both the planning horizon and episode.
Noting that the regret of the predictedvariance is controlled by the variance of variance, one can expect to reduce the total regret using thisinformation, as done in [27].
We begin by introducing relevant quantities that are parallel with thosein linear bandits.
Let us ﬁrst introduce the following lemma.
Lemma 7.
Let x 1 ,...,x T ∈
R d be a sequence of vectors with ￿ x t ￿ ≤
X for all t ∈ [
T ] .
Let
V t =  I + ∑ ts = 1 x s x ￿ s .
Let= ⌧< ⌧< ⌧< ...
< ⌧ z =
T where ⌧ i marks the last time step of the
i -thblock formed by { ⌧ i − + 1 ,..., ⌧ i } for all i ∈ { 0 ,...,z } .
Let anc ( t ) be the ‘anchor’ of t , the last timestep of the i -th block such that the ( i + ) -th block contains t : anc ( t )
=
max { ⌧ i ∶ i ∈ { 0 ,...,z } , ⌧ i < t } .
Let r > 0 .
Deﬁne J ⊆
T ] to be the set of indices t such that ￿ x t ￿ 2 V − 1 anc ( t ) >
r ￿ x t ￿ 2 V − 1 t − 1 is true for the ﬁrsttime in the block containing t .
Then,￿
J ￿ ≤ ln ( r ) d
ln ￿ + ￿ e ln ( r )
X ￿ .
To keep track of episode-horizon index pairs concisely, we use a ﬂat index t ∈ [
T ] where T ∶ =
HK .
Speciﬁcally, an episode k and a horizon h corresponds to the ﬂat index t =
( k − )
H + h .
Lett ( k,h ) ∶ =
( k − )
H + h .
Let k ( t ) and h ( t ) be the mapping from t to its corresponding episode andhorizon index respectively
so that k = k ( t ( k,h )) and h = h ( t ( k,h )) .
By taking ⌧ k in Lemma 7 ast ( k,H ) , we have that anc ( t ) ∶ = t ( t − 1 ,H ) .
We deﬁne
T m,it ∶ = { t ( k ′ ,h ′ )
∶ ( k ′ ,h ′ ) ∈
T m,ik,h } andµ mt ∶ = µ m k ( t ) , h ( t ) .
Similarly, we deﬁne x mt , etc., by replacing the subscript k,h with t .
Hereafter,any appearance of subscript k,h can be replaced with t such that t =
t ( k,h ) without changing themeaning.
Given m , k and h , we deﬁne ` mk,h as the integer ` such that ( x mk,h ) ￿ µ mk,h ∈ ( ⋅ − ` ,⋅ − ` + ] whereµ mk,h ∶ = ✓ mk,h − ✓ ∗ and ✓ mt = arg max ✓ ∈ ⇥ k − ￿ { ✓ ￿ x m + 1 t − ( ✓ ￿ x mt ) } ￿ .
For simplicity, we abbreviate` mk,h by ` .
Deﬁne W m,i, ` t ( µ ) ∶ = − ` I + ∑ s ∈ T m,it ￿ ∧ 2 − ` ￿ ( x ms ) ￿ µ ￿ ￿ x ms ( x ms ) ￿ and introduce a shorthand
W m,i, ` anc ( t ) ∶ =
W m,i, ` anc ( t ) ( µ mt ) as before.
With the deﬁnition above we have the following:
[Equation]
` ￿ µ mt ￿ + ￿ s ∈ T m,i anc ( t ) ￿ ( x ms ) ￿ µ mt ￿ ` ( x ms ) ￿ µ mt =
￿ µ mt ￿ 2 W m,i, ` anc ( t ) .
We now show the key result of the conﬁdence set of VarLin2 that parallels Lemma 3 for bandits.
Lemma 8.
Fix m ∈ { 0 ,...,L m } and i ∈ [
L ] .
Let t ∈ T m,iT .
Then, with ` = ` mt ,￿ µ m t ￿
[Equation]
W m,i,
[Equation]
anc ( t [Equation] ≤ c M ⋅ max { √ − i ◆ , √ −
[Equation]
◆ } where c M > 0 is an absolute constant.
What is different from the linear bandit problem is that we do not update ✓ until the planning horizonis over and an additional layer for peeling is imposed on variance.
In [27], the authors introduce anindicator I kh to characterize episode-horizon pairs for which growth of the norm of µ with respect to
W t is controlled by the norm with respect to W anc ( t ) with d 2 growth rate, i.e.,
I kh ∶ = { ￿ µ t ￿
W m,i, ` t ≤ ( d + ) ￿ µ t ￿
W m,i, ` anc ( t ) } .
where t = t ( k,h ) .
Our novelty novelty lies in being able to replace( d + ) 2 above by a constant rater that is set to 2 later (modulo some differences due to technical reasons).
To distinguish we denotesuch a set by I k,h .
See B.5 for the deﬁnition of I k,h and the proof of the following lemma.
Lemma 9.
[Equation]
for r > 0 .
Note here that once we ﬁx r such as r = 2 , the bound can be replaced by O ( d log( dHK )) .
We nowuse the following regret decomposition due to [27] which just come from replacing I kh by I k,h .
Lemma 10.
(Zhang et al.
R
[Equation] [Equation]
where Reg= ∑ k,h (
[Equation]
+
[Equation]
[Equation] + ))
[Equation]
k
[Equation]
h , Reg= ∑ k,h (
V kh ( s kh )
− r kh −
P s kh ,a kh V kh + )
I k,h , and Reg= ∑
Kk =
( ∑
Hh = 1 r kh −
V ⇡ k( s k)) .
Let ˘ x mk,h ∶ = x mk,h
I k,h and deﬁne R m , M m as
R m ∶ = ￿ k,h ( ˘ x mk,h ) ￿ µ mk,h , and M m ∶ =
￿ k,h (
P s kh ,a kh (
V kh + ) 2 m − (
V kh + ( s kh + )) 2 m )
I k,h .
We have that Reg=
M 0 and Reg≤
R 0 since
Q
[Equation]
( ✓ − ✓ ∗ ) .
To proceed, we ﬁrst note that ∑ k,h (
I k,h −
I k,h + ) and Reg 3 are bounded by O ( d log( dHK )) and
O ( ￿
K log ( ￿  )) respectively from Lemma 9 and Lemma 15.
Since Reg+
Reg≤
R+
M 0 , itremains to ﬁnd a bound on R+
M 0 .
This, however, involves solving a series of recursive inequalities.
We leave the details in the appendix and provide a high-level description below.
Let us begin with Lemma 14 in the appendix that shows￿
[Equation]
+ [Equation]
[Equation]
(7) where the RHS is a function of √
M m + 1 and √
R 0 .
Taking Proposition 1 below for granted andcombining it with the relation from Zhang et al.
[27, Eq.
(57)] showing￿ k
[Equation]
I k
[Equation]
one arrives at R m
[Equation]
This bound is the keyimprovement we obtain via our peeling-based regret analysis.
Speciﬁcally, the bound on R m obtainedby [27] has d 4 and d 6 in place of d￿ 2 and d above.
We ﬁrst show how our regret bound helps in obtaining the stated regret bound and then present
Proposition 1.
Noting that both R L and M L are trivially bounded by HK , one can solve the series ofinequalities on R m and ￿
M m ￿ to obtain a bound on R 0 :
[Equation]
Solving it for R 0 , we obtain R
[Equation]
￿  ) ￿ .
One can now plug in R 0 to the bound (7) and obtain a bound on ￿
M￿ in a similar way as follows, which concludes the proof: ￿
[Equation]
We now show the key proposition that allows us to improve the bound on R m .
In the paper by Zhanget al.
[27], d 4 was derived while we propose the following.
Proposition 1.
Let ˘ ⌘ mk,h
[Equation]
⌘ mk,h I k,h .
Then, we have
[Equation]
Proof.
Deﬁne
T m,i, `
[Equation]
∶
[Equation]
) ￿ µ mt ∈ ( ⋅ − ` ,⋅ 2− ` ]} and split the time steps
T m,i, ` by
T m,i, ` ,
￿ ￿ ∶ =
￿￿￿￿￿￿￿ t ∈ T m,i, ` ∶ ￿ µ mk,h ￿
W m,i, ` anc ( a ) ≤ c M √ − i ◆ ￿￿￿￿￿￿￿ and
T m,i, ` , ￿ ￿ ∶ = T m,i, ` ￿ T m,i, ` , ￿ ￿ .
Having deﬁned I k,h with r = 2 , we also denote ˘
T m,i, ` , ￿ z ￿ ∶ = T m,i, ` , ￿ z ￿ ∩ ￿ t ∈ [
T ] ∶
I t =
￿ .
Now wedecompose R m as
R m =
￿ t ∈
T ] ( ˘ x mt ) ￿ µ mt =
￿ i, ` ￿ t ∈ ˘
T m,i, ` , ￿ ￿ ( ˘ x mt )
￿ µ mt + ￿ i, ` ￿ t ∈ ˘
T m,i, ` , ￿ ￿ ( ˘ x mt ) ￿ µ mt .
Fix m , i and ` and focus on ∑ t ∈ ˘
T m,i, ` , ￿ z ￿ ( ˘ x mt )
￿ µ mt for z = 1 , 2 .
Hereafter, we omit the superscriptsand subscripts of ( m,i, ` ) to avoid clutter, unless there is a need.
Note that for t ∈ ˘
T ￿ ￿ ,n and b suchthat t < b ∈ T ￿ ￿ ,
W anc ( t ) ( µ b ) = − ` I + ￿ t ′ ∈ T m,i [Equation] ` anc ( t ) ( ∧ − ` ￿ x ￿ t ′ µ b ￿ ) x t ′ x ￿ t ′ ￿ − ` I + ￿ t ′ ∈ T ￿ ￿ ,n anc ( t ) ( ∧ − ` − ` + n + ) x t ′ x ￿ t ′ ￿ − ` I + − n − ￿ t ′ ∈ T ￿ ￿ ,n anc ( t ) x t ′ x ￿ t ′ ￿ c− n V ￿ ￿ ,n anc ( t ) .
For the same t , letting b = arg max t ≤ b ′ ∈ ˘
T ￿ ￿ ￿ x ￿ t µ b ′ ￿ , − ` + n ≤ ￿ x ￿
[Equation]
µ b ￿ ≤ ￿ x t ￿
W − 1 anc ( t ) ( µ b ) ￿ µ b ￿
W anc ( t ) ( µ b ) ≤ √ 2 n ￿ x t ￿ (
V ￿ 1 ￿ ,n anc ( t ) )
− ￿ µ b ￿
W anc ( b ) ( µ b ) ≤ c √ r √ 2 n ￿ x t ￿ (
V ￿ 1 ￿ ,n t − ) − 1 √ 2 − i ◆ (by b ∈ ˘
T ￿ ￿ )
This implies that ￿ x t ￿ (
V ￿ 1 ￿ ,n t − ) − ≥ c 2 − 2 ` + n r− i ◆ .
Thus,￿ t ∈ ˘
T ￿ ￿ x ￿ t µ t ≤ c− ` ￿ t ∈ ˘
T ￿ ￿ ≤ c− ` ￿￿ ˘
T ￿ ￿ ￿ ￿ t ∈ ˘
T ￿ ￿ ≤ c− ` ￿￿￿￿￿￿ ˘
T ￿ ￿ ￿ ` ￿ n = ￿ t ∈ ˘
T ￿ ￿ ,n ≤ c− ` ￿￿￿￿￿￿ ˘
T ￿ ￿ ￿ ` ￿ n = ￿ t ∈ ˘
T ￿ ￿ ,n ￿￿ x t ￿ (
V ￿ 1 ￿ ,n t − ) − ≥ c 2 − 2 ` + n r− i ◆ ￿ ≤ c− ` ￿￿￿￿￿ ˘
T ￿ ￿ ￿ ` ￿ n = ￿ t ∈ T ￿ ￿ ,n ￿￿ x t ￿ (
V ￿ 1 ￿ ,n t − ) − ≥ c 2 − 2 ` + n r− i ◆ ￿ ( ˘
T ￿ ￿ ,n ⊆ T ￿ ￿ ,n ) ≤ c− ` ￿￿￿￿￿ ˘
T ￿ ￿ ￿ ` ￿ n = r− i ◆ − 2 ` + n d ln ￿ + c r− i ◆ − 2 ` + n− ` ￿ ≤ c ￿ dr ￿ ˘
T ￿ ￿ ￿ − i ◆ ln ￿ + cr ◆ 8 ` ￿ ≤ c ￿ dr ( + ￿ t ∈ ˘
T ￿ ￿ ⌘ mt ) ◆ ln ￿ + cr ◆ 8 ` ￿ where we use the fact that ￿ T ￿ ￿ ￿ ⋅ − i ≤
O ( + ∑ t ∈ ˘
T ￿ ￿ ⌘ m t ) , which is straightforward by the deﬁnition.
The summation over ˘
T ( ) can be handled in a similar way and the details of the proof is provided in
Section B.7 in our appendix.
Conclusion
In this work, we have made signiﬁcant improvements in the regret upper bounds for linear banditsand linear mixture MDPs by employing a novel peeling-based regret analysis based on the ellipticalpotential count lemma.
Our study opens up numerous future research directions.
First, the optimalregret rates are still not identiﬁed for these problems.
It would be interesting to close the gap betweenthe upper and lower bound.
Second, our algorithms are not computationally tractable.
We believecomputationally tractable algorithms, even at the price of increased regret, may lead to practicalalgorithms.
Finally, characterizing variance-dependent uncertainty in the linear regression settingwithout prior knowledge of variances is an interesting statistical problem on its own.
Identifyingnovel estimators for it and proving their optimal coverage would be interesting.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] There is no
negative impacts, to our knowledge.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[N/A](b)
Did you specify all the training details (e g., data splits, hyperparameters, how they werechosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[N/A]
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?
[N/A]
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable [Table] information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review Board [Table]
(IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount spenton participant compensation?
[N/A]