Computationally Efficient Horizon-Free Reinforcement Learning for Linear Mixture MDPs
Anonymous Author(s)
AffiliationAddress
Abstract
Recent studies have shown that episodic reinforcement learning (RL) is not moredifficult than contextual bandits, even with a long planning horizon and unknownstate transitions.
However, these results are limited to either tabular Markovdecision processes (MDPs) or computationally inefficient algorithms for linearmixture MDPs.
In this paper, we propose the first computationally efficient horizonfree algorithm for linear mixture MDPs, which achieves the optimal (cid:101) O
( d √ K + d 2 )regret up to logarithmic factors.
Our algorithm adapts a weighted least squareestimator for the unknown transitional dynamic, where the weight is both varianceaware and uncertainty-aware .
When applying our weighted least square estimatorto heterogeneous linear bandits, we can obtain an (cid:101) O
( d (cid:113)(cid:80)
Kk =
σ 2 k + d ) regret inthe first K rounds, where d is the dimension of the context and σ 2 k is the variance ofthe reward in the k -th round.
This also improves upon the best known algorithmsin this setting when σ 2 k ’s are known.
Introduction
How to design efficient algorithms is a central problem for reinforcement learning (RL).
Here,the efficiency includes both statistical efficiency , which requires the RL algorithm enjoy a lowregret/polynomial sample complexity for finding the near-optimal policy and computational efficiency
which expects the RL algorithm have polynomial running time.
When restricting to episodic RL withtotal reward upper bounded by 1 1 , a longstanding question is whether episodic RL is statistically andcomputationally more difficult than contextual bandits (Jiang and Agarwal, 2018), since episodic
RL can be seen as an extension of contextual bandits to have a long planning horizon and unknownstate transition.
For tabular RL, this questions has been fully resolved by a line of works (Wang et al.
,2020; Zhang et al., 2021b;
Li et al., 2022; Zhang et al., 2022), which propose various horizon-freealgorithms.
Here we say an algorithm is horizon-free if its regret/sample complexity has at most apolylogarithmic dependence on the planning horizon H .
In particular, (Zhang et al., 2021b)
proposedthe first computationally efficient algorithm for tabular RL whose regret enjoys a polylogarithmicdependence on the planning horizon, and (Zhang et al., 2022) further removed the polylogarithmicdependence on the planning horizon.
For RL with function approximation to deal with large state space, Zhang et al.
(2021c)
; Kim et al.(2021) have made some progress towards horizon-free RL for a class of MDPs called linear mixture
MDPs (Jia et al., 2020; Ayoub et al., 2020; Zhou et al., 2021a), whose transition dynamic can berepresented as a linear combination of d basic transition models.
More specifically, Zhang et al.(2021c) proposed a VARLin algorithm for linear mixture MDPs with an (cid:101) O
( d 4 . 5 √ K + d 9 ) regret for the first K episodes, and Kim et al.
(2021) proposed a VARLin2 algorithm with an improved regret(cid:101)
O ( d √ K + d 2 ) .
However, neither algorithm is computationally efficient, because both of them needto work with nonconvex confidence sets and do not provide a polynomial-time algorithm to solve themaximization problem over these sets.
So the following question remains open:
Can we design
computationally efficient horizon-free RL
algorithms
when
function
approximation
is
employed?
In this paper, we answer the above question affirmatively for linear function approximation byproposing the first computationally efficient horizon-free RL algorithm for linear mixture MDPs.
Ourcontributions are summarized as follows.
• As a warm-up, we consider the heterogeneous linear bandits where the variances of rewards ineach round are different.
Such a setting can be regarded as a special case of linear mixture MDPs.
We propose a computationally efficient algorithm WeightedOFUL + and prove that in the firstK -rounds, the regret of WeightedOFUL + is (cid:101) O
( d (cid:113)(cid:80)
Kk =1 σ 2 k + dR + d ) , where σ 2 k is the varianceof the reward in the k -th round and R is the magnitude of the reward noise.
Our regret is varianceaware ,
i
e., it only depends on the summation of variances and does not have a √ K term.
Thisdirectly improves the (cid:101) O
( d (cid:113)(cid:80)
Kk =1 σ 2 k + √ dK + d ) regret achieved by Zhou et al.
(2021a).• For linear mixture MDPs, when the total reward for each episode is upper bounded by 1 , we proposea HF-UCRL-VTR + algorithm and show that it has an (cid:101)
O ( d √ K + d 2 ) regret for the first K episodes,where d is the number of basis transition dynamic.
Our HF-UCRL-VTR + is computationallyefficient , horizon-free and near-optimal , as it matches the regret lower bound proved in our paperup to logarithmic factors.
Our regret is strictly better than the regret attained in previous works
Zhou et al., 2021a; Zhang et al., 2021c; Kim et al., 2021).
• At the core of both WeightedOFUL + and HF-UCRL-VTR + is a carefully designed weighted linearregression estimator, whose weight is both variance-aware and uncertainty-aware , in contrast toprevious weighted linear regression estimator that is only variance-aware (Zhou et al., 2021a).
Forlinear mixture MDPs, we further propose a HOME that constructs the variance-uncertainty-awareweights for high-order moments of the value function, which is pivotal to obtain a horizon-freeregret for linear mixture MDPs.
For a better comparison between our results and previous results, we summarize these results in Table1.
It is evident that our results improve upon all previous results in the respective settings 2 .
Notation We use lower case letters to denote scalars, and use lower and upper case bold faceletters to denote vectors and matrices respectively.
We denote by [ n ] the set { 1 , . . . , n } , by [ n ] theset { 0 , . . .
, n − 1 } .
For a vector x ∈ R d and a positive semi-definite matrix Σ ∈ R d × d , we denoteby ∥ x ∥ 2 the vector’s Euclidean norm and define ∥ x ∥ Σ = √ x ⊤ Σx .
For x , y ∈ R d , let x ⊙ y
be the
Hadamard (componentwise) product of x and y .
For two positive sequences { a n } and { b n } withn = 1 , 2 , . . .
, we write a n = O ( b n ) if there exists an absolute constant C > 0 such that a n ≤ Cb nholds for all n ≥ 1 and write a n = Ω( b n ) if there exists an absolute constant C > 0 such thata n ≥ Cb n holds for all n ≥ 1 .
We use (cid:101) O ( · ) to further hide the polylogarithmic factors.
We use 72{·} to denote the indicator function.
For a, b ∈ R satisfying a ≤ b , we use [ x ] [ a,b ] to denote thetruncation function x · 1 { a ≤ x ≤ b } + a · 1 { x < a } + b · 1 { x > b } .
Related work
In this section, we will review prior works that are most relevant to ours.
Heterogeneous linear bandits.
Linear bandits have been studied for a long time.
Most of existingworks focus on the homogeneous linear bandits where the noise distributions at different round are
[Table]
[Equation]
[Table]
Yes
Lower
[Table]
[Equation]
- - identical (Auer, 2002;
Chu et al., 2011; Li et al., 2010; Dani et al., 2008; Abbasi-Yadkori et al.
,2011; Li et al., 2019a,b).
Recently, a series of works focus on the heterogeneous linear banditswhere the noise distribution changes over time.
Lattimore et al.
(2015) assumed that the noisedistribution is Bernoulli and proposed an algorithm with an (cid:101)
O ( d √ K ) regret.
Kirschner and Krause(2018) assumed that the noise at k -th round is σ 2 k -sub-Gaussian, and they proposed a weighted ridgeregression-based algorithm with an (cid:101) O
( d (cid:113)(cid:80)
Kk =1 σ 2 k ) regret.
A recent line of works assume thevariance of the noise at k -th round is bounded by σ 2 k .
Under this assumption, Zhang et al.
(2021c)proposed a VOFUL algorithm with an (cid:101) O
( d 4 . 5 (cid:113)(cid:80)
Kk =1 σ 2 k + d 5 ) regret.
Kim et al.
(2021) proposeda VOFUL2 algorithm with an (cid:101)
O ( d 1 .
5 (cid:113)(cid:80)
Kk =1 σ 2 k + d 2 ) regret.
Both VOFUL and VOFUL2 do notneed to know the variance information.
However, they are computationally inefficient since theyneed to work with nonconvex confidence sets defined by a series of second-order constraints, and donot propose a polynomial-time algorithm to solve the maximization problem over these sets.
Withthe variance information, Zhou et al.
(2021a) proposed a computationally efficient WeightedOFULwith an (cid:101) O
( d (cid:113)(cid:80)
Kk =1 σ 2 k + √ dK + d ) regret.
Our work is under the same assumptions as Zhou et al.(2021a) and improves the regret for linear bandits.
Horizon-free tabular RL.
RL is widely believed to be harder than contextual bandits problem due toits long planning horizon and the unknown state transitions.
For tabular RL, under the assumptionthat the total reward obtained by any policy is upper bounded by 1 , Jiang and Agarwal (2018)conjectured that any algorithm to find an ϵ -optimal policy needs to have a polynomial dependenceon the planning horizon H in the sample complexity.
Such a conjecture was firstly refuted by Wanget al.
by proposing a horizon-free algorithm with an (cid:101) O ( |S| 5 |A| 4 ϵ − 2 polylog ( H )) samplecomplexity that depends on H polylogarithmically, where ϵ is the target sub-optimality of the policy,
S is the state space and A is the action space.
Zhang et al.
(2021b) proposed a near-optimal algorithmwith an improved regret O (( (cid:112) |S||A| K + |S| 2 |A| ) polylog ( H )) and sample complexity.
Similarregret/sample complexity guarantees with a polylogarithmic H dependence have also been establishedunder different RL settings (Zhang et al., 2020;
Ren et al., 2021; Tarbouriech et al., 2021).
Recently
Li et al.
; Zhang et al.
(2022) further proposed algorithms with H -independent regret/samplecomplexity guarantees.
However, all the above works are limited to tabular RL.
Our work proposesan algorithm with a regret bound that depends on H polylogarithmically for linear mixture MDPs,which extends these horizon-free tabular RL algorithms.
RL with linear function approximation.
Recent years have witnessed a trend on RL with linearfunction approximation (e.g., Jiang et al., 2017; Dann et al., 2018; Yang and Wang, 2019a; Jin et al.
,2020; Wang et al., 2019; Du et al., 2019; Sun et al., 2019; Zanette et al., 2020a,b; Weisz et al., 2020;
Yang and Wang, 2019b; Modi et al., 2020; Jia et al., 2020; Ayoub et al., 2020; Zhou et al., 2021b).
Allthese works assume that the MDP enjoys some linear representation and propose different statisticaland computational complexities which depend on the dimension of the linear representation.
Amongthese assumptions, our work falls into the category of linear mixture MDP which assumes that thetransition dynamic can be represented as a linear combination of several basis transition probabilityfunctions (Yang and Wang, 2019b; Modi et al., 2020;
Jia et al., 2020; Ayoub et al., 2020;
Zhou et al.,2021b).
Previous algorithms for linear mixture MDPs either suffer from a polynomial dependenceon the episode horizon H (Yang and Wang, 2019b;
Modi et al., 2020; Jia et al., 2020; Ayoub et al.
Zhou et al., 2021b;
Cai et al., 2019; He et al., 2021;
Zhou et al., 2021a) or do not have acomputationally efficient implementation (Zhang et al., 2021c; Kim et al., 2021).
Our work achievesthe best of both worlds for the first time.
Preliminaries
[Figure]
3.1 Heterogeneous linear bandits
[Figure]
We consider the same heterogeneous linear bandits as studied in Zhou et al.
(2021a).
Let {D k } ∞ k =1be decision sets that are fixed.
At each round k , the agent selects an action a k ∈ D k satisfying∥ a k ∥ 2 ≤ A , then receives a reward r k provided by the environment.
Specifically, r k is generated byr k =
⟨ θ ∗ , a k ⟩ + ϵ k , where θ ∗ ∈ R
d is an unknown vector, and ϵ k is a random noise satisfying
[Equation]
where σ k is an upper bound of the variance of the noise ϵ k that are observable to the agent.
Weassume that σ k is ( a 1: k , ϵ 1: k − 1 ) -measurable.
The agent aims to minimize the pseudo-regret definedas follows:
[Equation]
3.2 Episodic reinforcement learning
We also study RL with linear function approximation for episodic linear mixture MDPs.
We introducethe necessary definitions of MDPs here.
The reader can refer to Puterman (2014) for more details.
Episodic MDP.
We denote a homogeneous, episodic MDP by a tuple M = M ( S , A , H, r, P ) , where
S is the state space and A is the action space, H is the length of the episode, r : S ×
A → [0 , 1] is thedeterministic reward function, and P is the transition probability function.
For the sake of simplicity,we restrict ourselves to countable state space and finite action space.
A policy π = { π h } Hh =1 is acollection of H functions, where each of them maps a state s to an action a .
Value function and regret.
For ( s, a ) ∈ S × A , we define the action-value function Q πh ( s, a ) and(state) value function V πh ( s ) as follows:
[Equation]
The optimal value function V ∗ h ( · ) and the optimal action-value function Q ∗ h ( · , · ) are defined by
V ∗ h ( s ) =
sup π V πh ( s ) and Q ∗ h ( s, a ) =
sup π Q πh ( s, a ) , respectively.
For any function V : S → R ,we introduce the following shorthands to denote the conditional variance of V at P ( ·| s, a ) :
[Equation]
where V 2 stands for the function whose value at s is V 2 ( s ) .
Using this notation, the Bellmanequations for policy π and the Bellman optimality equation can be written as
[Equation]
The goal is to minimize the K -episode regret defined as follows:
[Equation]
In this paper, we focus on proving high probability bounds on the regret Regret ( K ) .
In this work we make the following assumptions.
The first assumption assumes that for any policy,the accumulated reward of an episode is upper bounded by 1, which has been considered in previousworks (Krishnamurthy et al., 2016; Jiang and Agarwal, 2018).
The accumulated reward assumptionensures that the only factor that can affect the final statistical complexity is the planning difficultybrought by the episode length, rather than the scale of the reward.
Assumption 3.1 (Bounded total reward) .
For any policy π , let { s h , a h } Hh =1 be any states and actionssatisfying a h
= π h
( s h ) and s h +1 ∼ P ( ·| s h , a h ) .
Then we have 0 ≤ (cid:80)
Hh =1 r ( s h , a h ) ≤ 1 .
Next assumption assumes that the transition dynamic enjoys a linearized representation w.r.t.
somefeature mapping.
We define the linear mixture MDPs (Jia et al., 2020; Ayoub et al., 2020; Zhou et al.,2021b) as follows.
Assumption 3.2 (Linear mixture MDP) .
M is an episodic B -bounded linear mixture MDP, such thatthere exists a vector θ ∗ ∈ R d and ϕ ( ·|· , · ) such that P ( s ′ | s, a ) = ⟨ ϕ ( s ′ | s, a ) , θ ∗ ⟩ for any state-actionnext-state triplet ( s, a, s ′ )
∈ S ×
A × S .
Meanwhile, ∥ θ ∗ ∥ 2 ≤ B and for any bounded function
V : S → [0 , 1] and any tuple ( s, a ) ∈ S × A ,
we have∥
ϕ V ( s, a ) ∥ 2 ≤ 1 , where ϕ V ( s, a ) =
(cid:88) s ′ ∈S ϕ (
s ′ | s, a ) V ( s ′ ) .
Lastly, for any V : S → [0 , 1] , ϕ V can be calculated efficiently within O time.
Remark 3.3.
A key property of linear mixture MDP is that for any function V : S → R and anystate-action pair ( s, a ) , the conditional expectation of V over P ( ·| s, a ) is a linear function of θ ∗ , i.e.,[ P V ]( s, a ) =
⟨ ϕ V ( s, a ) ,
θ ∗ ⟩ .
Meanwhile, the conditional variance of V over P ( ·| s, a ) is a quadraticfunction of θ ∗ , i.e., [ V V ]( s, a ) = ⟨ ϕ V 2 ( s, a ) , θ ∗ ⟩ − [ ⟨ ϕ V ( s, a ) , θ ∗ ⟩ ] 2 .
Remark 3.4.
For a general class of ϕ , ϕ V ( s, a ) : S ×
A → R d can be computed efficientlyfor any ( s, a ) ∈ S
× A if V : S
→ R can be computed efficiently.
For instance, ϕ ( s ′ | s, a ) =
e s ′ ,s,a ∈ R |S| 2 |A| and ϕ ( s ′ | s, a ) =
ψ ( s ′ ) ⊙ ξ ( s, a ) , where ψ , ξ are two sub feature functions.
Morediscussions are referred to Zhou et al.
(2021a).
Computationally efficient variance-aware linear bandits
In this section, we propose our algorithm WeightedOFUL + in Algorithm 1 for the heterogeneouslinear bandits introduced in Section 3.1. WeightedOFUL + adopts the weighted ridge regressionestimator used by WeightedOFUL (Zhou et al., 2021a), but uses a refined weight.
It first computes aweighted estimate of θ ∗ , denoted by (cid:98) θ k , based on previous contexts and rewards, where the weights¯ σ k are computed by the noise variance σ k .
Then WeightedOFUL + constructs the confidence setof θ ∗ , denoted by (cid:98) C k , estimates the reward ⟨ a , θ ⟩ for θ ∈ (cid:98) C k , and selects the arm that maximizesthe estimated reward optimistically.
The selection rule of a k is identical to selecting the best armw.r.t.
to their upper confidence bound, i.e., a k ← argmax a ∈D k ⟨ a , (cid:98) θ k ⟩ + (cid:98) β k
∥ a ∥ (cid:98) Σ − 1 k (Li et al., 2010).
WeightedOFUL
+ constructs ¯ σ 2 k as the maximization of the variance, a constant, and the uncertainty∥ a k ∥ (cid:98) Σ − 1 k , as defined in (4.1).
Note that Zhou et al.
(2021a) proposed a variance-aware weight¯ σ k = max { σ k , α } for heterogeneous linear bandits.
The additional uncertainty term in our weightenables us to build a tighter confidence set (cid:98) C k since the uncertainty of arms can be leveraged by atighter Bernstein-type concentration inequality (See Section 4.1 for more details).
Meanwhile, wenotice that He et al.
(2022) proposed a pure uncertainty-aware weight ¯ σ k = max { α, γ ∥ a k ∥ 1 / 2 (cid:98) Σ − 1 k } to
Require:
Regularization parameter λ > 0 , and B , an upper bound on the ℓ 2 -norm of θ ∗ , confidence radius (cid:98) β k 1: (cid:98) Σ 1 ← λ
I , (cid:98) b 1 ← 0 , (cid:98) θ 1 ← 0 , (cid:98) β 1 = √ λB 2: for k = 1 , . .
, K do 3:
Let (
cid:98) C
k ← { θ : ∥ (cid:98) Σ
1 / 2 k ( θ − (cid:98) θ k ) ∥ 2 ≤ (cid:98) β k } , observe D k 4:
Set ( a k , θ k ) ← argmax a ∈D k , θ ∈ (cid:98) C k ⟨ a , θ ⟩ 5:
Observe ( r k , σ k ) , set ¯ σ k as ¯ σ k ← max { σ k , α, γ ∥ a k ∥ 1 / 2 (cid:98) Σ − 1 k } (4.1) 6:
(cid:98) Σ k +1 ← (cid:98) Σ k + a k a ⊤ k / ¯ σ 2 k , (cid:98)
b k +1 ← (cid:98)
b k + r k a k / ¯
σ 2 k , (cid:98) θ k +1 ← (cid:98) Σ − 1 k +1 (cid:98)
b k + 7: end for deal with the corruption in contextual linear bandits, which serves a different purpose compared toour setting (there is no corruption in our bandit model).
The following theorem gives the regret bound of WeightedOFUL + .
Theorem 4.1.
Let 0 < δ < 1 .
Suppose that for all k ≥ 1 and all a ∈ D k , ⟨ a , θ ∗ ⟩ ∈ [ − 1 , 1] ,∥ θ ∗ ∥ 2 ≤ B and { (cid:98) β k } k ≥ 1 are set to(cid:98) β k = 12 (cid:112) d log(1 + kA 2 / ( α 2 dλ )) log(32(log( γ 2 /α ) + 1) k 2 /δ ) + 30 log(
[Equation]
Then with probability at least 1 − δ , the regret of WeightedOFUL + is bounded by
[Equation]
where ι = log(1 + KA 2 / ( dλα 2 ))
Moreover, setting α = 1 / √ K, γ = R 1 / 2 /d 1 / 4 and λ =
d/B 2yields a high probability regret Regret ( K )
= (cid:101) O
( d (cid:113)(cid:80)
Kk =1 σ 2 k + dR + d ) .
Remark 4.2.
Treating R as a constant, the regret of WeightedOFUL + becomes (cid:101) O
( d (cid:113)(cid:80)
Kk =
σ 2 k + d ) .
It strictly outperforms the (cid:101) O
( d (cid:113)(cid:80)
Kk =1 σ 2 k + √ dK + d ) regret achieved in Zhou et al.
(2021a).
Compared with the (cid:101) O
( d 4 . 5 (cid:113)(cid:80)
Kk =1 σ 2 k + d 5 ) regret by VOFUL (Zhang et al., 2021c) and the(cid:101) O ( d 1 . 5 (cid:113)(cid:80)
Kk =1 σ 2 k + d 2 ) regret by VOFUL2 (Kim et al., 2021), the regret of WeightedOFUL +has a better dependence on d , and WeightedOFUL + is computationally efficient.
It is worth notingthat both VOFUL and VOFUL2 do not need to know the variance σ 2 k while our algorithm does.
Whether there exists an algorithm that can achieve the (cid:101) O ( d (cid:113)(cid:80)
Kk =1 σ 2 k + √ dK + d ) regret withoutknowing the variance information remains an open problem.
4.1 Proof sketch
We give a proof sketch of Theorem 4.1 along with two key lemmas that are pivotal to obtain theimproved regret.
To begin with, we first show that θ ∗ belongs to confidence balls centering at (cid:98) θ kwith radius (cid:98) β k .
This can be proved by the following lemma, which is an improved version of the
Bernstein-type self-normalized martingale inequality proposed by Zhou et al.
(2021a).
Lemma 4.3.
Let {G k } ∞ k
=1 be a filtration, and { x k , η k } k ≥ 1 be a stochastic process
such that x k ∈ R dis G k -measurable and η k ∈ R is G k +1 -measurable.
Let L, σ, λ, ϵ > 0 , µ ∗ ∈ R d .
For k ≥ 1 , lety k = ⟨ µ ∗ , x k ⟩ + η k and suppose that η k , x k also satisfy
[Equation]
For k ≥ 1 , let Z k
= λ I + (cid:80)
ki
=1 x i x ⊤ i , b k = (cid:80) ki
=
y i x i , µ k = Z − 1 k b k , andβ k = 12 (cid:112) σ 2 d log(1 + kL 2 / ( dλ ))
log(32(log
[Equation]
R/
[Equation]
[Equation] [Equation]
[Equation] [Equation]
) ϵ.
Then, for any 0 < δ < 1 , we have with probability at least 1 − δ that,
[Equation]
Note that (cid:98) θ k can be regarded as µ k − 1 in Lemma 4.3 with ϵ = R/γ 2 , x k = a k / ¯ σ k , y k
= r k / ¯ σ k ,η k = ϵ k / ¯ σ k ,
Z k = (cid:98) Σ
k +1 and µ ∗ =
θ ∗ .
With the help of the weight ¯ σ k , the variance of η k isupper bounded by 1 (since ¯ σ k ≥ σ k ) and | η k | min { 1 , ∥ x k ∥
Z − 1 k } ≤ |
ϵ k |∥ a k ∥ (cid:98) Σ − 1 k / ¯ σ 2 k ≤ R/γ 2 (since¯ σ 2 k ≥ γ 2 ∥ a k ∥ (cid:98) Σ − 1 k ).
Therefore, by Lemma 4.3, w.h.p.
θ ∗ ∈ (cid:98) C k .
Following the standard procedure tobound the regret of the optimistic algorithm (Abbasi-Yadkori et al., 2011), we have
Regret ( K ) =
K
[Equation]
Next lemma gives an upper bound of (4.5).
Lemma 4.4.
Let { σ k , (cid:98) β k } k ≥ 1 be a sequence of non-negative numbers, α, γ > 0 , { a k } k ≥ 1 ⊂ R dand ∥ a k ∥ 2 ≤ A .
Let { ¯ σ k } k ≥ 1 and { (cid:98) Σ k } k ≥ 1 be (recursively) defined as follows: (cid:98) Σ 1 = λ I ,
[Equation]
Let ι = log(1 + KA 2 / ( dλα 2 )) .
Then we have
K
[Equation]
By Lemma 4.4, the regret of WeightedOFUL + can be bounded by
Regret ( K ) =
(cid:101) O (cid:0) d + √ d
[Equation]
β K
[Equation]
with (cid:98) β K = (cid:101) O ( √ d + R/γ 2 + √ λB ) , which finishes the proof.
Here we compare the regret of WeightedOFUL + and the regret of WeightedOFUL (Zhou et al.,2021a)
(which chooses the weight ¯ σ k = max { σ k , α } ) to see where the improvement comes from.
In 225 particular, Zhou et al.
(2021a) proved the regret of WeightedOFUL as follows
[Equation]
with (cid:98) β α K
=
(cid:101) O ( √ d
+ R/α + √ λB ) .
At the first glance, both (4.6) and (4.7) have a √ Kα 2 term.
However, thanks to the design of ¯ σ k and Lemma 4.3, the √ Kα 2 term in (4.6) can be shaved bychoosing a small enough α , while this term in (4.7) cannot be shaved due to the existence of a R/αterm in (cid:98) β αK .
Computationally efficient horizon-free RL for linear mixture MDPs
In this section, we propose a horizon-free RL algorithm HF-UCRL-VTR + in Algorithm 2.
We leavethe discussion of the computational complexity of HF-UCRL-VTR + to Appendix A.
HF-UCRL-VTR
+ follows the value targeted regression (VTR) framework proposed by Jia et al.(2020); Ayoub et al.
(2020) to learn the linear mixture MDP.
In detail, following the observation in
Remark 3.3, VTR estimates θ ∗ by solving a regression problem over predictors/contexts ϕ
k,h, 0 =ϕ
V k,h +1 ( s kh , a kh ) and responses V k,h +1 ( s kh +1 ) .
Specifically, HF-UCRL-VTR
+ takes the estimate (cid:98) θ k, 0 237 as the solution to the following weighted regression problem:
[Equation]
Require: Regularization parameter λ , an upper bound B of the ℓ 2 -norm of θ ∗ , confidence radius { (cid:98) β k } k ≥ 1 , level M , variance parameters α, γ , [ M ] = { 0 , . . .
, M − 1 } 1:
For m ∈ [ M ] , set (cid:98) θ 1 ,m ← 0 , (cid:101) Σ 0 ,H +1 ,m
← λ I , (cid:101) b 0 ,H +1 ,m ← 0 .
Set V 1 ,H +1 ( · ) ←2: for k = 1 , . .
, K do 3: for h = H, . . .
, 1 do 4:
Set Q k,h ( · , · ) ← (cid:104) r ( · , · ) + (cid:10)(cid:98) θ k, 0 , ϕ V k,h +1 ( · , · ) (cid:11) + (cid:98) β k
(cid:13)(cid:13)(cid:13)(cid:98)
Σ − 1 / 2 k, 0 ϕ V k,h +1 ( · , · ) (cid:13)(cid:13)(cid:13) 2 (cid:105) [0 , 1] 5:
Set π kh ( · ) ← argmax a ∈A Q k,h ( · , a ) 6:
Set V k,h ( · ) ← max a ∈A Q k,h ( · , a ) 7: end for 8:
Receive s k 1 .
For m ∈ [ M ] , set (cid:101) Σ k, 1 ,m ← (cid:101)
Σ
k − 1 ,H +1 ,m 9: for h = 1 , . .
, H do 10:
Take action a kh ← π kh ( s kh ) , receive s kh +1 ∼ P ( ·| s kh , a kh ) .
For m ∈ [ M ] , denote ϕ k,h,m =
ϕ V 2 mk,h +1 ( s kh , a kh ) .
Set { ¯ σ k,h,m } m ∈
[ M ] ← Algorithm 3( { ϕ k,h,m , (cid:98) θ k,m , (cid:101) Σ k,h,m , (cid:98) Σ k,m } m ∈
[ M ] , (cid:98) β k , α, γ ) 13:
For m ∈ [ M ] , set (cid:101) Σ k,h +1 ,m ← (cid:101) Σ k,h,m + ϕ k,h,m ϕ ⊤ k,h,m / ¯ σ 2 k,h,m 14:
For m ∈ [ M ] , set (cid:101) b k,h +1 ,m ← (cid:101) b k,h,m + ϕ k,h,m V 2 m k,h +1 ( s kh +1 ) / ¯ σ 2 k,h,m 15: end for 16:
For m ∈ [ M ] , set (cid:98) Σ
k +1 ,m ← (cid:101) Σ k,H +1 ,m , (cid:98)
b k +1 ,m ← (cid:101) b k,H +1 ,m , (cid:98) θ k +1 ,m ← (cid:98) Σ − 1 k
+1 ,m (cid:98)
b k +1 ,m 17: end for where ¯ σ j,h, 0 is the upper bound of the conditional variance [ V V k,h +1 ]( s kh , a kh ) .
Such a weightedregression scheme has been adapted by UCRL-VTR + in Zhou et al.
(2021a).
With (cid:98) θ k, 0 ,
HF-UCRL-VTR
+ then constructs the optimistic estimates Q k,h (resp.
V k,h ) of the optimal valuefunctions Q ∗ h (resp.
V ∗ h ) and takes actions optimistically.
Note that (cid:98) θ k, 0 is updated at the end ofeach episode.
We highlight several improved algorithm designs of HF-UCRL-VTR + compared to
UCRL-VTR
+ as follows.
Improved weighted linear regression estimator
[Equation]
,h, 0 similar to theweight used in WeightedOFUL + .
Assuming that the conditional variance [ V V k,
h +1 ]( s kh , a kh ) can becomputed for any value function V and state action pair ( s, a ) , then the weight can be set as
[Equation]
where (cid:101) Σ k,h, 0 is the weighted sample covariance matrix of ϕ k,h,
up to k -th episode and h -th stage.
However, the true variance is not accessible since P ( ·| s, a ) is unknown.
Therefore, HF-UCRL-VTR +replaces [ V V k,
h +1 ]( s kh , a kh ) with its estimate [¯ V k, 0 V k,h +1 ]( s kh , a kh ) and an error bound E k,h, 0 satisfying [¯ V k, 0 V k,h
+1 ]( s k h , a k h )+ E k,h, 0 ≥
[ V V k,h +1 ]( s k h , a k h ) with high probability.
Thanks to the fact that[ V V k,h +1 ]( s kh , a kh ) is a quadratic function of θ ∗ as illustrated in Remark 3.3, [¯ V k, 0 V k,h +1 ]( s kh , a kh )can be estimated as follows:
[Equation]
where (cid:98) θ k, 1 is the solution to some regression problem over predictors/contexts ϕ
k,h, 1 =ϕ V 2 k,h +1 ( s kh , a kh ) and responses V 2 k,
h +1 ( s kh +1 ) .
Higher-order moment regression.
To obtain a better estimate, it is natural to set (cid:98) θ k, 1 as the solutionto the weighted regression problem on ϕ k,h, 1 and V 2 k,
h +1 ( s kh +1 ) with weight ¯ σ k,h, 1 .
Here ¯ σ k,h, 1 isconstructed in a similar way to ¯ σ k,h, 0 , which relies on the conditional variance of [ V V 2 k,h +1 ]
( s k h , a k h ) .
By repeating this process, we recursively estimate the conditional 2 m -th moment of V k,h +1 by itsvariance, which is the conditional 2 m +1 -th moment of V k,h +1 .
It is worth noting that the idea ofhigh-order recursive estimation has been used in Li et al.
(2020) and later in Zhang et al.
(2021b,c) toachieve horizon-free regret/sample complexity guarantees.
Similar recursive analysis also appearedin Lattimore and Hutter (2012).
Require:
Features { ϕ k,h,m } m ∈
[ M ] , vector estimators { (cid:98) θ k,m } m ∈
[ M ] , covariance matrix { (cid:101) Σ k,h,m , (cid:98) Σ k,m } m ∈
[ M ] , confidence radius (cid:98) β k , α, γ 1: for m = 0 , . . .
, M − 2 do 2:
Set [¯ V k,m V 2 m k,h +1 ]( s kh , a kh ) ← (cid:2)(cid:10) ϕ
k,h,m +1 , (cid:98) θ k,m +1 (cid:11)(cid:3) [0 , 1] − (cid:2)(cid:10) ϕ
k,h,m , (cid:98) θ k,m (cid:11)(cid:3) 2 [0 , 1] 3:
Set E k,h,m ← min (cid:8) 1 , 2 (cid:98) β k (cid:13)(cid:13)(cid:98) Σ − 1 / 2 k,m ϕ k,h,m (cid:13)(cid:13) 2 (cid:9) + min (cid:8) 1 , (cid:98) β
k
(cid:13)(cid:13)(cid:98) Σ − 1 / 2 k,m +1 ϕ k,h,m +1 (cid:13)(cid:13) 2 (cid:9) 4:
Set ¯ σ 2 k,h,m ← max (cid:8) [¯ V k,m V 2 m k,h +1 ]( s kh , a kh )
+ E k,h,m , α 2 , γ 2 (cid:13)(cid:13)(cid:101) Σ − 1 / 2 k,h,m ϕ k,h,m (cid:13)(cid:13) 2 (cid:9) 5: end for 6: Set ¯ σ 2 k,h,M − 1 ← max (cid:8) 1 , α 2 , γ 2 (cid:13)(cid:13)(cid:13)(cid:101)
[Equation]
Ensure: { ¯ σ k,h,m } m ∈
[ M ]
The estimated conditional moment [¯ V k,m V 2 m k,h +1 ]( s kh , a kh ) relies on (cid:98) θ k,m +1 and (cid:98) θ k,m , and⟨ ϕ k,h,m +1 , (cid:98) θ k,m +1 ⟩ serves as the estimate of the higher-moment [ V V 2 m
+1 k,h +1 ]( s kh , a kh ) .
The detailedconstructions for the high-order moment estimator are summarized in Algorithm 3.
We provide the regret bound for HF-UCRL-VTR + here.
Theorem 5.1.
Set M = log(3 KH ) / log 2 .
For any δ > 0 , set { (cid:98) β k } k ≥ 1 as
[Equation]
then with probability at least 1 − (2 M + 1) δ , the regret of Algorithm 2 is bounded by
[Equation]
where ι = log(1 + KH/ ( dλα 2 )) , ζ = 4 log(4 log( KH ) /δ ) .
Moreover, setting α = 270 (cid:112) d/ ( KH ) , γ = 1 /d 1 / 4 and λ = d/B 2 yields a high-probability regret Regret ( K )
= (cid:101) O
( d √ K + d 2 ) .
Remark 5.2.
The regret of HF-UCRL-VTR + is strictly better than that of VOFUL (cid:101)
O
( d 4 . 5 √ K + d 9 )
Zhang et al.
(2021c),
and it matches the regret of VOFUL2 (Kim et al., 2021).
More importantly,
HF-UCRL-VTR
+ is computationally efficient, while there is no efficient implementation of VOFUL/VOFUL2.
Next theorem provides the regret lower bound and suggests that the regret obtained by
HF-UCRL-VTR
+ is near-optimal.
The lower bound is proved by constructing hard-instances oflinear mixture MDPs following Zhou et al.
(2021b,a); Zhang et al.
(2021a).
Theorem 5.3.
Let B > 1 .
Then for any algorithm, when K ≥ max { 3 d 2 , ( d − 1) / (192( B − 1)) } ,there exists a B -bounded linear mixture MDP satisfying Assumptions 3.1 and 3.2 such that itsexpected regret E [ Regret ( K )] is lower bounded by d √ K/ (16 √ 3) .
Remark 5.4.
When specialized to tabular MDPs where d = |S| 2 |A| , HF-UCRL-VTR + yields ahorizon-free regret (cid:101) O ( |S| 2 |A|√ K + |S| 4 |A| 2 ) .
Although the regret does not match the near-optimalresult (cid:101) O ( (cid:112) |S||A| K + |S| 2 |A| ) (Zhang et al., 2021b), it is not surprising since HF-UCRL-VTR + isdesigned for a more general MDP class.
We leave the design of algorithms that achieve near-optimalregret for both linear mixture MDPs and tabular MDPs simultaneously as a future work.
Conclusion
In this work, we propose a new weighted linear regression estimator that adapts variance-uncertaintyaware weights, which can be applied to both heterogeneous linear bandits and linear mixture MDPs.
For heterogeneous linear bandits, our WeightedOFUL + algorithm achieves an (cid:101) O
( d (cid:113)(cid:80)
Kk =
σ 2 k + d )regret in the first K rounds.
For linear mixture MDPs, our HF-UCRL-VTR + algorithm achieves thenear-optimal (cid:101) O
( d √ K + d 2 ) regret.
Both of our algorithms are computationally efficient and yieldthe state-of-the-arts regret results.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] Our paper proposes a computational efficient and horizon-free RL algorithm for linear mixture MDPs.
(b) Did you describe the limitations of your work?
[Yes] See Remark 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumption 3.1 and 3.2.(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix C andD.3.
If you ran experiments...
(a) Did you include the code, data and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]