N EAR - OPTIMAL O FFLINE R EINFORCEMENT L EARNING WITH L INEAR R EPRESENTATION : L EVERAGING V ARIANCE I NFORMATION WITH P ESSIMISM
Anonymous authors
Paper under double-blind review
A BSTRACT
Ofﬂine reinforcement learning , which seeks to utilize ofﬂine/historical data to optimize sequential decision-making strategies, has gained surging prominence in recent studies.
Due to the advantage that appropriate function approximators can help mitigate the sample complexity burden in modern reinforcement learning problems, existing endeavors usually enforce powerful function representation models ( e.g. neural networks) to learn the optimal policies.
However, a precise understanding of the statistical limits with function representations, remains elusive, even when such a representation is linear.
Towards this goal, we study the statistical limits of ofﬂine reinforcement learning with linear model representations.
To derive the tight ofﬂine learning bound, we design the variance-aware pessimistic value iteration (VAPVI), which adopts the conditional variance information of the value function for time-inhomogeneous episodic linear Markov decision processes (MDPs).
VAPVI uses the estimated variances of the value functions to reweight the Bellman residuals in the least-square pessimistic value iteration and provides improved ofﬂine learning bounds over the existing best-known results (whereas the Bellman residuals are equally weighted).
More importantly, our learning bounds are expressed in terms of system quantities, which provide natural instance-dependent characterizations that previous results are short of.
We hope our results draw a clearer picture of what ofﬂine learning should look like when linear representations are provided.
I NTRODUCTION
Ofﬂine reinforcement learning (ofﬂine RL or batch RL Lange et al.
; Levine et al.
(2020)) is the framework for learning a reward-maximizing policy in an unknown environment ( Markov
Decision Process or MDP) 1 using the logged data coming from some behavior policy µ .
Function approximations, on the other hand, are well-known for generalization in standard supervised learning.
Ofﬂine RL with function representation/approximation, as a result, provides generalization across large state-action spaces for the challenging sequential decision-making problems when no iteration is allowed (as opposed to online learning).
This paradigm is crucial to the success of modern RL problems as many deep RL algorithms ﬁnd their prototypes in the literature of ofﬂine RL.
For example, Xie and Jiang (2020) provides a view that Fitted Q-Iteration Gordon (1999); Ernst et al.
(2005) can be considered as the theoretical prototype of the deep Q -networks algorithm (DQN) Mnih et al.
(2015) with neural networks being the function representors.
On the empirical side, there are a huge body of deep RL-based algorithms Mnih et al.
; Silver et al.
; Fujimoto et al.
Kumar et al.
; Wu et al.
; Kidambi et al.
Yu et al.
; Kumar et al.
Janner et al.
; Chen et al.
(2021a) that utilize function approximations to achieve respective successes in the ofﬂine regime.
However, it is also realized that practical function approximation schemes can be quite sample inefﬁcient ( e.g. millions of samples are needed for deep Q -network to solve certain Atari games Mnih et al. (2015)).
To understand this phenomenon, there are numerous studies consider how to achieve sample efﬁciency with function approximation from the theoretical side, as researchers ﬁnd sample efﬁcient algorithms are possible with particular model representations, in either online RL ( e.g. Yang and Wang (2019; 2020);
Modi et al.
(2020); Jin et al.
(2020); Ayoub et al.
; Jiang et al.
; Du et al.
Sun et al.
; Zanette et al.
; Zhou et al.
; Jin et al.
(2021a);
Du et al.
or ofﬂine RL ( e.g. Munos (2003); Chen and Jiang (2019); Xie and Jiang (2020); Jin et al.
(2021b)
; Xie et al. (2021a); Duan et al.
Nguyen-Tang et al.
; Zanette et al.
Among them, the linear MDP model (Yang and Wang, 2020; Jin et al., 2020), where the transition is represented as a linear combinations of the given d -dimensional feature, is (arguably) the most studied setting in function approximation and there are plenty of extensions based upon it ( e.g. generalized linear model (Wang et al., 2021b), reward-free RL (Wang et al., 2020), gap-dependent analysis (He et al., 2021) or generative adversarial learning (Liu et al., 2021)).
Given its prosperity, however, there are still unknowns for understanding function representations in RL, especially in the ofﬂine case.
• While there are surging researches in showing provable sample efﬁciency (polynomial sample complexity is possible) under a variety of function approximation schemes, how to improve the sample efﬁciency for a given class of function representations remains understudied.
For instance, given a neural network approximation class, an algorithm that learns the optimal policy with complexity O
( H 10 ) is far worse than the one that can learn in O ( H 3 ) sample complexity, despite that both algorithms are considered sample efﬁcient.
Therefore, how to achieve the “optimal”/tight sample complexity when function approximation is provided is a valuable question to consider.
On the other hand, it is known that tight sample complexity, due to the limit of the existing statistical analysis tools, can be very tough to obtain when function representation has a very complicated form.
However, does this mean tight analysis is not hopeful even when the representation is linear?
Second, in the existing analysis of ofﬂine RL (with function approximation or simply the tabular MDPs), the learning bounds depend either explicitly on the data-coverage quantities ( e
g. uniform concentrability coefﬁcients Chen and Jiang (2019)
; Xie and Jiang (2020), uniform visitation measure Yin et al.
; Yin and Wang (2021a) and single concentrability Rashidinejad et al.
; Xie et al. (2021b)) or the horizon length H (Jin et al., 2021b; Uehara and Sun, 2021).
While those results are valuable as they do not depend on the structure of the particular problem (therefore, remain valid even for pathological MDPs), in practice, the empirical performances of ofﬂine reinforcement learning are often far better than those non-adaptive bounds would indicate.
Can the learning bounds reﬂect the nature of individual MDP instances when the MDP model has a certain function representation?
In this work, we think about ofﬂine RL from the above two aspects.
In particular, we consider the fundamental linear model representations and ask the following question of interest:
Can we achieve the statistical limits for ofﬂine RL when models have linear representations?
1. R ELATED THEORETICAL WORK
Ofﬂine RL with general function representations.
The ﬁnite sample analysis of ofﬂine RL with function approximation is initially conducted by Fitted Q -Iteration (FQI) type algorithms and can be dated back to (Munos, 2003; Szepesv´ari and Munos, 2005; Antos et al., 2008a;b).
Later, Chen and Jiang (2019); Le et al.
(2019); Xie and Jiang (2020) follow this line of research and derive the improved learning results.
However, owing to the aim for tackling general function approximation, those learning bounds are expressed in terms of the stringent concentrability coefﬁcients (therefore, are less adaptive to individual instances) and are usually only information-theoretical , due to the computational intractability of the optimization procedure over the general function classes.
Other works impose weaker assumptions ( e.g. partial coverage (Liu et al., 2020; Kidambi et al., 2020;
Uehara and Sun, 2021)), and their ﬁnite sample analysis are generally suboptimal in terms of H or the effective horizon (1 − γ ) − 1 .
Ofﬂine RL with tabular models.
For tabular MDPs, tight learning bounds can be achieved under several data-coverage assumptions.
For the class of problems with uniform data-visitation measure d m , the near-optimal sample complexity bound has the rate
O
( H 3 /d m (cid:15) 2 ) for time-inhomogeneous
MDPs (Yin et al., 2021) and O ( H 2 /d m (cid:15) ) for time-homogeneous MDPs (Yin and Wang, 2021a; Ren et al., 2021).
Under the single concentrability assumption, the tight rate O ( H 3 SC (cid:63) /(cid:15) 2 ) is obtained by Xie et al.
(2021b).
In particular, the recent study Yin and Wang (2021b) introduces the intrinsic ofﬂine learning bound that is not only instance-dependent but also subsumes previous optimal results.
Ofﬂine RL with linear model representations.
Recently, there is more focus on studying the provable efﬁcient ofﬂine RL under the linear model representations.
Jin et al.
(2021b)
ﬁrst shows ofﬂine RL with linear MDP is provably efﬁcient by the pessimistic value iteration .
Their analysis deviates from their lower bound by a factor of d · H (check their Theorem 4.4 and 4.6).
Later, Xie et al.
(2021a) considers function approximation under the Bellman-consistent assumptions, and, when realized to linear MDP setting, improve the sample complexity guarantee of Jin et al.
(2021b)
by an order O ( d ) (Theorem 3.2).
However, their improvement only holds for ﬁnite action space (due to the dependence log |A| ) and by the direct reduction (from Theorem 3.1) their result does not imply a computationally tractable algorithm.
Concurrently, Zanette et al.
(2021) considers the Linear
Bellman Complete model and designs the actor-critic style algorithm that achieves tight result under the assumption that the value function is bounded by 1 .
While their algorithm is efﬁcient (which is based on solving a sequence of second-order cone programs), the resulting learning bound requires the action space to be ﬁnite due to the mirror descent updates in the Actor procedure (Agarwal et al., 2021).
Besides, assuming the value function to be less than 1 simpliﬁes the challenges in dealing with horizon H since when rescaling their result to [0 , H ] , there is a H factor blow-up, which makes no horizon improvement comparing to Jin et al.
(2021b).
On the other hand, Wang et al.
(2021a);
Zanette (2021) study the statistical hardness of ofﬂine RL with linear representations by prooﬁng the exponential lower bounds.
As a result, none of the existing algorithms can achieve the statistical limit for the well-structured linear MDP model with the general (inﬁnite or continuous) state-action spaces.
Variance-aware studies.
Talebi and Maillard (2018)
ﬁrst incorporate the variance structure in online tabular MDPs and Zanette and Brunskill (2019) tighten the result.
For linear MDPs, Zhou et al.
ﬁrst uses variance structure to achieve the near-optimal result and the Weighted OFUL incorporates the variance structure explicitly in the regret bound.
Recently, Variance-awareness is also considered in Zhang et al.
(2021) for horizon-free setting and for OPE problem (Min et al., 2021).
1. O UR CONTRIBUTION
In this work, we study ofﬂine RL for time-inhomogeneous episodic linear Markov decision processes.
Linear MDPs serve as one critical step towards understanding function approximation in RL since: 1.
unlike general function representation, linear MDP representation has the well-structured form by the given feature representors, which makes delicate statistical analysis hopeful; 2.
unlike tabular representation, which only works for ﬁnite models, linear MDP provides generalization as it adapts to inﬁnite or continuous state-action spaces.
Especially, we design the variance-aware pessimistic value iteration (VAPVI, Algorithm 1) which incorporates the conditional variance information of the value function and, by the variance structure, Theorem 3.2 is able to improve over the aforementioned stateof-the-art guarantees.
In addition, we further improve the state-action guarantee by designing an even tighter bonus (3).
VAPVI-Improved (Theorem 3.3) is near-minimax optimal as indicated by our lower bound (Theorem 3.5).
Importantly, the resulting learning bounds from VAPVI/VAPVI-
Improved are able to characterize the adaptive nature of individual instances and yield different convergence rates for different problems.
Algorithmically, we use the estimated variances to reweight the Bellman residual learning objective so that the (training) samples with high uncertainty get less attention (Section 3).
This is the key to obtaining instance-adaptive guarantees.
P RELIMINARIES 2. P ROBLEM SETTINGS
Episodic time-inhomogeneous linear Markov decision process.
A ﬁnite-horizon Markov Decision
Process (MDP) is denoted as M = ( S , A , P, r, H, d 1 ) (Sutton and Barto, 2018), where S is the arbitrary state space and A is the arbitrary action space which can be inﬁnite or even continuous.
A time-inhomogeneous transition kernel P h : S × A (cid:55)→ ∆ S ( ∆ S represents a probability simplex) maps each state action ( s h , a h ) to a probability distribution P h ( ·| s h , a h ) and P h can be different across time.
In addition, r : S × A (cid:55)→ R is the mean reward function satisfying 0 ≤ r ≤ 1 .
dis the initial state distribution.
H is the horizon.
A policy π = ( π 1 , . . .
, π H ) assigns each state s h ∈ S a probability distribution over actions according to the map s h
(cid:55)→ π h ( ·| s h ) ∀ h ∈
[ H ] and induces a random trajectory s 1 , a 1 , r 1 , . . .
, s H , a H , r H , s H +1 with s 1 ∼ d 1 , a h ∼ π ( ·| s h ) , s h +1 ∼
P h ( ·| s h , a h ) , ∀ h ∈
[ H ] .
In particular, we adopts the linear MDP protocol from Jin et al.
(2020; 2021b), meaning that the transition kernel and the mean reward function admit linear structures in the feature map.
Deﬁnition 2.1 (Linear MDPs) .
An episodic MDP ( S , A , H, P, r ) is called a linear MDP with a known (unsigned) feature map φ : S ×
A → R d if there exist d unknown (unsigned) measures ν h
= ( ν (1) h , . .
, ν ( d ) h ) over S and an unknown vector θ h ∈ R d such that
P
[Equation]
where (cid:107) ν
h ( S ) (cid:107) 2 ≤ √ d and max( (cid:107) φ ( s, a ) (cid:107) 2 , (cid:107) θ h (cid:107) 2 ) ≤ 1 for all h ∈
[ H ] and ∀ s, a ∈ S × A .
(cid:107) µ h ( S ) (cid:107) =
(cid:82) S (cid:107) µ h ( s ) (cid:107) ds .
V -values and Q -values.
For any policy π , the V -value functions V πh ( · ) ∈ R S and Q-value functions
Q πh ( · , · ) ∈ R S
×
A are deﬁned as: V πh ( s ) =
E π
[ (cid:80)
Ht
= h r t | s h
= s ] , Q πh ( s, a ) =
E π [ (cid:80)
Ht
= h r t | s h , a h =
s, a ] , ∀ s, a, h ∈ S , A , [ H ] .
The performance measure is deﬁned as v π
[Equation]
(cid:104)(cid:80)
[Equation]
1 r
[Equation]
(cid:105) .
The Bellman (optimality) equations follow ∀ h ∈
[ H ] : Q πh = r h
+ P h V πh +1 ,
V πh =
E a ∼ π h [ Q πh ] , Q (cid:63)h = r h
+ P h V (cid:63)h +1 , V (cid:63)h = max a Q (cid:63)h ( · , a ) (where Q h , V h , P h are vectors).
By Deﬁnition 2.1, the Q -values also admit linear structures, i.e. Q πh = (cid:104) φ, w πh (cid:105) for some w πh ∈ R d (Lemma H.9).
Lastly, for a policy π , we denote the induced occupancy measure over the state-action space at any time h ∈
[ H ] to be: for any E ⊆ S
× A , d πh ( E ) := E [( s h , a h ) ∈
E | s 1 ∼ d 1 , a i ∼ π ( ·| s i ) , s i ∼ P i − 1 ( ·| s i − 1 , a i − 1 ) , 1 ≤ i ≤ h ] and E π,h
[ f ( s, a )] :
= (cid:82) S×A
f ( s, a ) d πh ( s, a ) dsda .
Here for notation simplicity we abuse d πh ( · ) to denote either probability measure or density function.
Ofﬂine learning setting.
The ofﬂine RL requires the agent to learn the policy π that maximizes v π , provided with the historical data D
= { ( s τh , a τh , r τh , s τh +1 ) } h ∈
[ H ] τ ∈
[ K ] rolled out from some behavior policy µ .
The ofﬂine nature requires we cannot change µ and in particular we do not know the data generating distribution of µ .
To sum up, the agent seeks to ﬁnd a policy π alg such that v (cid:63) − v π alg ≤ (cid:15) for the given batch data D and a given targeted accuracy (cid:15) > 0 .
2. A SSUMPTIONS
It is known that learning a near-optimal policy from the ofﬂine data D cannot be sample efﬁcient without certain data-coverage assumptions (Wang et al., 2021a;
Yin and Wang, 2021b).
To begin with, we deﬁne the population covariance matrix under the behavior policy µ for all h ∈
[ H ] :
[Equation]
(1) since Σ ph measure the coverage of state-action space for data D , we make the following assumption.
Assumption 2.2 (Feature Coverage) .
The data distributions µ satisfy the minimum eigenvalue condition: ∀ h ∈
[ H ] , κ h :=
λ min (Σ ph ) > 0 and denote κ = min h κ h .
Note κ is a system-dependent (non-universal) quantity as it is upper bounded by 1 /d (Assumption 2 in Wang et al.
(2021a)).
We make this assumption for the following reasons.
First of all, our ofﬂine learning guarantee (Theorem 3.2) provides simultaneously comparison to all the policies, which is stronger than competing with optimal policy only (whereas relaxed assumption sufﬁces, e.g. sup x ∈ R d x Σ π(cid:63)
x (cid:62) x Σ µ x (cid:62) < ∞ (Uehara and Sun, 2021)).
As a consequence, the behavior distribution µ must be able to explore each feature dimension for the result to be valid.
Second, even if Assumption 2.2 does not hold, we can always restrict our algorithmic design to the effective subspan of Σ ph , which causes the alternative notion of κ :=
min h ∈ [ H ] { κ h : s.t. κ h
= smallest positive eigenvalue at time h } (see Appendix G.1 for detailed discussions).
In this scenario, learning the optimal policy cannot be guaranteed as a constant suboptimality gap needs to be suffered due to the lack of coverage and this is called assumption-free
RL in Yin and Wang (2021b).
Lastly, previous works analyzing the linear MDPs impose very similar assumptions, e.g. Xie et al.
(2021a)
Theorem 3.2 where Σ − 1 D exists and Min et al.
(2021) for the OPE problem.
Next, for any function V h +1 ( · ) ∈ [0 , H − h ] , we deﬁne the conditional variance σ V
h
+1 :
S ×A → R + as σ V
h
+
( s, a ) 2 :=
max { 1 , Var P h
( V h +1 )
( s, a ) } .
Based on this deﬁnition, we can deﬁne the variance-involved population covariance matrices as:
[Equation]
In particular, when V h = V (cid:63)h , we use the notation Λ (cid:63)ph instead.
Since sup ( s,a ) ∈S×A σ V h
( s, a ) 2 ≤ H 2 , then by Assumption 2.2 we directly have the following corollary.
Corollary 2.3.
Deﬁne ι h :=
λ min (Λ ph ) , ι := min h ι h .
Then ι h ≥ κ h H 2 > 0 ∀ h ∈
[ H ] , and ι > 0 .
A LGORITHM
Least square regression is usually considered as one of “the default tools” for solving with problems with linear structures ( e.g. LinUCB algorithm for linear Bandits) and ﬁnds its popularity in RL as well since Least-Square Value Iteration (LSVI, Jin et al.
(2020)) is shown to be provably efﬁcient for linear MDPs, due to that V
h +1 ( s (cid:48) ) is an unbiased estimator of [ P h V h +1 ]
( s, a ) .
Concretely, it solves the ridge regression problems at each time steps (with λ > 0 being the regularization parameter):
[Equation]
and has the closed-form solution (cid:98) w h
= Σ − 1 h (cid:80)
Kk =1 φ ( s kh , a kh )
[ r k,h
+ V h +
( s (cid:48) kh )] with Σ − 1 h
= (cid:80)
Kk =1 φ ( s kh , a kh ) φ ( s kh , a kh ) (cid:62) + λI .
In ofﬂine RL, this has also been used in pessimistic value iteration (Jin et al., 2021b) and ﬁtted Q-evaluation (Duan et al., 2020).
Nevertheless, LSVI could only yield suboptimal guarantees, as illustrated by the following example.
Example 3.1.
Instantiate PEVI (Theorem 4.4 in Jin et al.
(2021b))
with φ ( s, a ) = 1 s,a ( i.e. tabular
MDPs) 5 , by direct calculation the learning bound has the form O ( dH · (cid:80) h,s,a d π (cid:63) h ( s, a ) (cid:113) 1 K · d µh ( s,a ) ) and the optimal result (Yin and Wang (2021b)
Theorem 4.1) gives O ( (cid:80) h,s,a d π (cid:63) h ( s, a ) (cid:114)
Var Ps,a ( r + V (cid:63)h +1 ) K · d µh ( s,a ) ) .
The former has the horizon dependence H 2 and the latter is H 3 / 2 by law of total variance.
Motivation.
By comparing the above two expressions, it can be seen that PEVI cannot get rid of the explicit H factor due to missing the variance information ( w.r.t V (cid:63) ).
If we go deeper, one could ﬁnd that it might not be all that ideal to put equal weights on all the training samples in the least square objective (2), since, unlike linear regression where the randomness coming from one source distribution, we are regressing over a sequence of distributions in RL ( i.e. each s h , a h corresponds to a different distribution P ( ·| s h , a h ) and there are possibly inﬁnite many of them).
Therefore, conceptually, the sample piece ( s h , a h , s h +1 )
that has higher variance distribution P ( ·| s h , a h ) tends to be less “reliable” than the one ( s (cid:48) h , a (cid:48) h , s (cid:48)
h +1 ) with lower variance (hence should not have equal weight in (2)).
This suggests reweighting scheme might help improve the learning guarantee and reweighting over the variance of the value function stands as a natural choice.
3. V ARIANCE -A WARE P ESSIMISTIC V ALUE I TERATION
Now we explain our framework that incorporates the variance information.
By the ofﬂine nature, we can use the independent episodic data D (cid:48)
= { (¯ s τh , ¯ a τh , ¯ r τh , ¯ s τ (cid:48) h ) } h ∈
[ H ] τ ∈
[ K ] (from µ ) to estimate the conditional variance of any V -values V h +1 via the deﬁnition [Var h V h
+1 ]
( s, a ) =
[ P h ( V h +1 ) 2 ]( s, a ) − ([ P h V h +1 ]( s, a )) 2 .
For the second order moment, by Deﬁnition 2.1, it holds (cid:2) P
[Equation]
Denote β h :=
(cid:82) S V 2 h +
( s (cid:48) )
d ν h ( s (cid:48) ) , then P h V 2 h +
= (cid:104) φ, β h (cid:105)
and we can estimator it via: ¯ β h = argmin β ∈ R d
K (cid:88) k
= (cid:2)(cid:10) φ (¯ s kh , ¯ a kh ) , β (cid:11) − V
2 h
+1 (cid:0) ¯ s kh +
(cid:1)(cid:3)
+ λ (cid:107) β (cid:107) 22 = ¯Σ − 1 h
K (cid:88) k
= φ (¯ s kh ,
¯ a kh ) V 2 h +1 (cid:0)
¯ s kh +1 (cid:1) and, similarly, the ﬁrst order moment P h V h
+1 :
= (cid:104) φ, θ h (cid:105) can be estimated via: ¯ θ h = argmin θ ∈ R d
K (cid:88) k
= (cid:2)(cid:10) φ (¯ s kh , ¯ a kh ) , θ (cid:11)
− V h +1 (cid:0)
¯ s kh +
(cid:1)(cid:3)
+ λ (cid:107) θ (cid:107) 22 = ¯Σ − 1 h
K (cid:88) k
= φ (¯ s kh , ¯
a kh ) V h
+1 (cid:0)
¯ s kh +1 (cid:1)
The ﬁnal estimator is deﬁned as (cid:98) σ 2 V h ( · , · ) :
= max { 1 , (cid:100)
Var
h
V h +
( · , · ) } with (cid:100)
Var
h
V h +
( · , · ) = (cid:104) φ ( · , · ) , ¯ β h (cid:105)
[0 , ( H − h +1) 2 ] − (cid:2) (cid:104) φ ( · , · ) , ¯ θ h (cid:105) [0 ,H − h +1] (cid:3) 2 .
In particular, when setting V h +1 = (cid:98) V h
+1 , it recovers (cid:98) σ h in Algorithm 1 line 8.
Here ¯Σ h
= (cid:80)
Kτ =1 φ (¯ s τh , ¯ a τh ) φ (¯ s τh , ¯ a τh ) (cid:62) + λI d .
Variance-weighted LSVI.
The idea of LSVI (2) is based on approximate the Bellman updates: T h ( V )( s, a ) =
r h ( s, a ) +
( P h V )( s, a ) .
With variance estimator (cid:98) σ h at hand, we can modify (2) to solve the variance-weighted LSVI instead (Line 10 of Algorithm 1)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
kh
[Equation]
kh
[Equation]
where (cid:98) Λ h
= (cid:80)
Kk =1 φ ( s kh , a kh ) φ ( s kh , a kh ) (cid:62) / (cid:98) σ 2 h ( s kh , a kh ) + λI d .
The estimated Bellman update (cid:98) T h (acts on (cid:98) V h +1 ) is deﬁned as: ( (cid:98) T h (cid:98) V h +1 )( · , · ) =
φ ( · , · ) (cid:62) (cid:98) w h and the pessimism Γ h is assigned to update (cid:98) Q h ≈ (cid:98) T h (cid:98) V h
+1 − Γ h , i.e. Bellman update + Pessimism
(Line 12-13 in Algorithm 1).
Tighter Pessimistic Design.
To improve the learning guarantee, we create a tighter penalty design that includes (cid:98) Λ − 1 h rather than ¯Σ − 1 h and an extra higher order O ( 1 K ) term:
[Equation]
Note such a design admits no explicit factor in H in the main term (as opposed to Jin et al.
(2021b))
therefore it is the key for achieving adaptive/problem-dependent results (as we shall discuss later).
The full algorithm VAPVI is stated in Algorithm 1.
In particular, we halve the ofﬂine data into two independent parts with D = { ( s τh , a τh , r τh , s τ (cid:48) h ) } h ∈
[ H ] τ ∈
[ K ] and D (cid:48) =
{ (¯ s τh , ¯ a τh , ¯ r τh , ¯ s τ (cid:48) h ) } h ∈
[ H ] τ ∈
[ K ] for different purposes (estimating variance and updating Q -values).
3. M AIN RESULT
We denote quantities M 1 , M 2 , M 3 , M 4 as in the notation list A. Then VAPVI provides the following result.
The complete proof is provided in Appendix C.
Theorem 3.2.
Let K be the number of episodes.
If K > max {M 1 , M 2 , M 3 , M 4 } and √ d > ξ , where ξ := sup V ∈ [0 ,H ] , s (cid:48) ∼ P
h
( s,a ) ,
h ∈
[ H ] (cid:12)(cid:12)(cid:12)(cid:12) r h
+ V ( s (cid:48) ) − ( T h V )
( s,a ) σ V ( s,a ) (cid:12)(cid:12)(cid:12)(cid:12) .
Then for any 0 < λ < κ , with probability 1 − δ , for all policy π simultaneously, the output (cid:98) π of Algorithm 1 satisﬁes
[Equation]
K where Λ h = (cid:80)
Kk =1 φ ( s kh ,a kh ) · φ ( s kh ,a kh ) (cid:62) σ(cid:98)
Vh +1( skh,akh ) + λI d .
In particular, we have with probability 1 − δ , v
[Equation]
φ
[Equation]
) (cid:21)(cid:1) +
[Equation]
√ d
K where Λ (cid:63)h = (cid:80)
Kk =1 φ ( s kh ,a kh ) · φ ( s kh ,a kh ) (cid:62) σ
V(cid:63)h +1( skh,akh ) + λI d and (cid:101)
O hides universal constants and the Polylog terms.
valid range.
[Table]
Theorem 3.2 provides several improvements over the existing best-known results and we now explain it.
However, before that, we ﬁrst discuss about our theorem condition.
Comparing to Zhou et al.
In the online regime, Zhou et al.
(2021) is the ﬁrst result that achieves optimal regret rate with O ( dH √ T ) in the linear (mixture) MDPs.
However, this tight rate is achieved based on the condition d ≥ H (see their Theorem 6 and Remark 7).
In ofﬂine RL, VAPVI only requires a milder condition √ d > ξ comparing to d ≥ H (since for any ﬁxed V ∈ [0 , H ] , the standardized quantity r + V ( s (cid:48) )
− ( T h V )( s,a ) σ V ( s,a ) is bounded by constant with high probability, e.g. by chebyshev inequality), which makes our result apply to a wider range of linear MDPs.
Comparing to Jin et al.
(2021b).
Jin et al.
(2021b)
ﬁrst shows pessimistic value iteration (PEVI) is provably efﬁcient for Linear MDPs in ofﬂine RL.
VAPVI improves PEVI over O ( √ d ) on the feature dimension, and improves the horizon dependence as Λ h (cid:60) 1 H
Σ h implies Λ − 1 h (cid:52)
H 2 Σ − 1 h .
In addition, when instantiate to the tabular case, i.e. φ ( s, a ) = 1 s,a , VAPVI gives
O ( √ d (cid:80)
[Equation] [Equation]
) , which enjoys O ( √ H ) improvement over PEVI (recall
Example 3.1) and the order O
( H 3 / 2 ) is tight (check Section G for the detailed derivation).
Comparing to Xie et al.
(2021a).
Their linear MDP guarantee in Theorem 3.2. enjoys the same rate as VAPVI in feature dimension but the horizon dependence is essentially the same as Jin et al.
(2021b)
(by translating H ≈ O
11 − γ ) ) therefore is not optimal.
Their general function approximation scheme provides nice characterizations for on-support error and off-support error, but the algorithmic framework is information-theoretical only (and the practical version PSPI will not yield the same learning guarantee).
Also, due to the use ﬁnite function class and policy class, the reduction to linear
MDP only works with ﬁnite action space.
As a comparison, VAPVI has no constraints on any of these.
Comparing to Zanette et al.
Concurrently, Zanette et al.
(2021) considers ofﬂine RL with the linear Bellman complete model, which is more general than linear MDPs and with the assumption
Q π ≤ 1 , and their PACLE algorithm provides near-minimax optimal guarantee in this setting.
However, when recovering to the standard setting Q π ∈ [0 , H ] , their bound will rescale by an H factor, 7 which is suboptimal due to the variance-unawareness.
The reason behind this is: when
Q π ≤ 1 , lack of variance information encoding will not matter, since in this case Var P ( V π ) ≤ 1 has constant order (therefore will not affect the optimal rate); when Q π ∈ [0 , H ] , Var P ( V π ) can be as large as H 2 , effectively leveraging the variance information can help improve the sample efﬁciency, e.g. via law of total variances , just like VAPVI does.
On the other hand, their guarantee also requires ﬁnite action space, due to the mirror descent style analysis.
Nevertheless, we do point out Zanette et al.
(2021) has improved state-action measure than VAPVI, as (cid:107) E π [ φ ( · , · )] (cid:107) M − 1 ≤
E π
[ (cid:107) φ ( · , · ) (cid:107) M − 1 ] by
Jensen’s inequality and that norm (cid:107)·(cid:107)
M − 1 is convex for some positive-deﬁnite matrix M .
Adaptive characterization and faster convergence.
Comparing to existing works, one major improvement is that the main term for VAPVI √
[Equation]
φ
[Equation]
[Equation] [Equation]
admits no explicit dependence on H , which provides a more adaptive/instance-dependent characterization.
For instance, if we ignore the technical treatment by taking λ = 0 and σ (cid:63)h ≈ Var P ( V (cid:63)h +1 ) , then for the partially deterministic systems (where there are t stochastic P h ’s and H − t deterministic P h ’s), the main term diminishes to √ d (cid:80) ti =1 E π (cid:63) (cid:2)(cid:113) φ ( · , · ) (cid:62) Λ (cid:63) − 1 h
i φ ( · , · ) (cid:3) with h i ∈
h : s.t.
P h is stochastic } and can be a much smaller quantity when t (cid:28) H .
Furthermore, for the fully deterministic system ,
VAPVI automatically provides faster convergence rate O ( 1 K ) from the higher order term, given that the main term degenerates to 0 .
Those adaptive/instance-dependent features are not enjoyed by (Xie et al., 2021a; Zanette et al., 2021), as they always provide the standard statistical rate
O ( 1 √ K )
(also check Remark C.9 for a related discussion).
3. VAPVI-I MPROVED : F URTHER IMPROVEMENT IN STATE - ACTION DIMENSION
Can we further improve the VAPVI?
Indeed, by deploying a carefully tuned tighter penalty, we are able to further improve the state-action dependence if the feature is non-negative ( φ ≥ 0 ).
Concretely, we replace the following Γ Ih in Algorithm 1 instead, and call the algorithm VAPVI-Improved (or VAPVI-I for short).
The proof can be found in Appendix D.
[Equation]
Theorem 3.3.
Let K be the number of episodes.
If K > max {M 1 , M 2 , M 3 , M 4 } and √ d > ξ .
Deploying Γ Ih (3) in Algorithm 1.
Then for any 0 < λ < κ , with probability 1 − δ , for all policy π simultaneously, the output (cid:98) π of Algorithm 1 (VAPVI-I) satisﬁes
[Equation]
In particular, when choosing π = π (cid:63) , the above guarantee holds true with Λ − 1 h replaced by Λ (cid:63) − 1 h .
Here Λ − 1 h , Λ (cid:63) − 1 h , ξ are deﬁned the same as Theorem 3.2.
Theorem 3.3 maintains nearly all the features of Theorem 3.2 (except higher order term is slightly worse) and the dominate term evolves from E π (cid:107) φ (cid:107) Λ − 1 h to (cid:107) E π [ φ ] (cid:107) Λ − 1 h .
Clearly, the two bounds differ by the magnitude of Jensen’s inequality.
To provide a concrete view of how much improvement is made, we check the parameter dependence in the context of tabular MDPs (where we ignore the higher order term for conciseness).
In particular, we consider the following assumption for our discussion.
Assumption 3.4 (Rashidinejad et al.
; Xie et al. (2021b)) .
There exists a optimal policy π (cid:63) , s.t. sup h,s,a d π (cid:63) h ( s, a ) /d µh
( s, a ) :
= C (cid:63) < ∞ , where d π is the marginal state-action probability under π .
In tabular RL, φ ( s, a ) = 1 s,a and d = S · A ( S, A be the ﬁnite state, action cardinality), then
Theorem 3 .
→√ SA H (cid:88)
[Equation]
Theorem 3 . 3 → √ SA H (cid:88)
[Equation]
Theorem 3.3 enjoys a S state improvement over Theorem 3.2 and nearly recovers the minimax rate (cid:113)
H 3 C (cid:63) S K (Xie et al., 2021b).
The detailed derivation can be found in Appendix G.
Also, to show our result is near-optimal, we provide the corresponding lower bound.
The proof is in Appendix E.
Theorem 3.5 (Minimax lower bound) .
There exist a pair of universal constants c, c (cid:48) > 0 such that given dimension d , horizon H and sample size K > c (cid:48)
d 3 , one can always ﬁnd a family of linear MDP instances M such that (where Λ
[Equation]
Theorem 3.5 nearly matches the main term in VAPVI-I (Theorem 3.3) and certiﬁes it is near-optimal.
P ROOF O VERVIEW
Due to the space constraint, we could only provide a brief overview of the key proving ideas of the theorems.
We begin with Theorem 3.2.
First, by the extended value difference lemma (Lemma H.7), we can convert bounding the suboptimality gap of v (cid:63) − v (cid:98) π to bounding (cid:80)
Hh =1 2 · E π [Γ h ( s h , a h )] , given that | ( T h (cid:98) V h
+1 − (cid:98) T h (cid:98)
V h +1 )
( s, a ) | ≤ Γ h ( s, a ) for all s, a, h .
To bound T h (cid:98) V h
+1 − (cid:98) T h
[Equation]
V h +1 , by decomposing it reduces to bounding the key quantity
[Equation]
The term is treated in two steps.
First, we bound the gap of (cid:13)(cid:13)(cid:13) σ 2 (cid:98) V h
+1 − (cid:98) σ 2 h (cid:13)(cid:13)(cid:13) so we can convert (cid:98) σ 2 h to σ 2 (cid:98)
V h
+1 .
Next, since Var (cid:104) r τh + (cid:98) V
h
+1 (cid:0) s τh +1 (cid:1) − (cid:16) T h (cid:98) V
[Equation]
+1 (cid:17) ( s τh , a τh ) | s τh , a τh (cid:105)
≈ σ 2 (cid:98) V h
+1 , therefore by the variance-weighted scheme in (equation 6), we can leverage the recent technical development
Bernstein inequality for self-normalized martingale (Lemma H.3) for acquiring the tight result, in contrast to the previous treatment of Hoeffding inequality for self-normalized martingale + Covering.
For the second part, one needs to further convert σ 2 (cid:98) V h
+1 to σ (cid:63) 2 h
( Λ − 1 h to Λ (cid:63) − 1 h ) with appropriate concentrations.
The proof of Theorem 3.3 is similar but with more complicated computations and relies on using the linear representation of φ in Γ Ih (3), so that the expectation over π is inside the square root by taking expectation over the linear representation at the beginning.
The lower bound proof uses a simple modiﬁcation of Zanette et al.
(2021) which consists of the reduction from learning to testing with Assouad’s method, and the use of standard information inequalities ( e.g. from total variation to KL divergence).
For completeness, we provide the full proof in Appendix E.
D ISCUSSION AND C ONCLUSION
This work studies ofﬂine RL with linear MDP representation and contributes Variance Aware
Pessimistic Value Iteration (VAPVI) which adopts the conditional variance information of the value function.
VAPVI uses the estimated variances to reweight the Bellman residuals in the least-square pessimistic value iteration and provides improved ofﬂine learning bounds over the existing best-known results.
VAPVI-I further improves over VAPVI in the state-action dimension and is near-minimax optimal.
One highlight of the theorems is that our learning bounds are expressed in terms of system quantities, which automatically provide natural instance-dependent characterizations that previous results are short of.
On the other hand, while VAPVI/VAPVI-I close the existing gap from previous literature (Jin et al., 2021b;
Xie et al., 2021a), the optimal guarantee is in the minimax sense.
Although our upper bounds possess instance-dependent characterizations, the lower bound only holds true for a class of hard instances.
In this sense, whether “instance-dependent optimality” can be achieved remains elusive in the current linear MDP setting (such a discussion is recently initiated in MAB problems (Xiao et al., 2021)).
We leave this as future work.
Ethics Statement
This draft does not have any ethics inﬂuence since it’s a theoretical paper that studies ofﬂine RL with linear representation.
Appendix A N OTATION L IST
Σ ph
E
[Table]
¯Σ h
[Table]
Mmax {
[Table] [Table]
[Table]
Mmax {
[Table] [Table]
[Table] [Table]
Mmax
[Table] [Table]
[Table] [Table]
M12 (cid:112)
[Table]
d
[Table]
(( λ + K ) H/λδ ) /
[Table]
B E XTENDED L ITERATURE R EVIEW B. L INEAR MODEL REPRESENTATION AND ITS EXTENSION IN ONLINE RL
There are numerous works in online RL that study linear model representations.
Yang and Wang (2019; 2020); Jin et al.
(2020) propose Linear MDP, which assumes the transition kernel and the reward are linear in given features.
Cai et al.
(2020); Ayoub et al.
(2020) propose Linear mixture
MDP, which assumes the transition probability is a linear combination of some base kernels.
Linear
Bellman Complete model (Zanette et al., 2020) generalizes linear MDP model by allowing linear functions to approximate the Q -function and the function class is closed under the Bellman update.
The notion of low Bellman rank (Jiang et al., 2017) subsumes not only linear MDPs but also models including linear quadratic regulator (LQR), Reactive POMDP (Krishnamurthy et al., 2016) and Block
MDP (Du et al., 2019).
There are also other models, e.g. factored MDP (Sun et al., 2019), Bellman
Eluder dimension (Jin et al., 2021a) and Bilinear class (Du et al., 2021).
With the linear MDP model itself, there are also fruitful extensions, e.g. gap-dependent analysis with logarithmic regret (He et al., 2021), low-switching cost RL (Gao et al., 2021), safe RL (Amani et al., 2021), reward-free RL (Wang et al., 2020), generalized linear model (GLM) (Wang et al., 2021b), two-player markov game
(Chen et al., 2021b) and generative adversarial learning (Liu et al., 2021).
In particular, Zhou et al.
(2021) shows UCRL-VTR+ is near-minimax optimal when feature dimension d ≥ H .
B. E XISTING RESULTS IN OFFLINE RL WITH MODEL REPRESENTATIONS
Ofﬂine RL with general function representations.
The ﬁnite sample analysis of ofﬂine RL with function approximation is initially conducted by Fitted Q -Iteration (FQI) type algorithms and can be dated back to (Munos, 2003; Szepesv´ari and Munos, 2005; Antos et al., 2008a;b).
Later, Chen and Jiang (2019); Le et al.
(2019); Xie and Jiang (2020) follow this line of research and derive the improved learning results.
However, owing to the aim for tackling general function approximation, those learning bounds are expressed in terms of the stringent concentrability coefﬁcients (therefore, are less adaptive to individual instances) and are usually only information-theoretical , due to the computational intractability of the optimization procedure over the general function classes.
Other works impose weaker assumptions ( e.g. partial coverage (Liu et al., 2020; Kidambi et al., 2020;
Uehara and Sun, 2021)), and their ﬁnite sample analysis are generally suboptimal in terms of H or the effective horizon (1 − γ ) − 1 .
Ofﬂine RL with tabular models.
For tabular MDPs, tight learning bounds can be achieved under several data-coverage assumptions.
For the class of problems with uniform data-visitation measure d m , the near-optimal sample complexity bound has the rate
O
( H 3 /d m (cid:15) 2 ) for time-inhomogeneous
MDPs (Yin et al., 2021) and O ( H 2 /d m (cid:15) ) for time-homogeneous MDPs (Yin and Wang, 2021a; Ren et al., 2021).
Under the single concentrability assumption, the tight rate O ( H 3 SC (cid:63) /(cid:15) 2 ) is obtained by Xie et al.
(2021b).
In particular, the recent study Yin and Wang (2021b) introduces the intrinsic ofﬂine learning bound that is not only instance-dependent but also subsumes previous optimal results.
Ofﬂine RL with linear model representations.
Recently, there are more focus on studying the provable efﬁcient ofﬂine RL under the linear model representations.
Jin et al.
(2021b)
ﬁrst shows ofﬂine RL with linear MDP is provably efﬁcient by the pessimistic value iteration (PEVI), which is an ofﬂine counterpart of LSVI-UCB in Jin et al.
Their analysis deviates from their lower bound by a factor of d · H (check their Theorem 4.4 and 4.6).
Later, Xie et al.
(2021a) considers function approximation under the Bellman-consistent assumptions, and, when realized to linear MDP setting, improve the sample complexity guarantee of Jin et al.
(2021b)
by a order O ( d ) (Theorem 3.2).
However, their improvement only holds for ﬁnite action space (due to the dependence log |A| ) and by the direct reduction (from Theorem 3.1) their result does not imply a computationally tractable algorithm.
In addition, there is no improvement on the horizon dependence.
Concurrently, Zanette et al.
(2021) considers the Linear Bellman Complete model (which originates from its online version
Zanette et al.
and designs the actor-critic style algorithm that achieves tight result under the assumption that the value function is bounded by 1 .
While their algorithm is efﬁcient (which is based on solving a sequence of second-order cone programs), the resulting learning bound requires the action space to be ﬁnite due to the mirror descent/natural policy gradient updates in the Actor procedure (Agarwal et al., 2021).
Besides, assuming the value function to be less than 1 simpliﬁes the challenges in dealing with horizon H since when rescale their result to [0 , H ] , there is a H factor blow-up, which makes no improvement in the horizon dependence comparing to Jin et al.
(2021b).
On the other hand, Wang et al.
(2021a)
; Zanette (2021) study the statistical hardness of ofﬂine RL with linear representations by prooﬁng the exponential lower bounds.
As a result, none of the existing algorithms can achieve the statistical limit for the well-structured linear MDP model with the general (inﬁnite or continuous) state-action spaces in the ofﬂine regime.
C P ROOFS IN S ECTION 3.
Instead of prooﬁng the result for v (cid:63) − v (cid:98) π , in most parts of the proof we deal with V (cid:63) 1 − V (cid:98) π 1 , which is more general.
C. S OME PREPARATIONS
Deﬁne the Bellman update error ζ h ( s, a ) :
= ( T h (cid:98) V h +1 )
( s, a ) − (cid:98) Q h
( s, a ) and recall (cid:98)
π h ( s )
=
arg max π h (cid:104) (cid:98) Q h ( s, · ) , π h ( · | s ) (cid:105) A , then by the direct application of Lemma H.
[Equation]
The next lemma shows it is sufﬁcient to bound the pessimistic penalty, which is the key in the proof.
Lemma C.1.
Suppose with probability 1 − δ , it holds for all h, s,
a ∈
[ H ] × S × A that | ( T h (cid:98) V h
+1 − (cid:98) T h (cid:98)
V h +1 )
( s, a ) | ≤ Γ h ( s, a ) , then it implies ∀ s, a, h ∈ S ×
A ×
[ H ] , 0 ≤ ζ h ( s, a ) ≤ 2Γ h
( s, a ) .
Furthermore, it holds for any policy π simultaneously, with probability 1 − δ ,
[Equation]
Proof of Lemma C.1.
We ﬁrst show given | ( T h (cid:98)
V h
+1 − (cid:98) T h (cid:98)
V h +1 )
( s, a ) | ≤ Γ h ( s, a ) , then 0 ≤ ζ h ( s, a ) ≤ 2Γ h ( s, a ) , ∀ s, a, h ∈ S ×
A ×
[ H ] .
Step1: we ﬁrst show 0 ≤ ζ h ( s, a ) , ∀ s, a, h ∈ S ×
A ×
[ H ] .
Indeed, if ¯ Q h ( s, a ) ≤ 0 , then by deﬁnition (cid:98) Q h ( s, a ) = 0 and in this case ζ h ( s, a ) :
= ( T h (cid:98) V h +1 )
( s, a ) − (cid:98) Q h
( s, a ) =
( T h (cid:98) V h +1 )
( s, a ) ≥ 0 ; if ¯ Q h ( s, a ) > 0 , then (cid:98) Q h ( s, a ) ≤ ¯ Q h ( s, a ) and
[Equation]
Step2: next we show ζ h ( s, a ) ≤ 2Γ h ( s, a ) , ∀ s, a, h ∈ S ×
A ×
[ H ] .
Indeed, we have (cid:98) Q h ( s, a ) = max( ¯ Q h ( s, a ) , 0) and this is because: ¯ Q h ( x, a ) =
( (cid:98) T h (cid:98) V h +1 )
( x, a ) − Γ h ( x, a ) ≤ ( T h (cid:98) V h +1 )
( x, a ) ≤ H − h
+ 1 .
Therefore, in this case we have:
[Equation]
For the last statement, denote F := { 0 ≤ ζ h ( s, a ) ≤ 2Γ h ( s, a ) , ∀ s, a, h ∈ S
×
A ×
[ H ] } .
Note conditional on F , then by equation 7, V π 1 ( s ) − V (cid:98)
π 1 ( s ) ≤ (cid:80)
Hh =1 2 · E π [Γ h ( s h , a h ) | s 1 = s ] holds for any policy π almost surely.
Therefore,
[Equation]
≥ P
[Equation]
which ﬁnishes the proof.
C.
[Equation]
By Lemma C.1, it remains to bound | ( T h (cid:98) V h +1 )
( s, a ) − ( (cid:98) T h (cid:98) V h +1 )
( s, a ) | .
Suppose w h is the coefﬁcient corresponding to the T h (cid:98)
V h
+1 (such w h exists by Lemma H.9), i.e. T h (cid:98) V h
+1 =
φ (cid:62) w h , and recall
( (cid:98) T h (cid:98) V h +1 )
( s, a ) =
φ ( s, a ) (cid:62) (cid:98) w h , then: (cid:16)
[Equation]
h (cid:98) V h
+1 (cid:17) ( s, a ) − (cid:16)(cid:98) T h (cid:98) V
h +1 (cid:17) ( s, a ) =
[Equation]
( s, a ) (cid:62) ( w h
[Equation]
(cid:98)
[Equation]
h )
[Equation]
( s,
[Equation]
(cid:62)
[Equation]
[Equation] [Equation]
(cid:98)
[Equation] [Equation]
(cid:32)
[Equation]
(cid:88)
[Equation]
[Equation] [Equation]
(cid:16)
[Equation]
(cid:98)
[Equation]
(cid:0)
[Equation]
(cid:1)(cid:17)
[Equation]
(cid:98)
[Equation] [Equation]
[Equation]
) (cid:33)
[Equation]
) (cid:33) (cid:124)
[Equation]
The term (i) is dealt by the following lemma.
Lemma C.2.
Recall κ in Assumption 2.2.
Suppose K ≥ max
[Equation] [Equation]
λH
[Equation]
, then with probability 1 − δ , for all s, a, h ∈ S ×
A ×
[ H ]
[Equation]
Proof.
Recall T h (cid:98) V h
+1 =
φ (cid:62) w h and apply Lemma H.6, we obtain with probability 1 − δ , for all s, a, h ∈ S
×
A ×
[ H ] ,
[Equation]
where ˜Λ ph := E µ,h (cid:2)(cid:98) σ h ( s, a ) − 2 φ ( s, a ) φ ( s, a ) (cid:62) (cid:3) and the second inequality is by Lemma H.6 (with φ (cid:48)
= φ/ (cid:98) σ h and (cid:107) φ/ (cid:98) σ h (cid:107) ≤ (cid:107) φ (cid:107) ≤ 1 := C ) and the third inequality uses √ a (cid:62) · A · a ≤ (cid:112) (cid:107) a (cid:107) 2 (cid:107) A (cid:107) 2 (cid:107) a (cid:107) 2 = (cid:107) a (cid:107) 2 (cid:112) (cid:107) A (cid:107) 2 with a to be either φ or w h .
Moreover, λ min (˜Λ ph ) ≥ κ/ max h,s,a (cid:98) σ h ( s, a ) 2 ≥ κ/H 2 implies (cid:13)(cid:13)(cid:13) (˜Λ ph ) − 1 (cid:13)(cid:13)(cid:13) ≤ H 2 /κ , therefore for all s, a, h ∈ S ×
A ×
[ H ] , with probability 1 − δ
[Equation]
K .
For term (ii), denote: x
[Equation]
then by Cauchy inequality it follows
[Equation]
C.2. A
[Table]
[Equation]
Recall (in Theorem
[Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
, a τh ) + λ · I and Λ h = (cid:80)
Kτ =1 φ ( s τh , a τh ) (cid:62) φ ( s τh , a τh ) /σ 2 (cid:98) V h
+
( s τh , a τh ) + λI .
Then we have the following lemma to control the term (cid:113) φ ( s, a ) (cid:98) Λ − 1 h φ ( s, a ) .
Lemma C.3.
Denote the quantities C 1 = max { 2 λ, 128 log(2 d/δ ) , 128 H 4 log(2 d/δ ) /κ 2 } and
C 2 = max { λ 2 κ log(( λ + K ) H/λδ ) , 96 2 H 12 d log(( λ + K ) H/λδ ) /κ 5 } .
Suppose the number of episode
K satisﬁes K > max { C 1 , C 2 } , then with probability 1 − δ ,
[Equation]
Proof of Lemma C.3.
By deﬁnition (cid:113) φ ( s, a ) (cid:98) Λ − 1 h φ ( s, a ) = (cid:107) φ ( s, a ) (cid:107) (cid:98) Λ − 1 h .
Then denote
[Equation]
where Λ h = (cid:80)
Kτ =1 φ ( s τh , a τh ) (cid:62) φ ( s τh , a τh ) /σ 2 (cid:98) V h
+
( s τh , a τh ) + λI .
Under the condition of K , by
Lemma C.7, with probability 1 − δ
[Equation]
Λ
[Equation]
Λ
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Table]
[Equation] [Equation]
Next by Lemma H.5 (with φ to be φ/σ (cid:98) V h
+1 and C = 1 )
, it holds with probability 1 − δ ,
[Equation]
Λ
[Equation]
h −
[Equation]
E µ,h [ φ
[Equation]
s, a
[Equation]
Therefore by
Weyl’s spectrum theorem and the condition
K > max { 2 λ, 128 log(2 d/δ ) , 128 H 4 log(2 d/δ ) /κ 2 } , the above implies
[Equation]
V h
+1 ( s, a )] (cid:17) −√√ K (cid:18) log 2 d δ (cid:19) 1 /≥ κ
H 2 − 4 √ 2 √
K (cid:18) log 2 d δ (cid:19) 1 /≥ κ 2 H 2 .
Hence with probability 1 − δ , (cid:107) Λ (cid:48) h (cid:107) ≤ 2 and (cid:13)(cid:13) Λ (cid:48)− 1 h (cid:13)(cid:13)
= 1 /λ min (Λ (cid:48) h ) ≤ 2 H 2 /κ .
Similarly, one can show (cid:13)(cid:13)(cid:13)(cid:98) Λ (cid:48)− 1 h (cid:13)(cid:13)(cid:13) ≤ 2 H 2 /κ with high probability.
Now apply Lemma H.4 to (cid:98) Λ (cid:48) h and Λ (cid:48) h and a union bound, we obtain with probability 1 − δ , for all s, a (cid:107) φ ( s, a ) (cid:107)
[Equation]
Λ (cid:48)− 1 h ≤
[Equation]
1 h ≤ (cid:34) 1 + (cid:114)
[Equation]
where the third inequality uses equation 10 and the last and the second last inequality use
K > max {
[Equation]
) /κ 5 } .
Note the above is equivalent to (cid:113)
[Equation] [Equation]
a
[Equation]
s, a ) by multiplying 1 / √ K on both sides.
C.2. A NALYZING THE TERM
[Equation]
Lemma C.4.
Recall x τ = φ ( s τh ,a τh ) (cid:98) σ ( s τh ,a τh ) and η τ = (cid:16) r τh + (cid:98) V h
+1 (cid:0) s τh +1 (cid:1) − (cid:16) T h (cid:98)
V h
+1 (cid:17) ( s τh , a τh ) (cid:17) / (cid:98) σ ( s τh , a τh ) .
Let C H,d,κ := 12 (cid:112) H 4 d log(( λ + K ) H/λδ ) /κ and denote ξ :
= sup V ∈ [0 ,H ] , s (cid:48) ∼ P
h
( s,a ) ,
h ∈
[ H ] (cid:12)(cid:12) (cid:12)(cid:12) r h
+ V ( s (cid:48) ) − ( T h V )
( s, a ) σ V ( s, a ) (cid:12)(cid:12) (cid:12)(cid:12) .
If K ≥ 4 C 2 H,d,κ , then with probability 1 − δ , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K (cid:88) τ
= x τ η τ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:98) Λ −≤(cid:115)
d log (cid:18) 1 + K λd (cid:19) · log (cid:18) 4 Kδ (cid:19) + 4 ξ log (cid:18) 4 Kδ (cid:19) ≤ (cid:101)
O max (cid:8) √ d, ξ (cid:9) , where (cid:101)
O absorbs the constants and Polylog terms.
Proof of Lemma C.4.
By construction, we have (cid:107) x
τ (cid:107) ≤ (cid:107) φ/ (cid:98) σ (cid:107) ≤ 1 and by Lemma C.7, with probability 1 − δ/ 3 ,
[Equation]
K
Therefore, when K ≥ 4 C 2 H,d,κ , C H
[Equation]
d,κ (cid:113) 1 K ≤ 1 / 2 ≤ σ (cid:98) V h
+
( s τh , a τh ) / 2 and hence |
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Next, note (cid:98) V h +1 is estimated using the transition from h
+ 1 to H and (cid:98) σ h is estimated using independent episodes D (cid:48) , therefore conditional on (cid:98) V h
+1 , we receive
Var [ η τ |F τ − 1 ] =
Var (cid:104)
η τ | s τh , a τh , (cid:98) V h
+1 (cid:105) =
Var
[Equation]
+
[Equation]
−
[Equation]
where the ﬁrst inequality is by Lemma C.7 and the second one is by deﬁnition of σ h,V ( s, a ) 2 :=
max { 1 , Var P h ( V )( s, a ) } .
Thus, by Bernstein inequality for self-normalized martingale (Lemma H.3), 10 with probability 1 − δ ,
[Equation]
where (cid:101) O absorbs the constants and Polylog terms.
Recall M 1 , M 2 , M 3 , M 4 in List A. Based on the above results, we have the following key lemma:
Lemma C.5.
Assume K > max {M 1 , M 2 , M 3 , M 4 } , for any 0 < λ < κ , suppose √ d > ξ , where ξ := sup V ∈ [0 ,H ] , s (cid:48) ∼ P
h
( s,a ) ,
h ∈
[ H ] (cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
+ V
( s
[Equation] [Equation]
T
[Equation]
V
[Equation]
s
[Equation]
a
[Equation]
V ( s,a ) (cid:12)(cid:12)(cid:12)(cid:12) .
Then with probability 1 − δ , for all h, s, a ∈ [ H ] × S × A ,
[Equation]
where Λ h = (cid:80)
Kτ =1 φ ( s τh , a τh ) (cid:62) φ ( s τh , a τh ) /σ 2 (cid:98) V h
+
( s τh , a τh )
+ λI and (cid:101)
O absorbs the universal constants and Polylog terms.
Proof of Lemma C.5.
Combing equation 8, Lemma C.2, equation 9, Lemma C.3 and C.4 and a union bound to ﬁnish the proof.
C. P ROOF OF THE FIRST PART OF T HEOREM 3.
Theorem C.6 (First part of Theorem 3.2) .
Let K be the number of episodes.
Suppose √ d > ξ , where
[Equation]
Then for any 0 < λ < κ , with probability 1 − δ , for all policy π simultaneously, the output (cid:98) π of
Algorithm 1 satisﬁes
[Equation]
where Λ h = (cid:80)
Kτ =1 φ ( s τh ,a τh )
[Equation]
φ ( s
[Equation]
Vh
[Equation]
sτh,aτh ) + λI d
and (cid:101)
O absorbs the universal constants and the Polylog terms.
Proof of Theorem C.6.
Combing Lemma C.1 and Lemma C.5, we directly have with probability 1 − δ , for all policy π simultaneously,
[Equation]
now take the initial distribution d 1 on both sides to get the stated result.
C. T WO I NTERMEDIATE RESULTS
The next two lemmas provide intermediate results in ﬁnishing the whole proofs.
C.4. B OUNDING THE VARIANCE
Lemma C.7.
Recall the deﬁnition (cid:98) σ h ( · , · ) 2 = max { 1 , (cid:100)
Var P h (cid:98) V h
+1 ( · , · ) } + 1 and σ (cid:98) V h
+1 ( · , · ) 2 :=
max { 1 , Var P h (cid:98) V h
+1 ( · , · ) } + 1 .
Moreover, (cid:2)(cid:100) Var h (cid:98) V h
+1 (cid:3) ( · ,
[Equation] [Equation]
(cid:2)(cid:10) φ ( · , · ) , ¯ θ h (cid:11) [0 ,H − h +1] (cid:3) 2 (where ¯ β h and ¯ θ h are deﬁned in Algorithm 1).
Let K ≥ max (cid:8) 512(1 /κ )
2 log (cid:0) 4 Hdδ (cid:1) , 4 λ/κ (cid:9) , then with probability 1 − δ ,
[Equation]
|
[Equation]
Proof.
Step1: we ﬁrst show for all h, s, a ∈
[ H ] × S × A , with probability 1 − δ
[Equation]
Proof of Step1. Note
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
For 2 , since K ≥ max (cid:8) 512(1 /κ ) 2 log (cid:0) 4 Hdδ (cid:1) , 4 λ/κ (cid:9) , by Lemma H.6 and a union bound over h ∈
[ H ] , with probability 1 − δ for all h, s, a ∈
[ H ] × S × A ,
[Table]
, a
[Table]
For 1 , we have
[Table]
¯Σ − 1 h (13)
Note (cid:98) V h
+1 leverages the random transition from time t + 1 to H , therefore conditional on (cid:98) V h
+1 , the transitions (¯ s τh , ¯ a τh , ¯ s τh +1 ) are conditional independent w.r.t to different τ .
Thus, deﬁne x τ = φ (¯ s τh , ¯ a τh ) ( (cid:107) φ (cid:107) 2 ≤ 1 ) and η τ = (cid:98) V h
+
(¯ s τh +1 )
2 − P h
( (cid:98) V h +1 ) 2 (¯ s τh , ¯ a τh ) is H 2 -subgaussian, by Lemma H.2 (where t = K and L = 1 ) with probability 1 − δ ,
[Table]
Denote A := (cid:26)(cid:13)(cid:13)(cid:13)(cid:80) Kτ =1 φ (¯ s τh , ¯ a τh ) · (cid:16)(cid:98)
V h
+
(¯ s τh +1 )
2 − P h
( (cid:98)
[Equation] [Equation]
(¯ s τh
[Equation]
¯ a τh
[Equation]
(cid:17)(cid:13)(cid:13)(cid:13)
[Equation]
then P ( A ) =
E [ 1 ( A )]
=
E [ E [ 1 ( A ) | (cid:98) V h +1 ]]
=
E [ P ( A | (cid:98) V h +1 )]
≥ E [1 − δ ] = 1 − δ .
Apply a union bound for h ∈ [ H ] , we have with probability 1 − δ , for all h ∈
[ H ] ,
[Equation] [Equation]
and similar to 2 , with probability 1 − δ for all h, s,
a ∈ [ H ] × S × A ,
[Equation]
[Equation]
Combing equation 12, equation 13, equation 14 and equation 15 we obtain with probability 1 − δ for all h, s,
a ∈
[ H ] × S × A , (cid:12)(cid:12)(cid:12) (cid:104) φ ( s, a ) , ¯ β
[Equation]
Step2: we show for all h, s, a ∈ [ H ] × S × A , with probability 1 − δ
[Equation]
(cid:104)
[Equation]
a
[Equation]
The proof of Step2 follows nearly the identical way as Step1 except (cid:98) V 2 h is replaced by (cid:98) V h .
Step3: We prove sup
h
|| (cid:98) σ 2 h − σ 2 (cid:98) V h |
[Equation]
∞ ≤
[Equation]
Proof of Step3.
By equation 16,
[Equation]
Combining this with Step1 we receive ∀ h, s, a ∈ [ H ] × S × A , with probability 1 − δ
[Equation]
Var
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Finally, by the non-expansiveness of operator max { 1 , ·} + 1 , we have the stated result.
C.4. A CRUDE BOUND ON sup h || V (cid:63)h − (cid:98) V h || ∞ .
Lemma
C.8.
Deﬁne
[Equation] [Equation]
Var P
[Equation]
V
[Equation]
+
[Equation]
s
[Equation]
a
[Equation]
+
[Equation]
if
[Equation]
≥ max {M 1 , M 2 , M 3 , M 4 } and K > C · H 4 κ 2 , then with probability at least 1 − δ ,
[Equation]
Proof.
Step1: We show with probability at least 1 − δ ,
sup h (cid:13)(cid:13) V (cid:63)h − V
[Equation]
∞ ≤
[Equation]
H
[Equation]
√
[Equation]
√ κK (cid:17) .
Indeed, combing Lemma C.1 and Lemma C.5, similar to the proof of Theorem C.6, we directly have with probability 1 − δ , for all policy π simultaneously, and for all s ∈ S ,
h ∈
[ H ]
V πh ( s ) − V (cid:98) πh ( s ) ≤ (cid:101) O (cid:32) √ d ·
H (cid:88) t = h
E π (cid:104)(cid:0) φ (
[Equation] [Equation]
Next, since K ≥ max (cid:8) 512(1 /κ ) 2 log (cid:0) 4 Hdδ (cid:1) , 4 λ/κ (cid:9) , by Lemma H.6 and a union bound over h ∈ [ H ] , with probability 1 − δ
[Equation]
Lastly, taking π = π (cid:63) in equation 17 to obtain
[Equation]
This implies by using the condition K > C · H 4 κ 2 , we ﬁnish the proof of Step1.
Step2: We show with probability 1 −
[Equation] [Equation]
[Table]
Indeed, applying Extended Value Difference Lemma H.7 for π = π (cid:48)
= (cid:98) π , then with probability 1 − δ , for all s, h
[Equation]
where the second inequality uses Lemma C.5 and the last inequality follows the same procedure as
Step
[Equation]
Step3: Combine Step1 and Step2, by triangular inequality and a union bound we ﬁnish the proof of the lemma.
Remark C.9.
Note as an intermediate calculation, equation 18 ensures a learning bound with order (cid:101) O
( H 2 √ d √ κK ) .
Here, the convergence rate is the standard statistical rate 1 √ K and the H 2 dependence is loose.
However, the feature dependence (cid:112) d/κ is roughly tight, since, in the well-explored case (Assumption 2 of Wang et al.
(2021a))
, κ = 1 /d and the (cid:112) d/κ = √ d 2 recovers the optimal feature dependence dH √ T in the online setting (Zhou et al., 2021).
If κ (cid:28) 1 /d , then doing ofﬂine learning requires sample size proportional to d/κ , which reveals ofﬂine RL is harder when the exploration of behavior policy is insufﬁcient.
When κ = 0 , learning the optimal policy accurately cannot be guaranteed even if the sample/episode size K → ∞ .
C. P ROOF OF THE SECOND PART OF T HEOREM 3.
Lemma C.10.
Recall (cid:98) σ h =
(cid:114) max (cid:110) 1 , (cid:100)
Var P h (cid:98) V h
+1 (cid:111) + 1 and σ (cid:63)h = (cid:113) max (cid:8) 1 ,
Var P h V (cid:63)h +1 (cid:9) + 1 .
Let K ≥ max (cid:8) 512(1 /κ )
2 log (cid:0) 4 Hdδ (cid:1) , 4 λ/κ (cid:9) and K ≥ max {M 1 , M 2 , M 3 , M 4 } , then with probability 1 − δ , sup
[Equation]
Proof.
By deﬁnition and the non-expansiveness of max { 1 , ·} + 1 , we have
[Equation]
[Table]
with probability 1 − δ for all h ∈
[ H ]
[Table]
where the last inequality comes from Lemma C.8.
Combining this with Lemma C.7, we have the stated result.
Lemma C.11.
Denote the quantities C 1 = max { 2 λ, 128 log(2 d/δ ) , 128 H 4 log(2 d/δ ) /κ 2 } and
C 2 = max { λ 2 κ log(( λ + K ) H/λδ ) , 96 2 H 12 d log(( λ + K ) H/λδ ) /κ 5 } .
Suppose the number of episode
K satisﬁes K > max { C 1 , C 2 } , then with probability 1 − δ ,
[Equation]
Proof of Lemma C.11.
By deﬁnition (cid:113) φ ( s, a )Λ − 1 h φ
[Equation] [Equation]
=
[Equation]
φ
[Equation] [Equation]
h .
Then denote
[Equation]
where Λ h = (cid:80)
Kτ =1 φ ( s τh , a τh ) (cid:62) φ ( s τh , a τh ) /σ 2 V (cid:63)h +1 ( s τh , a τh ) + λI .
Under the condition of K , by
Lemma C.10, with probability 1 − δ
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
κK
[Equation] [Equation]
Next by Lemma H.5 (with φ to be φ/σ V (cid:63) h +1 and C = 1 ), it holds with probability 1 − δ ,
[Equation]
Therefore by
Weyl’s spectrum theorem and the condition
K > max { 2 λ, 128 log(2 d/δ ) , 128 H 4 log(2 d/δ ) /κ 2 } , the above implies
[Equation]
[Table]
1 /λ min (Λ (cid:63) (cid:48) h ) ≤ 2 H 2 /κ .
Similarly, (cid:13)(cid:13)(cid:13) Λ (cid:48) − 1 h (cid:13)(cid:13)(cid:13) ≤ 2 H 2 /κ with high probability.
Now apply Lemma H.4 to Λ (cid:63) (cid:48) h and Λ (cid:48) h and a union bound, we obtain with probability 1 − δ , for all s, a (cid:107) φ ( s, a ) (cid:107) Λ (cid:48)− 1 h ≤ (cid:20)
[Equation]
+
[Equation]
− 1 h
[Equation]
(cid:48)
[Equation]
a
[Equation]
1 h ≤
[Equation]
1 +
[Equation]
H
[Equation]
H
[Equation]
h
[Equation]
(cid:13)
[Equation]
1 h ≤ 
[Equation]
Hκ
[Equation]
(cid:34) (cid:101)
O (cid:32) H
[Equation]
√ d √ κK
[Equation]
where the third inequality uses equation 19 and the last inequality uses K > max {
[Equation]
The claimed result follows straightforwardly by multiplying 1 / √ K on both sides of the above.
Proof of Theorem 3.2.
The ﬁrst part of the theorem has been shown in Theorem C.6.
For the second part, apply Theorem C.6 with π = π (cid:63) , then with probability 1 − δ ,
[Equation]
Now apply Lemma C
[Equation]
11 and a union bound, with probability 1 − δ ,
[Equation]
D P ROOF OF T HEOREM 3.
First of all, we show the following lemma.
Lemma D.1.
Suppose K > max {M 1 , M 2 , M 3 , M 4 } .
Plug Γ Ih ( s, a ) ← φ ( s, a ) (cid:62) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:98) Λ − 1 h
K (cid:88)
[Equation]
K ) in Algorithm 1 and let T h be the Bellman operator and (cid:98) T h be the approximated Bellman operator.
Then we have with probability 1 − δ :
[Equation]
Proof of Lemma D.1.
Suppose w h is the coefﬁcient corresponding to the T h (cid:98)
V h
+1 (such w h exists by
Lemma H.9), i.e. T h (cid:98) V h
+
=
φ (cid:62) w h , and recall ( (cid:98) T h (cid:98) V h +1 )
( s, a ) =
φ ( s, a ) (cid:62) (cid:98) w h , then: (cid:16)
[Equation]
(cid:98)
[Equation]
+1 (cid:17) ( s, a ) − (cid:16)(cid:98)
[Equation]
h (cid:98) V h
+1 (cid:17) ( s, a
[Equation]
( s,
[Equation]
(cid:62) ( w h
[Equation]
(cid:98) w h ) =
φ ( s,
[Equation]
(cid:62) w
[Equation]
( s,
[Equation]
) (cid:62) (cid:98) Λ − 1 h (cid:32) K (cid:88) τ
= φ ( s τh , a τh ) · (cid:16) r τh + (cid:98) V h
+1 (cid:0) s τh +1 (cid:1)(cid:17) / (cid:98) σ 2 h ( s τh , a τh ) (cid:33) =
φ ( s, a ) (cid:62) w h − φ ( s, a ) (cid:62) (cid:98) Λ − 1 h (cid:32) K (cid:88)
τ
= φ ( s τh , a τh ) · (cid:16) T h (cid:98) V h
+1 (cid:17) ( s τh , a τh ) / (cid:98) σ 2 h ( s τh , a τh ) (cid:33) (cid:124) (cid:123)(cid:122) (cid:125)
(i) + φ ( s, a ) (cid:62) (cid:98) Λ − 1 h (cid:32) K (cid:88) τ
= φ ( s τh , a τh ) · (cid:16) r τh + (cid:98) V h
+1 (cid:0) s τh +1 (cid:1) − (cid:16)(cid:98) T h (cid:98)
V h
+1 (cid:17) ( s τh , a τh ) (cid:17) / (cid:98) σ 2 h ( s τh , a τh ) (cid:33) (cid:124) (cid:123)(cid:122) (cid:125) (ii) + φ ( s, a ) (cid:62) (cid:98) Λ − 1 h (cid:32) K (cid:88) τ
= φ ( s τh , a τh ) · (cid:16)(cid:16)(cid:98) T h (cid:98) V h
+1 (cid:17) ( s τh , a τh ) − (cid:16) T h (cid:98) V
h +1 (cid:17) ( s τh , a τh
[Equation]
For term (i), by Lemma C.2 it is bounded by 2 λH
[Table]
[Equation]
[Table]
K with probability 1 − δ/ 2 .
For term (ii), it is bounded by
[Table]
For term (iii), by Cauchy inequality
[Table]
where the ﬁrst inequality is by Lemma H.6 (with φ (cid:48)
= φ/ (cid:98) σ h and (cid:107) φ/ (cid:98) σ h (cid:107) ≤ (cid:107) φ (cid:107) ≤ 1 := C ) and the third inequality uses √ a (cid:62) · A · a ≤ (cid:112) (cid:107) a (cid:107) 2 (cid:107) A (cid:107) 2 (cid:107) a (cid:107) 2 = (cid:107) a (cid:107) 2 (cid:112) (cid:107) A (cid:107) 2 with a to be either φ or w h .
Moreover, λ min (˜Λ ph ) ≥ κ/ max h,s,a (cid:98) σ h ( s, a ) 2 ≥ κ/H 2 implies (cid:13)(cid:13)(cid:13) (˜Λ ph ) − 1 (cid:13)(cid:13)(cid:13) ≤ H 2 /κ .
The second inequality is true by denoting x τ = φ ( s τh , a τh ) / (cid:98) σ ( s τh , a τh ) and
[Equation]
and use Lemma H.10 as the condition for applying Lemma H.2.
By collecting those three terms together we have the result.
D. P ROOF OF T HEOREM 3.
Proof.
Use Lemma D.1 as the condition for Lemma C.1 and average over initial distribution d 1 , we obtain with probability 1 − δ ,
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
+ (cid:101)
O ( H 4 d/κ K ) (21)
Denote A h := (cid:80)
Kτ =1 φ ( s τh ,a τh ) · ( r τh + (cid:98) V h
+
( s τh +1 )
− ( T h (cid:98) V h +1 )
( s τh ,a τh ) )
(cid:98) σ 2 h ( s τh ,a τh ) , then
E π h [ φ ( s, a )] (cid:62) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:98) Λ − 1 h
K (cid:88) τ
= φ ( s τh , a τh ) · (cid:16) r τh + (cid:98) V h
+1 (cid:0) s τh +1 (cid:1) − (cid:16)(cid:98) T h (cid:98)
V h
+1 (cid:17) ( s τh , a τh ) (cid:17) (cid:98) σ 2 h ( s τh , a τh ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
≤ E π h
[ φ ] (cid:62) · (cid:12) (cid:12)(cid:12) (cid:98) Λ − 1 h
A h (cid:12) (cid:12)(cid:12) + E π h [ φ ] (cid:62) (cid:12)(cid:12) (cid:12) (cid:12) (cid:12)(cid:12)(cid:98) Λ − 1 h
K (cid:88) τ
= φ ( s τh , a τh ) · (cid:16) T h (cid:98) V h
+
( s τh , a τh ) − (cid:98) T h (cid:98)
[Table]
For the second term, it can be bounded similar to term (iii) in Lemma D.1 and for the ﬁrst term by
Cauchy’s inequality it is bounded by
[Equation]
where the ﬁrst inequality is identical to the analysis in Section C.2.2 and the second inequality is identical to the analysis in Section C.2.1 with φ replaced by E [ φ ] .
Plug this back to equationwe ﬁnish the proof for the ﬁrst part.
For the second part, converting Λ − 1 h to Λ (cid:63) − 1 h is identical to
Section C.5.
This ﬁnishes the proof.
E P ROOF OF M INIMAX L OWER BOUND T HEOREM 3.
The proof follows the lower bound proof of Zanette et al.
For completeness, we provide all the details in below.
E. C ONSTRUCTION
Similar to the proof of [Zanette et al.
(2021), Theorem 2], we construct a family of MDPs, each parameterized by a Boolean vector u
= ( u 1 , . .
, u H ) with each u h ∈ {− 1 , +1 } d − 2 for h ∈
[ H ] .
The MDPs share the same transition kernel and are only different in the reward observations.
State space: At each time step h , there are two states S
=
{ +1 , − 1 } .
Action space: The action space A
= {− 1 , 0 , +1 } d − 2 .
Feature map: The feature map φ : S × A (cid:55)→ R d is given by
[Equation]
The construction ensures the condition (cid:107) φ ( s, a ) (cid:107) 2 ≤ 1 for any ( s, a ) ∈ S × A .
Transition kernel: The transition probability P h ( s (cid:48) | s, a ) is independent of action a .
In other words, the Markov decision process reduces to a homogeneous Markov chain with transition matrix
[Equation]
By letting
[Equation]
we have P h ( s (cid:48) | s, a )
= (cid:104) φ ( s, a ) , ν h
( s (cid:48) )
(cid:105) to be a valid probability transition.
Reward observations: For any MDP M u , at each times step h , the reward follows a Gaussian distribution with
[Equation]
where δ ∈ (cid:2) 0 , 1 √ 3 d (cid:3) determines to what extent the MDP models are different from each other.
The mean reward function satisﬁes r u,h ( s, a ) =
(cid:104) φ ( s, a ) , θ u,h (cid:105) with θ u,h =  
[Equation]

[Equation]
∈ R d .
Ofﬂine data collection Scheme: The dataset D = { ( s τh , a τh , r τh , s τh +1 )
} h ∈
[ H ] τ ∈
[ K ] consist of K i.i.d.
trajectories.
All the trajectories initiate from uniform distribution.
We take a behavior policy µ ( · | s ) that is independent of state s .
Let { e 1 , e 2 , . . .
, e d − 2 } be the canonical bases of R d − 2 andd −
2 ∈ R
d − 2 be the zero vector.
The behavior policy µ is set as µ ( e j | s )
= 1 d for any j ∈ [ d − 2] and µ ( 0 d − 2 | s ) = 2 d .
E. O VERVIEW OF PROOF
The proof of the theorem is based on Assouad’s method, where we ﬁrst reduce the problem to binary hypothesis tests (Lemmas E.1 and E.2) and then connect the testing error to the uncertainty quantity in the upper bound (Lemma E.3).
Lemma E.1 (Reduction to testing) .
There exists a universal constant c 1 > 0 such thata
[Equation] [Equation]
where (cid:98) π denotes the output of any algorithm that maps from observations to an estimated policy.
ψ is any test function for parameter u and D H is the hamming distance.
Lemma E.2.
There exists a universal constant c 2 > 0 such that when taking δ :=
c 2 d √ K , we have
[Equation]
When K (cid:38) d 3 , δ :=
c 2 d √ K ensures that δ ≤ 1 / √ 3 d .
Combining Lemmas E.1 and E.2 yields a lower bound
[Equation]
[Equation]
where c > 0 is a universal constant.
We then use the following Lemma E.3 to connect the above lower bound to the uncertainty term √ d · (cid:80) Hh
[Equation]
[ φ ] for the chosen linear MDP instances class M .
Lemma E.3.
There exists a universal constant c 3 > 0 such that for all M ∈ M ,
[Equation]
Plugging inequality equation E.3 into the bound equation 24, we obtain the minimax lower bound equation 5 in the statement of theorem.
E. R EDUCTION TO TESTING VIA A SSOUAD ’ S METHOD
Proof of Lemma E.1.
For any index vector u = ( u 1 , . . .
, u H ) ∈
U
= {− 1 , +1 } ( d − 2) × H , the optimal policy for MDP instance M u is simply π (cid:63)h ( · ) =
u h for h ∈
[ H ] .
Similar to the proof of Lemma 9 in Zanette et al.
(2021), we can show that the value suboptimality of policy π on MDP M u is given by
[Equation]
Deﬁne u
π
=
( u π 1 , . . .
, u πH ) with u πh :=
sign (cid:0) E π [ a h ] (cid:1) , then the (cid:96) 1 -norm is lower bounded as
[Equation]
where D H ( · ; · ) denotes the Hamming distance.
It follows that
[Equation] [Equation]
We then apply Assouad’s method (Lemma 2.12 in Sampson and Guttorp (1992)) and obtain that
[Equation]
where ψ is any test functions mapping from observations to { u, u (cid:48) } .
Combining inequalities equation 26 and equation 27, we ﬁnish the proof.
E. L OWER BOUND ON THE TESTING ERROR
Proof of Lemma E.2.
The proof of Lemma E.2 is similar to that of Lemma 10 in Zanette et al.
We ﬁrst apply Theorem 2.12 in Sampson and Guttorp (1992) to lower bound the testing error using
KullbackLeibler divergence and obtain
[Equation]
It only remains to estimate D KL ( Q u (cid:107)Q u (cid:48) )
[Equation]
The probability density Q u takes the form
[Equation]
where ξ 1 = (cid:2) 12 , 12 (cid:3) is the initial distribution.
It follows that
[Equation]
If we take δ = c 2 d √ K , then inequality equation 28 ensures inequality equation 23, as claimed in the statement of the lemma.
E. C ONNECTION TO THE UNCERTAINTY TERM
Proof of Lemma E.3.
We ﬁrst calculate the explicit form of the inverse of variance-rescaled covariance matrix Λ (cid:63),ph .
For each time step h ∈
[ H ] , the value function V (cid:63)u,h +1 takes the form
[Equation]
Since (cid:0) P h +1 V (cid:63)u,h +2 (cid:1) (+1) = (cid:0) P h +1 V (cid:63)u,h +2 (cid:1) ( − 1) and r u,h
+1 (+1 , a ) − r u,h +1 ( − 1 , a ) = 2 / √ 6 , we have
[Equation]
Similarly,
[Equation]
By routine calculation, we ﬁnd that the population-level rescaled covariance matrix takes the form
[Equation]
R
[Equation]
for any h ∈ [ H ] .
Applying Gaussian elimination on Λ (cid:63),ph , we have
[Equation]
[Table]
[Equation]
Recall that by our construction,
[Equation]
It follows that
[Equation]
Therefore,
[Equation]
Taking the summation over h ∈ [ H ] , we obtain the bound equation 25 as claimed in the lemma statement.
F A N UMERICAL S IMULATION F. A L INEAR MDP CONSTRUCTION
We consider a synthetic linear
MDP example that is similar to Min et al.
but with some modiﬁcations for the ofﬂine learning task.
The MDP instance we use consists of |S| = 2 states and |A| = 100 actions, and feature dimension d = 100 .
We set S = { 0 , 1 } and A = { 0 , 1 , . . .
, 99 } respectively.
For each action a ∈ { 0 , 1 , . . .
, 99 } , we use binary encoding to obtain a vector a ∈ R 8 using its binary representation ( i.e. each coordinate is either 0 or 1 ).
we interchangebly use a and and its vector representation a for the ease of explanation.
We ﬁrst deﬁne δ ( s, a ) =
(cid:26)if 1 { s = 0 } =
{ a = 0 } otherwise , then the non-stationary linear MDP is speciﬁed by the following conﬁguration
• Feature mapping: φ ( s,
[Equation] [Equation]
a (cid:62) , δ ( s, a ) , 1 − δ ( s, a ) (cid:1) (cid:62) ∈ R•
The true measure ν h
ν
h
( s ) =
, 0 , (1 − s ) ⊕ α h , s ⊕ α h ) ,
where { α h } h ∈ [ H ] is a sequence of integers taking values 0 or 1 and ⊕ is the standard XOR operator
We deﬁne θ h ≡ (0 , . . .
, 0 , r, 1 − r )
∈
Rwith the choice of r = 0 .
The transition follows P h ( s (cid:48) | s, a )
= (cid:104) φ ( s, a ) , ν h
( s (cid:48) )
(cid:105) and the mean reward function r h ( s, a ) =
(cid:104) φ ( s, a ) , θ h (cid:105) .
• Behavior policy: always choose action a = 0 with probability p , and other actions uniformly with probability (1 − p ) / 99 .
The initial distribution chooses s = 0 and s = 1 with equal probability 1 / 2 .
We use p = 0 . 6 .
[Figure]
F. E MPIRICAL COMPARISON BETWEEN PEVI AND VAPVI ON THE CONSTRUCTED LINEAR MDP
We compare Pessimistic Value Iteration (PEVI) in Jin et al.
(2021b)
and our VAPVI Algorithm 1 in Figure 1, with horizon to be H = 20 , 30 , 50 .
In addition, we add the non-pessimistic version for both algorithms, i.e. least-square value iteration (LSVI) and variance-aware value iteration (VAVI).
The true optimal value v (cid:63) is computed via value iteration using the underlying transition kernels.
For the empirical validation of VAPVI, we do not split the data and, in particular, in all the methods we choose λ = 0 .
01 (instead of λ = 1 used in theory (Jin et al., 2021b) which causes over-regularization in the simulation).
We can observe VAPVI outperforms PEVI and the gap becomes larger when horizon H increases.
One main reason for this to happen is due to the bonus used in PEVI (Jin et al., 2021b)
[Equation]
is overly pessimistic comparing to our
[Equation]
when H becomes larger and this could potentially make the learning less accurate.
In addition, both non-pessimistic algorithms exhibit similar accuracy, and this is partially owing to our truncation scheme (cid:98) σ h ( · , · ) 2 ← max { 1 , (cid:100)
Var P h (cid:98) V h
+1 ( · , · ) } so (cid:98) σ h ( · , · ) 2 will just be 1 when the estimated variance is small.
Lastly, variance-aware pessimism eventually outperforms non-pessimism algorithms when sample size is large and this might come from that the pessimistic bonus is estimated more accurately when more samples are collected.
G S OME MISSING DERIVATIONS AND DISCUSSIONS G. R EGARDING COVERAGE ASSUMPTION
Now we discuss the feature coverage assumption.
Indeed, even if Assumption 2.2 is not satisﬁed, we can still learn in the effective subspan of Σ ph :=
E µ,h (cid:2) φ ( s, a ) φ ( s, a ) (cid:62) (cid:3) .
Indeed, since Σ ph is symmetric, by orthogonal decomposition we have Σ
ph =
Z
h Λ Z (cid:62) h , where Z h (can be estimated using the samples for practical purpose) consists of orthogonal basis and Λ consists of eigenvalues of Σ ph in the diagonal.
Suppose we do not have a full coverage, i.e.
[Equation]
[Equation]
0] with d (cid:48) <
d
[Equation]
then we can create transformed features φ (cid:48)
h ( s, a ) =
Z h · φ h ( s, a ) , and then
[Equation]
Then we can do learning w.r.t.
the truncated features φ (cid:48)
h | 1: d (cid:48) ’s only instead of the original φ .
It reduces to the weaker notion of κ :=
min h ∈ [ H ] { κ h : s.t. κ h
= smallest positive eigenvalue at time h } .
G. D ERIVATION OF EQUATION
When reducing Theorem 3.2,3.3 to the tabular case, set φ ( s, a ) =
1 s,a , d = SA , λ = 0 , and recall by Assumption 3.4 (let’s assume π (cid:63) is a deterministic policy as it always exists in tabular MDP)
C (cid:63) :
= sup h,s,a d π (cid:63) h ( s, a ) /d µh
( s, a ) , then for Theorem 3.
[Equation]
[Table]
where the ﬁrst inequality is by Chernoff bound and the last one is by Lemma
[Table]
4. of Yin and Wang (2020)
(Law of total variances).
The rest of them are from Cauchy’s inequality.
Similarly, for
Theorem 3.
[Equation]
we also have
[Equation]
[Table]
H A UXILIARY L EMMAS
Lemma H.1 (Matrix McDiarmid inequality / Matrix Chernoff bound (Tropp, 2012)) .
Let z k , k = 1 , . .
, K be independent random vectors in R d , and let H be a mapping that maps K vectors to a d × d symmetric matrix.
Assume there exists a sequence of ﬁxed symmetric matrices { A k } k ∈
[ K ] such that for z k , z (cid:48)
k ranges over all possible values for each k ∈
[ K ] , it holds
[Equation]
[Equation] [Equation] [Equation] z
[Equation]
[Equation]
A 2 k .
Deﬁne σ 2 := (cid:13)(cid:13)(cid:80) k
A 2 k (cid:13)(cid:13) .
Then for any t > 0 ,
P {(cid:107) H ( z 1 , . . . , z K ) − E H z 1 , . .
. , z K ) (cid:107) ≥ t } ≤ d · exp (cid:18) − t8 σ(cid:19)
Lemma H.2 (Hoeffding inequality for self-normalized martingales (Abbasi-Yadkori et al., 2011)) .
Let { η t } ∞ t =1 be a real-valued stochastic process.
Let {F t } ∞ t
=
be a ﬁltration, such that η t is F tmeasurable.
Assume η t also satisﬁes η t given F t − 1 is zero-mean and R -subgaussian, i.e. ∀ [Equation] ∈ R
[Equation]
Let { x t } ∞ t =1 be an R d -valued stochastic process where x t is F t − 1 measurable
and (cid:107) x t (cid:107) ≤ L .
Let Λ t = λI d
+ (cid:80) ts =
1 x s x (cid:62) s .
Then for any δ > 0 , with probability 1 − δ , for all t > 0 ,
[Equation]
Lemma H.3 (Bernstein inequality for self-normalized martingales (Zhou et al., 2021)) .
Let { η t } ∞ t = be a real-valued stochastic process.
Let {F t } ∞ t
=
be a ﬁltration, such that η t is F t -measurable.
Assume η t also satisﬁes
[Equation]
Let { x t } ∞ t =1 be an R d -valued stochastic process where x t is F t − 1 measurable
and (cid:107) x t (cid:107) ≤ L .
Let Λ t = λI d
+ (cid:80) ts =
1 x s x (cid:62) s .
Then for any δ > 0 , with probability 1 − δ , for all t > 0 , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) t (cid:88) s =
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
{+1 t ≤ 8 σ (cid:115) d log (cid:18) 
+}
+ tLλd (cid:19) · log (cid:18) 4 tδ (cid:19)
+ 4 R log (cid:18) 4 tδ (cid:19)
Lemma H.4 (Converting the variance under the matrix norm) .
Let Λ 1 and Λ 2 ∈ R d
×
d are two positive semi-deﬁnite matrices.
Then:
[Equation]
and
[Equation]
for all φ ∈ R d .
Proof.
For the ﬁrst part, note
[Equation]
For the second one,
[Equation]
[Table]
≤
[Table]
[Equation]
[Table]
[Equation]
[Table]
Lemma H.5 (Lemma H.4 of Min et al.
let φ : S ×
A → R d satisﬁes (cid:107) φ ( s, a ) (cid:107) ≤ C for all s, a ∈ S × A .
For any K > 0 , λ > 0 , deﬁne ¯ G K =
(cid:80)
Kk =1 φ ( s k , a k ) φ ( s k , a k ) (cid:62) + λI d where ( s k , a k ) ’s are i.i.d samples from some distribution ν .
Then with probability 1 − δ , (cid:13)(cid:13)(cid:13)(cid:13)
¯ G K K − E ν (cid:20) ¯ G K K (cid:21)(cid:13)(cid:13)(cid:13)(cid:13) ≤√ 2 C 2 √ K (cid:18) log 2 d δ (cid:19) 1 /.
Proof of Lemma H.5.
For completeness, we provide the proof of Lemma H.5.
Let x k = φ ( s k , a k ) .
Denote (cid:101) Σ h as the matrix obtained by replacing the k-th vector x k in (cid:98) Σ h by (cid:101) x k and leaving the rest
K 1 vectors unchanged.
Then
[Equation]
Notice that (cid:13)(cid:13)(cid:13)(cid:80)
Kk A 2 k (cid:13)(cid:13)(cid:13) =
4 C 4 K , by Lemma H.1 we have the result.
Lemma H.6 (Lemma H.5. of Min et al. (2021)) .
Let φ : S ×
A → R d be a bounded function s.t.
(cid:107) φ (cid:107) 2 ≤ C .
Deﬁne ¯ G K
=
(cid:80)
Kk =1 φ ( s k , a k ) φ ( s k , a k ) (cid:62) + λI d where ( s k , a k ) ’s are i.i.d samples from some distribution ν .
Let G
= E ν [ φ ( s, a ) φ ( s, a ) (cid:62) ] .
Then for any δ ∈ (0 , 1) , if K satisﬁes
[Equation]
Then with probability at least 1 − δ , it holds simultaneously for all u ∈ R d that
[Equation]
Lemma H.7 (Extended Value Difference (Section B.1 in Cai et al.
Let π = { π h } Hh =1 and π (cid:48)
= { π (cid:48)
h }
Hh =1 be two arbitrary policies and let { (cid:98) Q h } Hh =1 be any given Q-functions.
Then deﬁne (cid:98) V h
( s ) :
= (cid:104) (cid:98) Q h ( s, · ) , π h ( · | s ) (cid:105) for all s ∈ S .
Then for all s ∈ S ,
[Equation]
29) where ( T h V )( · , · ) :
= r h ( · , · ) + ( P h V )( · , · ) for any V ∈ R S .
Proof.
Denote ξ h = (cid:98) Q h − T h (cid:98) V h
+1 .
For any h ∈ [ H ] , we have
[Equation]
recursively apply the above for (cid:98) V h
+1 − V π [Equation]
h +1 and use the E π (cid:48) notation (instead of the inner product of P h , π (cid:48) h ) we can ﬁnish the prove of this lemma.
Lemma H.8.
Let (cid:98) π
= { (cid:98) π h
Hh =1 and (cid:98) Q h ( · , · ) be the arbitrary policy and Q-function and also (cid:98) V h
( s ) =
(cid:104)
(cid:98) Q h ( s, · ) , (cid:98) π h ( ·| s ) (cid:105) ∀ s ∈ S .
and ζ h ( s, a ) :
=
( T h (cid:98) V h +1 )
( s, a ) − (cid:98) Q h
( s, a ) (element-wisely) to be the Bellman update error.
Then for any arbitrary π , we have
[Equation]
where the expectation are taken over s h , a h .
Proof.
Note the gap can be rewritten as
[Equation]
By Lemma H.7 with π = (cid:98) π , π (cid:48)
=
π
[Equation]
we directly have
V π 1 ( s ) − (cid:98) V 1 ( s ) =
H (cid:88)
[Equation]
[Equation]
Next apply Lemma H.7 again with π = π (cid:48)
= (cid:98) π , we directly have
[Equation]
[Equation]
Combine the above two results we prove the stated result.
Lemma H.9.
For a linear MDP, for any 0 ≤ V ( · ) ≤ H , then there exists a w h ∈ R d s.t.
T h
V = (cid:104) φ, w h (cid:105) and (cid:107) w h (cid:107)
≤ 2 H √ d for all h ∈
[ H ] .
Here T h ( V )( s, a ) =
r h ( x, a ) + ( P h V )( s, a ) .
Similarly, for any π , there exists w π h ∈ R d , such that Q π h = (cid:104) φ, w π h (cid:105) with (cid:107) w π h (cid:107) 2 ≤ 2( H − h + 1) √ d .
Proof.
By deﬁnition,
[Equation]
therefore (cid:107) w h (cid:107) 2 ≤ (cid:107) θ h (cid:107) [-
-]2 + H · (cid:107) ν
h
( S ) (cid:107) ≤ 1 + H √ d ≤ 2 H √ d .
The proof of the second part is similar by backward induction and the fact V πh ≤ H − h + 1 for any π .
Lemma H.10.
For any pessimistic bonus design Γ h , suppose K > max {M 1 , M 2 , M 3 , M 4 } , then with probability 1 − δ , Algorithm 1 yields
[Equation]
Proof of Lemma H.10.
Suppose w h is the coefﬁcient corresponding to the T h (cid:98)
V h
+1 (such w h exists by Lemma H.9), i.e. T h (cid:98) V h
+1 =
φ (cid:62) w h , and recall ( (cid:98) T h (cid:98) V h +1 )
( s, a ) =
φ ( s, a ) (cid:62) (cid:98) w h , then:
[Equation]
T h (cid:98) V h
+1 (cid:17) ( s, a ) − (cid:16)(cid:98) T h (cid:98) V
h +1 (cid:17) ( s, a ) =
φ ( s, a ) (cid:62)
[Equation]
For term (i), it is bounded by 2 λH 3 √ d/κ K with probability 1 − δ by Lemma C.2.
For term (ii), by Cauchy inequality it is bounded by (cid:107)
[Table]
τh
[Table]
a
[Table]
[Table]
κK · ˜ O ( H √ d ) =
[Equation]
O ( H(cid:112) d/κ √
K ) , where the ﬁrst inequality is by Lemma H.6 (with φ (cid:48)
= φ/ (cid:98) σ h and (cid:107) φ/ (cid:98) σ h (cid:107) ≤ (cid:107) φ (cid:107) ≤ 1 := C ) and the third inequality uses √ a (cid:62) · A · a ≤ (cid:112) (cid:107) a (cid:107) 2 (cid:107) A (cid:107) 2 (cid:107) a (cid:107) 2 = (cid:107) a (cid:107) 2 (cid:112) (cid:107) A (cid:107) 2 with a to be either φ or w h .
Moreover, λ min (˜Λ ph ) ≥ κ/ max h,s,a (cid:98) σ h ( s, a ) 2 ≥ κ/H 2 implies (cid:13)(cid:13)(cid:13) (˜Λ ph ) − 1 (cid:13)(cid:13)(cid:13) ≤ H 2 /κ .
The second inequality comes from Lemma H.2 with R =
H since | η τ | = | ( r τ h + (cid:98) V h
+1 (cid:0) s τ h +1 (cid:1) − ( T h (cid:98) V h +1 )
( s τh , a τh )) / (cid:98) σ h ( s τh , a τh ) | ≤ H and | x τ | = | φ ( s τh , a τh ) / (cid:98) σ h ( s τh , a τh ) | ≤ 1 .
The ﬁnal result is obtained by absorbing the term (i) via the condition K > max {M 1 , M 2 , M 3 , M 4 } .