O FFLINE R EINFORCEMENT L EARNING WITH D IFFERENTIABLE F UNCTION A PPROXIMATION IS P ROVABLY E FFICIENT
A BSTRACT
Ofﬂine reinforcement learning , which aims at optimizing sequential decisionmaking strategies with historical data, has been extensively applied in real-life applications.
State-Of-The-Art algorithms usually leverage powerful function approximators ( e g. neural networks) to alleviate the sample complexity hurdle for better empirical performances.
Despite the successes, a more systematic understanding of the statistical complexity for function approximation remains lacking.
Towards bridging the gap we take a step by considering ofﬂine reinforcement learning with differentiable function class approximation (DFA).
This function class naturally incorporates a wide range of models with nonlinear/nonconvex structures.
We show ofﬂine RL with differentiable function approximation is provably efﬁcient by analyzing the pessimistic ﬁtted Q-learning (PFQL) algorithm and our results provide the theoretical basis for understanding a variety of practical heuristics that rely on Fitted Q-Iteration style design.
In addition, we further improve our guarantee with a tighter instance-dependent characterization.
We hope our work could draw interest in studying reinforcement learning with differentiable function approximation beyond the scope of current research
I NTRODUCTION
Ofﬂine reinforcement learning (Lange et al., 2012; Levine et al., 2020) refers to the paradigm of learning a policy in the sequential decision making problems, where only the logged data are available and were collected from an unknown environment ( Markov Decision Process / MDP).
Inspired by the success of scalable supervised learning methods, modern reinforcement learning algorithms ( e.g. Silver et al.
(2017)) incorporate high-capacity function approximators to acquire generalization across large state-action spaces and have achieved excellent performances along a wide range of domains.
For instance, there are a huge body of deep RL-based algorithms that tackle challenging problems such as the game of Go and chess (Silver et al., 2017; Schrittwieser et al., 2020), Robotics (Gu et al., 2017; Levine et al., 2018), energy control (Degrave et al., 2022) and Biology (Mahmud et al., 2018;
Popova et al., 2018).
Nevertheless, practitioners also noticed that algorithms with general function approximators can be quite data inefﬁcient, especially for deep neural networks where the models may require million of steps for tuning the large number of parameters they contain.
On the other hand, statistical analysis has been actively conducted to understand the sample/statistical efﬁciency for reinforcement learning with function approximation, and fruitful results have been achieved under the respective model representations (Munos, 2003; Chen and Jiang, 2019;
Yang and Wang, 2019; Du et al., 2019; Sun et al., 2019; Modi et al., 2020; Jin et al., 2020b; Ayoub et al., 2020; Zanette et al., 2020; Jin et al., 2021a; Du et al., 2021; Jin et al., 2021b;
Zhou et al., 2021a; Xie et al., 2021a; Min et al., 2021; Nguyen-Tang et al., 2022; Li et al., 2021; Zanette et al., 2021;
Yin et al., 2022; Uehara et al., 2022; Cai et al., 2022).
However, most works consider linear model approximators ( e.g. linear (mixture) MDPs) or its variants.
While the explicit linear structures make the analysis trackable (linear problems are easier to analyze), they are unable to reveal the sample/statistical complexity behaviors of practical algorithms that apply powerful function approximations (which might have complex structures).
In addition, there is an excellent line of works tackling provably efﬁcient ofﬂine RL with general function approximation ( e.g. (Chen and Jiang, 2019; Xie et al., 2021a; Zhan et al., 2022)).
Due to the generic function approximation class considered, those complexity bounds are usually expressed in the standard worst-case fashion O ( V 2max qn ) which lack the characterizations of individual instance behaviors.
However, as mentioned in Zanette and Brunskill (2019), practical reinforcement learning algorithms often perform far better than what these problem-independent bounds would suggest.
These observations motivate us to consider function approximation schemes that can help address the existing limitations.
In particular, in this work we consider ofﬂine reinforcement learning with differentiable function class approximations.
Its deﬁnition is given in below.
Deﬁnition 1.1 (Parametric Differentiable Function Class) .
Let S , A be arbitrary state, action spaces and a feature map ϕ ( (cid:1) , (cid:1) ) :
S (cid:2) A ! Ψ (cid:26) R m .
The parameter space Θ 2 R d .
Both Θ and Ψ are compact spaces.
Then the parametric function class (for a model f : R d (cid:2) R m ! R ) is deﬁned as
F := f f ( θ, ϕ ( (cid:1) , (cid:1) )) : S (cid:2) A !
R , θ 2 Θ g that satisﬁes differentiability/smoothness condition: 1. for any ϕ 2 R m , f ( θ, ϕ ) is third-time differentiable with respect to θ ; 2. f, ∂ θ f, ∂ 2 θ,θ f, ∂ 3 θ,θ,θ f are jointly continuous for ( θ, ϕ ) .
Remark 1.2.
Differentiable Function Class was recently proposed for studying Off-Policy Evaluation (OPE)
Problem (Zhang et al., 2022a) and we adopt it here for the policy learning task.
Note by the compactness of Θ , Ψ and continuity, there exists constants C Θ , B F , κ 1 , κ 2 , κ 3 > 0 that bounds: k θ k 2 (cid:20) C Θ , j f ( θ, ϕ ( s, a ))
j (cid:20) B F , kr θ f ( θ, ϕ
( s, a ))
k 2 (cid:20) κ 1 , (cid:13)(cid:13) r 2 θθ f ( θ, ϕ ( s, a )) (cid:13)(cid:13) 2 (cid:20) κ 2 , (cid:13)(cid:13) r 3 θθθ f ( θ, ϕ ( s, a )) (cid:13)(cid:13) 2 (cid:20) κ 3 for all θ 2 Θ , s, a 2 S (cid:2) A .
Why consider differentiable function class (Deﬁnition 1.1)?
There are two main reasons why differentiable function class is worth studying for reinforcement learning.
• Due to the limitation of statistical tools, existing analysis in reinforcement learning usually favor basic settings such as tabular MDPs (where the state space and action space are ﬁnite (Azar et al., 2013; 2017; Sidford et al., 2018; Jin et al., 2018; Cui and Yang, 2020; Agarwal et al., 2020; Yin et al., 2021a;b; Li et al., 2020; Ren et al., 2021; Xie et al., 2021b; Li et al., 2022; Zhang et al., 2022b; Qiao et al., 2022; Cui and Du, 2022)) or linear MDPs (Yang and Wang, 2020; Jin et al., 2020b; Wang et al., 2020; Jin et al., 2021b; Ding et al., 2021; Wang et al., 2021a; Min et al., 2021) / linear Mixture MDPs (Modi et al., 2020;
Cai et al., 2020; Zhang et al., 2021a;
Zhou et al., 2021b;a) (where the transition dynamic admits linear structures) so that well-established techniques ( e.g. from linear regression) can be applied.
In addition, subsequent extensions are often based on linear models ( e.g. Linear Bellman Complete models (Zanette et al., 2020) and Eluder dimension (Russo and Van Roy, 2013; Jin et al., 2021a)).
Differentiable function class strictly generalizes over the previous popular choices, i.e. by choosing f ( θ, ϕ ) =
h θ, ϕ i or specifying ϕ to be one-hot representations, and is far more expressive as it encompasses nonlinear approximators.
• Practically speaking, the ﬂexibility of selecting model f provides the possibility for handling a variety of tasks.
For instance, when f is speciﬁed to be neural networks, θ corresponds to the weights of each network layers and ϕ ( (cid:1) , (cid:1) ) corresponds to the state-action representations (which is induced by the network architecture).
When facing with easier tasks, we can deploy simpler model f such as polynomials.
Yet, our statistical guarantee is not affected by the speciﬁc choices as we can plug the model f into Theorem 3.2 to obtain the respective bounds (we do not need separate analysis for different tasks).
1. R ELATED WORKS
Reinforcement learning with function approximation.
RL with function approximation has a long history that can date back to Bradtke and Barto (1996); Tsitsiklis and Van Roy (1996).
Later, [Table] it draws signiﬁcant interest for the ﬁnite sample analysis (Jin et al., 2020b; Yang and Wang, 2019)
[Table]
Since then, people put tremendous efforts towards generalizing over linear function approximations and examples include Linear Bellman complete models (Zanette et al., 2020), Eluder dimension (Russo and Van Roy, 2013; Jin et al., 2021a), linear deterministic Q ⋆ (Wen and Van Roy, 2013) or Bilinear class (Du et al., 2021).
While those extensions are valuable, the structure conditions assumed usually make the classes hard to track beyond the linear case.
For example, the practical instances of Eluder Dimension are based on the linear-in-feature (or its transformation) representations (Section 4.1 of Wen and Van Roy (2013)).
As a comparison, differentiable function class contains a range of functions that are widely used in practical algorithms (Riedmiller, 2005).
Ofﬂine RL with general function approximation (GFA).
Another interesting thread of work considered ofﬂine RL with general function approximation (Ernst et al., 2005; Chen and Jiang, 2019;
Liu et al., 2020; Xie et al., 2021a) which only imposes realizability and completeness/concentrability assumptions.
The major beneﬁt is that the function hypothesis can be arbitrary with no structural assumptions and it has been shown that ofﬂine RL with GFA is provably efﬁcient.
However, the generic form of functions in GFA makes it hard to go beyond worst-case analysis and obtain ﬁnegrained instance-dependent learning bounds similar to those under linear cases.
On the contrary, our results with DFA can be more problem adaptive by leveraging gradients and higher order information.
In addition to the above, there are more connected works.
Zhang et al.
(2022a) ﬁrst considers the differentiable function approximation (DFA) for the off-policy evaluation (OPE) task and builds the asymptotic theory, Fan et al.
(2020) analyzes the deep Q-learning with the speciﬁc ReLu activations, and Kallus and Uehara (2020) considers semi-parametric / nonparametric methods for ofﬂine RL (as opposed to our parametric DFA in 1.1).
These are nice complementary studies to our work.
Our contribution.
We provide the ﬁrst Instance-dependent ofﬂine learning bound under non-linear function approximation.
Informally, we show that (up to a lower order term) the natural complexity measure is proportional to P
[Equation]
[Equation]
s, a [Equation] is the gradient w.r.t.
the parameter θ ⋆ at feature ϕ and Σ
h
= P i g
( s i,h , a i,h ) g
( s i,h , a i,h ) ⊤ is the Fisher information matrix of the observed data at b θ .
This is achieved by analyzing the pessimistic ﬁtted Q-learning (PFQL) algorithm (Theorem 3.2).
In addition, we further analyze its variancereweighting variant, which recovers the variance-dependent structure and can yield faster sample convergence rate.
Last but not least, existing ofﬂine RL studies with tabular models, linear models and GLM models can be directly indicated by the appropriate choice of our model F .
P RELIMINARIES
Episodic Markov decision process.
Let M =
( S , A , P, r, H, d 1 ) to denote a ﬁnite-horizon Markov
Decision Process (MDP), where S is the arbitrary state space and A is the arbitrary action space which can be inﬁnite or continuous.
The transition kernel P h : S (cid:2) A 7! ∆ S ( ∆ S represents a distribution over states) maps each state action ( s h , a h ) to a probability distribution P h ( (cid:1)j s h , a h ) and
P h can be different for different h (time-inhomogeneous).
H is the planning horizon and d 1 is the initial state distribution.
Besides, r : S (cid:2) A 7! R is the mean reward function satisfying {+
 (cid:20)+} r (cid:20) 1 .
A policy π = ( π 1 , . . .
, π H ) assigns each state s h 2 S a probability distribution over actions by mapping s h 7! π h ( (cid:1)j s h ) {+
+} h {+
+}
[ H ] and induces a random trajectory s 1 , a 1 , r 1 , . . .
, s H , a H , r H , s H + with s 1 (cid:24) d 1 , a h (cid:24) π ( (cid:1)j s h ) , s h +1 (cid:24) P h
( (cid:1)j s h , a h ) , 8 h 2 [ H ] .
Given a policy π , the V -value functions and state-action value function (Q-functions) Q πh ( (cid:1) , (cid:1) )
R S × A are deﬁned as: V πh ( s ) =
E π
[Equation]
|
[Equation]
=
[Equation]
[Equation]
s, a ] , ∀ s, a, h ∈ S , A , [ H ] .
The Bellman (optimality) equations follow 8 h 2 [ H ] , s, a 2 S (cid:2)
A : Q πh ( s, a ) = r h ( s, a ) + R S V πh +1 ( · ) dP h ( ·| s, a ) , V πh ( s ) = E a ∼ π h ( s )
[ Q πh ( s, a )] , Q ⋆h ( s, a ) = r h ( s, a ) + R S V ⋆h +1 ( · ) dP h ( ·| s, a ) , V ⋆h ( s ) = max a Q ⋆h ( s, a ) .
We deﬁne Bellman operator P h for any function V 2 R S as P h ( V ) =
r h + R S V dP h , then P h
( V πh +1 ) =
Q πh and P h
( V ⋆h +1 )
=
Q ⋆h .
The performance measure is v π := E [Equation] [Equation]
] = [Equation]
hP [Equation] t i .
Lastly, the induced stateaction marginal occupancy measure for any h 2 [ H ] is deﬁned to be: for any E (cid:18) S (cid:2) A , d πh ( E ) :
=
E [( s h , a h ) 2 E j s 1 (cid:24) d 1 , a i (cid:24) π ( (cid:1)j s i ) , s i (cid:24) P i − 1 ( (
cid:1)j s i − 1 , a i − 1 ) , 1 (cid:20) i (cid:20) h ] and E π,h
[ f ( s, a )] :=
R
S×A f ( s, a ) d πh ( s, a ) dsda .
Ofﬂine Reinforcement Learning.
The goal of Ofﬂine RL is to learn the policy
π ⋆ := arg max π v π using only the historical data D
= { ( s τh , a τh , r τh , s τh +1 ) } h ∈
[ H ] τ ∈ [ K ] .
The data generating behavior policy is denoted as µ .
In the ofﬂine regime, we have neither the knowledge about µ nor the access to further exploration for a different policy.
The agent is asked to ﬁnd a policy b π such that v ⋆ (cid:0) v b π (cid:20) ϵ for the given batch data D and a speciﬁed accuracy ϵ > 0 .
2. A SSUMPTIONS
Function approximation in ofﬂine RL requires sufﬁcient expressiveness of F .
In fact, even under the realizability and concentrability conditions, sample efﬁcient ofﬂine RL might not be achievable (Foster et al., 2021).
Therefore, under the differentiable function setting (Deﬁnition 1.1), we make the following assumptions.
Assumption 2.1 (Realizability+Bellman Completeness) .
The differentiable function class F in Definition 1.1 satisﬁes:
• Realizability: for optimal Q ⋆h , there exists θ ⋆h 2 Θ such that Q ⋆h ( (cid:1) , (cid:1) ) =
f ( θ ⋆h , ϕ ( (cid:1) ))
8 h ; • Bellman Completeness: Let G :=
f V ( (cid:1) ) 2 R S : such that k V k ∞ (cid:20) H g .
Then in this case sup V ∈G inf f ∈F k f (cid:0) P h ( V ) k ∞ (cid:20) ϵ F for some ϵ F (cid:21) 0 .
Realizability and Bellman Completeness are widely adopted in the ofﬂine RL analysis with general function approximations (Chen and Jiang, 2019; Xie et al., 2021a) and Assumption 2.1 states its differentiable function approximation version. There are other forms of completeness, e.g. optimistic closure deﬁned in Wang et al. (2021b).
Data coverage assumption.
Furthermore, in the ofﬂine regime, it is known that function approximation cannot be sample efﬁcient for learning a ϵ -optimal policy without data-coverage assumptions when ϵ is small ( i.e. high accuracy)
(Wang et al., 2021a).
In particular, we consider two types of coverage assumptions and provide guarantees for them separately.
Assumption 2.2 (Concentrability Coverage) .
For any ﬁxed policy π , deﬁne the marginal stateaction occupancy ratio as d πh ( s, a ) /d µh ( s, a ) 8 s, a .
Then the concentrability coefﬁcient is deﬁned as C eff :
= sup π sup h ∈
[ H ] k d πh /d µh k 22 ,d µh , where k g ( (cid:1) , (cid:1) ) k 2 ,d µ := p E d µ [ g ( (cid:1) , (cid:1) ) 2 ] and C eff < 1 .
This is the standard coverage assumption that has has been widely assumed in (Ernst et al., 2005;
Szepesvári and Munos, 2005; Chen and Jiang, 2019; Xie and Jiang, 2020a), and 2.2 is fully characterized by the MDPs.
In addition, we can make an alternative assumption 2.3 that depends on both the MDPs and the function approximation class F .
It assumes a curvature condition for F .
Assumption 2.3 (Uniform Coverage) .
We have 8 h {+
+}
[ H ] , there exists κ > 0 ,
• E µ,h (cid:2) ( f ( θ 1 , ϕ ( · , · )) − f ( θ 2 , ϕ ( · , · )))
2 (cid:3) ≥ κ ∥ θ 1 − θ 2 ∥ 22 , ∀ θ 1 , θ 2 ∈ Θ; ( ⋆ ) • E µ,h (cid:2) ∇ f ( θ, ϕ ( s, a )) · ∇ f ( θ, ϕ ( s, a ))
⊤ (cid:3) ≻ κI, ∀ θ ∈ Θ .
( ⋆⋆ )
In the linear function approximation regime, Assumption 2.3 reduces to 2.4 since ( ⋆ ) and ( ⋆⋆ ) are identical assumptions.
Concretely, if f ( θ, ϕ ) = h θ, ϕ i , then ( ⋆ ) E
[Equation]
Θ ⇔ 2.4 ⇔ ( ⋆⋆ [Equation]
(cid:2) ∇ f ( θ, ϕ ( s, a )) · ∇ f ( θ, ϕ ( s, a ))
⊤ (cid:3) (cid:31) κI .
Therefore, 2.3 can be considered as a natural extension of 2.4 for differentiable class.
We do point that 2.3 can be violated for function class F that is “not identiﬁable” by the data distribution µ ( i.e. , there exists f ( θ 1 ) , f ( θ 2 ) 2 F , θ 1 6 = θ 2 s.t.
E µ,h [( f ( θ 1 , ϕ ( (cid:1) , (cid:1) ))
(cid:0) f
( θ 2 , ϕ ( (cid:1) , (cid:1) )))
2 ] = 0 ).
Nevertheless, there are representative non-linear differentiable classes ( e.g. generalized linear model (GLM)) satisfying 2.3.
Example 2.4 (Linear function coverage assumption (Wang et al., 2021a;
Min et al., 2021;
Yin et al., 2022; Xiong et al., 2022)) .
Σ feature h :=
E µ,h (cid:2) ϕ ( s, a ) ϕ ( s, a ) ⊤ (cid:3) (cid:31) κI 8 h {+
+}
[ H ] with some κ > 0 .
Example 2.5 (ofﬂine generalized linear model (Li et al., 2017; Wang et al., 2021b)) .
For a known feature map ϕ : S (cid:2) A !
B d and link function f : [ (cid:0) 1 , 1] 7! [ (cid:0) 1 , 1] the class of GLM is F GLM :
= f ( s, a ) 7! f ( h ϕ ( s, a ) , θ i ) : θ 2 Θ g satisfying E µ,h (cid:2) ϕ
( s, a ) ϕ
( s, a ) ⊤ (cid:3) (cid:31) κI .
Furthermore, f ( (cid:1) ) is either monotonically increasing or decreasing and 0 < κ (cid:20) j f ′ ( z ) j (cid:20) K < 1 , j f ′′ ( z ) j (cid:20) M <for all j z j (cid:20) 1 and some κ, K, M .
Then F GLM satisﬁes 2.3, see Appendix B.
D IFFERENTIABLE F UNCTION A PPROXIMATION IS P ROVABLY E FFICIENT
In this section, we present our solution for ofﬂine reinforcement learning with differentiable function approximation.
As a warm-up, we ﬁrst analyze the vanilla ﬁtted Q-learning (VFQL, Algorithm 2), which only requires the concentrability Assumption 2.2.
The algorithm is presented in Appendix I.
Theorem 3.1.
Choose 0 < λ (cid:20) 1 / 2 C 2Θ in Algorithm 2.
Suppose Assumption 2.1,2.2.
Then if
K (cid:21) max [Equation]
o , with probability 1 (cid:0) δ , the output b π of VFQL guarantees: v
[Equation]
If the model capacity is insufﬁcient, 3.1 will induce extra error due to the large ϵ F .
If ϵ F ! 0 , the parametric rate 1 √ K can be recovered and similar results are derived with general function approximation (GFA) (Chen and Jiang, 2019).
However, using concentrability coefﬁcient conceals the problem-dependent structure and omits the speciﬁc information of differentiable functions in the complexity measure.
Owing to this, we switch to the stronger “uniform” coverage 2.3 and analyze the pessimistic ﬁtted Q-learning (PFQL, Algorithm 1).
Motivation of PFQL.
The PFQL algorithm mingles the two celebrated algorithmic choices: Fitted
Q-Iteration (FQI) and Pessimism.
However, before going into the technical details, we provide some interesting insights that motivate our analysis.
First of all, the square error loss used in FQI (Gordon, 1999; Ernst et al., 2005) naturally couples with the differentiable function class as the resulting optimization objective is more computationally tractable (since stochastic gradient descent (SGD) can be readily applied) comparing to other information-theoretical algorithms derived with general function approximation ( e.g. the maxmin objective in Xie et al.
(2021a), eqn (3.2)).
4 In particular, FQI with differentiable function approximation resembles the theoretical prototype of neural FQI algorithm (Riedmiller, 2005) and DQN algorithm
(Mnih et al., 2015; Fan et al., 2020) when instantiating the model f to be deep neural networks.
Furthermore, plenty of practical algorithms leverage ﬁtted-Q subroutines for updating the critic step ( e.g. (Schulman et al., 2017; Haarnoja et al., 2018)) with different differentiable function choices.
In addition, we also incorporate pessimism for the design.
Indeed, one of the fundamental challenges in ofﬂine RL comes from the distributional shift .
When such a mismatch occurs, the estimated/optimized Q -function (using batch data D ) may witness severe overestimation error due to the extrapolation of model f
(Levine et al., 2020).
Pessimism is the scheme to mitigate the error / overestimation bias via penalizing the Q-functions at state-action locations that have high uncertainties (as opposed to the optimism used in the online case), and has been widely adopted ( e.g.
(Buckman et al., 2020; Kidambi et al., 2020; al , 2021b)).
Algorithm 1 description.
Inside the backward iteration of PFQL, Fitted Q-update is performed to optimize the parameter (Line 4).
b θ h is the root of the ﬁrst-order stationarity equation P
Kk = [Equation] f
[Equation] θ, ϕ [Equation] k
[Equation]
λθ = 0 and Σ h is the Gram matrix with respect to r θ f j θ = b θ h .
Note for any s, a 2 S (cid:2) A , m ( s, a ) :
= ( r θ f ( b θ h , ϕ
( s, a ))
⊤ Σ − 1 h r θ f ( b θ h , ϕ ( s, a ))) − 1 measures the effective sample size that explored s, a along the gradient direction r θ f j θ =
b θ h , and β/ p m ( s, a ) is the estimated uncertainty at ( s, a ) .
However, the quantity m ( s, a ) depends on b θ h , and b θ h needs to be close to the true θ ⋆h ( i.e. b Q h (cid:25)
f ( b θ h , ϕ ) needs to be close to Q ⋆h ) for the uncertainty estimation
Γ h to be valid, since putting a random θ into m ( s, a ) can cause an arbitrary Γ h that is useless (or might even deteriorate the algorithm).
Such an “implicit” constraint over b θ h imposes the extra difﬁculty for the theoretical analysis due to that general differentiable functions encode nonlinear structures.
Besides, the choice of β is set to be e O ( dH ) in Theorem 3.2 and the extra term e O ( 1 K ) in Γ h is for theoretical reason only.
Algorithm
[Table] +1 (cid:1)(cid:9) K,H k,h =1 .
Require β . Denote ϕ h,k := ϕ ( s k h , a k h ) . 2: Initialization: Set b V H +1 ( · ) ← 0 and λ > 0 .
3: for h = H, H − 1 , . . .
, 1 do 4:
Set b θ h ← arg min [Equation] Θ [Equation]
Set Σ
h ← P [Equation]
∇ [Equation] b
[Equation] λI d .
Set Γ h ( · , · ) ← β q
[Equation]
Set ¯ Q h ( · , · ) ← f ( b θ h , ϕ ( · , · )) − Γ h ( · , · ) 8:
Set b Q h ( · , · ) ← min (cid:8) ¯ Q h ( · , · ) , H − h + [Equation] (cid:9) + 9: Set b π h (
[Equation] ← [Equation] ←
[Equation] 10: end for 11: Output: { b π h } Hh =1 .
Model-Based vs. Model-Free.
PFQL can be viewed as the strict generalization over the previous value iteration based algorithms, e.g. PEVI algorithm (Jin et al.
(2021b), linear MDPs) and the VPVI algorithm (Yin and Wang (2021), tabular MDPs).
On one hand, approximate value iteration (AVI) algorithms (Munos, 2005) are usually model-based algorithms (
for instance the tabular algorithm
VPVI uses empirical model b P for planning).
On the other hand, FQI has the form of batch Qlearning update ( i.e. Q-learning is a special case with batch size equals to one), therefore is more of model-free ﬂavor.
Since FQI is a concrete instantiation of the abstract AVI procedure (Munos, 2007), PFQL draws a uniﬁed view of model-based and model-free learning.
Now we are ready to state our main result for PFQL and the full proof can be found in Appendix D,E,F.
Theorem 3.2.
Let β = 8 dHι and choose 0 < λ (cid:20) 1 / 2 C 2Θ in Algorithm 1.
Suppose Assumption 2.1,2.3 with ϵ F = 0 .
Then if K (cid:21) max [Equation]
with probability 1 (cid:0) δ , for all policy π simultaneously, the output of PFQL guarantees
[Equation]
where ι is a Polylog term and the expectation of π is taken over s h , a h .
In particular, if further
K (cid:21) max f e O ( [Equation] [Equation]
[Equation]
g we have
[Equation]
Here Σ ⋆h = P
Kk =1 r θ f
( θ ⋆h , ϕ ( s kh , a kh ))
r ⊤ θ f ( θ ⋆h , ϕ ( s kh , a kh ))
+ λI d and the deﬁnition of higher order parameter C hot , C ′ hot can be found in List A.
Corollary 3.3 (Ofﬂine Generalized Linear Models (GLM)) .
Consider the GLM function class de- ﬁned in 2.5.
Suppose β, λ, K are deﬁned the same as Theorem 3.2.
ϵ F = 0 .
Then with probability 1 (cid:0) δ , for all policy π simultaneously, PFQL guarantees
[Equation]
PFQL is provably efﬁcient.
Theorem 3.2 veriﬁes PFQL is statistically efﬁcient.
In particular, by
Lemma L.5 we have kr θ f
( θ ⋆h , ϕ ) k Σ − 1 h ≲ 2 κ 1 √ κK , resulting the main term to be bounded by 32 dH 2 κ 1 √ κK that recovers the standard statistical learning convergence rate 1 √ K .
Comparing to Jin et al.
(2021b).
Theorem 3.2 strictly subsumes the linear MDP learning bound in Jin et al.
(2021b).
In fact, in the linear case r θ f ( θ, ϕ ) =
r θ h θ, ϕ i = ϕ and 3.2 reduces to
O ( [Equation]
[Equation]
Instance-dependent learning.
Previous studies for ofﬂine RL with general function approximation (GFA) (Chen and Jiang, 2019; Xie and Jiang, 2020b) are more of worst-case ﬂavors as they usually rely on the concentrability coefﬁcient C .
The resulting learning bounds are expressed in the form 6 O ( V 2max q Cn ) that is unable to depict the behavior of individual instances.
In contrast, the guarantee with differentiable function approximation is more adaptive due to the instancedependent structure P
Hh =1 E π ⋆ [Equation]
[Equation]
This Fisher-Information style quantity characterizes the learning hardness of separate problems explicitly as for different MDP instances M 1 , M 2 , the coupled θ ⋆h,M 1 , θ ⋆h,M 2 will generate different performances via the measure P
[Equation]
E
[Equation]
i [Equation]
Standard worst-case bounds ( e.g. from GFA approximation) cannot explicitly differentiate between problem instances.
Feature representation vs. Parameters.
One interesting observation from Theorem 3.2 is that the learning complexity does not depend on the feature representation dimension m but only on parameter dimension
d as long as function class F satisﬁes differentiability deﬁnition 1.1 (not even in the higher order term).
This seems to suggest, when changing the model f with more complex representations, the learning hardness will not grow as long as the number of parameters need to be learned does not increase.
Note in the linear MDP analysis this phenomenon is not captured since the two dimensions are coupled ( d = m ).
Therefore, this heuristic might help people rethink about what is the more essential element (feature representation vs. parameter space) in the representation learning RL regime ( e.g. low rank MDPs (Uehara et al., 2022)).
We leave the concrete understanding the connection between features and parameters as the future work.
Technical challenges with differentiable function approximation (DFA).
Informally, one key step for the analysis is to bound j f ( b θ h , ϕ ) (cid:0)
f ( θ ⋆h , ϕ ) j .
This can be estimated by the ﬁrst order approximation r f ( b θ h , ϕ ) ⊤ (cid:1) ( b θ h (cid:0) θ ⋆h ) .
However, different from the least-square value iteration (LSVI) objective (Jin et al., 2020b; 2021b), the ﬁtted Q-update (Line 4, Algorithm 1) no longer admits a closed-form solution for b θ h .
Instead, we can only leverage b θ h is a stationary point of
Z h ( θ ) : = P Kk = [Equation]
θ,[-ϕ h,k ) − r h,k − (cid:98) V h +1 (cid:0) s kh +
-]
[Equation]
[Equation]
λ (cid:1) θ (since Z h ( b θ h ) = 0 ).
To measure the difference b θ h (cid:0) θ ⋆h , for any θ 2 Θ , we do the Vector Taylor expansion Z h ( θ ) (cid:0)
Z h ( b θ h ) =
Σ sh ( θ (cid:0) b θ h )
+ R K ( θ ) (where R K ( θ ) is the higher-order residuals) at the point b θ h with
[Equation]
The perturbation term ∆ Σ sh encodes one key challenge for solving b θ h (cid:0) θ ⋆h since it breaks the positive deﬁniteness of Σ sh , and, as a result, we cannot invert the Σ sh in the Taylor expansion of Z h .
This is due to DFA (Deﬁnition 1.1) is a rich class that incorporates nonlinear curvatures.
In the linear function approximation regime, this hurdle will not show up since r 2 θθ f (cid:17) 0 and ∆ Σ sh is always invertible as long as λ > 0 .
Moreover, for the off-policy evaluation (OPE) task, one can overcome this issue by expanding over the population counterpart of Z h at underlying true parameter of the given behavior target policy (Zhang et al., 2022a).
However, for the policy learning task, we cannot use either population quantity or the true parameter θ ⋆h since we need a computable/data-based pessimism Γ h to make the algorithm practical.
3. S KETCH OF THE PFQL A NALYSIS
Due to the space constraint, here we only overview the key components of the analysis.
To begin with, by following the result of general MDP in Jin et al.
(2021b), the suboptimality gap can be bounded by (Appendix D)
P Hh =
1 2 E π [Γ h ( s h , a h )] if j ( P h b V h +1 (cid:0) f ( b θ h , ϕ )) ( s, a ) j (cid:20) Γ h
( s, a ) .
To deal with P h b V h +1 , by Assumption 2.1 we can leverage the parameter Bellman operator T (Deﬁnition D.1) so that P h b V h
+1 =
f ( θ T b V h
+1 , ϕ ) .
Next, we apply the second-order approximation to obtain P h b V h
{++
 (cid:0) f ( b θ h , ϕ ) (cid:25) r f ( b θ h , ϕ ) ⊤ ( θ T b V h +1 (cid:0) b θ h ) + 12 ( θ T b V+}
{+h +
 (cid:0) b θ h ) ⊤ r 2 θθ f ( b θ h , ϕ )( θ T b V+}
{+h +1 (cid:0) b θ h ) . Later, we use (1) to represent Z h ( θ T b V h +1 ) (cid:0) Z h ( b θ h ) = Σ sh ( θ T b V h +
+}
(cid:0) b θ h ) + R K ( θ T b V h +1 )
=
Σ h ( θ T b V h
{++
 (cid:0) b+} θ h )
+ e R K
( θ T b V h +1 ) by denoting e R K ( θ T b V h +1 ) =
∆ Σ sh ( b θ h (cid:0) θ T b V h +1 )
+ R K ( θ T b V h +1 ) .
Now that Σ − 1 h is invertible thus provides the estimation (note Z h ( b θ h ) = 0 )
[Equation]
However, to handle the higher order terms, we need the explicit ﬁnite sample bound for ∥ θ T b V
h +1 − b θ h ∥ 2 (or ∥ θ ⋆h − b θ h ∥ 2 ).
In the OPE literature, Zhang et al.
(2022a) uses asymptotic theory (Prohorovs Theorem) to show the existence of B ( δ ) such that k b θ h (cid:0) θ ⋆h k (cid:20) B ( δ ) √ K .
However, this is insufﬁcient for ﬁnite sample/non-asymptotic guarantees since the abstraction of B ( δ ) might prevent the result from being sample efﬁcient.
For example, if B ( δ ) has the form e H log( 1 δ ) , then e H log( 1 δ ) √ K is an inefﬁcient bound since K needs to be e H /ϵ 2 large to guarantee ϵ accuracy.
To address this, we use a novel reduction to general function approximation (GFA) learning proposed in Chen and Jiang (2019).
We ﬁrst bound the loss objective E µ [ ℓ h ( b θ h )]
(cid:0) E µ [ ℓ h ( θ T b V h +1 )] via a “orthogonal” decomposition and by solving a quadratic equation.
The resulting bound can be directly used to further bound ∥ θ T b V h
+1 − b θ h ∥ 2 for obtaining efﬁcient guarantee e O ( dH √ κK ) .
During the course, the covering technique is applied to extend the ﬁnite function hypothesis to all the differentiable functions in 1.1.
See Appendix G and Appendix D,E,F for the complete proofs.
I MPROVED L EARNING VIA V ARIANCE A WARENESS
In addition to knowing the provable efﬁciency for differentiable function approximation (DFA), it is of great interest to understand what is the statistical limit with DFA, or equivalently, what is the “optimal” sample/statistical complexity can be achieved in DFA (measured by minimaxity criteria)?
Towards this goal, we further incorporate variance awareness to improve our learning guarantee.
Variance awareness is ﬁrst designed for linear Mixture MDPs (Talebi and Maillard, 2018;
Zhou et al., 2021a) to achieve the near-minimax sample complexity and it uses estimated conditional variances Var P ( ·| s,a ) ( V ⋆h +1 ) to reweight each training sample in the LSVI objective.
8 Later, such a technique is leveraged by Min et al.
; Yin et al.
(2022) to obtained the instance-dependent results.
Intuitively, conditional variances σ 2 ( s, a ) :=
Var P ( ·| s,a ) ( V ⋆ h +1 ) serves as the uncertainty measure of the sample ( s, a, r, s ′ )
that comes from the distribution P ( (cid:1)j s, a ) .
If σ 2 ( s, a ) is large, then the distribution P ( (cid:1)j s, a ) has high variance and we should put less weights in a single sample ( s, a, r, s ′ ) , a )) − r − V πh +1 ( s ′ )) ∇ f ( θ, ϕ ( s, a ))] , and the corresponding ∆ Σ sh in ∂∂θ Z h ( θ ) | θ = θ πh is zero by Bellman equation.
We mention Zhang et al.
(2021b) uses variance-aware conﬁdence sets in a slightly different way.
rather than weighting all the samples equally. In the differentiable function approximation regime, the update is modiﬁed to
[Equation]
with σ 2 h ( (cid:1) , (cid:1) ) estimated by the ofﬂine data.
Notably, empirical algorithms have also shown uncertainty reweighting can improve the performances for both online RL (Mai et al., 2022) and
ofﬂine RL (Wu et al., 2021).
These motivates our variance-aware ﬁtted Q-learning (VAFQL) algorithm 3.
Theorem 4.1.
Suppose Assumption 2.1,2.3 with ϵ F = 0 .
Let β = 8 dι and choose 0 < λ (cid:20) 1 / 2 C 2Θ in Algorithm 3.
Then if K (cid:21) K 0 and p d (cid:21) e O ( ζ ) , with probability 1 (cid:0) δ , for all policy π simultaneously, the output of VAFQL guarantees
[Equation]
where ι is a Polylog term and the expectation of π is taken over s h , a h .
In particular, we have
[Equation]
(cid:21) (cid:1) ι + e O ( ¯ C ′ hot K ) . Here Λ ⋆h = P
Kk =1 r θ f
( θ ⋆ h , ϕ h,k ) r ⊤ θ f ( θ ⋆ h , ϕ h,k ) /σ ⋆ h
( s k h , a k h ) 2 + λI d and the σ ⋆ h (cid:1) , (cid:1) ) 2 := max f 1 , Var P
h V ⋆h
+1 ( (cid:1) , (cid:1) ) g .
The deﬁnition of K 0 , ¯ C hot , ¯ C ′
hot
, ζ can be found in List A.
In particular, to bound the error for u h , v h and b σ 2 h , we need to deﬁne an operator J that is similar to the parameter Bellman operator D.1.
The Full proof of Theorem 4.1 can be found in Appendix J.
Comparing to Theorem 3.2, VAFQL enjoys a net improvement of the horizon dependence since
Var P ( V ⋆h ) (cid:20) H 2 .
Moreover, VAFQL provides better instance-dependent characterizations as the main term is fully depicted by the system quantities except the feature dimension d .
For instance, when the system is fully deterministic (transition P h ’s are deterministic),
σ ⋆h (cid:25) Var P
h V ⋆h
+1 ( (cid:1) , (cid:1) ) (cid:17)(if ignore the truncation) and Λ ⋆ − 1 ! 0 .
This yields a faster convergence with rate O ( 1 K ) .
Lastly, when reduced to linear MDPs, 4.1 recovers the results of Yin et al.
except an extra p d .
On the statistical limits.
To complement the study, we incorporate a minimax lower bound via a reduction to Zanette et al.
The following theorem reveals we cannot improve over Theorem 4. by more than a factor of p d in the most general cases.
The full discussion is in K.
Theorem 4.2 (Minimax lower bound) .
Specifying the model to have linear representation f = h θ, ϕ i .
There exist a pair of universal constants c, c ′ > 0 such that given dimension d , horizon H and sample size K > c ′ d 3 , one can always ﬁnd a family of MDP instances such that for any algorithm b π (where Λ ⋆,ph = E
[Equation]
C ONCLUSION , L IMITATION AND F UTURE D IRECTIONS
In this work, we study ofﬂine RL with differentiable function approximation and show the sample efﬁcient learning.
We further improve the horizon dependence via a variance aware variant.
However, the dependence of the parameter space still scales with d (whereas for the linear case it is p d ), and this is due to applying covering argument for the rich class of differentiable functions.
For large deep models, the parameter dimension is huge, therefore it would be interesting to know if certain algorithms can further improve the parameter dependence, or whether this d is essential.
Also, how to relax uniform coverage 2.3 is unknown under the current analysis.
In addition, understanding the connections between the differentiable function approximation and overparameterized neural networks approximation Nguyen-Tang and Arora (2023); Xu and Liang (2022) is important.
We leave these open problems as future work.
Lastly, the differentiable function approximation setting provides a general framework that is not conﬁned to ofﬂine RL.
Understanding the sample complexity behaviors of online reinforcement learning (Jin et al., 2020b; Wang et al., 2021b), reward-free learning (Jin et al., 2020a; Wang et al., 2020) and representation learning (Uehara et al., 2022) might provide new and uniﬁed views over the existing studies. A CKNOWLEDGMENTS
The authors would like to thank anonymous reviewers for their helpful suggestions.
Ming Yin would like to thank Chi Jin for the helpful suggestions regarding the assumption for differentiable function class and Andrea Zanette, Xuezhou Zhang for the friendly discussion.
Mengdi Wang gratefully acknowledges funding from Ofﬁce of Naval Research (ONR) N00014-21-1-2288, Air Force Ofﬁce of
Scientiﬁc Research (AFOSR) FA9550-19-1- and NSF 19-589, CMMI-1653435.
Ming Yin and Yu-Xiang Wang are gratefully supported by National Science Foundation (NSF) Awards # and #2003257. Appendix A N OTATION L IST
[Table]
B F URTHER I LLUSTRATION THAT G ENERALIZED L INEAR M ODEL E XAMPLE SATISFIES 2.
Recall the deﬁnition in 2.5, then:
For ( ⋆⋆ ) ,
[Equation] (cid:2) ϕ ( (cid:1) , (cid:1) )
(cid:1) ϕ
( (cid:1) , (cid:1) ) ⊤ (cid:3) (cid:31) κ 3 I, 8 θ 2 Θ
For ( ⋆ ) , by Taylor’s Theorem,
E µ,h h ( f ( θ 1 , ϕ
( (cid:1) , (cid:1) ))
(cid:0) f ( θ 2 , ϕ ( (cid:1) , (cid:1) )))
i = E µ,h [ f ′ ( θ s,a , ϕ ( (cid:1) , (cid:1) ))
2 ( θ 1 (cid:0) θ 2 ) ⊤ ϕ
( (cid:1) , (cid:1) )
ϕ
( (cid:1) , (cid:1) )
⊤ ( θ 1 (cid:0) θ 2 )] (cid:21) κ 2 E µ,h [( θ 1 (cid:0) θ 2 ) ⊤ ϕ ( (cid:1) , (cid:1) )
ϕ ( (cid:1) , (cid:1) ) ⊤ ( θ 1 (cid:0) θ 2 )]
= κ 2 ( θ 1 (cid:0) θ 2 )
⊤ E µ,h [ ϕ ( (cid:1) , (cid:1) ) ϕ ( (cid:1) , (cid:1) )
⊤ ]( θ 1 (cid:0) θ 2 ) (cid:21) κ 3 k θ 1 (cid:0) θ 2 kand choose κ 3 as κ in 2.3.
C O N THE COMPUTATIONAL COMPLEXITY
For storage of Pessimistic Fitted Q-learning, at each time step h 2 [ H ] in Algorithm 1, we need to store b θ h , Σ h and r f ( b θ h , ϕ h,k ) .
Therefore, the total space complexity is O ( dH + d 2 H + dKH ) .
For computation, assuming b θ h is solved via SGD and let M denote the number of gradient steps, then the complexity is dominated by computing b θ h , Σ h and Σ − 1 h , which results in O ( MH + KdH + d 3 H ) complexity (where H comes from h = H, . . .
The space complexity and computational complexity for VAFQL has the same order as PFQL except that the constant factors are larger.
D S OME BASIC CONSTRUCTIONS
First of all, Recall in the ﬁrst-order condition, we have
[Equation]
Therefore, if we deﬁne the quantity Z h ( (cid:1) , (cid:1) ) 2 R d as Z
[Equation]
, then we have (recall b θ h 2 Int(Θ) )
[Equation]
= [Equation] In addition, according to Bellman completeness Assumption 2.1, for any bounded V ( (cid:1) ) 2 R S with k
V k ∞ (cid:20) H , inf f ∈F k f (cid:0) P h ( V ) k ∞ (cid:20) ϵ F , 8 h (recall P h ( V ) =
r h + R S V dP h ).
Therefore, we can deﬁne the parameter Bellman operator T as follows.
Deﬁnition D.1 (parameter Bellman operator) .
By the Bellman completeness Assumption 2.1, for any k V k ∞ (cid:20) H , we can deﬁne the parameter Bellman operator T : V ! θ T V 2 Θ such that
[Equation]
Denote δ V :=
f ( θ T V , ϕ ) (cid:0) P h ( V ) , then we have k f ( θ T V , ϕ ) (cid:0) P h ( V ) k ∞ =
k δ V k ∞ (cid:20) ϵ F .
In particular, by realizability Assumption 2.1 it holds θ T V
⋆h
+
= θ ⋆h
and this is due to f ( θ T V ⋆h +1 , ϕ ) =
P h
( V ⋆h +1 )
=
V ⋆h =
f ( θ ⋆h , ϕ ) .
D. S UBOPTIMALITY DECOMPOSITION
Denote ι h ( s, a ) := P h b V
h +1 ( s, a ) (cid:0) b Q h ( s, a ) , by Jin et al.
(2021b)
we have the following decomposition.
Lemma D.2 (Lemma 3.1 of Jin et al.
(2021b)) .
Let b π = f b π h g
Hh =1 a policy and b Q h be any estimates with b V h = h b Q h ( s, (cid:1) ) , b π h ( (cid:1) j s ) i A .
Then for any policy π , we have
[Equation]
In particular, if we choose b π h ( (cid:1)j s ) :
= arg max π h b Q h ( s, (cid:1) ) , π ( (cid:1) j s ) i A , then
[Equation]
Lemma D.3.
Let b P h be the general estimated Bellman operator.
Suppose with probability 1 (cid:0) δ , it holds for all h, s, a 2 [ H ] (cid:2) S (cid:2)
A that j ( P h b V h
+1 (cid:0) b
P h b V h +1 )
( s, a ) j (cid:20) Γ h ( s, a ) , then it impliess, a, h 2 S (cid:2)
A (cid:2)
[ H ] , 0 (cid:20) ζ h ( s, a ) (cid:20) 2Γ h
( s, a ) .
Furthermore, it holds for any policy π simultaneously, with probability 1 (cid:0) δ ,
[Equation]
Proof of Lemma D.3.
This is a generic result that holds true for the general MDPs and was ﬁrst raised by Theorem 4.2 of Jin et al.
(2021b).
Later, it is summarized in Lemma C.1 of Yin et al.
With Lemma D.3, we need to bound the term jP
h b V h +1 ( s, a ) (cid:0) b P h b
V h +1 ( s, a ) j .
[Equation]
Throughout this section, we suppose ϵ F = 0 , i.e. f ( θ T V , ϕ ) =
P h ( V ) .
According to the regression oracle (Line 4 of Algorithm 1), the estimated Bellman operator b P h maps b V h +1 to b θ h ,
i.e. b P h b V h +1 =
f ( b θ h , ϕ ) .
Therefore (recall Deﬁnition D.1)
[Equation]
where we apply the ﬁrst-order Taylor expansion for the differentiable function f at point b θ h and
Hot h, 1 is a higher-order term.
Indeed, the following Lemma E.1 bounds the Hot h, 1 term with e
O ( 1 K ) .
Lemma E.1.
Recall the deﬁnition (from the above decomposition)
Hot h, 1 := f
[Equation]
b V h +1 , ϕ
( s, a ))
(cid:0) f [Equation] b θ h , ϕ ( s, a )) (cid:0) r f [Equation]
b
[Equation]
h , ϕ ( s, a ) [Equation] T b V [Equation]
+ [Equation] b θ h (cid:17) , then with probability 1 (cid:0) δ , j
[Equation]
Proof of Lemma E.1.
By second-order Taylor’s Theorem
[Equation]
there exists a point ξ (lies in the line segment of b θ h and θ T b V h +1 ) such that
[Equation]
Therefore
[Equation]
by directly applying Theorem G
[Equation] [Equation]
with probability 1 (cid:0) δ , for all h 2 [ H ] ,
[Equation]
E. A NALYZING r f ( b θ h , ϕ ( s, a ))
[Equation]
VIA Z h .
From (3) and Lemma E.1, the problem further reduces to bounding r f ( b θ h , ϕ
( s, a )) (cid:16)
[Equation]
To begin with, we ﬁrst provide a characterization of θ T b V
h +1 (cid:0) b θ h .
Indeed, by ﬁrst-order Vector Taylor expansion (Lemma L.1), we have (note Z h ( b θ h j b V h +1 )
= 0 )
for any θ 2 Θ ,
[Equation]
[Equation]
where R K ( θ ) is the higher-order residuals and Σ
sh :=
∂∂θ Z h ( θ j b θ h +1 ) (cid:12)(cid:12)(cid:12) θ = b θ h with Σ sh :=
[Equation]
∂θZ h ( θ [Equation]
b
[Equation] +
[Equation]
) (cid:12)(cid:12)(cid:12)(cid:12)
θ [Equation] X [Equation]
i r
[Equation]
(cid:1) θ !
[Equation]
b
[Equation]
h =
K X k = n(cid:16)
[Equation]
o | {z } :
[Equation]
+
K X k
= r θ [Equation] ⊤ θ [Equation]
[Equation]
h
[Equation]
| [Equation] [Table] Note ∆ Σ sh is not desirable since it could prevent Σ sh from being positive-deﬁnite (and it could cause Σ sh to be singular).
Therefore, we ﬁrst deal with ∆ Σ sh in below.
Lemma E.2.
With probability 1 (cid:0) δ , for all h 2 [ H ] , [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
Proof of Lemma E.2.
Step1: We prove for ﬁxed ¯ θ 2 Θ , with probability 1 (cid:0) δ , for all h {+
+}
[ H ] ,
[Equation]
K
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
K .
Indeed, we have
[Equation]
On one hand, by Theorem G.2 with probability 1 (cid:0) δ/ 2 for all h {+
+}
[ H ]
[Equation]
K
K X
[Equation]
On other hand, recall the deﬁnition of T
[Equation]
we have
[Equation]
1 ) (cid:12)(cid:12)(cid:12) s kh , a kh i(cid:17) (cid:1) r 2 θθ f (¯ θ, ϕ h,k ) = (cid:16) ( P h b
V h +1 ) ( s kh , a kh ) (cid:0) ( P h b V h +1 ( s kh +1 ))
( s kh , a kh ) (cid:17) (cid:1) r 2 θθ f (¯ θ, ϕ h,k ) = 0 .
Also, since (cid:13)(cid:13)(cid:13)(cid:16) f
( θ T b V h +1 , ϕ h,k ) (cid:0) r h,k (cid:0) b V h
+1 ( s kh +1 ))
(cid:17) (cid:1) r 2 θθ f (¯ θ, ϕ h ) (cid:13)(cid:13)(cid:13) 2 (cid:20) Hκ 2 , denote σ 2 := K (cid:1)
H 2 κ 22 , then by Vector Hoeffding’s inequality (Lemma L.2),
[Equation]
t
[Equation]
[Equation]
:= δ which is equivalent to
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] g , then by law of total expectation P ( A ) = E [ 1 A ] = E [ E [
[Equation]
jf s
[Equation] [Equation]
=
[Equation] [Equation]
E [ P [ A jf
[Equation]
, a
[Equation]
=1 ]] [Equation]
E [1 (cid:0) δ ] = 1 (cid:0) δ , i.e. with probability at least 1 (cid:0) δ/ 2 (and a union bound),
[Equation]
Using above and equation 6, equation 7 and a union bound, w.p. 1 (cid:0) δ , for all h {+
+}
[ H ] , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
K
Step2: we ﬁnish the proof of the lemma.
Consider the function class n
[Equation]
o
[Equation]
then by triangular inequality j
[Equation]
sup
[Equation]
By Lemma L.8, the covering number C of the ϵ -net of the above function class satisﬁes log C (cid:20) d log(1+ 2 C Θ Hκ 3 ϵ ) .
By choosing ϵ = 1 /K , by a union bound over C cases we obtain for all h {+
+}
[ H ]
[Equation]
Combing Lemma E.2 and Theorem G.2 (and a union bound), we directly have
Corollary E.3.
With probability 1 (cid:0) δ ,
[Equation]
[Equation]
K
[Equation]
Here e O absorbs all the constants and Polylog terms.
Now we select θ = θ T b V h +1 in equation 4, and denote e R K
( θ T b V h +1 ) =
∆ Σ sh ( b θ h (cid:0) θ T b V h +1 )
+
R K ( θ T b V h +1 ) , then equation 4 is equivalent to
[Equation]
R
[Equation]
Note λ > 0 implies Σ h is invertible, then we have (recall Z h ( b θ h j b θ h +1 ) = 0 )
[Equation]
Plug it back to equation 3 to get
[Equation]
We will bound second term Hot 2 to have higher order O ( 1 K ) in Section E.5 and focus on the ﬁrst term.
By direct decomposition,
[Equation]
[Equation]
] | {z } :
[Equation]
f
[Equation]
] | {z } :
[Equation]
[Equation] # | {z } :
[Equation]
E. B OUNDING THE TERM I
We ﬁrst bound the term I 3 .
We have the following Lemma.
Lemma E.4.
For any ﬁxed V ( (cid:1) ) 2 R S with k V k ∞ (cid:20) H and any ﬁxed θ such that k θ T V (cid:0) θ k 2 (cid:20) q 36 H 2 (log( H/δ )+ C d, log K )+2 λC 2Θ κK . Let e I 3 :=
[Equation]
f
[Equation]
b θ
[Equation]
, ϕ ( s, a ) [Equation] Σ −
[Equation]
h
[Equation]
) # , and if K (cid:21) max n 512 κ 41 κ(cid:16) log( 2 d δ ) + d log(1 + 4 κ 1 D 2 κ 2 C Θ K 3 λ) (cid:17) , 4 λ κ o , then with probability 1 (cid:0) δ , (where D = max f κ 1 , q (144 dH 2 κ 22 ( H 2 log( H/δ )+ C d, log K )
+8 dH 2 κ 22 λC 2Θ )log
( d/δ ) κ g )
j e I 3 j (cid:20) 4 κr (144 dH 2 κ 22 ( H 2 log( H/δ ) + C d, log K ) + 8 dH 2 κ 22 λC 2Θ ) log( d/δ ) κ
K + O ( 1 K 3 / 2 ) .
Proof of Lemma E
[Table]
Indeed, with probability 1 (cid:0) δ/ 2 ,
[Table]
[Equation]
[Table]
[Equation]
where, under the condition K (cid:21) max n
[Equation]
) (cid:17)
[Equation]
λκ o , we applied Lemma L.5 .
Next, on one hand,
[Equation]
On the other hand,
[Equation]
Therefore by Vector Hoeffding’s inequality (Lemma L.2) (also note the condition for boundedness (cid:13)(cid:13)(cid:0)
[Equation]
[Equation]
probability
[Equation]
and this implies with probability 1 (cid:0) δ/ 2 ,
[Equation]
K X k =
[Equation]
r
[Equation]
κ choose u = P
Kk =1 (cid:0) f
( θ T V , ϕ h,k ) (cid:0) r h,k (cid:0) V ( s kh +1 ) (cid:1) (cid:1) ( r θ f ( θ T V , ϕ h,k ) (cid:0) r θ f
( θ, ϕ h,k )) in
Lemma L.5, by a union bound we obtain with probability 1 (cid:0)
δ |
[Equation]
[Equation]
# (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤
[Equation]
√
[Equation]
(cid:19)
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) K X k = (cid:16) f
[Equation]
[Equation]
r h
[Equation]
Lemma E.5.
Under the same condition as Lemma E.4.
With probability 1 (cid:0) δ ,
[Equation]
Here
[Equation]
absorbs Polylog terms.
Proof of Lemma E
[Equation]
Deﬁne
[Equation]
then
[Equation]
[Table] [Equation] Let C a be the ϵ/ 3 (2 κ 21 + Hκ 2 ) K -covering net of f θ : k θ k 2 (cid:20) C Θ g , C V be the ϵ 6 κ 1 K -covering net of V deﬁned in Lemma L.9 and C b be the ϵ 3 Hκ 2 K -covering net of f θ : k θ k 2 (cid:20) C Θ g , then by Lemma L. and Lemma L.9, log
[Equation]
K ϵ
[Equation]
K ϵ ) log C {+V (cid:20) d log (cid:18) 
+}
+ 288 C Θ [Equation] [Equation] C
[Equation]
ϵ
[Equation] (cid:19)
[Equation]
d
[Equation]
log 1 +p dBκ 41 Kϵ! . Further notice with probability 1 (cid:0) δ/ 2 (by Lemma L.5), for all ﬁxed sets of parameters θ, V satisﬁes k θ
[Equation]
q
[Equation]
simultaneously,
[Equation]
−
[Equation]
h
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
h and (cid:13)(cid:13)(cid:13) θ T b V h +
[Equation]
(cid:0) b θ [Equation] (cid:13)(cid:13)(cid:13)
[Equation]
(cid:20) q
[Equation]
with probability 1 (cid:0) δ/ 2 by Theorem G.2.
Now, choosing ϵ =
O (1 /K 2 ) and by Lemma E.4 and union bound over covering instances
, we obtain with probability 1 (cid:0) δ | I [Equation] κ
[Equation]
K
[Equation]
O ( 1 K 3 / 2 ) .
E. B OUNDING THE SECOND TERM I
In this section, we bound the term
[Equation]
The following Lemma shows that I 2 is a higher-order error term with rate e O
( 1 K ) .
Lemma E.6 (Bounding I 2 ) .
If K satisﬁes K (cid:21) 512 κ
[Equation] [Equation]
, and K (cid:21) 4 λ/κ , then with probability 1 (cid:0) δ
[Equation]
Here e O absorbs constants and Polylog terms.
Proof of Lemma E.6.
Step1.
Deﬁne η k
( V ) :
= f ( θ T V , ϕ h,k ) [Equation] f ( θ
[Equation]
V ⋆h +1 , ϕ h,k ) (cid:0) V ( s kh +1 )
+
V ⋆h +1 ( s kh +1 ) and let k V ( (cid:1) ) k ∞ (cid:20) H be any ﬁxed function such that sup s kh ,a kh ,s kh +1 j η k ( V ) j (cid:20) e O ( κ 1 Hq d 2 κK ) , i.e. arbitrary ﬁxed V function in the neighborhood (measured by η k ) of V ⋆h +1 .
Then by deﬁnition of T it holds E [ η k ( V, θ ) j s kh , a kh ] = 0 .
Let the ﬁxed θ 2 Θ be arbitrary and deﬁne x k ( θ ) =
r θ f ( θ, ϕ h,k ) .
Next, deﬁne G h ( θ ) =
P Kk =1 r f ( θ, ϕ ( s kh , a kh )) (cid:1) r f ( θ, ϕ ( s kh , a kh ))
⊤ + λI d , since k x k k 2 (cid:20) κ 1 and j η k j (cid:20) e O ( κ 1 Hq d 2 κK ) , by self-normalized Hoeffding’s inequality (Lemma L.3), with probability 1 (cid:0) δ (recall t := K in Lemma L.3), (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) K X
[Equation]
=
[Equation]
) η [Equation] ( V ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation]
[Equation]
λδ (cid:19) .
Step2.
Deﬁne h
( V, θ ) :
= P Kk =1 x k ( θ ) η k ( V ) and H ( V, θ ) :
= (cid:13)(cid:13)(cid:13)P Kk =1 x k ( θ ) η k ( V ) (cid:13)(cid:13)(cid:13)
G h ( θ ) − 1 , then note by deﬁnition j η k ( V ) j (cid:20) 2 H , which implies k h ( V, θ ) k 2 (cid:20) 2 KHκ 1 and j η k ( V 1 ) (cid:0) η k
( V 2 ) j (cid:20)
jP h
V 1 (cid:0) P h V 2 j + k V 1 (cid:0) V 2 k ∞ (cid:20) 2 k V 1 (cid:0) V 2 k ∞ and
[Equation]
[-∥ x k ( θ 1 ) − x k ( θ 2 ) ∥ 
 + κ 1 | η k ( V 1 )-]Furthermore, (cid:13)(cid:13)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] h(cid:13)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
i
[Table]
[Equation]
[Table]
[Equation]
2 κ 1 κ
[Equation]
K
[Equation]
∥ θ
[Equation]
θ
[Equation]
All the above imply
[Equation]
Then a ϵ -covering net of f H ( V, θ ) g can be constructed by the union of ϵ4 (cid:16) 4 p K
[Equation]
H
[Equation]
[Equation]
[Equation]
+ q 8 K 3 H 2 κ 31 κ
[Equation]
[Equation] 2 (cid:17) 2 -covering net of f θ 2 Θ g and ϵ4(4 p [Equation] 3 κ 21 H
[Equation]
) 2 -covering net of V in Lemma L.9.
The covering number N ϵ satisﬁes
[Equation]
[Equation] A . [Equation]
K X
[Equation]
and with probability 1 (cid:0) δ j
[Equation]
where the second inequality uses θ T V ⋆h
+
= θ ⋆h
and the third inequality uses Theorem G.2 and Theorem G.3.
The last equal sign is due to C d, log K (cid:20) e O ( d 2 ) (recall Lemma G.1).
Now choosing ϵ
=
O (1 /K ) in Step2 and union bound over both equation 9 and covering number in
Step2, we obtain with probability 1 (cid:0) δ ,
[Equation]
where we absorb all the Polylog terms.
Meanwhile, by Lemma L.5 with probability 1 (cid:0)
[Equation]
Finally
[Equation]
by equation 10 and equation 11 and a union bound, we have with probability 1 (cid:0) δ , [Equation] |
[Equation]
where the ﬁrst inequality is CauchySchwarz inequality
[Equation]
E. B OUNDING THE MAIN TERM I
In this section
[Equation]
we bound the dominate term
[Equation]
First of all, by CauchySchwarz inequality, we have
[Equation]
Then we have the following Lemma to bound I 1 .
Lemma E.7.
With probability 1 (cid:0) δ ,
[Equation]
where C δ, log K only contains Polylog terms.
Proof of Lemma E.7.
Step1.
Let the ﬁxed θ 2 Θ be arbitrary and deﬁne x k ( θ ) =
r θ f ( θ, ϕ h,k ) .
Next, deﬁne G h ( θ ) =
P Kk =1 r f ( θ, ϕ ( s kh , a kh )) (cid:1) r f ( θ, ϕ ( s kh , a kh ))
⊤ + λI d , then k x k k 2 (cid:20) κ 1 .
Also denote η k :=
f ( θ T V ⋆h +1 , ϕ h,k ) (cid:0) r h,k (cid:0) V ⋆h +
( s kh +1 ) , then E [ η k j s kh , a kh ] = 0 and j η k j (cid:20) H .
Now by self-normalized Hoeffding’s inequality (Lemma L.3), with probability 1 (cid:0) δ (recall t :=
K in
Lemma L.3),
[Equation]
Step2.
Deﬁne h ( θ ) :=
P Kk
=
x k ( θ ) η k and H ( θ ) :
= (cid:13)(cid:13)(cid:13)P Kk =1 x k ( θ ) η k (cid:13)(cid:13)(cid:13) G h ( θ ) − 1 , then note by deﬁnition j η k j (cid:20) H , which implies k h ( θ ) k 2 (cid:20) KHκ 1 and by x k ( θ 1 ) (cid:0) x k ( θ 2 ) =
r 2 θθ f ( ξ, ϕ ) (cid:1) ( θ 1 (cid:0) θ 2 ) , k h ( θ 1 ) (cid:0) h ( θ 2 ) k 2 (cid:20) K max k
( H k x k ( θ 1 ) (cid:0) x k ( θ 2 ) k 2 ) (cid:20) HKκ[-
-]
2 k θ 1 (cid:0) θ 2 k 2 .
Furthermore, (cid:13)(cid:13)
G h ( θ 1 ) − 1 (cid:0) G h ( θ 2 ) − 1 (cid:13)(cid:13) 2 (cid:20) (cid:13)(cid:13)[-G h ( θ 1 ) − 1 (cid:13)(cid:13) 
-]
G h ( θ 1 ) − 1 (cid:13)(cid:13)
[Equation]
All the above imply
[Equation]
Then a ϵ -covering net of f H ( θ ) g can be constructed by the union of ϵ 2 (cid:16) p 4 K 2 H
[Equation]
1 κ
[Equation]
λ + p
[Equation]
K 3 H 2 κ 31 κ 2 /λ 2 (cid:17) 2 -covering net of f θ 2 Θ g .
By Lemma L.8, the covering number N ϵ satisﬁes
[Equation]
[Table]
[Equation]
Now choosing ϵ
=
O (1 /K ) in Step2 and union bound over the covering number in Step
[Equation]
we obtain with probability 1 (cid:0) δ ,
[Equation]
where we absorb all the Polylog terms.
Combing above with equation
[Equation]
we obtain with probability 1 (cid:0) δ ,
[Equation]
where C δ, log K only contains Polylog terms.
E. A NALYZING Hot 2 IN EQUATION
Lemma E.8.
Recall Hot 2 :=
r f ( b
[Equation]
, ϕ
[Equation]
s, a )) Σ −
[Equation] h [Equation] λθ
[Equation]
i .
If the number of episode K satisﬁes K (cid:21) max n
[Equation]
dδ )
+ d log(1 +
[Equation]
[Equation]
) (cid:17) , 4 λκ o , then with probability 1 (cid:0) δ ,
[Equation]
K
A where e O absorbs all the constants and Polylog terms.
Proof of Lemma E.8.
Step1: we ﬁrst show with probability 1 (cid:0) δ
[Equation]
Recall by plug in θ T b V h +1 in equation 4, we have [Equation]
[Equation]
and by second-order Taylor’s Theorem we have
[Equation]
[Table]
[Equation]
Therefore
[Equation]
we can bound κ z 2 with κ z 2 (cid:20) ( Hκ 3 +3 κ 1 κ 2 ) K and this implies with probability 1 (cid:0) δ/ 2 ,
[Equation]
And by Corollary E.3 with probability [Equation] e
[Equation]
Therefore [Table] by Lemma L.5 and a union bound with probability 1 (cid:0) δ ,
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] K
A where e O absorbs all the constants and Polylog terms.
Here the last inequality uses bound for (cid:13)(cid:13)(cid:13)
R K ( θ T b V h +1 )
(cid:13)(cid:13)(cid:13) 2 and
[Equation]
Step2: By Lemma L.5, with probability 1 (cid:0) δ ,
[Equation]
F P ROOF OF T HEOREM 3.
Now we are ready to prove Theorem 3.2.
In particular, we prove the ﬁrst part.
Also, recall that we consider the exact Bellman completeness ( ϵ F = 0 ).
F. T HE FIRST PART
Proof of Theorem 3.2 (ﬁrst part).
First of all, from the previous calculation (3), (8), we have (cid:12)(cid:12)(cid:12)
[Table]
[Equation]
Now by Lemma E.5, Lemma E.6, Lemma E.7, Lemma E.8 and Lemma E.1 (and a union bound), with probability 1 (cid:0) δ , [Equation] Finally
[Equation]
Plug the above into Lemma D
[Equation]
3, by a union bound over all h 2 [ H ] , we have with probability 1 (cid:0) δ , for any policy π ,
[Equation]
where ι = C δ, log K only contains Polylog terms and
[Equation]
F. T HE SECOND PART
Next we prove the second part of Theorem 3.2. Proof of Theorem 3.2 (second part)
[Equation]
Step1.
Choose π = π
⋆ in the ﬁrst part, we have
[Equation]
Next, by the triangular inequality of the norm to obtain
[Equation]
since with probability 1 (cid:0) δ ,
[Equation]
where the last inequality uses part three of Theorem G
[Equation]
Then by a union bound and Lemma L.5,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
O κ
[Equation]
K ! Second, by Lemma L.6 with probability 1 (cid:0) δ
[Equation]
Σ ⋆h
[Equation]
This implies
[Equation]
and also by Weyl’s spectrum theorem and under the condition K (cid:21) 128 κ 41 log(2 d/δ ) κ, with probability 1 (cid:0) δ λ [Equation] Σ ⋆h K
[Equation]
λ min (cid:0) E µ [ r [Equation] f ( [Equation]
⋆h , ϕ ) r
[Equation]
f
[Equation]
⋆h , ϕ ) ⊤
[Equation] (cid:1) [Equation] [Equation]
(cid:19) [Equation] (cid:21) [Equation]
p
[Equation]
K (cid:18) log 2 d δ (cid:19) 1 /(cid:21)
κ then (cid:13)(cid:13)(cid:13) ( Σ ⋆h K ) − 1 (cid:13)(cid:13)(cid:13) (cid:20) 2 κ .
Similarly, with probability 1 (cid:0) δ , (cid:13)(cid:13)
( Σ h K ) − 1 (cid:13)(cid:13) (cid:20) 2 κ .
Then by Lemma L.7, ∥∇ θ f
[Equation]
⋆h , ϕ (
[Equation]
, a ))
[Equation]
K Σ − 1 h ≤
[Equation]
as long as K (cid:21) e O ( ( κ 21 + λ ) 2 κ 22 κ 21 H 4 d 2 κ) .
The above is equivalently to
[Equation]
Combining Step1, Step2 and a union bound, we have with probability 1 (cid:0) δ ,
[Equation]
[Table] [Table]
[Equation]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
) (cid:21)
[Table]
[Equation]
) where C ′ hot =
C hot + κ 1 κ 2 H 4 d 2 κ 3 /.
G P ROVABLE E FFICIENCY BY REDUCTION TO G ENERAL F UNCTION A PPROXIMATION
In this section, we bound the accuracy of the parameter difference (cid:13)(cid:13)(cid:13)b θ h (cid:0) θ T b V h +1 (cid:13)(cid:13)(cid:13) 2 via a reduction to General Function Approximation scheme in Chen and Jiang (2019).
Recall the objective
[Equation]
Then by deﬁnition, b θ h :=
arg min θ ∈ Θ ℓ h ( θ ) and θ T b V h
+1 satisﬁes f ( θ T b V h
+1 , ϕ )
= P h b V h
{++
+} + δ b V h
+1 .
Therefore, in this case, we have the following lemma:
Lemma G.1.
Fix h
2 [ H ] .
With probability 1 (cid:0) δ ,
E µ [ ℓ h ( b θ h )] (cid:0) E µ [ ℓ h
[Equation]
θ T b V h
[Equation]
(cid:20)
[Equation]
where the expectation over µ is taken w.r.t.
( s kh , a kh
[Equation]
s kh [Equation] [Equation] 1 , ..., K only (i.e., ﬁrst compute E µ [ ℓ h ( θ )] for a ﬁxed θ , then plug-in either b θ {+h +
+}
[-h +
-] or θ T b V h +1 ).
Here
C d
[Equation]
log
[Equation]
κ
[Equation]
d 2 log (cid:16)
[Equation]
[Equation]
λ (cid:17) .
Proof of Lemma G.1.
Step1: we ﬁrst prove the case where λ = 0 .
Indeed, ﬁx h 2 [ H ] and any function V ( (cid:1) ) 2 R S .
Similarly, deﬁne f V ( s, a ) :
= f ( θ T V , ϕ ) =
P h V + δ V .
For any ﬁxed θ 2 Θ , denote g ( s, a ) =
f ( θ, ϕ
( s, a )) .
Then deﬁne
[Equation]
Since all episodes are independent of each other,
X
[Equation]
f
[Equation]
:= X [Equation] g
[Equation] s k h , a k h ) , V ( s k h +1 ) , f V ( s k h , a k h )) are independent r.v.s and it holds
[Equation]
[Equation]
Next, the variance of X is bounded by:
[Equation]
where the step ( (cid:3) ) comes from E
[Equation]
where the last step uses law of total expectation and the deﬁnition of f V
[Equation]
Therefore, by Bernstein inequality
[Equation]
with probability 1 (cid:0) δ ,
[Equation]
Now, if we choose g ( s, a ) :
= f ( b θ h , ϕ ( s, a )) , then b θ h minimizes ℓ h ( θ ) , therefore, it also minimizes 1 K
P Kk =1 X
i ( θ, b V h
+1 , f b V h +1 )
and this implies
[Equation]
Therefore, we obtain
E µ [ X ( b θ h , b V h +1 , f b V
[Equation]
)] (cid:20) [Equation] However, the above does not hold with probability 1 (cid:0) δ since b θ h and b V h + :=
min f max a f ( b θ h +1 , ϕ ( (cid:1) , a )) (cid:0) q r f ( b θ h
+1 , ϕ ( (cid:1) , a ))
⊤ A (cid:1) r f ( θ, ϕ ( (cid:1) , a )) , H g (where A is certain symmetric matrix with bounded norm) depend on b θ h and b θ h +1 which are data-dependent.
Therefore, we need to further apply covering Lemma L.10 and choose ϵ =
O (1 /K ) and a union bound to obtain with probability 1 (cid:0) δ , E
[Equation]
[ X ( b θ h , b V h +1 , f b V h +1 )] (cid:20)
[Equation]
where C d, log(
[Equation] := log [Equation] 24 C
[Equation] κ [Equation]
+ [Equation] log[-(cid:16) 
-]
[Equation] d 2 log [Equation]
11 Solving this quadratic equation to obtain with probability 1 (cid:0) δ ,
E
[Equation]
Now according to equation 18, by deﬁnition we ﬁnally have with probability 1 (cid:0) δ (recall the expectation over µ is taken w.r.t.
( s kh , a kh , s kh +1 ) k
= 1 , ..., K only)
[Equation]
Step2.
If λ > 0 , there is only extra term λK [Equation]
[Equation] [Equation] (cid:20)
[Equation]
{+(cid:13)(cid:13)(cid:13)b θ h (cid:13)(cid:13)(cid:13) 
+}
(cid:20) λC 2Θ K in addition to above.
This ﬁnishes the proof.
Theorem G.2 (Provable efﬁciency (Part I)) .
Let C d, log K be the same as Lemma G.1.
Then denote b d,K,ϵ F := q
[Equation]
with probability
[Equation]
Proof of Theorem G.2.
Apply a union bound in Lemma G.1, we have with probability 1 (cid:0) δ ,
[Equation]
) Now we prove for all h 2 [ H ] ,
E µ (cid:20)(cid:16)
[Equation]
Indeed, similar to equation
[Equation]
by deﬁnition we have
[Equation]
=
[Equation]
where the third identity uses µ is taken w.r.t.
s h , a h , s h
+
(recall Lemma G.1) and law of total expectation.
The ﬁrst inequality uses the deﬁnition of θ T b V h +1 .
Now apply Assumption 2.3, we have
[Equation]
Combine the above with equation 21 and equation 22, we obtain the stated result.
Theorem
G.
(Provable efﬁciency (Part
II)) .
Let
C d, log K be the same as
Lemma G.1 and suppose ϵ F = 0 .
Furthermore, suppose λ (cid:20) 1 / 2 C 2Θ and K (cid:21) max
[Equation]
o .
Then, with probability 1 (cid:0) δ , 8 h 2 [ H ] ,
[Equation]
Furthermore, we have with probability 1 (cid:0) δ
[Equation]
where e O absorbs Polylog terms and higher order terms.
Lastly, it also holds for all h 2 [ H ] , w.p.
[Equation]
Proof of Theorem G.3.
Step1: we show the ﬁrst result.
We prove this by backward induction.
When h = H + 1 , by convention f ( b θ h , ϕ
( s, a ))
=
f ( θ ⋆h , ϕ ( s, a ))
=
so the base case holds.
Suppose for h + 1 , with probability 1 (cid:0) ( H (cid:0) h ) δ , it holds true that sup s,a (cid:12)(cid:12)(cid:12) f ( b θ h +1 , ϕ
( s, a ))
(cid:0) f
( θ ⋆h +1 , ϕ ( s, a ))
(cid:12)(cid:12)(cid:12) (cid:20) C h + q 1 K + a ( h + 1) , we next consider the case for t = h .
On one hand, by Theorem G.2, we have with probability
[Table]
d, log K ) + 2 λC 2 Θ κK + (cid:13)(cid:13)(cid:13)b V h +1 (cid:0) V ⋆h +1 (cid:13)(cid:13)(cid:13) ∞ ,
Recall b V h +1 ( (cid:1) ) :
[-+
-]=
min f max a f ( b θ h +1 , ϕ ( (cid:1) , a )) (cid:0) Γ h ( (cid:1) , a ) , H g and
V ⋆h +1 ( (cid:1) ) = max a f ( θ ⋆h +1 , ϕ ( (cid:1) , a ))
=
min f max a f ( θ ⋆h +1 , ϕ ( (cid:1) , a )) , H g , we obtain (cid:13)(cid:13)(cid:13)b V h +1 (cid:0) V ⋆h[-+
-]
+1 (cid:13)(cid:13)(cid:13) ∞ (cid:20) sup s,a (cid:12)(cid:12)(cid:12) f ( b θ h +1 , ϕ
( s, a ))
(cid:0) f
( θ ⋆h +1 , ϕ ( s, a ))
(cid:12)(cid:12)(cid:12) + sup h,s,a Γ h ( s, a ) (23)
Note the above holds true for any generic Γ h ( s, a ) .
In particular, according to Algorithm 1, we specify
[Equation]
and by Lemma L.5, with probability 1 (cid:0) δ ,
[Equation]
and by a union bound this implies with probability 1 (cid:0) ( H (cid:0) h +
[Equation]
δ , sup s,a
[Equation]
Solving for C h , we obtain C h (cid:20) κ 1 H
[Equation]
for all H .
By a union bound (replacing δ by δ/H ), we obtain the stated result.
Step2: Utilizing the intermediate result equation 23, we directly have with probability 1 (cid:0) δ , sup h
[Equation]
b
[Equation]
[Equation]
O
[Equation]
where sup s,a (cid:12)(cid:12)(cid:12) f ( b θ h , ϕ ( s, a )) (cid:0) f ( θ ⋆h , ϕ ( s, a ))
(cid:12)(cid:12)(cid:12) can be bounded using Step1.
Step3: Denote M :=
[Equation]
) , then by
Step1 we have with probability 1 (cid:0) δ (here ξ is some point between b θ h and θ ⋆h ) for all h 2 [ H ]
[Equation]
where the last inequality is by Assumption 2.3.
Solve this to obtain the stated result.
H W ITH POSITIVE B ELLMAN COMPLETENESS COEFFICIENT ϵ F >
In Theorem 3.2, we consider the case where ϵ F = 0 .
If ϵ F > 0 , similar guarantee can be achieved with the measurement of model misspeciﬁcation.
For instance, the additional error q
[Equation]
+ 4 Hϵ F will show up in Lemma G.1 (as stated in the current version), q b d,K,ϵ
F κ + q 2 Hϵ F κ will show up in Lemma G.2.
Then the decomposition in equation 3 will incur the extra δ b V h +1 term with δ b V
h +1 might not be 0 .
The analysis with positive ϵ F > 0 will make the proofs more intricate but incurs no additional technical challenge.
Since the inclusion of this quantity is not our major focus, as a result, we only provide the proof for the case where ϵ F =
so the readers can focus on the more critical components that characterize the hardness of differentiable function class.
I VFQL AND ITS ANALYSIS
We present the vanilla ﬁtted Q-learning (VFQL)
Algorithm 2 as follows.
For VFQL, no pessimism is used and we assume b θ h 2 Θ without loss of generality.
1: Input: Ofﬂine Dataset D =
{+(cid:8)(cid:0) s kh , a kh , r kh , s kh +1 (cid:1)(cid:9) K,H k,h =1 . Denote ϕ h,k := ϕ ( s kh , a kh ) . 2: Initialization: Set b V H +
+}
( (cid:1) ) 0 and λ > 0 . 3: for h = H, H (cid:0) 1 , . . .
, 1 do 4:
Set b θ h arg min θ ∈ Θ (cid:26)P
Kk =
h f ( θ, ϕ h,k ) (cid:0) r h,k (cid:0)
b V h +1 ( s kh +1 )
i+ λ (cid:1)
k θ k(cid:27) 5:
Set b Q h ( (cid:1) , (cid:1) ) min n f ( b θ h , ϕ ( (cid:1) , (cid:1) )) , H (cid:0) h
+o + 6:
Set b π h ( (cid:1) j (cid:1) )
arg max π h (cid:10) b Q h ( (cid:1) , (cid:1) ) , π h ( (cid:1) j (cid:1) ) (cid:11) A , b V h ( (cid:1) ) max π h (cid:10) b Q h ( (cid:1) , (cid:1) ) , π h ( (cid:1) j (cid:1) ) (cid:11)
A 7: end for 8: Output: f
b π h g Hh =1 .
Recall ι h ( s, a ) :=
P h b V h +1 ( s, a ) (cid:0) b Q h ( s, a ) and the deﬁnition of Bellman operator D.1.
Note min f(cid:1) , H (cid:0) h + 1 g + is a non-expansive operator, therefore we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
By Lemma D.2, we have for any π ,
[Equation]
where the second inequality uses Cauchy inequality and the third one uses the deﬁnition of concentrability coefﬁcient 2.2.
Next, for VFQL, there is no pessimism therefore the quantity B in Lemma L.10 is zero, hence the covering number applied in Lemma G.1 is bounded by C d, log( K ) (cid:20)
e O ( d ) and
[Equation]
Now leveraging equation 21 and equation 22 in Theorem G.2 to obtain
[-(cid:21) ≤ E µ   ℓ h ( (cid:98) θ h ) − λ (cid:13)(cid:13)(cid:13)(cid:98) θ h (cid:13)(cid:13)(cid:13) 
-]
E µ (cid:20)(cid:16)
[Equation]
Plug the above into equation 24, we obtain with probability 1 (cid:0) δ , for all policy π , v
[Equation]
This ﬁnishes the proof of Theorem 3.1.
J P ROOFS FOR VAFQL
In this section, we present the analysis for variance-aware ﬁtted Q learning (VAFQL).
Throughout the whole section, we assume ϵ F = 0 , i.e. the exact Bellman-Completeness holds.
The algorithm is presented in the following.
Before giving the proofs of Theorem 3, we ﬁrst prove some useful lemmas. − 1 , . . , 1 do 4: Set u h ← arg min θ ∈ Θ
[Equation]
Set v h ← arg min θ ∈ Θ
[Equation]
Set (cid:2)d Var h b V h +1 (cid:3) ( · , · ) = f ( v h
[Equation]
ϕ ( · , · ))
[Equation]
H
[Equation]
h +
[Equation] ϕ [Equation] [Equation]
+ [Equation] [Equation] Set b σ h ( · , · ) 2 ← max { 1 , d Var P h b
V h {++
 Set b+} θ h ← arg min θ
[Equation] Θ [Equation] Set Λ h ← P
[Equation] Set Γ h ( · , · ) ← β q [Equation] (cid:16)
[Equation] (cid:17) 11: Set ¯ Q h ( · , · ) ← f ( b θ h , ϕ ( · , · )) − Γ h ( · , · ) , b Q h ( · , · ) ← min (cid:8) ¯ Q h ( · , · ) , H − h + 1 (cid:9) + 12: Set b π h ( · | · ) ← arg max π h (cid:10) b Q h ( · , · ) , π h ( · | · ) (cid:11) A , b V h ( · ) ← max π h (cid:10) b Q h ( · , · ) , π h ( · | · ) (cid:11)
A 13: end for 14: Output: { b π h } Hh =1 .
J. P ROVABLE E FFICIENCY FOR V ARIANCE -A WARE F ITTED Q L EARNING
Recall the objective
[Equation]
Then by deﬁnition, b θ h := arg min θ ∈ Θ ℓ h ( θ ) and θ T b V h
+1 satisﬁes f ( θ T b V h
+1 , ϕ )
= P h b V h
{++
+} ( s kh +1 )
(recall ϵ F = 0 ).
Therefore, in this case, we have the following lemma:
Lemma J.1.
Fix h
2 [ H ] .
With probability 1 (cid:0) δ ,
E µ [ ℓ h ( b θ h )] (cid:0) E µ [ ℓ h ( θ T b V h +1 )] (cid:20) 36 H 2 (log(
[Equation]
δ ) + C d, log K ) + λC
[Equation]
K where the expectation over µ is taken w.r.t.
[Equation]
K only (i.e., ﬁrst compute E µ [ ℓ h ( θ )] for a ﬁxed θ , then plug-in either b θ h +
or θ T b V h +1 ).
Here
[Equation]
Proof of Lemma J.1.
Step1: Consider the case where λ = 0 .
Indeed, ﬁx h 2 [ H ] and any function
V ( (cid:1) ) 2 R S .
Similarly, deﬁne f V ( s, a ) :
= f ( θ T V , ϕ )
=
P h V .
For any ﬁxed θ 2 Θ , denote g ( s, a ) =
f ( θ, ϕ
( s, a )) .
Moreover, for any u, v 2 Θ , deﬁne
[Equation]
Then deﬁne (we omit the subscript u, v of σ 2 u,v for the illustration purpose when there is no ambiguity)
[Equation]
Since all episodes are independent of each other,
[Equation]
V ) :=
X ( g ( s kh , a kh ) , V ( s kh +1 ) , f V ( s kh , a kh ) , σ 2 ( s kh , a kh )) are independent r.v.s and it holds
[Equation]
Next, the variance of X is bounded by
Var
[Equation]
s h , a h [Equation] =4 H
[Equation]
E
[Equation]
[ X ( g, f, f V , σ 2 )] ( (cid:3) ) follows from that
[Equation]
Therefore, by Bernstein inequality, with probability 1 (cid:0) δ ,
[Equation]
r 2Var [ X ( g, f, f V , σ 2 )] log(1 /δ )
[Equation]
Now, if we choose g ( s, a ) :
= f ( b θ h , ϕ ( s, a )) and u = u h , v = v h from Algorithm 3, then b θ h minimizes ℓ h ( θ ) , therefore, it also minimizes 1 K P Kk =1 X i ( θ, b
V h +1 , f b V h
+1 , b σ 2 h ) and this implies
K
[Equation]
Thus, we obtain
[Equation]
However, the above does not hold with probability 1 (cid:0) δ since b θ h , b σ 2 h and b V h + := min f max a f ( b θ h +1 , ϕ ( (cid:1) , a ))
[Equation]
q
[Equation]
ϕ
[Equation]
[Equation]
[Equation]
f ( θ,
[Equation]
[Equation] , a )) , H g (where A is certain symmetric matrix with bounded norm) depend on b θ h , b θ h +1 which are data-dependent.
Therefore, we need to further apply covering Lemma L.11 and choose ϵ =
O (1 /K ) and a union bound to obtain with probability 1 (cid:0) δ , [Equation] h
[Equation]
where C d, log( K ) :
= d log(1+24 C Θ ( H + 1) κ 1 K )+ d log
[Equation]
288 H
[Equation]
p [Equation] [Equation] K
[Equation] (cid:17) + d 2 log (cid:16) [Equation]
[Equation]
(cid:17)
[Equation]
log
[Equation]
log( [Equation] K ) (where we let B = 1 /λ since (cid:13)(cid:13) Λ − 1 h (cid:13)(cid:13) 2 (cid:20) 1 /λ ).
Solving this quadratic equation to obtain with probability 1 (cid:0)
[Equation]
[Equation] Now according to equation 25, by deﬁnition we ﬁnally have with probability 1 (cid:0) δ (recall the expectation over µ is taken w.r.t.
( s kh , a kh , s kh +1 ) k
= 1 , ..., K only)
[Equation] K
[Equation]
where we used f ( θ T b V h +1 , ϕ )
= P h b V h
{++
+} = f b V h +1 .
Step2.
If λ > 0 , there is only extra term λK
[Equation]
K in addition to above.
This ﬁnishes the proof.
Theorem J.2 (Provable efﬁciency for VAFQL) .
Let C d, log K be the same as Lemma J.1.
Then, with probability 1 (cid:0) δ
[Equation]
Proof of Theorem J.2.
Apply a union bound in Lemma J.1, we have with probability 1 (cid:0) δ ,
[Equation]
b [Equation] T
[Equation] V h [Equation] [Equation]
[Equation]
log
[Equation]
Now we prove for all h 2 [ H ] , E µ
[Equation]
K 75 .
Indeed, identical to equation 26,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
i
[Equation]
where the third identity uses law of total expectation and that µ is taken w.r.t.
s h , a h , s h +1 only (recall Lemma J.1)
so the b σ 2 h can be move outside of the conditional expectation.
The fourth identity uses the deﬁnition of θ T b V h +1 since f ( θ T b V h +1 , ϕ
( s, a ))
=
P h,s,a b V h +1 .
Then we have
E
[Equation]
where the third identity uses µ is over s h , a h only and the last one uses b σ 2 h ( (cid:1) , (cid:1) ) (cid:20) H 2 .
Combine the above with equation 27 and equation 28, we obtain the stated result.
Theorem J.3 (Provable efﬁciency of VAFQL (Part II)) .
Let C d, log K be the same as Lemma J.1.
Furthermore, suppose λ (cid:20) 1 / 2 C 2Θ and K (cid:21) max
[Equation]
Then, with probability 1 (cid:0) δ , 8 h 2 [ H ] sup s,a (cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
, a )
[Equation]
Furthermore, we have with probability 1 (cid:0) δ , sup h
[Equation]
where e O absorbs Polylog terms and higher order terms.
Lastly, it also holds for all h 2 [ H ] , w
[Equation] p.
[Equation]
Proof of Theorem J.3.
Step1: we show the ﬁrst result.
We prove this by backward induction.
When h = H + 1 , by convention f ( b θ h , ϕ
( s, a ))
=
f ( θ ⋆h , ϕ ( s, a ))
=
so the base case holds.
Suppose for h + 1 , with probability 1 (cid:0) ( H (cid:0) h ) δ , sup s,a (cid:12)(cid:12)(cid:12) f
( b θ h , ϕ ( s, a )) (cid:0) f
( θ ⋆h , ϕ ( s, a ))
(cid:12)(cid:12)(cid:12) (cid:20) C h + q 1 K , we next consider the case for t = h .
On one hand, by Theorem J.2, we have with probability 1 (cid:0) δ/ 2 , sup s,a (cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
a
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
=
[Equation]
a
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Note the above holds true for any generic Γ h ( s, a ) .
In particular, according to Algorithm 3, we specify
[Equation]
and by Lemma L.5, with probability 1 (cid:0) δ (note here Σ − 1 h is replaced by Λ − 1 h and (cid:13)(cid:13)
Λ − 1 h (cid:13)(cid:13) {+
 (cid:20)+} H 2 /κ ),
[Equation]
and by a union bound this implies with probability 1 (cid:0) ( H (cid:0) h + 1) δ , sup s,a
[Equation]
Solving for C h , we obtain C h (cid:20) κ 1 H
[Equation]
for all H .
By a union bound (replacing δ by δ/H ), we obtain the stated result.
Step2: Utilizing the intermediate result equation 29, we directly have with probability 1 (cid:0) δ , sup h (cid:13)(cid:13)(cid:13)b V h (cid:0) V ⋆h (cid:13)(cid:13)(cid:13) ∞ (cid:20) sup s,a (cid:12)(cid:12)(cid:12) f
( b θ h , ϕ ( s, a )) (cid:0) f
( θ ⋆h , ϕ ( s, a ))
(cid:12)(cid:12)(cid:12) + 2 dH 2 κ 1 p κK + O ( 1 K ) , where sup s,a (cid:12)(cid:12)(cid:12) f ( b θ h , ϕ
( s, a ))
(cid:0) f ( θ ⋆h , ϕ ( s, a ))
(cid:12)(cid:12)(cid:12) can be bounded using Step1.
Step3: Denote M :=
(cid:18)
[Equation]
(cid:19) [Equation] 1 K + O ( 1 K ) , then by
Step1 we have with probability 1 (cid:0) δ (here ξ is some point between b θ h and θ ⋆h ) for all h 2 [ H ]
M 2 (cid:21) sup s,a (cid:12)(cid:12)(cid:12) f ( b θ h , ϕ ( s, a )) (cid:0) f ( θ ⋆h , ϕ ( s, a )) (cid:12)(cid:12)(cid:12)(cid:21) E µ [ (cid:16) f
( b θ h , ϕ ( s, a )) (cid:0) f ( θ ⋆h , ϕ ( s, a )) (cid:17)] (cid:21) κ (cid:13)(cid:13)(cid:13)b θ h (cid:0) θ ⋆h (cid:13)(cid:13)(cid:13)  where the last step is by Assumption 2.3.
Solving this to obtain the stated result.
J.
[Equation]
Recall the deﬁnition σ ⋆ 2 h ( (cid:1) , (cid:1) ) = max f 1 , [
Var P h V ⋆h +1 ]( (cid:1) , (cid:1) ) g .
In this section, we bound the term j
[Equation]
and
[Equation]
where
[Equation]
and true parameters u ⋆h , v ⋆h satisfy f ( u ⋆h , ϕ ( (cid:1) , (cid:1) ))
[Equation]
E P ( s ′ |· , · ) [ V ⋆ 2 h ( s ′ )] .
Furthermore, we deﬁne
[Equation]
and the parameter Expectation operator J : V 2 R S ! θ J V 2 Θ such that:
[Equation]
Note θ J V 2 Θ by Bellman completeness, reward r is constant and differentiability (Deﬁnition 1.1) is an additive closed property.
By deﬁnition,
[Equation]
and
[Equation]
We ﬁrst give the following result.
Lemma J.4.
Suppose λ (cid:20) 1 / 2 C 2Θ and K (cid:21) max n 512 κ 41 κ(cid:16) log(
[Equation]
) + d log(
[Equation]
κ
[Equation]
{+) (cid:17) , 4 λκ o . Then, with probability 1 (cid:0) δ , 8 h 2 [ H ] , (cid:13)(cid:13)(cid:13) u h (cid:0) θ J b V h +
 (cid:13)(cid:13)(cid:13) 2 (cid:20)+}
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
V 2 {+h +1 (cid:13)(cid:13)(cid:13) 
+}
(cid:20)
[Equation]
and sup s,a | f ( u h , ϕ
( s, a ))
− f ( u ⋆h , ϕ
( s, a ))
| ≤
[Equation]
sup s,a | f ( v h , ϕ ( s, a )) − f ( v ⋆h , ϕ ( s, a ))
| ≤
[Equation]
log
[Equation]
δ )
[Equation]
d
[Equation]
The above directly implies for all h 2 [ H ] , with probability 1 (cid:0) δ , j σ ⋆ 2 h (cid:0) b σ 2 h j (cid:20) @ 3 κ 1 Hs
[Equation]
Proof of Lemma J.4.
In fact, the proof follows a reduction from the provable efﬁciency procedure conducted in Section G.
This is due to the regression procedure in equation 30 is the same as the procedure equation 17 except the parameter Bellman operator T is replaced by the parameter Expectation operator J (recall here ¯ ϕ h,k uses the independent copy D ′ and e O ( d 2 ) comes from the covering argument.).
Concretely, the X ( g, V, f V ) used in Lemma G.1 will be modiﬁed to X ( g, V, f V ) =
( g ( s, a ) (cid:0) V ( s ′ ))
2 (cid:0) ( f ( θ J V , ϕ ( s, a ))
(cid:0) V ( s ′ )) 2 by removing reward information and the decomposition
[Equation]
holds true.
Then with probability 1 (cid:0) δ ,
[Equation]
and
[Equation]
[Table]
[Equation]
[Table]
J. P ROOF OF T HEOREM 4.
In this section, we sketch the proof of Theorem 4.1 since the most components are identical to
Theorem 3.2.
We will focus on highlighting the difference for obtaining the tighter bound.
First of all, Recall in the ﬁrst-order condition, we have
[Equation]
θ
[Equation]
Therefore, if we deﬁne the quantity Z h ( (cid:1) , (cid:1) ) 2 R d as
Z h ( θ j V, σ 2 ) =
K X
[Equation]
then we have
[Equation]
According to the regression oracle (Line 8 of Algorithm 3), the estimated Bellman operator b P h maps b V h +1 to b θ h , i.e. b P h b V h
+1 =
f ( b θ h , ϕ ) .
Therefore (recall Deﬁnition D.1)
[Equation]
where we apply the ﬁrst-order Taylor expansion for the differentiable function f at point b θ h and
Hot h, 1 is a higher-order term.
Indeed, the following Lemma E.1 bounds the Hot h, 1 term with e
O ( 1 K ) .
Lemma J.5.
Recall the deﬁnition (from the above decomposition)
Hot
[Equation]
(cid:0)
[Equation]
(cid:17) , then with probability 1 (cid:0) δ , j Hot h, 1 j (cid:20) e O ( 1 K ) , 8 h 2 [ H ] .
Proof.
The proof is identical to that of Lemma E.1 but with the help of Lemma J.2.
Next, according to the expansion of Z h ( θ j b V h
+1 , b σ 2 h ) , we have
[Equation]
where
[Equation]
) where R K ( θ T b V h +1 ) is the second order residual that is bounded by e O (1 /K ) and I 1 = ∇ f [Equation] b θ
[Equation] [Equation] s, a )
[Equation]
I
[Equation] = ∇ f [Equation] Similar to the PFQL case, I 2 , I 3 , Hot 2 can be bounded to have order O (1 /K ) via provably efﬁciency theorems in Section J.1 and in particular, the inclusion of σ 2 u,v will not cause additional order in d .
Now we prove the result for the dominate term I 1 .
Lemma J.6.
With probability 1 (cid:0) δ ,
[Equation]
4 Hd (cid:13)(cid:13)(cid:13)
[Equation]
f
[Equation]
h
[Equation]
( s, a ))
(cid:13)(cid:13)(cid:13)
[Equation] h (cid:1) C δ [Equation] log K + e O ( κ 1 p κK ) , where C δ, log K only contains Polylog terms.
Proof of Lemma J.6.
First of all, by CauchySchwarz inequality, we have j I 1 j (cid:20) [Equation] Λ −h [Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)
[Equation]
X
[Equation] = [Equation] (cid:16)
[Equation]
) (cid:17) (cid:1)
[Equation] ) b [Equation] ( s
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13) Λ − 1 h . Recall that σ 2 u,v ( (cid:1) , (cid:1) ) : = max f 1 , f ( v, ϕ ( (cid:1) , (cid:1) ))
[0 , ( H − h +1) 2 ] (cid:0) (cid:2) f ( u, ϕ ( (cid:1) , (cid:1) ))
[0 ,H − h +1] (cid:3) 2 g .
Step1.
Let the ﬁxed θ 2 Θ be arbitrary and ﬁxed u, v such that σ 2 u,v ( (cid:1) , (cid:1) ) (cid:21) 12 σ 2 u ⋆h , v ⋆h ( (cid:1) , (cid:1) ) = 12 σ ⋆ 2 h ( (cid:1) , (cid:1) ) and deﬁne x k ( θ, u, v ) =
r θ f ( θ, ϕ h,k ) /σ
u,v ( s kh , a kh ) .
Next, deﬁne G u,v ( θ ) = P
[Equation] = [Equation] f ( θ,
[Equation] s kh , a kh [Equation] ) (cid:1) r f ( θ, ϕ ( s kh , a kh ))
⊤ /σ 2 u,v ( s kh , a kh ) + λI d , then k x k k 2 (cid:20) κ 1 .
Also denote η k :=
[ f ( θ T V ⋆h +1 , ϕ h,k ) (cid:0) r h,k (cid:0) V ⋆h +
( s kh +1 )]
/σ u,v ( s kh , a kh ) , then E [ η k j s kh , a kh ] = 0 and
[Equation]
then by Self-normalized Bernstein’s inequality (Lemma L.4), with probability 1 (cid:0) δ ,
[Equation]
K X
[Equation]
where j η k j (cid:20) ζ with ζ = 2 max s,a,s ′ | f ( [-θ,u,v ) −≤(cid:115) d log (cid:18) 
-]θ T V⋆h +1 ,ϕ
( s,a ))
− r − V ⋆h +
( s ′ )
| σ ⋆h ( s,a ) and the last inequality uses p d (cid:21) e O ( ζ ) .
Step2.
Deﬁne h
( θ, u, v ) :
= P Kk =1 x k ( θ, u, v ) η k ( u, v ) and H ( θ, u, v ) :
= k h ( θ, u, v ) k
G u,v ( θ ) − 1 , k h [Equation] [Equation]
[Equation]
max
[Equation]
[Table]
[Equation]
All the above imply
[Equation]
[Table]
note
[Equation]
Then a ϵ -covering net of f H ( θ, u, v ) g can be constructed by the union of covering net for θ, u, v and by Lemma L.8, the covering number N ϵ satisﬁes (where e O absorbs Polylog terms)
[Equation]
[Table] [Table]
[Equation]
Now choosing ϵ
=
O (1 /K ) in Step2 and union bound over the covering number in Step
[Equation]
we obtain with probability 1 (cid:0) δ (recall
[Equation]
where we absorb all the Polylog terms.
Combing above with equation 33, we obtain with probability 1 (cid:0) δ ,
[Equation]
Combing dominate term I 1 (via Lemma J.6) and all other higher order terms we can obtain the ﬁrst result together with Lemma D.3.
The proof of the second result is also very similar to the proofs in Section F.2.
Concretely, when picking π = π ⋆ , we can convert the quantity q
[Equation]
to q
[Equation]
using Theorem J.3, and convert q
[Equation]
to
[Equation]
using Lemma J.4.
K T HE LOWER BOUND
Theorem K.1 (Restatement of Theorem 4.2) .
Specifying the model to have linear representation f = h θ, ϕ i .
There exist a pair of universal constants c, c ′ > 0 such that given dimension d , horizon
H and sample size K > c ′ d 3 , one can always ﬁnd a family of MDP instances such that for any algorithm b π
[Equation]
where Λ ⋆,ph = E
[Equation]
Remark K.2.
Note Theorem 4.2 is a valid lower bound for comparison.
This is because the upper bound result holds true for all model f such that the corresponding F satisﬁes Assumption 2.1, 2.3.
Therefore, for the lower bound construction it sufﬁces to ﬁnd one model f such that the lower bound equation 34 holds.
Here we simply choose the linear function approximation.
K. R EGARDING THE PROOF OF LOWER BOUND
The proof of Theorem 4.2 can be done via a reduction to linear function approximation lower bound.
In fact, it can be directly obtained from Theorem 3.5 of Yin et al.
and the original proof comes from Theorem 2 of Zanette et al.
Concretely, all the proofs in Theorem 3.5 of Yin et al.
(2022) follows and the only modiﬁcation is to replace q
[Equation]
in Section E.5 by
[Equation]
and the ﬁnal result holds with ϕ ( (cid:1) , (cid:1) ) = r θ f ( θ ⋆h , ϕ ( (cid:1) , (cid:1) )) by the reduction f = h θ, ϕ i .
L A UXILIARY LEMMAS
Lemma L.1 ( k -th Order Mean Value Form of Taylor’s Expansion) .
Let k (cid:21) 1 be an integer and let function f : R d ! R be k times differentiable and continuous over the compact domain Θ (cid:26) R d .
Then for any x, θ 2 Θ , there exists ξ in the line segment of x and θ , such that
[Equation]
⊤ ( x (cid:0) θ ) + 1 2! ( x (cid:0) θ ) ⊤ r 2 θθ f ( θ )( x (cid:0) θ ) + . . .
+ ( k (cid:0) 1)!
r k − 1 f ( θ ) (cid:16)O ( x (cid:0) θ ) (cid:17)
k −+k !
r k f ( ξ ) (cid:16)O ( x (cid:0) θ ) (cid:17) k .
Here r k f ( θ ) denotes k -dimensional tensor and N denotes tensor product.
Lemma L.2 (Vector Hoeffding’s Inequality) .
Let X =
( X 1 , . . .
, X d ) be d -dimensional vector Random Variable with E [ X ] = 0 and k X k 2 (cid:20) R .
X (1) , . . .
, X ( n ) ’s are n samples.
Then with probability 1 (cid:0) δ , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)n n X i =
X ( i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:20) r 4 dRn log( d δ ) .
Proof of Lemma L.2.
Since k X k 2 (cid:20) R implies j X j j (cid:20) R , by the univariate Hoeffding’s inequality, for a ﬁxed j 2 f 1 , ..., d g , denote Y j := 1 n P ni =1 X
( i ) j .
Then with probability 1 (cid:0) δ (note j X ( i ) j j (cid:20) R ),
P
[Equation]
r
[Equation]
(cid:20) δ.
By a union bound,
[Equation]
Lemma L.3 (Hoeffding inequality for self-normalized martingales (Abbasi-Yadkori et al., 2011)) .
Let f η t g ∞ t =1 be a real-valued stochastic process.
Let fF t g ∞ t
=
be a ﬁltration, such that η t is F tmeasurable.
Assume η t also satisﬁes η t given F t − 1 is zero-mean and R -subgaussian, i.e.
[Equation]
Let f x t g ∞ t = 1 be an R d -valued stochastic process where x t is F t − 1 measurable and k x t k (cid:20) L .
Let Λ t = λI d +
P ts =1 x s x ⊤ s .
Then for any δ > 0 , with probability 1 (cid:0) δ , for all t > 0 ,
[Equation]
Lemma L.4 (Bernstein inequality for self-normalized martingales (Zhou et al., 2021a)) .
Let f η t g ∞ t = be a real-valued stochastic process.
Let fF t g ∞ t
=
be a ﬁltration, such that η t is F t -measurable.
Assume η t also satisﬁes
[Equation]
Let f x t g ∞ t = 1 be an R d -valued stochastic process where x t is F t − 1 measurable and k x t k (cid:20) L .
Let Λ t = λI d +
P ts =1 x s x ⊤ s .
Then for any δ > 0 , with probability 1 (cid:0) δ , for all t > 0 ,
[Equation]
t X
[Equation]
Lemma L.5.
Let r f ( θ, ϕ ( (cid:1) , (cid:1) )) :
S (cid:2) A !
R d be a bounded function s.t. sup θ ∈ Θ kr f
( θ, ϕ ( (cid:1) , (cid:1) ))
k 2 (cid:20) κ 1 .
If K satisﬁes
[Equation]
Then with probability at least 1 (cid:0) δ , for all k u k 2 (cid:20) B simultaneously, it holds that
[Equation]
where Σ h =
[Equation]
Proof of Lemma L.5.
For a ﬁxed θ , deﬁne ¯ G = P
Kk =1 r f ( θ, ϕ ( s kh , a kh )) (cid:1) r f ( θ, ϕ ( s kh , a kh )) ⊤ + λI d , and G = E µ [ r f ( θ, ϕ ( s h , a h )) (cid:1) r f
( θ, ϕ ( s h , a h ))
⊤ ] , then by Lemma H.5.
of Min et al.
(2021), as long as
[Equation]
[Equation]
then with probability 1 (cid:0) δ , for all u 2 R d simultaneously, k u k ¯ G − 1 (cid:20) 2 √ K k u k G − 1 .
As a corollary, if we constraint u to the subspace k u k 2 (cid:20) B , then we have: with probability 1 (cid:0) δ , for all f u 2 R d : k u k 2 (cid:20) B g simultaneously, [Equation]
[Equation]
Next, for any θ , deﬁne h u ( θ ) :
= k u k ¯ G − 1 =
p u ⊤ ¯ G − 1 u = vuut u ⊤
K X k = r f ( θ, ϕ ( s kh , a kh )) (cid:1) r f ( θ, ϕ ( s kh , a kh )) ⊤ + λI d !
−u and ¯ G ( θ ) =
P Kk =1 r f ( θ, ϕ ( s kh , a kh )) (cid:1) r f ( θ, ϕ ( s kh , a kh ))
⊤ + λI d , we have for any θ 1 , θ(cid:13)(cid:13) ¯ G ( θ 1 ) (cid:0) ¯ G ( θ 2 ) (cid:13)(cid:13) {+
 (cid:20) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)+}
K X k = (cid:0) r f ( θ 1 , ϕ
( s k h , a k h ))
(cid:0) r f
( θ 2 , ϕ
( s k h , a k h )) (cid:1) (cid:1) r f ( θ 1 , ϕ ( s
[Equation]
Use the basic inequality for a, b
[Equation]
[Equation]
Therefore, the ϵ -covering net of f h ( θ ) : θ 2 Θ g is implies by the
[Equation]
covering net of f θ : θ 2 Θ g , so by Lemma L.8, the covering number N ϵ satisﬁes
[Equation]
Select θ = b θ h .
Choose ϵ
= O (1 /K )
and by a union bound over equation 36 to get with probability 1 (cid:0) δ , for all k u k 2 (cid:20) B (note By Assumption 2.
[Equation]
if (union bound over the condition equation 35)
[Equation]
where this condition is satisﬁed by the Lemma statement.
Lemma L.6.
let ϕ : S (cid:2)
A ! R d satisﬁes k ϕ
( s, a ) k (cid:20) C for all s, a 2 S (cid:2) A .
For any
K > 0 , λ > 0 , deﬁne ¯ G K = P
Kk =1 ϕ ( s k , a k ) ϕ
( s k , a k ) ⊤ + λI d where ( s k , a k ) ’s are i.i.d samples from some distribution ν .
Then with probability 1 (cid:0) δ ,
[Equation]
Proof of Lemma L.6.
See Lemma H.5 of Yin et al.
(2022) or Lemma H.4 of Lemma Min et al.
(2021) for details.
Lemma L.7 (Lemma H.4 in Yin et al.
Let Λ 1 and Λ 2 2 R d × d are two positive semi-deﬁnite matrices.
Then:
[Equation]
1 1 (cid:13)(cid:13) (cid:20) (cid:13)(cid:13) Λ − 1 2 (cid:13)(cid:13) + (cid:13)(cid:13) Λ − 1 1 (cid:13)(cid:13) (cid:1) (cid:13)(cid:13) Λ − 1 2 (cid:13)(cid:13) (cid:1) k Λ 1 (cid:0) Λ 2 k and k ϕ k Λ −
1 1 (cid:20) (cid:20) 1 + q(cid:13)(cid:13) Λ − 1 2 (cid:13)(cid:13) k Λ 2 k (cid:1) (cid:13)(cid:13) Λ − 1 1 (cid:13)(cid:13) (cid:1) k Λ 1 (cid:0) Λ 2 k (cid:21) (cid:1) k ϕ k Λ − 1 2 .
for all ϕ 2 R d .
L. C OVERING A RGUMENTS
Lemma L.8.
(Covering Number of Euclidean Ball)
For any ϵ > 0 , the ϵ -covering number of the
Euclidean ball in R d with radius R > 0 is upper bounded by (1 + 2 R/ϵ ) d .
Lemma L.9.
Deﬁne V to be the class mapping S to R with the parametric form
[Equation]
where the parameter spaces are f θ : k θ k 2 (cid:20) C Θ g and f A : k A k 2 (cid:20) B g .
Let N V ϵ be the covering number of ϵ -net with respect to l ∞ distance, then we have
[Equation]
(cid:19)
+ d 2 log 1 +p dBκϵ! .
Proof of Lemma L.9.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] Here k(cid:1)k F is Frobenius norm.
Let C θ be the ϵ
[Equation]
[Table]
[Equation]
[Table] [Equation] -net of space f θ : k θ k 2 (cid:20) C Θ g and C w be the ϵ 2 4 κ 21 -net of the space f A : k A k F (cid:20) p dB g , then by Lemma L.8,
[Equation]
Therefore, the covering number of space V satisﬁes log
[Equation]
Lemma L.10 (Covering of E µ ( X ( g, V, f )) ) .
Deﬁne
[Equation]
where f V := P h V + δ V and V ( s ) has form V θ ( s ) that belongs to V (as deﬁned in Lemma L.9).
Here
X ( θ, θ ′ ) is a function of s, a, r, s ′ as well [Equation] and we suppress the notation for conciseness only.
Then the function class H =
f h ( θ, θ ′ ) :
= E µ [ X ( θ, θ ′ )] j k θ k 2 (cid:20) C Θ , V θ 2 Vg has the covering number of ( ϵ + 4 Hϵ F ) -net bounded by
[Equation]
Proof of Lemma L.10.
First of all,
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
where the second inequality comes from f V = P h V + δ V .
Note the above holds true for all s, a, r, s ′ , therefore it implies
[Equation]
Now let C 1 be the ϵ 12( H +1) κ 1 -net of f θ : k θ k 2 (cid:20) C Θ g and C 2 be the ϵ/ 6 H -net of V , applying
Lemma L.8 and Lemma L.9 to obtain
[Equation]
which implies the covering number of H to be bounded by
[Equation]
Lemma L.11.
Denote σ 2 u,v ( [Equation] [Equation] [Equation]
[Equation]
and deﬁne
[Equation]
where f V := P h V and V ( s ) has form V θ ( s ) that belongs to V (as deﬁned in Lemma L.9).
Here ¯
X ( θ, θ ′ , u, v ) is a function of s, a, r, s ′ as well, and we suppress the notation for conciseness only.
Then the function class H =
f h ( θ, θ ′ , u, v ) :
= E µ [ ¯ X ( θ, θ ′ , u, v )] j k θ k 2 (cid:20) C Θ , V θ 2 Vg has the covering number of ϵ -net bounded by d log [Equation] [Equation]
H
[Equation]
Proof of Lemma L.11.
Recall
[Equation]
f
[Equation]
u, ϕ ( (cid:1) , (cid:1) ))
[0 ,H
[Equation]
2 g , and since max , truncation are non-expansive operations, then we can achieve for any s, a
[Equation]
[Table]
[Equation]
Note the above holds true for all s, a, r, s ′ , therefore it implies
[Equation]
and similar to Lemma L.10, the covering number of ϵ -net will be bounded by
[Equation]
Comparing to Lemma L.
[Equation]
the last two terms are incurred by covering u, v arguments.