O FFLINE R EINFORCEMENT L EARNING WITH D IFFERENTIABLE F UNCTION A PPROXIMATION IS P ROVABLY E FFICIENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Offline reinforcement learning , which aims at optimizing sequential decisionmaking strategies with historical data, has been extensively applied in real-life applications.
State-Of-The-Art algorithms usually leverage powerful function approximators ( e.g. neural networks) to alleviate the sample complexity hurdle for better empirical performances.
Despite all that, a more systematic understanding of the statistical complexity for function approximation remains lacking.
Towards bridging the gap, we take a step by considering offline reinforcement learning with differentiable function class approximation (DFA).
This function class naturally incorporates a wide range of models with nonlinear/nonconvex structures.
Most importantly, we show offline RL with differentiable function approximation is provably efficient by analyzing the pessimistic fitted Q-learning (PFQL) algorithm, and our results provide the theoretical basis for understanding a variety of practical heuristics that rely on Fitted Q-Iteration style design.
In addition, we further improve our guarantee with a tighter instance-dependent characterization.
We hope our work could draw interest in studying reinforcement learning with differentiable function approximation beyond the scope of current research.
I NTRODUCTION
Offline reinforcement learning (Lange et al., 2012; Levine et al., 2020) refers to the paradigm of learning a policy in the sequential decision making problems, where only the logged data are available and were collected from an unknown environment ( Markov Decision Process / MDP).
Inspired by the success of scalable supervised learning methods, modern reinforcement learning algorithms ( e.g.
Silver et al.
(2017)) incorporate high-capacity function approximators to acquire generalization across large state-action spaces and have achieved excellent performances along a wide range of domains.
For instance, there are a huge body of deep RL-based algorithms that tackle challenging problems such as the game of Go and chess (Silver et al., 2017; Schrittwieser et al., 2020), Robotics (Gu et al., 2017; Levine et al., 2018), energy control (Degrave et al., 2022) and Biology (Mahmud et al., 2018;
Popova et al., 2018).
Nevertheless, practitioners also noticed that algorithms with general function approximators can be quite data/sample inefficient, especially for deep neural networks where the models may require million of steps for tuning the large number of parameters they contain.
On the other hand, statistical analysis has been actively conducted to understand the sample/statistical efficiency for reinforcement learning with function approximation, and fruitful results have been achieved under the respective model representations (Munos, 2003; Chen and Jiang, 2019;
Yang and
Wang, 2019; Du et al., 2019; Sun et al., 2019; Modi et al., 2020; Jin et al., 2020; Ayoub et al., 2020;
Zanette et al., 2020; Jin et al., 2021a; Du et al., 2021; Jin et al., 2021b;
Zhou et al., 2021; Xie et al., 2021;
Min et al., 2021; Nguyen-Tang et al., 2021; Zanette et al., 2021;
Yin et al., 2022; Uehara et al., 2022).
However, most works consider linear model approximators ( e.g. linear (mixture) MDPs) or its variants.
While the explicit linear structures make the analysis trackable (linear problems are easier to analyze), they are unable to reveal the sample/statistical complexity behaviors of practical algorithms that apply powerful function approximations (which might have complex structures).
In addition, there is an excellent line of works tackling provably efficient offline RL with general function approximation ( e.g. (Chen and Jiang, 2019; Xie et al., 2021; Zhan et al., 2022)).
Due to the generic function approximation class considered, those complexity bounds are usually expressed in the standard worst-case fashion O ( V 2max (cid:113) 1 n ) which lack the characterizations of individual instance behaviors.
However, as mentioned in Zanette and Brunskill (2019), practical reinforcement learning algorithms often perform far better than what these problem-independent bounds would suggest.
These observations motivate us to consider function approximation schemes that can help address the existing limitations.
In particular, in this work we consider offline reinforcement learning with differentiable function class approximations.
Its definition is given in below.
Definition 1.1 (Parametric Differentiable Function Class) .
Let S , A be arbitrary state, action spaces and a feature map ϕ ( · , · ) : S ×
A → Ψ ⊂ R m .
The parameter space Θ ∈ R d .
Both Θ and Ψ are compact spaces.
Then the parametric function class (for a model f : R d × R m → R ) is defined as
[Equation]
that satisfies differentiability/smoothness condition: 1. for any ϕ ∈ R m , f ( θ, ϕ ) is third-time differentiable with respect to θ ; 2. f, ∂ θ f, ∂ 2 θ,θ f, ∂ 3 θ,θ,θ f are jointly continuous for ( θ, ϕ ) .
Remark 1.2.
Differentiable Function Class was recently proposed for studying Off-Policy Evaluation (OPE)
Problem (Zhang et al., 2022) and we adopt it here for the policy learning task.
Note by the compactness of Θ , Ψ and continuity, there exists constants C Θ , B F , κ 1 , κ 2 , κ 3 > 0 that bounds: ∥ [Equation]
a [Equation]
[Table]
Why consider differentiable function class (Definition 1.1)?
There are two main reasons why differentiable function class is worth studying for reinforcement learning.
• Due to the limitation of statistical tools, existing analysis in reinforcement learning usually favor basic settings such as tabular MDPs (where the state space and action space are finite (Azar et al., 2017; Sidford et al., 2018; Agarwal et al., 2020)) or linear MDPs (Yang and Wang, 2020; Jin et al., 2020) / linear Mixture MDPs (Modi et al., 2020; Cai et al., 2020) (where the transition dynamic admits linear structures) so that well-established techniques ( e.g. from linear regression) can be applied.
In addition, subsequent extensions are often based on linear models ( e.g. Linear Bellman Complete models (Zanette et al., 2020) and Eluder dimension (Russo and Van Roy, 2013)).
Differentiable function class strictly generalizes over the previous popular choices, i.e. by choosing f ( θ, ϕ ) =
⟨ θ, ϕ ⟩ or specifying ϕ to be one-hot representations, and is far more expressive as it encompasses nonlinear approximators.
• Practically speaking, the flexibility of selecting model f provides the possibility for handling a variety of tasks.
For instance, when f is specified to be neural networks, θ corresponds to the weights of each network layers and ϕ ( · , · ) corresponds to the state-action representations (which is induced by the network architecture).
When facing with easier tasks, we can deploy simpler model f such as polynomials.
Yet, our statistical guarantee is not affected by the specific choices as we can plug the concrete form of model f into Theorem 3.2 to obtain the respective bounds (we do not need separate analysis for different tasks).
R ELATED WORKS
Reinforcement learning with function approximation.
RL with function approximation has a long history that can date back to Bradtke and Barto (1996); Tsitsiklis and Van Roy (1996).
Later, it draws significant interest for the finite sample analysis (Jin et al., 2020; Yang and Wang, 2019).
Since then, people put tremendous efforts towards generalizing over linear function approximations and examples include Linear Bellman complete models (Zanette et al., 2020), Eluder dimension (Russo and Van Roy, 2013; Jin et al., 2021a), linear deterministic Q ⋆ (Wen and Van Roy, 2013) or Bilinear class (Du et al., 2021).
While those extensions are valuable, the structure conditions assumed usually make the classes hard to track beyond the linear case.
For example, the practical instances of Eluder
Dimension are based on the linear-in-feature (or its transformation) representations (Section 4. of Wen and Van Roy (2013)).
As a comparison, differentiable function class contains a range of functions that are widely used in practical algorithms (Riedmiller, 2005).
Offline RL with general function approximation (GFA).
Another interesting thread of work considered offline RL with general function approximation (Ernst et al., 2005; Chen and Jiang, 2019;
Liu et al., 2020; Xie et al., 2021) which only imposes realizability and completeness/concentrability assumptions.
The major benefit is that the function hypothesis can be arbitrary with no structural assumptions and it has been shown that offline RL with GFA is provably efficient.
However, the generic form of functions in GFA makes it hard to go beyond worst-case analysis and obtain finegrained instance-dependent learning bounds similar to those under linear cases.
On the contrary, our results with DFA can be more problem adaptive by leveraging gradients and higher order information.
In addition to the above, there are more connected works.
Zhang et al.
(2022) first considers the differentiable function approximation (DFA) for the off-policy evaluation (OPE) task and builds the asymptotic theory, Fan et al.
(2020) analyzes the deep Q-learning with the specific ReLu activations, and Kallus and Uehara (2020) considers semi-parametric / nonparametric methods for offline RL (as opposed to our parametric DFA in 1.1).
These are nice complementary studies to our work.
1. O UR CONTRIBUTION
We provide the first Instance-dependent offline learning bound under non-linear function approximation.
Informally, we show that (up to a lower order term) the natural complexity measure is proportional to (cid:80)
Hh =1 E π ⋆ ,h [ (cid:113) g
θ ( s, a ) ⊤ Σ − 1 h g θ ( s, a )] where g θ ( s, a ) :=
∇ f
( θ, ϕ ( s, a )) is the gradient w.r.t.
the parameter θ ⋆ at feature ϕ and Σ h
= (cid:80) i g
( s i,h , a i,h ) g
( s i,h , a i,h ) ⊤ is the Fisher information matrix of the observed data at (cid:98) θ .
This is achieved by analyzing the pessimistic fitted
Q-learning (PFQL) algorithm (Theorem 3.2).
In addition, we further analyze its variance-reweighting variant, which enjoys a more instance-dependent nature and can yield faster sample convergence rate.
Last but not least, existing offline RL studies with tabular models, linear models and GLM models can be directly indicated by the choice of our model F .
P RELIMINARIES
Episodic Markov decision process.
Let M =
( S , A , P, r, H, d 1 ) to denote a finite-horizon Markov
Decision Process (MDP), where S is the arbitrary state space and A is the arbitrary action space which can be infinite or continuous.
The transition kernel P h : S × A (cid:55)→ ∆ S ( ∆ S represents a distribution over states) maps each state action ( s h , a h ) to a probability distribution P h ( ·| s h , a h ) and
P h can be different for different h (time-inhomogeneous).
H is the planning horizon and d 1 is the initial state distribution.
Besides, r : S × A (cid:55)→ R is the mean reward function satisfying 0 ≤ r ≤ 1 .
A policy π = ( π 1 , . . .
, π H ) assigns each state s h ∈ S a probability distribution over actions by mapping s h (cid:55)→ π h ( ·| s h ) ∀ h ∈
[ H ] and induces a random trajectory s 1 , a 1 , r 1 , . . .
, s H , a H , r H , s H + with s 1 ∼ d 1 , a h ∼ π ( ·| s h ) , s h +1 ∼ P h ( ·| s h , a h ) ,
∀ h ∈
[ H ] .
Given a policy π , the V -value functions and state-action value function (Q-functions) Q πh ( · , · ) ∈
R S × A are defined as: V πh ( s ) =
E [Equation]
[ (cid:80)
Ht
= h r t | s h
= s ] , Q πh ( s [Equation] a ) [Equation]
E π [ (cid:80)
Ht
= h r t | s h , a h =
s, a ] , ∀ s, a, h ∈
S , A , [ H ] .
The Bellman (optimality) equations follow ∀ h ∈
[ H ] , s, a ∈ S × A :
[Equation]
We define Bellman operator P h for any function V ∈ R S as P h ( V ) =
r h + (cid:82) S V dP h , then
P h ( V πh +1 ) =
Q πh and P h ( V ⋆h +1 ) =
Q ⋆h .
The performance measure is v π := E d 1 [ V π 1 ] =
E π,d 1 (cid:104)(cid:80)
Ht =1 r t (cid:105) .
Lastly, the induced state-action marginal occupancy measure for any h ∈
[ H ] is defined to be: for any E ⊆ S × A , d πh ( E ) :
=
E [( s h , a h ) ∈
E | s 1 ∼ d 1 , a i ∼ π ( ·| s i ) , s i ∼
P i − 1 ( ·| s i − 1 , a i − 1 ) , 1 ≤ i ≤ h ] and E π,h [ f ( s, a )] : = (cid:82) S×A f ( s, a ) d π h ( s, a ) dsda .
Offline Reinforcement Learning.
The goal of Offline RL is to learn the policy
π ⋆ := arg max π v π using only the historical data D
= { ( s τh , a τh , r τh , s τh +1 ) } h ∈
[ H ] τ ∈ [ K ] .
The data generating behavior policy is denoted as µ .
In the offline regime, we have neither the knowledge about µ nor the access to further exploration for a different policy.
The agent is asked to find a policy (cid:98) π such that v ⋆ − v (cid:98) π ≤ ϵ for the given batch data D and a specified accuracy ϵ > 0 .
2. A SSUMPTIONS
Function approximation in offline RL requires sufficient expressiveness of F .
In fact, even under the realizability and concentrability conditions, sample efficient offline RL might not be achievable (Foster et al., 2021).
Therefore, under the differentiable function setting (Definition 1.1), we make the following assumptions.
Assumption 2.1 (Realizability+Bellman Completeness) .
The differentiable function class F in
Definition 1.1 satisfies:
• Realizability: for optimal Q ⋆h , there exists θ ⋆h ∈ Θ such that Q ⋆h ( · , · ) =
f ( θ ⋆h , ϕ ( · )) ∀ h ; • Bellman Completeness: Let G :=
{ V ( · ) ∈ R S : such that ∥ V ∥ ∞ ≤ H } .
Then sup V ∈G inf f ∈F ∥ f − P h ( V ) ∥ ∞ ≤ ϵ F for some ϵ F ≥ 0 .
Realizability and Bellman Completeness are widely adopted in the offline RL analysis with general function approximations (Chen and Jiang, 2019; Xie et al., 2021) and Assumption 2.1 states its differentiable function approximation version.
Data coverage assumption.
Furthermore, in the offline regime, it is known that function approximation cannot be sample efficient for learning a ϵ -optimal policy without data-coverage assumptions when ϵ is small ( i.e. high accuracy)
(Wang et al., 2021a).
In particular, we consider two types of coverage assumptions and provide guarantees for them separately.
Assumption 2.2 (Concentrability Coverage) .
For any fixed policy π , define the marginal stateaction occupancy ratio as d πh ( s, a ) /d µh ( s, a ) ∀ s, a .
Then the concentrability coefficient is defined as
C eff :=
sup π sup h ∈ [ H ] ∥ d πh /d µh ∥ 22 ,d µh , where ∥ g ( · , · ) ∥ 2 ,d µ := (cid:112) E d µ [ g ( · , · ) 2 ] and C eff < ∞ .
This is the standard coverage assumption that has has been widely assumed in (Ernst et al., 2005;
Szepesvári and Munos, 2005; Chen and Jiang, 2019; Xie and Jiang, 2020a), and 2.2 is fully characterized by the MDPs.
In addition, we can make an alternative assumption 2.3 that depends on both the MDPs and the function approximation class F . {+
 It assumes a curvature condition for F .+}
Assumption 2.3 (Uniform Coverage) .
We have ∀ h ∈
[ H ] , there exists κ > 0 ,
• E µ,h (cid:104) ( f ( θ 1 , ϕ ( · , · )) − f ( θ 2 , ϕ ( · , · )))
2 (cid:105) ≥ κ ∥ θ 1 − θ 2 ∥ 22 , ∀ θ 1 , θ 2 ∈ Θ; ( ⋆ ) • E µ,h (cid:2) ∇ f ( θ, ϕ ( s, a )) · ∇ f ( θ, ϕ ( s, a ))
⊤ (cid:3) ≻ κI, ∀ θ ∈ Θ .
( ⋆⋆ )
In the linear function approximation regime, Assumption 2.3 reduces to Example 2.4 since ( ⋆ ) and ( ⋆⋆ ) are identical assumptions.
Therefore, 2.3 can be considered as a natural extension of 2.4 for differentiable class.
Concretely, if f ( θ, ϕ ) = ⟨ θ, ϕ ⟩ , then ( ⋆ )
E µ,h [( f ( θ 1 , ϕ ( · , · )) − f ( θ 2 , ϕ ( · , · ))) 2 ]
= ( θ 1 − θ 2 )
⊤ E µ,h [ ϕ ( · , · ) ϕ ( · , · ) ⊤ ]( θ 1 − θ 2 ) ≥ κ
∥ θ 1 − θ 2 ∥∀ θ 1 , θ 2 ∈ Θ ⇔ 2.4 ⇔ ( ⋆⋆ )
E µ,h (cid:2) ∇ f ( θ, ϕ ( s, a )) · ∇ f ( θ, ϕ ( s, a )) ⊤ (cid:3) ≻ κI .
We do point that 2.3 can be violated for function class F that is “not identifiable” by the data distribution µ ( i.e. , there exists f ( θ 1 ) , f ( θ 2 ) ∈ F , θ 1 ̸ = θ 2 s.t.
E µ,h [( f ( θ 1 , ϕ ( · , · )) − f ( θ 2 , ϕ ( · , · ))) 2 ] = 0 ).
Nevertheless, there are representative non-linear differentiable classes ( e.g. generalized linear model (GLM)) satisfying 2.3.
Example 2.4 (Linear function coverage assumption (Wang et al., 2021a;
Min et al., 2021;
Yin et al., 2022; Xiong et al., 2022)) .
Σ feature h :=
E µ,h (cid:2) ϕ ( s, a ) ϕ ( s, a ) ⊤ (cid:3) ≻ κI ∀ h ∈
[ H ] with some κ > 0 .
Example 2.5 (offline generalized linear model (Li et al., 2017; Wang et al., 2021b)) .
For a known feature map ϕ : S ×
A → B d and link function f : [ − 1 , 1] (cid:55)→ [ − 1 , 1] the class of GLM is F GLM :=
{ ( s, a ) (cid:55)→ f ( ⟨ ϕ ( s, a ) , θ ⟩ ) : θ ∈ Θ } satisfying E µ,h (cid:2) ϕ ( s, a ) ϕ ( s, a ) ⊤ (cid:3) ≻ κI .
Furthermore, f ( · ) is either monotonically increasing or decreasing and 0 < κ ≤ | f ′ ( z ) | ≤ K < ∞ , | f ′′ ( z ) | ≤ M < ∞ for all | z | ≤ 1 and some κ, K, M .
Then F GLM satisfies 2.3, see Appendix B.
D IFFERENTIABLE F UNCTION A PPROXIMATION IS P ROVABLY E FFICIENT
In this section, we present our solution for offline reinforcement learning with differentiable function approximation.
As a warm-up, we first analyze the vanilla fitted Q-learning (VFQL, Algorithm 2), which only requires the concentrability Assumption 2.2.
The algorithm is presented in Appendix I.
Theorem 3.1.
Choose 0 < λ ≤ 1 / 2 C 2Θ in Algorithm 2.
Suppose Assumption 2.1,2.2.
Then if
K ≥ max (cid:110) 512 κ 41 κ 2 (cid:16) log( 2 Hdδ )
+ d log(1 + 4 κ 31 κ 2 C Θ K 3 λ) (cid:17) , 4 λκ (cid:111) , with probability 1 − δ , the output (cid:98) π of VFQL guarantees
[Equation]
If the model approximation capacity is insufficient, 3.1 will induce extra error due to the large ϵ F .
If ϵ F → 0 , the standard statistical rate 1 √ K can be recovered and similar results are derived with general function approximation (GFA) (Chen and Jiang, 2019; Xie and Jiang, 2020a).
However, using concentrability coefficient conceals the problem-dependent structure and omits the specific information of differentiable functions in the complexity measure.
Owing to this, we switch to the stronger “uniform” coverage 2.3 and analyze the pessimistic fitted Q-learning (PFQL, Algorithm 1) to arrive at the conclusion that offline RL with differentiable function approximation is provably efficient.
Motivation of PFQL.
The PFQL algorithm mingles the two celebrated algorithmic choices: Fitted
Q-Iteration (FQI) and Pessimism.
However, before going into the technical details, we provide some interesting insights that motivate our analysis.
First of all, the square error loss used in FQI (Gordon, 1999; Ernst et al., 2005) naturally couples with the differentiable function class as the resulting optimization objective is more computationally tractable (since stochastic gradient descent (SGD) can be readily applied) comparing to other information-theoretical algorithms derived with general function approximation ( e.g. the maxmin objective in Xie et al.
(2021), eqn (3.2)).
4 In particular, FQI with differentiable function approximation resembles the theoretical prototype of neural FQI algorithm (Riedmiller, 2005) and DQN algorithm
(Mnih et al., 2015; Fan et al., 2020) when instantiating the model f to be deep neural networks.
Furthermore, plenty of practical algorithms leverage fitted-Q subroutines for updating the critic step ( e.g. (Schulman et al., 2017; Haarnoja et al., 2018)) with different differentiable function choices.
In addition, we also incorporate pessimism for the design.
Indeed, one of the fundamental challenges in offline RL comes from the distributional shift .
When such a mismatch occurs, the estimated/optimized Q -function (using batch data D ) may witness severe overestimation error due to the extrapolation of model f
(Levine et al., 2020).
Pessimism is the scheme to mitigate the error / overestimation bias via penalizing the Q-functions at state-action locations that have high uncertainties (as opposed to the optimism used in the online case), and has been widely adopted ( e.g.
(Buckman et al., 2020; Kidambi et al., 2020; Jin et al., 2021b)).
Algorithm 1 description.
Inside the backward iteration of PFQL, Fitted Q-update is performed to optimize the parameter (Line 4).
(cid:98) θ h is the root of the first-order stationarity equation (cid:80)
Kk =1 (cid:16) f
( θ, ϕ h,k ) − r h,k − (cid:98) V h
+1 ( s kh +1 )
(cid:17) · ∇ ⊤ θ f ( θ, ϕ h,k ) + λθ = 0 and Σ h is the Gram matrix with respect to ∇ θ f | θ = (cid:98) θ h .
Note for any s, a ∈ S × A , m ( s, a ) :
= ( ∇ θ f ( (cid:98) θ h , ϕ ( s, a ))
⊤ Σ − 1 h ∇ θ f ( (cid:98) θ h , ϕ ( s, a )))
− 1 measures the effective sample size that explored s, a along the gradient direction ∇ θ f | θ = (cid:98) θ h , and β/ (cid:112) m
( s, a ) is the estimated uncertainty at ( s, a ) .
However, the quantity m ( s, a ) depends on (cid:98) θ h , and (cid:98) θ h needs to be close to the true θ ⋆h ( i.e. (cid:98) Q h
≈ f ( (cid:98) θ h , ϕ ) needs to be close to Q ⋆h ) for the uncertainty estimation
Γ h to be valid, since putting a random θ into m ( s, a ) can cause an arbitrary Γ h that is useless (or might even deteriorate the algorithm).
Such an “implicit” constraint over (cid:98) θ h imposes the extra difficulty for the theoretical analysis due to that general differentiable functions encode nonlinear structures.
As a direct comparison, in the simpler linear MDP case, the uncertainty measure Γ h := (cid:113) ϕ ( · , · ) ⊤ (Σ linear h ) − 1 ϕ ( · , · ) is always valid since it does not depend on the least-square regression weight (cid:98) w h (Jin et al., 2021b).
Besides, the choice of β is set to be (cid:101) O ( dH ) in Theorem 3.2 and the extra higher order term (cid:101) O
( 1 K ) in Γ h is for theoretical reason only.
Algorithm
Set [-(cid:98) V H +
-]θ ∈ Θ (cid:26)(cid:80)
Kk =
(cid:104) f
( θ ϕ h k − r h,k − (cid:98) V h
+1 ( s kh +
(cid:105) 2 + λ · ∥ θ ∥(cid:27)
Set
←
= θ f ( θ h , h,k ∇ ⊤ θ f ( θ h ,
,k ) + λI d .
Set Γ h ( · , · ) ← β (cid:113) ∇ θ f ( (cid:98) θ h , ϕ ( · , · ))
⊤ Σ − 1 h ∇ θ f ( (cid:98) θ h , ϕ ( · , · )) (cid:16) + (cid:101) O
( 1 K ) (cid:17) 7:
Set ¯ Q h ( · , · ) ← f ( (cid:98) θ h , ϕ ( · , · ))
− Γ h ( · , · ) 8:
Set (cid:98) Q h ( · , · ) ← min (cid:8) ¯ Q h ( · , · ) , H − h + 1 (cid:9) + 9:
Set (cid:98) π h ( · | · ) ← arg max π h (cid:10) (cid:98) Q h ( · , · ) , π h ( · | · ) (cid:11) A , (cid:98) V h ( · ) ← max π h (cid:10) (cid:98) Q h ( · , · ) , π h ( · | · ) (cid:11)
A 10: end for 11: Output: { (cid:98) π h
Hh =1 .
Model-Based vs. Model-Free.
PFQL can be viewed as the strict generalization over the previous value iteration based algorithms, e.g. PEVI algorithm (Jin et al.
(2021b), linear MDPs) and the VPVI algorithm (Yin and Wang (2021), tabular MDPs).
On one hand, approximate value iteration (AVI) algorithms (Munos, 2005) are usually model-based algorithms (
for instance the tabular algorithm
VPVI uses empirical model (cid:98) P for planning).
On the other hand, FQI has the form of batch Q-learning update ( i.e. Q-learning is a special case with batch size equals to one), therefore is more of model-free flavor.
Since FQI is a concrete instantiation of the abstract AVI procedure (Munos, 2007), PFQL draws a unified view of model-based and model-free learning.
Now we are ready to state our main result for PFQL and the full proof can be found in Appendix D,E,F.
Theorem 3.2.
Let β = 8 dHι and choose 0 < λ ≤ 1 / 2 C 2Θ in Algorithm 1.
Suppose Assumption 2.1,2.3 with ϵ F = 0 .
Then if K ≥ max (cid:110) 512 κ 41 κ 2 (cid:16) log( 2 Hd δ )
+ d log(1 + 4 κ 31 κ 2 C Θ K 3 λ) (cid:17) , 4 λ κ (cid:111) , with probability 1 − δ , for all policy π simultaneously, the output of PFQL guarantees
[Equation]
where ι is a Polylog term and the expectation of π is taken over s h , a h .
In particular, if further
K ≥ max {
[Equation]
} we have
[Equation]
Here Σ ⋆h = (cid:80)
Kk =1 ∇ θ f
( θ ⋆h , ϕ ( s kh , a kh )) ∇ ⊤ θ f ( θ ⋆h , ϕ ( s kh , a kh ))
+ λI d and the definition of higher order parameter C hot , C ′ hot can be found in List A
Corollary 3.3 (Offline Generalized Linear Models (GLM)) .
Consider the GLM function class defined in 2.5.
Suppose β, λ, K are defined the same as Theorem 3.2.
ϵ F = 0 .
Then with probability 1 − δ , for all policy π simultaneously, PFQL guarantees
[Equation]
PFQL is provably efficient.
Theorem 3.2 verifies PFQL is statistically efficient.
In particular, by
Lemma L.5 we have ∥∇ θ f
( θ ⋆h , ϕ ) ∥ Σ
[Equation]
≲
[Equation]
√ κK , resulting the main term to be bounded by 32 dH
[Equation]
that recovers the standard statistical learning convergence rate 1 √ K .
Comparing to Jin et al.
(2021b).
Theorem 3.2 strictly subsumes the linear MDP learning bound in Jin et al.
(2021b).
In fact, in the linear case ∇ θ f
( θ, ϕ ) =
∇ θ ⟨ θ, ϕ ⟩ = ϕ and 3.2 reduces to O ( dH (cid:80)
[Equation]
Instance-dependent learning.
Previous studies for offline RL with general function approximation (GFA) (Chen and Jiang, 2019; Xie and Jiang, 2020b) are more of worst-case flavors as they usually rely on the concentrability coefficient C .
The resulting learning bounds are expressed in the form 7 O ( V 2max (cid:113) Cn ) that is unable to depict the behavior of individual instances.
In contrast, the guarantee with differentiable function approximation is more adaptive due to the instancedependent structure (cid:80)
Hh =1 E π ⋆ (cid:20)(cid:113) ∇ ⊤ θ
f
( θ ⋆h , ϕ )Σ
[Equation]
− 1 h ∇ θ f ( θ ⋆h , ϕ ) (cid:21) . This Fisher-Information style quantity characterizes the learning hardness of separate problems explicitly as for different MDP instances M 1 , M 2 , the coupled θ ⋆h,M 1 , θ ⋆h,M 2 will generate different performances via the measure (cid:80) Hh =1 E π ⋆ (cid:104)(cid:113) ∇ ⊤ θ f ( θ ⋆h,M i , ϕ )Σ
[Equation]
h ∇ [Equation] f ( θ ⋆h,M i , ϕ ) (cid:105) ( i = 1 , 2 ). Standard worst-case bounds ( e.g. from GFA approximation) cannot explicitly differentiate between problem instances.
Feature representation vs. Parameters.
One interesting observation from Theorem 3.2 is that the learning complexity does not depend on the feature representation dimension m but only on parameter dimension
d as long as function class F satisfies differentiability definition 1.1 (not even in the higher order term).
This seems to suggest, when changing the model f with more complex representations, the learning hardness will not grow as long as the number of parameters need to be learned does not increase.
Note in the linear MDP analysis this phenomenon is not captured since the two dimensions are coupled ( d = m ).
Therefore, this heuristic might help people rethink about what is the more essential element (feature representation vs. parameter space) in the representation learning RL regime ( e.g. low rank MDPs (Uehara et al., 2022)).
We leave the concrete understanding between features and parameters as the future work.
Technical challenges with differentiable function approximation (DFA).
Informally, one key step for the analysis is to bound | f ( (cid:98) θ h , ϕ )
− f ( θ ⋆h , ϕ ) | .
This can be estimated by the first order approximation ∇ f ( (cid:98) θ h , ϕ ) ⊤ · ( (cid:98) θ h − θ ⋆h ) .
However, different from the least-square value iteration (LSVI) objective (Jin et al., 2020; 2021b), the fitted Q-update (Line 4, Algorithm 1) no longer admits a closed-form solution for (cid:98) θ h .
Instead, we can only leverage (cid:98) θ h is a stationary point of
Z h ( θ ) := (cid:80) Kk =1 (cid:104) f
( θ, ϕ h,k ) − r h,k − (cid:98) V h
+1 (cid:0) s kh +
(cid:1)(cid:105) ∇ f ( θ, ϕ h,k ) + λ · θ (since Z h ( (cid:98) θ h ) = 0 ).
To measure the difference (cid:98) θ h − θ ⋆h , for any θ ∈ Θ , we do the Vector Taylor expansion
Z h ( θ ) − Z h ( (cid:98) θ h ) =
Σ sh ( θ − (cid:98) θ h ) + R K ( θ ) (where R K ( θ ) is the higher-order residuals) at the point (cid:98) θ h with Σ sh : = ∂ ∂θZ h ( θ ) (cid:12)(cid:12)(cid:12)(cid:12) [Equation]
=
[Equation]
h
[Equation]
(cid:88) [Equation] = [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
=
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
(cid:124)
[Table]
+
[Table]
The perturbation term ∆ Σ sh encodes one key challenge for solving (cid:98) θ h − θ ⋆h since it breaks the positive definiteness of Σ sh , and, as a result, we cannot invert the Σ sh in the Taylor expansion of Z h .
This is due to DFA (Definition 1.1) is a rich class that incorporates nonlinear curvatures.
In the linear function approximation regime, this hurdle will not show up since ∇ 2 θθ f ≡ 0 and ∆ Σ sh is always invertible as long as λ > 0 .
Moreover, for the off-policy evaluation (OPE) task, one can overcome this issue by expanding over the population counterpart of Z h at underlying true parameter of the given behavior target policy (Zhang et al., 2022).
However, for the policy learning task, we cannot use either population quantity or the true parameter θ ⋆h since we need a computable/data-based pessimism Γ h to make the algorithm practical.
Check the following section for more discussions of the analysis.
3. S KETCH OF THE PFQL A NALYSIS
Due to the space constraint, here we only overview the key components of the analysis.
To begin with, by following the result of general MDP in Jin et al.
(2021b), the suboptimality gap can be bounded by (Appendix D) (cid:80)
[Equation]
] if
[Equation]
To deal with P h (cid:98) V h
+1 , by Assumption 2.1 we can leverage the parameter Bellman operator T (Definition D.1) so that P h (cid:98) V h
+1 =
f ( θ T (cid:98) V h
+1 , ϕ ) .
Next, we apply the second-order approximation to obtain P
[Equation]
Later, we use (1) to represent
[Equation]
+ R K ( θ T (cid:98) V h +1 )
= Σ h ( θ T (cid:98) V h +1 − (cid:98) θ h ) + (cid:101) R K
( θ T (cid:98) V h +1 ) by denoting (cid:101) R K
( θ T (cid:98) V h +1 )
=
∆ Σ sh ( (cid:98) θ h − θ T (cid:98) V h +1 )
+ R K ( θ T (cid:98)
V h +1 ) .
Now that Σ − 1 h is invertible thus provides the estimation (note Z h ( (cid:98) θ h )
= 0 )
[Equation]
However, to handle the higher order terms, we need the explicit finite sample bound for ∥ θ T (cid:98) V h
+1 − (cid:98) θ h ∥(or ∥ θ ⋆h − (cid:98) θ h ∥ 2 ).
In the OPE literature, Zhang et al.
(2022) uses asymptotic theory (Prohorov’s Theorem) to show the existence of B ( δ ) such that ∥ (cid:98) θ h − θ ⋆h ∥ ≤ B ( δ ) √ K .
However, this is insufficient for finite sample/non-asymptotic guarantees since the abstraction of B ( δ ) might prevent the result from being sample efficient.
For example, if B ( δ ) has the form e H log( 1 δ ) , then e H log( 1 δ ) √ K is an inefficient bound since K needs to be e H /ϵ 2 large to guarantee ϵ accuracy.
To address this technicality, we use a novel reduction to general function approximation (GFA) learning proposed in Chen and Jiang (2019).
Concretely, we first bound the loss objective
E µ [ ℓ h ( (cid:98) θ h )] − E µ [ ℓ h ( θ T (cid:98) V h +1 )] via a “orthogonal” decomposition and by solving a quadratic equation.
The resulting bound can be directly used to further bound ∥ θ T (cid:98)
V h
+1 − (cid:98) θ h ∥ 2 for obtaining efficient guarantee (cid:101) O ( dH √ κK ) .
During the course, the covering technique is applied to extend the finite function hypothesis in Chen and Jiang (2019) to all the differentiable functions in Definition 1.1.
See
Appendix G for the complete proofs.
The full proof can be found in Appendix D,E,F.
I MPROVED L EARNING VIA V ARIANCE A WARENESS
In addition to knowing the provable efficiency for differentiable function approximation (DFA), it is of great interest to understand what is the statistical limit with DFA, or equivalently, what is the “optimal” sample/statistical complexity can be achieved in DFA (measured by minimaxity criteria)?
Towards this goal, we further incorporate variance awareness to improve our learning guarantee.
Variance awareness is first designed for linear Mixture MDPs (Talebi and Maillard, 2018;
Zhou et al., 2021) to achieve the near-minimax sample complexity and it uses estimated conditional variances Var P ( ·| s,a ) ( V ⋆h +1 ) to reweight each training sample in the LSVI objective.
9 Later, such a technique is leveraged by Min et al.
; Yin et al.
(2022) to obtained the instance-dependent results.
Intuitively, conditional variances σ 2 ( s, a ) :=
Var P ( ·| s,a ) ( V ⋆h +1 ) serves as the uncertainty measure of the sample ( s, a, r, s ′ )
that comes from the distribution P ( ·| s, a ) .
If σ 2 ( s, a ) is large, then the distribution P ( ·| s, a ) has high variance and we should put less weights in a single sample ( s, a, r, s ′ ) rather than weighting all the samples equally.
In the differentiable function approximation regime, the update is modified to
[Equation]
with σ 2 h ( · , · ) estimated by the offline data.
Notably, empirical algorithms have also shown uncertainty reweighting can improve the performances for both online RL (Mai et al., 2022) and
offline RL (Wu et al., 2021).
These motivates our variance-aware fitted Q-learning (VAFQL) algorithm 3.
Theorem 4.1.
Suppose Assumption 2.1,2.3 with ϵ F = 0 .
Let β = 8 dι and choose 0 < λ ≤ 1 / 2 C 2Θ in
Algorithm 3.
Then if K ≥ K 0 and √ d ≥ (cid:101) O ( ζ ) , with probability 1 − δ , for all policy π simultaneously, the output of VAFQL guarantees
[Equation]
where ι is a Polylog term and the expectation of π is taken over s h , a h .
In particular, we have
[Equation]
Here Λ ⋆h = (cid:80)
Kk =1 ∇ θ f
( θ ⋆h , ϕ h,k ) ∇ ⊤ θ f ( θ ⋆h , ϕ h,k ) /σ ⋆h ( s kh , a kh ) 2 + λI d and the σ ⋆h ( · , · ):= max { 1 , Var P h V ⋆h +1 ( · , · ) } .
The definition of K 0 , ¯ C hot , ¯ C ′ hot , ζ can be found in List A.
In particular, to bound the error for u h , v h and (cid:98) σ 2 h , we need to define an operator J that is similar to the parameter Bellman operator D.1.
The Full proof of Theorem 4.1 can be found in Appendix J.
Comparing to Theorem 3.2, VAFQL enjoys a net improvement of the horizon dependence since
Var P ( V ⋆ h ) ≤ H 2 .
Moreover, VAFQL provides better instance-dependent characterizations as the main term is fully depicted by the system quantities except the feature dimension d .
For instance, when the system is fully deterministic (transition P h ’s are deterministic), σ ⋆h ≈ Var P h V ⋆h
+1 ( · , · ) ≡(if ignore the truncation) and Λ ⋆ − 1 → 0 .
This yields a faster convergence with rate O ( 1 K ) .
Lastly, when reduced to linear MDPs, 4.1 recovers the results of Yin et al.
except an extra factor of √ d .
On the statistical limits.
To complement the study, we incorporate a minimax lower bound via a reduction to Zanette et al.
; Yin et al.
The following theorem reveals we cannot improve over Theorem 4.1 by more than a factor of √ d in the most general cases.
The full discussion is deterred to Appendix K.
Theorem 4.2 (Minimax lower bound) .
Specifying the model to have linear representation f = ⟨ θ, ϕ ⟩ .
There exist a pair of universal constants c, c ′ > 0 such that given dimension d , horizon H and sample size K > c ′ d 3 , one can always find a family of MDP instances such that for any algorithm (cid:98) π
[Equation]
where Λ ⋆,ph = E (cid:104) (cid:80) Kk =1 ∇ θ f ( θ ⋆h ,ϕ ( s kh ,a kh )) ·∇ θ f ( θ ⋆h ,ϕ ( s kh ,a kh ))
⊤
Var h ( V ⋆h +1 )
( s kh ,a kh ) (cid:105) .
C ONCLUSION , L IMITATION AND F UTURE D IRECTIONS
In this work, we study offline reinforcement learning with differentiable function approximation and show the sample efficiency of differentiable function learning.
We further improve the sample complexity with respect to the horizon dependence via a variance aware variant.
However, the dependence of the parameter space still scales with d (whereas for linear function approximation this dependence is √ d ), and this is due to applying covering argument for the rich class of differentiable functions.
For large deep models, the dimension of the parameter can be huge, therefore it would be interesting to know if certain algorithms can further improve the parameter dependence, or whether this d is essential.
Also, how to relax uniform coverage assumption 2.3 is unknown under the current analysis.
We leave these open questions as future work.
Lastly, the differentiable function approximation setting provides a general framework that is not confined to offline RL.
Understanding the sample complexity behaviors of online reinforcement learning (Jin et al., 2020), reward-free learning and representation learning (Uehara et al., 2022) might provide new and unified views over the existing studies.
Appendix A N OTATION L IST
[Table]
B F URTHER I LLUSTRATION THAT G ENERALIZED L INEAR M ODEL E XAMPLE SATISFIES 2.
Recall the definition in 2.5, then:
For ( ⋆⋆ ) ,
E µ,h (cid:2) ∇ f ( θ, ϕ ( s, a )) · ∇ f ( θ, ϕ ( s, a )) ⊤ (cid:3) =
E µ,h (cid:2) f ′ ( ⟨ θ, ϕ ( s, a ) ⟩ ) 2 ϕ ( · , · ) · ϕ ( · , · ) ⊤ (cid:3) ≻ κ 2 E µ,h (cid:2) ϕ ( · , · ) · ϕ ( · , · ) ⊤ (cid:3) ≻ κ 3 I, ∀ θ ∈ Θ
For ( ⋆ ) , by Taylor’s Theorem,
E µ, [Equation]
θ [Equation] , ϕ ( · , · )) −
[Equation]
( θ 2 , ϕ ( · , · )))
[Equation]
E [Equation] ,h [ f ′ ( θ s,a , ϕ ( · , · )) 2 ( θ 1 − θ 2 ) ⊤ ϕ ( · , · ) ϕ ( · , · ) ⊤ ( θ 1 − θ 2 )] ≥ κ 2 E µ,h [( θ 1 − θ 2 ) ⊤ ϕ ( · , · ) ϕ ( · , · ) ⊤ ( θ 1 − θ 2 )]
= κ 2 ( θ 1 − θ 2 ) ⊤ E µ,h [ ϕ ( · , · ) ϕ ( · , · ) ⊤ ]( θ 1 − θ 2 ) ≥ κ 3 ∥
θ 1 − θ 2 ∥and choose κ 3 as κ in 2.3.
C O N THE COMPUTATIONAL COMPLEXITY
Pessimistic Fitted Q-learning .
For storage, at each time step h ∈
[ H ] in Algorithm 1, we need to store (cid:98) θ h , Σ h and ∇ f ( (cid:98) θ h , ϕ h,k ) .
Therefore, the total space complexity is O ( dH + d 2 H + dKH ) .
For computation, assuming (cid:98) θ h is solved via SGD and let M denote the number of gradient steps, then the complexity is dominated by computing (cid:98) θ h , Σ h and Σ − 1 h , which results in O ( MH + KdH + d 3 H ) complexity.
The space complexity and computational complexity for VAFQL has the same order as PFQL except that the constant factors are larger.
D S OME BASIC CONSTRUCTIONS
First of all, Recall in the first-order condition, we have
[Equation]
[Table]
(cid:98) θ h = 0 , ∀ h ∈
[ H ] .
Therefore, if we define the quantity Z h ( · , · ) ∈ R d as
[Equation]
then we have (recall (cid:98) θ h ∈ Int(Θ) )
[Equation]
In addition, according to Bellman completeness Assumption 2.1, for any bounded V ( · ) ∈ R S with ∥ V ∥ ∞ ≤ H , inf f ∈F ∥ f − P h ( V ) ∥ ∞ ≤ ϵ F , ∀ h (recall P h ( V ) =
r h
+ (cid:82) S V dP h ).
Therefore, we can define the parameter Bellman operator T as follows.
Definition D.1 (parameter Bellman operator) .
By the Bellman completeness Assumption 2.1, for any ∥ V ∥ ∞ ≤ H , we can define the parameter Bellman operator T : V → θ T V ∈ Θ such that
[Equation]
Denote δ V :=
f ( θ T V , ϕ ) − P h ( V ) , then we have ∥ f ( θ T V , ϕ ) − P h ( V ) ∥ ∞ =
∥ δ V ∥ ∞ ≤ ϵ F .
In particular, by realizability Assumption 2.1 it holds θ T V [-(cid:63)h +
-]⋆h
{++
+}
= θ ⋆h and this is due to f ( θ T V ⋆h +1 , ϕ ) =
P h
( V ⋆h +1 )
=
V ⋆h =
f ( θ ⋆h , ϕ ) .
D. S UBOPTIMALITY DECOMPOSITION
Denote ι h ( s, a ) :=
P h (cid:98) V h
+1 ( s, a ) − (cid:98) Q h ( s, a ) , by Jin et al.
(2021b)
we have the following decomposition.
Lemma D.2 (Lemma 3.1 of Jin et al.
(2021b)) .
Let (cid:98) π
= { (cid:98) π h
Hh =1 a policy and (cid:98) Q h be any estimates with (cid:98) V h = ⟨ (cid:98) Q h ( s, · ) , (cid:98) π h ( · | s ) ⟩ A .
Then for any policy π , we have v
[Equation]
[Equation]
In particular, if we choose (cid:98) π h ( ·| s ) :
= arg max π ⟨ (cid:98) Q h ( s, · ) , π ( · | s ) ⟩ A , then
[Equation]
Lemma D.3.
Let (cid:98) P h be the general estimated Bellman operator.
Suppose with probability 1 − δ , it holds for all h, s,
a ∈
[ H ] × S × A that | ( P h (cid:98) V
h
+1 − (cid:98) P h (cid:98)
V h +1 )
( s, a ) | ≤ Γ h ( s, a ) , then it implies ∀ s, a, h ∈ S ×
A ×
[ H ] , 0 ≤ ζ h ( s, a ) ≤ 2Γ h
( s, a ) .
Furthermore, it holds for any policy π simultaneously, with probability 1 − δ
[Equation]
Proof of Lemma D.3.
This is a generic result that holds true for the general MDPs and was first raised by Theorem 4.2 of Jin et al.
(2021b).
Later, it is summarized in Lemma C.1 of Yin et al.
With Lemma D.3, we need to bound the term |P
h (cid:98) V h
+1 ( s, a ) − (cid:98) P h (cid:98) V h
+1 ( s, a ) | .
[Equation]
Throughout this section, we suppose ϵ F = 0 , i.e. f ( θ T V , ϕ ) =
P h ( V ) .
According to the regression oracle (Line 4 of Algorithm 1), the estimated Bellman operator (cid:98) P h maps (cid:98) V h +1 to (cid:98) θ h , i.e. (cid:98) P h (cid:98) V h
+1 =
f ( (cid:98) θ h , ϕ ) .
Therefore (recall Definition D.1)
[Equation]
where we apply the first-order Taylor expansion for the differentiable function f at point (cid:98) θ h and
Hot h, 1 is a higher-order term.
Indeed, the following Lemma E.1 bounds the Hot h, 1 term with (cid:101) O
( 1 K ) .
Lemma E.1.
Recall the definition (from the above decomposition)
Hot h, 1 := f
( θ T (cid:98) V h
+1 , ϕ
( s, a )) −
[Equation]
T (cid:98) V h
+1 − (cid:98) θ h (cid:17) , then with probability 1 − δ , | Hot h, 1 | ≤ 18 H 2 κ 2 (log( H/δ ) + C d, log K )
+ κ 2 λC 2 Θ κK , ∀ h ∈
[ H ] .
Proof of Lemma E.1.
By second-order Taylor’s Theorem, there exists a point ξ (lies in the line segment of (cid:98) θ h and θ T (cid:98) V h +1 ) such that [Equation] (cid:98)
[Equation]
(cid:98)
[Equation]
(cid:17)
Therefore, by directly applying Theorem G.2, with probability 1 − δ , for all h ∈
[ H ] ,
[Equation]
θ
[Equation]
[Equation]
2 κ 2 (log( H/δ ) + C d, log K ) + κ 2 λC 2Θ κK
E.
A NALYZING
[Equation]
f
[Equation]
VIA Z h
[Equation]
From (3) and Lemma E.1, the problem further reduces to bounding ∇ f ( (cid:98) θ h , ϕ ( s, a )) (cid:16) θ T (cid:98) V h
+1 − (cid:98) θ h (cid:17) .
To begin with, we first provide a characterization of θ T (cid:98) V
h
+1 − (cid:98) θ h .
Indeed, by first-order Vector Taylor expansion (Lemma L.1), we have (note Z h ( (cid:98) θ h | (cid:98) V h +1 )
= 0 )
for any θ ∈ Θ ,
[Equation]
(4) where R K ( θ ) is the higher-order residuals and Σ
sh :=
∂∂θ Z h ( θ | (cid:98) θ h +1 ) (cid:12)(cid:12)(cid:12) θ = (cid:98) θ h with Σ sh :=
∂ ∂θZ h ( θ | (cid:98) V h +1 ) (cid:12)(cid:12)(cid:12)(cid:12) θ
= (cid:98) θ h
= ∂
∂θ (cid:32) K (cid:88) k
= (cid:104) f
( θ, ϕ h,k ) − r h,k − (cid:98) V h
+1 ( s kh +1 )
(cid:105) ∇ f ( θ, ϕ h,k ) + λ · θ (cid:33) θ = (cid:98) θ h
[Equation]
here ∇ 2 = ∇ [Table] ∇ denotes outer product of gradients.
Note ∆ Σ sh is not desirable since it could prevent Σ sh from being positive-definite (and it could cause Σ sh to be singular).
Therefore, we first deal with ∆ Σ sh in below.
Lemma E.2.
With probability 1 − δ , for all h ∈
[ H ] ,
[Equation]
Proof of Lemma E.2.
Step1: We prove for fixed ¯ θ ∈ Θ , with probability 1 − δ , for all h ∈
[ H ] ,
[Equation]
K .
Indeed, we have
[Equation]
On one hand
[Equation]
by Theorem G.2 with probability 1 − δ/ 2 for all h ∈
[ H ]
[Equation]
K
[Equation]
On other hand
[Equation]
recall the definition of T
[Equation]
we have
[Equation]
) (cid:12)(cid:12)(cid:12) s kh , a kh (cid:105)(cid:17) · ∇ 2 θθ f (¯ θ, ϕ h,k ) = (cid:16) ( P h (cid:98) V h +1 )
( s kh , a kh ) − ( P h (cid:98) V h +1 ( s kh +1 ))
( s kh , a kh ) (cid:17) · ∇ 2 θθ f (¯ θ, ϕ h,k ) = 0 .
Also, since (cid:13)(cid:13)(cid:13)(cid:16) f
( θ T (cid:98) V h
+1 , ϕ h,k ) − r h,k − (cid:98) V h
+1 ( s kh +1 ))
(cid:17) · ∇ 2 θθ f (¯ θ, ϕ h ) (cid:13)(cid:13)(cid:13) 2 ≤ Hκ 2 , denote σ 2 := K ·
H 2 κ 22 , then by Vector Hoeffding’s inequality (Lemma L.2),
P
[Equation]
(cid:13) (cid:13)
K
[Equation]
k =
[Equation]
(cid:16) f
[Equation]
θ
[Equation]
V h
+1 , ϕ h,k ) − r h,k − (cid:98) V h
+
[Equation]
s
[Equation]
[Equation]
· ∇ 2 θθ f (¯ θ, ϕ h ) (cid:13)(cid:13) (cid:13) (cid:13) (cid:13) 2 ≥ t/K (cid:12)(cid:12) (cid:12) (cid:12) (cid:12) { s k h , a k h } K k = (cid:33) ≤ d · e − t 2 / 8 dKH 2 κ 22 := δ which is equivalent to
P (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
Define
[Equation] [Table]
[Equation]
[Table]
} , then by law of total expectation P
[Equation]
A
[Equation] [Equation]
E [ P [ A
[Equation] [Equation]
, a kh } Kk =1 ]]
[Equation]
E [1 − δ ] = 1 − δ , i.e. with probability at least 1 − δ/ 2 (and a union bound), [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] Using above and equation 6, equation 7 and a union bound, w.p. 1 − δ , for all h ∈
[ H ] ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
K
Step2: we finish the proof of the lemma.
Consider the function class (cid:110)
[Equation]
) (cid:13)(cid:13)(cid:13)
[Equation]
[Table] ¯ θ ∈ Θ (cid:111) , then by triangular inequality [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
By Lemma L.8, the covering number C of the ϵ -net of the above function class satisfies log C ≤ d log(1 + 2 C Θ Hκ 3 ϵ ) .
By choosing ϵ = 1 /K , by a union bound over C cases we obtain for all h ∈
[ H ]
[Equation]
K +
[Equation]
Combing Lemma E.2 and Theorem G.2 (and a union bound)
[Equation]
we directly have
Corollary E
[Equation]
With probability 1 − δ ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
K
[Equation]
Here (cid:101)
O absorbs all the constants and Polylog terms.
Now we select θ = θ T (cid:98) V h
+1 in equation 4, and denote (cid:101) R K
( θ T (cid:98) V h +1 )
=
∆ Σ sh ( (cid:98) θ h − θ T (cid:98) V h +1 )
+
R K ( θ T (cid:98) V h +1 ) , then equation 4 is equivalent to
Z
[Equation]
R
[Equation]
T (cid:98) V
[Equation]
Note
[Table]
implies
[Table]
, then
[Table]
[Table] [Table]
0 ) θ
[Equation]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We will bound second term Hot 2 to have higher order O ( 1 K ) in Section E.5 and focus on the first term.
By direct decomposition,
[Equation]
h (cid:34) K (cid:88) k
= (cid:16) f
( θ T (cid:98) V h
+1 , ϕ h,k ) − r h,k − (cid:98) V h
+1 ( s kh +1 )
(cid:17) · (cid:16) ∇ ⊤ θ f ( θ T (cid:98) V h
+1 , ϕ h,k ) − ∇ ⊤ θ f ( (cid:98) θ h , ϕ h,k ) (cid:17)(cid:35) (cid:124) (cid:123)(cid:122) (cid:125) :
= I
E. B OUNDING THE TERM I
We first bound the term I 3 .
We have the following Lemma.
Lemma E.4.
For any fixed V ( · ) ∈ R S with ∥ V ∥ ∞ ≤ H and any fixed θ such that ∥ θ T V − θ ∥ 2 ≤ (cid:113)
[Equation]
Let (cid:101) I 3 :=
∇ f ( (cid:98) θ h , ϕ ( s, a ))
⊤ Σ
[Equation] 1 h (cid:34) [Equation] (cid:88)
[Equation]
[Equation]
)) (cid:35) , and if K ≥ max (cid:110) 512 κ 41 κ 2 (cid:16) log( 2 d δ ) + d log(1 + 4 κ 1 D 2 κ 2 C Θ K 3 λ) (cid:17) , 4 λ κ (cid:111) , then with probability 1 − δ , (where D = max { κ 1 , (cid:113) (144 dH 2 κ [-
-]22 ( H 2 log( H/δ )
+ C d, log K )
+8 dH 2 κ 22 λC 2Θ )log
( d/δ ) κ } ) | (cid:101) I 3 | ≤ 4 κ(cid:114)
(144 dH 2 κ 22 ( H 2 log( H/δ ) + C d, log K ) + 8 dH 2 κ 22 λC 2Θ ) log( d/δ ) κ
K + O ( 1 K 3 / 2 ) .
Proof of Lemma E
[Table]
Indeed, with probability 1 − δ/ 2 ,
[Table]
Σ − [Equation] where, under the condition K ≥ max (cid:110) [-
-]512 κ 41 κ 2 (cid:16) log(
[Equation]
dδ
[Equation]
+ d log [Equation] 1 +
[Equation]
κ
[Equation]
K
[Equation]
) (cid:17) , 4 λκ (cid:111) , we applied Lemma L.5 .
Next, on one hand,
[Equation]
On the other hand,
[Equation]
Therefore by Vector Hoeffding’s inequality (Lemma L.2) (also note the condition for boundedness (cid:13)(cid:13)(cid:0)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Table] probability [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
and this implies with probability 1 − δ/ 2 , [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table]
θ, ϕ h,k ))
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:114) (144 dH 2 κ 2 2 ( H 2 log( H/δ )
+ C d, log K )
+ 8 dH 2 κ 2 2 λC 2Θ ) log( d/δ ) κ choose u = (cid:80) Kk =1 (cid:0)
f ( θ T V , ϕ h,k ) − r h,k − V ( s kh +1 ) (cid:1) · ( ∇ θ f ( θ T V , ϕ h,k ) − ∇ θ f
( θ, ϕ h,k )) in
Lemma L.5, by a union bound we obtain with probability 1 − δ |
[Equation]
I
[Equation]
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:16)
[Equation]
[Equation]
[Equation]
) (cid:17)
[Equation]
( ∇
[Equation]
f ( θ
[Equation]
V , ϕ
[Equation]
,k ) − ∇
[Equation]
f ( θ, ϕ h,k ))
(cid:35)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤
[Equation]
(cid:19)
[Equation]
=
[Equation]
(cid:16)
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
θ
[Equation]
[Equation]
[Equation]
[Equation]
θ f
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
[Equation]
Lemma E.5.
Under the same condition as Lemma E
[Equation]
With probability 1 − δ ,
[Equation]
Here
[Equation]
absorbs Polylog terms.
Proof of Lemma E
[Equation]
Define
[Equation]
then
[Equation]
≤
[Equation]
[Table]
[Equation]
Let C a be the ϵ/ 3 (2 κ 21 + Hκ 2 ) K -covering net of { θ : ∥ θ ∥ 2 ≤ C Θ } , C V be the ϵ 6 κ 1 K -covering net of V defined in Lemma L.9 and C b be the ϵ 3 Hκ 2 K -covering net of { θ :
∥ θ ∥ 2 ≤ C Θ } , then by Lemma L. and Lemma L.9,
[Equation]
Further notice with probability 1 − δ/ 2 (by Lemma L.5), for all fixed sets of parameters θ, V satisfies ∥ θ T V − θ
[Equation]
≤ [-C Θ } , C V be the (cid:15) 6 κ 1 K -covering net of V deﬁned in Lemma L.9 and C b be the (cid:15) 
 Hκ 2 K -covering net of { θ : (cid:107) θ (cid:107) 2 ≤ C Θ } , then by Lemma L. and Lemma L.9,-](cid:113) 36 H
[Equation]
log
[Equation]
2 H/δ )+ C d, log K )+2 λC 2Θ κK simultaneously,
[Equation]
θ h , ϕ
[Equation]
s, a )) (cid:13)(cid:13)(cid:13) Σ − 1 h · (cid:13)(cid:13)(cid:13)
[Equation]
( (cid:98)
[Equation]
h +
[Equation]
, θ
[Equation]
(cid:98) V h
+
[Equation]
(cid:98)
[Equation]
[Equation]
θ T
[Equation]
θ ) (cid:13)(cid:13)(cid:13) Σ − 1 h
[Equation]
(cid:13)(cid:13)(cid:13) Σ − 1 h and (cid:13)(cid:13) (cid:13) θ T (cid:98) V
h
+1 − (cid:98) θ h (cid:13)(cid:13) (cid:13) 2 ≤
[Equation]
with probability 1 − δ/ 2 by Theorem G.2.
Now, choosing ϵ =
O (1 /K 2 ) and by Lemma E.4 and union bound over covering instances
, we obtain with probability 1 − δ | I 3 | ≤
[Equation]
[Equation]
E. B OUNDING THE SECOND TERM I
In this section, we bound the term
[Equation]
The following Lemma shows that I 2 is a higher-order error term with rate (cid:101)
O ( 1 K ) .
Lemma E.6 (Bounding I 2 ) .
If K satisfies K ≥ [Equation] [Equation] ) (cid:17) , and
K ≥ 4 λ/κ , then with probability 1 − δ
[Equation]
Here (cid:101)
O absorbs constants and Polylog terms.
Proof of Lemma E.6.
Step1.
Define η k
( V ) :
= f ( θ T V , ϕ h,k ) − f ( θ T V ⋆h
+1 , ϕ h,k ) − V ( s kh +1 )
+
V ⋆h +1 ( s kh +1 ) and let ∥ V ( · ) ∥ ∞ ≤ H be any fixed function such that sup s kh ,a kh ,s kh +1 | η k ( V ) | ≤ (cid:101) O ( κ 1 H 2 (cid:113) d 2 κK ) , i.e. arbitrary fixed V function in the neighborhood (measured by η k ) of V ⋆ h
+1 .
Then by definition of T it holds E [ η k ( V, θ ) | s kh , a kh ] = 0 .
Let the fixed θ ∈ Θ be arbitrary and define x k ( θ )
=
∇ θ f
( θ, ϕ h,k ) .
Next, define G h ( θ ) =
(cid:80)
Kk =1 ∇ f ( θ, ϕ ( s kh , a kh )) · ∇ f ( θ, ϕ ( s kh , a kh ))
⊤ + λI d , since ∥ x k ∥ 2 ≤ κ 1 and | η k | ≤ (cid:101) O
( κ 1 H 2 (cid:113) d 2 κK ) , by self-normalized Hoeffding’s inequality (Lemma L.3), with probability 1 − δ (recall t := K in Lemma L.3),
[Equation]
) (cid:115)
[Equation]
log (cid:18) λ
[Equation]
λδ (cid:19) .
Step2.
Define h
( V, θ ) :
= (cid:80)
Kk =1 x k ( θ ) η k ( V ) and H ( V, θ ) :
= (cid:13)(cid:13)(cid:13)(cid:80)
Kk =1 x k ( θ ) η k ( V ) (cid:13)(cid:13)(cid:13) G h ( θ ) − 1 , then note by definition | η k ( V ) | ≤ 2 H , which implies ∥ h ( V, θ ) ∥ 2 ≤ 2 KHκ 1 and | η k ( V 1 ) − η k
( V 2 ) | ≤ |P
h V 1 − P
[-(2 H (cid:107) x k ( θ 1 ) − x k ( θ 2 ) (cid:107) 
-]
h V 2 | + ∥ V 1 − V 2 ∥ ∞ ≤ 2 ∥ V 1 − V 2 ∥ ∞ and
[Equation]
{+∥ x k ( θ 1 ) − x k ( θ 2 ) ∥ 
+}
+ κ 1 | η k ( V 1 )
[Equation]
η k ( V 2 ) | )
[Equation]
∞ ) .
Furthermore, (cid:13)(cid:13)
[Equation]
θ
[Equation]
θ
[Equation]
[Equation]
[Equation]
[Equation]
h
[Equation] [Equation]
h
[Equation]
[Equation]
2 (cid:13)(cid:13)
[Equation]
h
[Equation]
[Equation]
[Equation]
All the above imply
[Equation]
Then a ϵ -covering net of { H ( V, θ ) } can be constructed by the union of
[Equation]
-covering net of { θ ∈ Θ } and
[Equation]
covering net of V in Lemma L.9.
The covering number N ϵ satisfies
[Equation]
Step3.
First note by definition in Step
[Equation]
and with probability 1 − δ | η k ( (cid:98) V h +
[Equation]
where the second inequality uses θ T V ⋆h +
= θ ⋆h and the third inequality uses Theorem G.2 and
Theorem G.3.
The last equal sign is due to C d, log K ≤ (cid:101) O
( d 2 ) (recall Lemma G.1).
Now choosing ϵ =
O (1 /K ) in Step2 and union bound over both equation 9 and covering number in
Step2, we obtain with probability 1 − δ ,
[Equation]
where we absorb all the Polylog terms.
Meanwhile, by Lemma L.5 with probability 1 − δ ,
[Equation]
Finally
[Equation]
by equation 10 and equation 11 and a union bound, we have with probability 1 − δ ,
[Equation]
where the first inequality is Cauchy–Schwarz inequality
[Equation]
E. B OUNDING THE MAIN TERM I
In this section
[Equation]
we bound the dominate term
[Equation]
First of all, by –Schwarz inequality, we have
[Equation]
Then we have the following Lemma to bound I 1 .
Lemma E.7.
With probability 1 − δ ,
[Equation]
where C δ, log K only contains Polylog terms.
Proof of Lemma E.7.
Step1.
Let the fixed θ ∈ Θ be arbitrary and define x k ( θ )
=
∇ θ f
( θ, ϕ h,k ) .
Next, define G h ( θ ) =
(cid:80)
Kk =1 ∇ f ( θ, ϕ ( s kh , a kh )) · ∇ f ( θ, ϕ ( s kh , a kh ))
⊤ + λI d , then ∥ x k ∥ 2 ≤ κ 1 .
Also denote η k :=
f ( θ T V ⋆h +1 , ϕ h,k ) − r h,k − V {+⋆h +
+} ( s kh +1 ) , then E [ η k | s kh , a kh ] = 0 and | η k | ≤ H .
Now by self-normalized Hoeffding’s inequality (Lemma L.3), with probability 1 − δ (recall t :=
K in
Lemma L.3), (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K (cid:88) k
= x k ( θ ) η k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
G h ( θ ) −≤ 2 H (cid:115) d log (cid:18) λ + Kκλδ (cid:19) .
Step2.
Define h ( θ ) :
= (cid:80)
Kk =1 x k ( θ ) η k and H ( θ ) :
= (cid:13)(cid:13)(cid:13)(cid:80)
Kk =1 x k ( θ ) η k (cid:13)(cid:13)(cid:13) G h ( θ ) − 1 , then note by definition | η k | ≤ H , which implies ∥ h ( θ ) ∥ 2 ≤ KHκ 1 and by x k ( θ 1 ) − x k ( θ 2 ) = ∇ 2 θθ f ( ξ, ϕ ) · ( θ 1 − θ 2 ) , ∥ h ( θ 1 ) − h ( θ 2 ) ∥ 2 ≤ K max k
( H ∥ x k ( θ 1 ) − x k ( θ 2 ) ∥ 2 ) ≤ HKκ {+
 ∥+} θ 1 − θ 2 ∥ 2 .
Furthermore, (cid:13)(cid:13)
G h ( θ 1 ) − 1 − G h ( θ 2 ) − 1 (cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)
G h ( θ 1 ) − 1 (cid:13)(cid:13) {+
 ∥ G h ( θ 1 ) − G h ( θ 2 ) ∥ 2 (cid:13)(cid:13) G h ( θ 2 ) − 1 (cid:13)(cid:13)≤λ 2 K sup k (cid:13)(cid:13) ∇ f ( θ 1 , ϕ h,k )+}
[Equation]
All the above imply
[Equation]
Then a ϵ -covering net of { H ( θ ) } can be constructed by the union of ϵ
[Equation]
√ 4 K [Equation]
[Equation]
κ [Equation] [Equation] + √
[Equation]
K 3 H 2 κ 31 κ 2 /λ 2 (cid:17) 2 -covering net of { θ ∈ Θ } .
By Lemma L.8, the covering number N ϵ satisfies
[Equation]
Step3. First note by definition in Step
[Equation]
Now choosing ϵ =
O (1 /K ) in Step2 and union bound over the covering number in Step [Equation] we obtain with probability 1 − δ ,
[Equation]
where we absorb all the Polylog terms.
Combing above with equation
[Equation]
we obtain with probability 1 − δ ,
[Equation]
where C δ, log K only contains Polylog terms
[Equation]
E. A NALYZING Hot 2 IN EQUATION
Lemma E.8.
Recall Hot
[Equation]
If K ≥ max (cid:110) 512 κ 41 κ 2 (cid:16) log
[Equation]
2 dδ ) + d log [Equation] 1 + 4 κ
[Equation] K [Equation] ) (cid:17) , 4 λκ (cid:111) , then with probability 1 − δ , (cid:12)(cid:12)(cid:12) ∇ f ( (cid:98) θ h
[Equation]
ϕ ( s
[Equation]
a
[Equation]
)Σ
[Equation]
[Equation]
R K [Equation] where (cid:101)
O absorbs all the constants and Polylog terms.
Proof of Lemma E.8.
Step1: we first show with probability 1 − δ
[Equation]
Recall by plug in θ T (cid:98) V h
+1 in equation 4, we have
[Equation]
and by second-order Taylor’s Theorem we have
[Equation]
[Table] [Equation]
[Equation]
Therefore, we can bound κ z 2 with κ z 2 ≤ ( Hκ 3 +3 κ 1 κ 2 ) K and this implies with probability 1 − δ/ 2 ,
[Equation]
And by Corollary E.3 with probability 1 − δ/ 2 ,
[Equation]
Therefore, by Lemma L.5 and a union bound with probability 1 − δ ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
K   where (cid:101)
O absorbs all the constants and Polylog terms.
Here the last inequality uses bound for
[Table]
h + [Table] 2 and (cid:13)(cid:13)(cid:13) ∆ Σ sh
[Table]
Step2:
By Lemma L.5, with probability 1 − δ ,
[Equation]
[Table]
[Equation]
F P ROOF OF T HEOREM 3.
Now we are ready to prove Theorem 3.2.
In particular, we prove the first part.
Also, recall that we consider the exact Bellman completeness ( ϵ F = 0 ).
F. T HE FIRST PART
Proof of Theorem 3.2 (first part).
First of all, from the previous calculation (3), (8), we have (cid:12)(cid:12)(cid:12) P h (cid:98) V h
+1 ( s, a ) − (cid:98) P h (cid:98) V h
+1 ( s, a ) (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12) ∇ f ( (cid:98) θ h , ϕ ( s, a ))
(cid:16) θ T (cid:98) V h
+1 − (cid:98) θ h (cid:17)(cid:12)(cid:12)(cid:12) + |
Hot h, 1 | ≤| I 1 | + | I 2 | + | I 3 | + | Hot h, 2 | + | Hot h, 1 |
Now by Lemma E.5, Lemma E.6, Lemma E.7, Lemma E.8 and Lemma E.1 (and a union bound), with probability 1 − δ ,
[Equation]
Finally, Plug the above into Lemma D [Equation] 3, by a union bound over all h ∈
[ H ] , we have with probability 1 − δ , for any policy π ,
[Equation]
where ι = C δ, log K only contains Polylog terms and
[Equation]
F. T HE SECOND PART
Next we prove the second part of Theorem
[Equation]
Proof of Theorem 3.2 (second part).
Step1.
Choose π = π ⋆ in the first part, we have
[Equation]
Next, by the triangular inequality of the norm to obtain
[Equation]
since with probability 1 − δ
[Equation]
where the last inequality uses part three of Theorem G
[Equation]
Then by a union bound and Lemma L.5,
[Equation]
[Table]
[Equation]
Second, by Lemma L.6 with probability 1 − δ
[Equation]
This implies
[Equation]
and also by Weyl’s spectrum theorem and under the condition K ≥ 128 κ 41 log(2 d/δ ) κ, with probability
[Equation]
[-, φ ) (cid:62) ] (cid:1) + λ K −√ 2 κ 21 √ K (cid:18) log 2 d δ (cid:19) 
-]then (cid:13)(cid:13)(cid:13) ( Σ ⋆h
[Equation]
[-K (cid:18) log 2 d δ (cid:19) 
 /≥ κ then (cid:13)(cid:13)(cid:13) ( Σ (cid:63)h K ) − 1-] (cid:13)(cid:13)(cid:13) ≤ 2 κ .
Similarly, with probability 1 − δ , (cid:13)(cid:13)
( Σ h K ) − 1 (cid:13)(cid:13) ≤ 2 κ .
Then by Lemma L.7, ∥∇ θ f
( θ ⋆h , ϕ ( s, a ))
[Equation]
K [Equation] s, a )) [Equation]
− 1 h [Equation] as long as K ≥ (cid:101) O ( ( κ 21 + λ ) 2 κ 22 κ 21 H 4 d 2 κ) .
The above is equivalently to ∥∇ θ f
[Equation]
⋆h , ϕ ( s h , a h ))
∥ Σ
[Equation]
1 h ≤ 2 ∥∇ θ f
[Equation]
θ ⋆h , ϕ
( s h , a h ))
∥ Σ ⋆ − 1 h .
Combining Step1, Step2 and a union bound, we have with probability 1 − δ ,
[Table]
[Equation] [Table] [Equation]
[Table]
where C ′ hot =
C hot + κ 1 κ
[Table]
2 κ
[Table]
G P ROVABLE E FFICIENCY BY REDUCTION TO G ENERAL F UNCTION A PPROXIMATION
In this section, we bound the accuracy of the parameter difference (cid:13)(cid:13)(cid:13)(cid:98) θ h − θ T (cid:98) V h
+1 (cid:13)(cid:13)(cid:13) 2 via a reduction to
General Function Approximation scheme in Chen and Jiang (2019).
Recall the objective
[Equation]
θ, [Equation] s [Equation]
, a kh [Equation] , a [Equation]
[Equation] Then by definition, (cid:98) θ h := arg min θ ∈ Θ ℓ h ( θ ) and θ T (cid:98) V h
+1 satisfies f ( θ T (cid:98) V h
+1 , ϕ )
= P h (cid:98) V h
+1 + δ (cid:98) V h
+1 .
Therefore, in this case, we have the following lemma:
Lemma G.1.
Fix h ∈
[ H ] .
With probability 1 − δ ,
[Equation]
where the expectation over µ is taken w.r.t.
[Equation]
1 , ..., K only (i.e., first compute E µ [ ℓ h ( θ )] for a fixed θ , then plug-in either (cid:98) θ
h +
or θ T (cid:98)
V h +1 ).
Here
[Equation]
[-+ d 2 log (cid:16) 
-]
Proof of Lemma G.1.
Step1: we first prove the case where λ = 0 .
Indeed, fix h ∈ [ H ] and any function V ( · ) ∈ R S .
Similarly, define f V ( s, a ) :
= f ( θ T V , ϕ ) =
P h V + δ V .
For any fixed θ ∈ Θ , denote g ( s, a ) =
f ( θ, ϕ ( s, a )) .
Then define
X ( g, V, f V ) :=
( g ( s, a )
[Equation]
Since all episodes are independent of each other,
X k ( g, V, f V ) :=
X ( g ( s k h , a k h ) , V ( s k h +1 ) , f V ( s k h , a k h )) are independent r.v.s and it holds
[Equation]
Next, the variance of X is bounded by:
[Equation]
where the step ( ∗ ) comes from
[Equation]
where the last step uses law of total expectation and the definition of f V .
Therefore, by Bernstein inequality [Equation] with probability 1 − δ ,
[Equation]
Now [Equation] if we choose g ( s, a ) :
= f ( (cid:98) θ h , ϕ ( s, a )) , then (cid:98) θ h minimizes ℓ h ( θ ) , therefore, it also minimizes
K (cid:80) Kk
= [Equation]
i ( θ, (cid:98) V h
+1 , f (cid:98) V h +1 )
and this implies
[Equation]
Therefore, we obtain
[Equation]
However, the above does not hold with probability 1 − δ since (cid:98) θ h and (cid:98) V h
+ := min { max a f ( (cid:98) θ h +1 , ϕ ( · , a )) − (cid:113) ∇ f ( (cid:98) θ h +1 , ϕ ( · , a )) ⊤ A · ∇ f ( θ, ϕ ( · , a )) , H } (where A is certain symmetric matrix with bounded norm) depend on (cid:98) θ h and (cid:98) θ h +1 which are data-dependent.
Therefore, we need to further apply covering Lemma L.10 and choose ϵ =
O (1 /K ) and a union bound to obtain with probability 1 − δ ,
[Equation]
where C d, log( K ) :
= log(1+24 C Θ ( H + 1) κ 1 K )+ d log (cid:16) 1 + 288 H 2 C Θ ( κ 1 √ C Θ + 2 (cid:112) κ 1 κ 2 /λ )
2 K 2 (cid:17)
+ d 2 log (cid:16)
+ 288 H 2 √ dκ 21 K 2 /λ (cid:17) .
12 Solving this quadratic equation to obtain with probability 1 − δ ,
E
[Equation]
Now according to equation 18, by definition we finally have with probability 1 − δ (recall the expectation over µ is taken w.r.t.
( s kh , a kh , s kh +1 ) k
= 1 , ..., K only)
[Equation]
Step2.
If λ > 0 , there is only extra term λK (cid:16)(cid:13)(cid:13)(cid:13)(cid:98) θ h (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13) θ T (cid:98) V h
+1 (cid:13)(cid:13)(cid:13) 2 (cid:17)
[Equation]
in addition to above.
This finishes the proof.
Theorem G.2 (Provable efficiency (Part I)) .
Let C d, log K be the same as Lemma G.1.
Then denote
[Equation]
probability
[Equation]
Proof of Theorem G.2.
Apply a union bound in Lemma G
[Equation]
1, we have with probability 1 − δ ,
[Equation]
) Now we prove for all h ∈
[ H ] ,
[Equation]
Indeed, similar to equation
[Equation]
by definition we have
E µ
[Equation] [Table]
[Table]
[Table]
where the third identity uses µ is taken w.r.t.
s h , a h , s h
+
(recall Lemma G.1) and law of total expectation.
The first inequality uses the definition of θ T (cid:98) V h
+1 .
Now apply Assumption 2.3, we have
[Equation]
Combine the above with equation 21 and equation 22, we obtain the stated result.
Theorem
G.
(Provable efficiency (Part
II)) .
Let
C d, log K be the same as
Lemma G.1 and suppose ϵ F = 0 .
Furthermore, suppose λ ≤ 1 / 2 C 2Θ and K ≥ max
[Equation]
[Equation] 4 λκ (cid:111) .
Then, with probability 1 − δ , ∀ h ∈ [ H ] ,
[Equation]
Furthermore, we have with probability 1 − δ , sup
[Equation]
where (cid:101) O absorbs Polylog terms and higher order terms.
Lastly, it also holds for all h ∈
[ H ] , w.p.
[Equation]
Proof of Theorem G.3.
Step1: we show the first result.
We prove this by backward induction.
When h = H + 1 , by convention f ( (cid:98) θ h , ϕ ( s, a )) =
f ( θ ⋆h , ϕ ( s, a ))
=
so the base case holds.
Suppose for h + 1 , with probability 1 − ( H − h ) δ , sup s,a (cid:12)(cid:12)(cid:12) f
( (cid:98) θ h +1 , ϕ ( s, a ))
− f
( θ ⋆h +1 , ϕ ( s, a ))
(cid:12)(cid:12)(cid:12) ≤ C h +1 (cid:113) 1 K + a ( h + 1) , we next consider the case for t = h .
On one hand, by Theorem G.2, we have with probability 1 − δ/ 2 ,
[Equation]
+1 (cid:12)(cid:12)(cid:12) ≤ κ(cid:114)
36 H 2 (log( H/δ ) + C d, log K ) + 2 λC 2 Θ κK + (cid:13) (cid:13)(cid:13) (cid:98) V h
+1 − V ⋆h +1 (cid:13) (cid:13)(cid:13) ∞ ,
Recall (cid:98) V h
+
( · ) := min { max a f ( (cid:98) θ h +1 , ϕ ( · , a ))
− Γ h ( · , a ) , H } and
V ⋆h +1 ( · ) = max a f ( θ ⋆h +1 , ϕ ( · , a ))
=
min { max a f ( θ ⋆h +1 , ϕ ( · , a )) , H } , we obtain (cid:13)(cid:13)(cid:13)(cid:98) V h
+1 − V {+⋆h +
+} (cid:13)(cid:13)(cid:13) ∞ ≤ sup s,a (cid:12)(cid:12)(cid:12) f ( (cid:98) θ h +1 , ϕ
( s, a ))
− f ( θ ⋆h +1 , ϕ ( s, a )) (cid:12)(cid:12)(cid:12) + sup h,s,a Γ h ( s, a ) (23)
Note the above holds true for any generic Γ h ( s, a ) .
In particular, according to Algorithm 1, we specify
[Equation]
and by Lemma L.5, with probability 1 − δ ,
[Equation]
and by a union bound this implies with probability 1 − ( H − h + 1) δ , [Equation]
[Equation]
≤ C
[Equation]
[Equation]
Solving for C h , we obtain C h ≤ κ 1 H (cid:113) 36 H 2 (log [Equation] H/δ [Equation]
+ [Equation] d
[Equation]
log K
[Equation] +2 λC [Equation] κ + H 2 dHκ 1 √ κ for all H .
By a union bound (replacing δ by δ/H ), we obtain the stated result.
Step2: Utilizing the intermediate result equation 23, we directly have with probability 1 − δ ,
[Equation]
[Equation] [Equation] [Equation] , where sup s,a (cid:12)(cid:12)(cid:12) f ( (cid:98) θ h , ϕ ( s, a )) − f ( θ ⋆h , ϕ ( s, a ))
(cid:12)(cid:12)(cid:12) can be bounded using Step1.
Step3: Denote M :=
(cid:18)
[Equation]
) , then by
Step1 we have with probability 1 − δ (here ξ is some point between (cid:98) θ h and θ ⋆h ) for all h ∈
[ H ]
[Equation]
where the last inequality is by Assumption 2.3.
Solve this to obtain the stated result.
H W ITH POSITIVE B ELLMAN COMPLETENESS COEFFICIENT ϵ F >
In Theorem 3.2, we consider the case where ϵ F = 0 .
If ϵ F > 0 , similar guarantee can be achieved with the measurement of model misspecification.
For instance, the additional error (cid:113)
[Equation]
will show up in Lemma G.1 (as stated in the current version), (cid:113) b d,K,ϵ
F κ + (cid:113) 2 Hϵ F κ will show up in Lemma G.2.
Then the decomposition in equation 3 will incur the extra δ (cid:98) V h +1 term with δ (cid:98)
V h
+1 might not be 0 .
The analysis with positive ϵ F > 0 will make the proofs more intricate but incurs no additional technical challenge.
Since the inclusion of this quantity is not our major focus, as a result, we only provide the proof for the case where ϵ F =
so the readers can focus on the more critical components that characterize the hardness of differentiable function class.
I VFQL AND ITS ANALYSIS
We present the vanilla fitted Q-learning (VFQL)
Algorithm 2 as follows.
For VFQL, no pessimism is used and we assume (cid:98) θ h ∈ Θ without loss of generality.
1: Input Denote : Set
3: for
[-+
-]=
1 do 4:
Set (cid:98) [Table] arg
[Table]
(cid:26)(cid:80)
Kk =
[Table]
(cid:104)
[Table]
[Table]
[Table]
[Table]
) (cid:105)
[Table]
(cid:27)
[Table]
Set
[Table]
[Table]
[Table]
) (cid:11)
[Table]
7: end for 8: Output: { (cid:98)
[Table]
h } Hh =1 .
I. A NALYSIS FOR VFQL (T HEOREM 3.1)
Recall ι h ( s, a ) :
=
P h (cid:98) V h
+1 ( s, a ) − (cid:98) Q h ( s, a ) and the definition of Bellman operator D.1.
Note min {· , H − h + 1 } + is a non-expansive operator, therefore we have | ι h ( [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
By Lemma D
[Table]
2, we have for any π ,
[Table]
[Equation]
where the second inequality uses Cauchy inequality and the third one uses the definition of concentrability coefficient 2. [Equation]
Next [Equation] for VFQL, there is no pessimism therefore the quantity B in Lemma L.10 is zero, hence the covering number applied in Lemma G.1 is bounded by C d, log( K ) ≤ (cid:101) O
( d ) and
E
[Equation]
Now leveraging equation 21 and equation 22 in Theorem G.2 to obtain
E
[Equation]
[Table]
(cid:21) ≤ E µ {+  ℓ h ( (cid:98) θ h ) − λ (cid:13)(cid:13)(cid:13)(cid:98) θ h (cid:13)(cid:13)(cid:13) 
 K  +}
[Equation]
E µ   ℓ h ( θ T (cid:98) V h +1 ) − λ (cid:13)(cid:13)(cid:13) θ T (cid:98) V h + (cid:13) (cid:13)(cid:13)
[Equation]
Plug the above into equation 24, we obtain with probability 1 − δ , for all policy π , v
[Equation]
This finishes the proof of Theorem 3.1.
J P ROOFS FOR VAFQL
In this section
[Equation]
we present the analysis for variance-aware fitted Q learning (VAFQL).
Throughout the whole section
[Equation]
we assume ϵ F = 0 , i [Equation] e. the exact Bellman-Completeness holds.
The algorithm is presented in the following.
Before giving the proofs of Theorem
[Equation]
we first prove some useful lemmas.
[Table]
/ (cid:98) σ 2 ( s kh , a kh )
+ λ · I , 10: Set Γ h ( · , · ) ← β (cid:113) ∇ θ f ( (cid:98) θ h , ϕ ( · , · )) ⊤ Λ − 1 h ∇ θ f ( (cid:98) θ h , ϕ ( · , · )) (cid:16) + (cid:101) O ( 1 K ) (cid:17) 11:
Set ¯ Q h ( · , · ) ← f ( (cid:98) θ h , ϕ ( · , · )) − Γ h ( · , · ) , (cid:98) Q h ( · , · ) ← min (cid:8) ¯ Q h ( · , · ) , H − h + 1 (cid:9) + 12: Set (cid:98) π h ( · | · ) ← arg max π h (cid:10) (cid:98) Q h ( · , · ) , π h ( · | · ) (cid:11) A , (cid:98) V h ( · ) ← max π h (cid:10) (cid:98) Q h ( · , · ) , π h ( · | · ) (cid:11) A 13: end for 14: Output: { (cid:98) π h } Hh =1 .
J. P ROVABLE E FFICIENCY FOR V ARIANCE -A WARE F ITTED Q L EARNING
Recall the objective
[Equation]
Then by definition, (cid:98) θ h := arg min θ ∈ Θ ℓ h ( θ ) and θ T (cid:98) V h
+1 satisfies f ( θ T (cid:98) V h
+1 , ϕ )
= P h (cid:98) V h
+1 ( s kh +1 )
(recall ϵ F = 0 ).
Therefore, in this case, we have the following lemma:
Lemma J.1.
Fix h ∈
[ H ] .
With probability 1 − δ ,
[Equation]
K where the expectation over µ is taken w.r.t.
( s kh , a kh , s kh +1 ) k
= 1 , ..., K only (i.e., first compute E µ [ ℓ h ( θ )] for a fixed θ , then plug-in either (cid:98) θ h {++
+} or θ T (cid:98) V h +1 ).
Here
[Equation]
Proof of Lemma J.1.
Step1: Consider the case where λ = 0 .
Indeed, fix h ∈ [ H ] and any function
V ( · ) ∈ R S .
Similarly, define f V ( s, a ) :
= f ( θ T V , ϕ )
= P h V .
For any fixed θ ∈ Θ , denote g ( s, a ) =
f ( θ, ϕ ( s, a )) .
Moreover, for any u, v ∈ Θ , define
[Equation]
Then define (we omit the subscript u, v of σ 2 u,v for the illustration purpose when there is no ambiguity)
[Equation]
Since all episodes are independent of each other,
X
[Equation]
g
[Equation]
:=
X ( g ( s kh , a kh ) , V ( s kh +1 ) , f V ( s kh , a kh ) , σ 2 ( s kh , a kh )) are independent r.v.s and it holds
[Equation]
Next, the variance of X is bounded by
Var
[Equation]
=
[Equation]
E
[Equation]
[ X [Equation] )] ( ∗ ) follows from that E µ
[Equation]
=
[Equation]
Therefore, by Bernstein inequality, with probability 1 − δ ,
[Equation]
Now, if we choose g ( s, a ) :
= f ( (cid:98) θ h , ϕ ( s, a )) and u = u h , v = v h from Algorithm 3, then (cid:98) θ h minimizes ℓ h ( θ ) , therefore, it also minimizes 1 K (cid:80)
Kk =1 X i ( θ, (cid:98) V h
+1 , f (cid:98) V h
+1 , (cid:98) σ 2 h ) and this implies K K (cid:88)
[Equation]
Thus, we obtain
[Equation]
However, the above does not hold with probability 1 − δ since (cid:98) θ h , (cid:98) σ 2 h and (cid:98) V h
+1 :=
[Equation]
[Equation] [Equation] [Equation] [Equation]
A
[Equation]
f
[Equation]
θ, ϕ [Equation] · , a )) , H } (where A is certain symmetric matrix with bounded norm) depend on (cid:98) θ h , (cid:98) θ h +1 which are data-dependent.
Therefore, we need to further apply covering Lemma L.11 and choose ϵ =
O (1 /K ) and a union bound to obtain with probability 1 − δ ,
[Equation]
where C d, log( K ) :
= d log(1+24 C Θ ( H + 1) κ 1 K )+ d log (cid:16)
[Equation]
+ 288 H 2 C Θ
[Equation]
κ 1 √ C Θ + 2 (cid:112) κ 1 κ 2 /λ )
2 K 2 (cid:17)
+ d 2 log (cid:16)
[Equation]
+ 288 H 2 √ dκ 21 K
[Equation]
[-+ d 2 log (cid:16) 
 + 288 H 2 √ dκ 21 K 2-] /λ (cid:17) + d log(1 + 16 C Θ H 2 κ 1 K ) + d log(1 + 32 C Θ H 3 κ 1 K ) (where we let B = 1 /λ since (cid:13)(cid:13) Λ − 1 h (cid:13)(cid:13) 2 ≤ 1 /λ ).
Solving this quadratic equation to obtain with probability 1 − δ ,
[Equation]
Now according to equation 25, by definition we finally have with probability 1 − δ (recall the expectation over µ is taken w.r.t.
( s kh , a kh , s kh +1 ) k
= 1 , ..., K only)
E µ
[Equation]
26) where we used f ( θ T (cid:98) V h
+1 , ϕ )
= P h (cid:98) V h
+1 = f (cid:98) V h
+1 .
Step2.
If λ > 0 , there is only extra term λK (cid:16)(cid:13)(cid:13)(cid:13)(cid:98) θ h (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13) θ T (cid:98) V h +1 (cid:13)(cid:13)(cid:13) 2 (cid:17) ≤ λK (cid:13)(cid:13)(cid:13)(cid:98)
[Equation]
h (cid:13)(cid:13)(cid:13) 2 ≤ λC 2Θ K in addition to above.
This finishes the proof.
Theorem J.2 (Provable efficiency for VAFQL) .
Let C d, log K be the same as Lemma J.1.
Then, with probability
[Equation]
Proof of Theorem J.2.
Apply a union bound in Lemma J.1, we have with probability 1 − δ ,
[Equation]
Now we prove for all h ∈
[ H ] ,
[Equation]
Indeed, identical to equation 26, [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
where the third identity uses law of total expectation and that µ is taken w.r.t.
s h , a h , s h +1 only (recall
Lemma J.1)
so the (cid:98) σ 2 h can be move outside of the conditional expectation.
The fourth identity uses the definition of θ T (cid:98) V h
+1 since f ( θ T (cid:98) V h
+1 , ϕ ( s, a ))
=
P h,s,a (cid:98) V h
+1 .
Then we have
[Equation]
where the third identity uses µ is over s h , a h only and the last one uses (cid:98) σ 2 h ( · , · ) ≤ H 2 .
Combine the above with equation 27 and equation 28, we obtain the stated result.
Theorem J.3 (Provable efficiency of VAFQL (Part II)) .
Let C d, log K be the same as Lemma J.1.
Furthermore, suppose λ ≤ 1 / 2 C 2Θ and K ≥ max (cid:110)
[Equation]
[Equation]
[Equation]
log
[Equation]
2 dδ
[Equation]
+ d
[Equation]
Then, with probability 1 − δ , ∀ h ∈
[ H ]
[Equation]
(cid:114)
K + O ( 1 K ) , Furthermore, we have with probability 1 − δ , sup h (cid:13)(cid:13)(cid:13)(cid:98) [Equation] K
[Equation]
where (cid:101) O absorbs Polylog terms and higher order terms.
Lastly, it also holds for all h ∈
[ H ] , w.p.
1 − δ
[Equation]
Proof of Theorem J.3.
Step1: we show the first result.
We prove this by backward induction.
When h = H + 1 , by convention f ( (cid:98) θ h , ϕ ( s, a )) =
f ( θ ⋆h , ϕ ( s, a ))
=
so the base case holds.
Suppose for h + 1 , with probability 1 − ( H − h ) δ , sup s,a (cid:12)(cid:12)(cid:12) f
( (cid:98) θ h , ϕ ( s, a )) − f ( θ ⋆h , ϕ ( s, a )) (cid:12)(cid:12)(cid:12) ≤ C h +1 (cid:113) 1 K , we next consider the case for t = h .
On one hand, by Theorem J.2, we have with probability 1 − δ/ 2 ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Recall we have the form (cid:98) V
h
+1 ( · ) :
= min { max a f ( (cid:98) θ h +1 , ϕ ( · , a ))
− Γ h ( · , a ) , H } and V ⋆h +1 ( · ) = max a f ( θ ⋆h +1 , ϕ ( · , a ))
=
min { max a f ( θ ⋆h +1 , ϕ ( · , a )) , H } , we obtain (cid:13)(cid:13)(cid:13)(cid:98)
[Table]
h +1 −
[Table]
+
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Note the above ( s, a ) .
In particular, according to Algorithm 3, we specify
[Equation]
and by Lemma L.5, with probability 1 − δ (note here Σ − 1 h is replaced by Λ − 1 h and (cid:13)(cid:13)
Λ − 1 h (cid:13)(cid:13) 2 ≤ H 2 /κ ),
[Equation]
and by a union bound this implies with probability 1 − ( H − h + 1) δ , sup s,a (cid:12)(cid:12)(cid:12) f
[Equation]
≤ C h + (cid:114)
[Equation]
K + κ
[Equation]
Solving for C h , we obtain C h ≤ κ 1 H (cid:113) 36 H 4 (log( H/δ )+ C d, log K )
+2 λC 2Θ κ
[Equation]
H
[Equation]
√ κ for all H .
By a union bound (replacing δ by δ/H ), we obtain the stated result.
Step2: Utilizing the intermediate result equation 29, we directly have with probability 1 − δ , sup h (cid:13)(cid:13)(cid:13)(cid:98)
V
h − V ⋆h (cid:13)(cid:13)(cid:13) ∞ ≤ sup s,a (cid:12)(cid:12)(cid:12)
[Equation] [Equation] [Equation] + O
[Equation]
where sup s,a (cid:12)(cid:12)(cid:12) f ( (cid:98) θ h , ϕ ( s, a )) − f ( θ ⋆h , ϕ ( s, a ))
(cid:12)(cid:12)(cid:12) can be bounded using Step1.
Step3: Denote M :=
[Equation]
, then by
Step1 we have with probability 1 − δ (here ξ is some point between (cid:98) θ h and θ ⋆h ) for all h ∈
[ H ]
[Equation]
where the last step is by Assumption 2.3.
Solving this to obtain the stated result.
J. B OUNDING |
[Equation]
Recall the definition σ ⋆ 2 h ( · , · ) = max { 1 , [Var P h V ⋆h +1 ]( · , · ) } .
In this section, we bound the term | (cid:98)
[Equation]
and
[Equation]
where
[Equation]
and true parameters u ⋆h , v ⋆h satisfy f ( u ⋆h , ϕ ( · , · ))
[Equation]
[Equation] [Equation] ⋆h
[Equation]
=
E P ( s ′ |· , · ) [ V ⋆ 2 h ( s ′ )] .
Furthermore, we define
[Equation]
and the parameter Expectation operator J : V ∈ R S → θ J V ∈ Θ such that:
[Equation]
Note θ J V ∈ Θ by Bellman completeness, reward r is constant and differentiability (Definition 1.1) is an additive closed property.
By definition,
[Equation]
and
[Equation]
We first give the following result.
Lemma J.4.
Suppose λ ≤ 1 / 2 C 2Θ and K ≥ max (cid:110) 512 κ [Equation] κ
[Equation] log [Equation] d
[Equation]
Then, with probability 1 − δ , ∀ h ∈
[ H ] , [Equation]
≤ (cid:115) 36 H 2 (log( H/δ ) + (cid:101) O ( d 2 ))
+ 2 λC 2Θ κK ,
∀ h ∈
[ H ] , (cid:13)(cid:13)(cid:13) v h − θ J (cid:98) V 2 h +1 (cid:13)(cid:13)(cid:13) 2 ≤ (cid:115) 36 H 4 (log( H/δ ) + (cid:101) O ( d 2 ))
+ 2 λC 2Θ κK ,
∀ h ∈
[ H ] .
and sup s,a | f ( u h , ϕ ( s, a ))
− f ( u ⋆h , ϕ ( s, a )) | ≤   κ 1 H (cid:115) 36 H 2 (log( H 2 /δ ) + (cid:101) O
( d 2 )) + 2 λC 2Θ κ + 2 H 2 dκ 1 √ κ 
 (cid:114)
K + O ( 1 K ) , sup s,a | f ( v h , ϕ
( s, a )) − f ( v ⋆h , ϕ ( s, a )) | ≤   κ 1 H (cid:115) 36 H 4 (log( H 2 /δ )
+ (cid:101) O ( d 2 )) + 2 λC 2Θ κ + 2 H 3 dκ 1 √ κ   (cid:114)
K + O ( 1 K ) .
The above directly implies for all h ∈
[ H ] , with probability 1 − δ , | σ ⋆ 2 h − [Equation] σ [Equation]
≤
[Equation]
(cid:115) 36 H 4 (log
[Equation]
κK .
Proof of Lemma J.4.
In fact, the proof follows a reduction from the provable efficiency procedure conducted in Section G.
This is due to the regression procedure in equation 30 is the same as the procedure equation 17 except the parameter Bellman operator T is replaced by the parameter
Expectation operator J (recall here ¯ ϕ h,k uses the independent copy D ′ and (cid:101)
O ( d 2 ) comes from the covering argument.).
Concretely, the X ( g, V, f V ) used in Lemma G.1 will be modified to
X ( g, V, f V ) =
( g ( s, a ) − V ( s ′ ))
2 − ( f ( θ J V , ϕ ( s, a ))
− V ( s ′ )) 2 by removing reward information and the decomposition
[Equation]
holds true.
Then with probability 1 − δ ,
[Equation]
and
[Equation]
h +1 , ϕ )
| ≤ [Equation] (cid:13)(cid:13)(cid:13) [Equation]
J (cid:98) V 2 h +1 (cid:13)(cid:13)(cid:13)
[Equation]
2 Hκ 1 (cid:13)(cid:13)(cid:13)
[Equation]
h − θ J (cid:98) V h
+1 (cid:13)(cid:13)(cid:13)
[Equation]
κK .
J. P ROOF OF T HEOREM 4.
In this section, we sketch the proof of Theorem 4.1 since the most components are identical to
Theorem 3.2.
We will focus on highlighting the difference for obtaining the tighter bound.
First of all, Recall in the first-order condition, we have
[Equation]

[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
Therefore, if we define the quantity Z h ( · , · ) ∈ R d as
Z h ( θ | V, σ 2 ) =
K (cid:88) k
=
[Equation]
H, then we have
[Equation]
According to the regression oracle (Line 8 of Algorithm 3), the estimated Bellman operator (cid:98) P h maps (cid:98) V h +1 to (cid:98) θ h , i.e. (cid:98) P h (cid:98) V h
+1 =
f ( (cid:98) θ h , ϕ ) .
Therefore (recall Definition D.1)
[Equation]
31) where we apply the first-order Taylor expansion for the differentiable function f at point (cid:98) θ h and
Hot h, 1 is a higher-order term.
Indeed, the following Lemma E.1 bounds the Hot h, 1 term with (cid:101) O
( 1 K ) .
Lemma J.5.
Recall the definition (from the above decomposition)
Hot h, [Equation] = f
[Equation]
−
[Equation] , then with probability 1 − δ ,
[Equation]
Proof.
The proof is identical to that of Lemma E.1 but with the help of Lemma J.2.
Next, according to the expansion of Z h ( θ | (cid:98) V h
+1 , (cid:98) σ 2 h ) [Equation] we have
[Equation]
where
[Equation]
where R K ( θ T (cid:98) V h +1 ) is the second order residual that is bounded by (cid:101) O
(1 /K ) and
[Equation]
Similar to the PFQL case, I 2 , I 3 , Hot 2 can be bounded to have order O (1 /K ) via provably efficiency theorems in Section J.1 and in particular, the inclusion of σ 2 u,v will not cause additional order in d .
Now we prove the result for the dominate term I 1 .
Lemma J.6.
With probability 1 − δ ,
[Equation]
where C δ, log K only contains Polylog terms.
Proof of Lemma J.6.
First of all, by Cauchy–Schwarz inequality, we have |
[Equation]
Recall that σ 2 u,v ( · , · ) := max { 1 , f ( v, ϕ ( · , · ))
[0 , ( H − h +1) 2 ] − [Equation]
[Equation] [Equation]
[Equation]
Step1.
Let the fixed θ ∈ Θ be arbitrary and fixed u, v such that σ 2 u,v ( · , · ) ≥ 12 σ 2 u ⋆h , v ⋆h ( · , · ) = 12 σ ⋆ 2 h ( · , · ) and define x k ( θ, u, v ) =
∇ θ f
( θ, ϕ h,k ) /σ
u,v ( s kh , a kh ) .
Next, define G u,v ( θ )
= (cid:80)
Kk =1 ∇ f ( θ, ϕ ( s kh , a kh )) · ∇ f ( θ, ϕ ( s kh , a kh ))
⊤ /σ 2 u,v ( s kh , a kh ) + λI d , then ∥ x k ∥ 2 ≤ κ 1 .
Also denote η k :=
[ f ( θ T V ⋆h +1 , ϕ h,k ) − r h,k − V {+⋆h +
+} ( s kh +1 )] /σ u,v ( s kh , a kh ) , then E [ η k | s kh , a kh ] = 0 and
Var[ η k | s kh , a kh ] =
Var[ [Equation] T V ⋆h +
[Equation]
, ϕ [Equation]
,k ) [Equation]
r
[Equation]
[Equation] [Equation] then by Self-normalized Bernstein’s inequality (Lemma L.4), with probability 1 − δ , (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) K (cid:88) k
{+= x k ( θ, u, v ) η k (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) G ( θ,u,v ) −≤(cid:115) d log (cid:18) 
 + Kκλd (cid:19) · log (cid:18) 4 Kδ (cid:19)+}
+4 ζ log (cid:18) 4 Kδ (cid:19) ≤ (cid:101) O ( √ d ) where | η k | ≤ ζ with ζ = 2 max s,a,s ′ | f ( θ T V⋆h +1 ,ϕ
( s,a )) − r − V {+⋆h +
+}
( s ′ ) | σ ⋆h ( s,a ) and the last inequality uses √ d ≥ (cid:101) O ( ζ ) .
Step2.
Define h
( θ, u, v ) :
= (cid:80)
Kk
=1 x k
( θ, u, v ) η k ( u, v ) and H ( θ, u, v ) :
=
∥ h ( θ, u, v ) ∥ G u,v ( θ ) − 1 , ∥ h ( θ 1 , u 1 , v 1 ) − h ( θ 2 , u 2 , v 2 ) ∥ 2 ≤ K max k ∥ ( x k · η k )( θ 1 , u 1 , v 1 ) − ( x k · η k )( θ 2 , u 2 , v 2 ) ∥≤ K max k (cid:40)
H
[Equation]
σ
[Equation]
[Table]
Furthermore,
[Table]
All the above imply
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
note
[Equation]
Then a ϵ -covering net of { H ( θ, u, v ) } can be constructed by the union of covering net for θ, u, v and by Lemma L.8, the covering number N ϵ satisfies (where (cid:101)
O absorbs Polylog terms)
[Equation]
Step3. First note by definition in Step
[Equation]
Now choosing ϵ =
O (1 /K ) in Step2 and union bound over the covering number in Step [Equation] we obtain with probability 1 − δ (recall √ d ≥ (cid:101) O ( ζ ) ),
[Equation]
where we absorb all the Polylog terms.
Combing above with equation 33, we obtain with probability 1 − δ ,
[Equation]
Combing dominate term I 1 (via Lemma J.6) and all other higher order terms we can obtain the first result together with Lemma D.3.
The proof of the second result is also very similar to the proofs in Section F.2.
Concretely, when picking π = π ⋆ , we can convert the quantity
[Equation]
to
[Equation]
using Theorem J.
[Equation]
and convert
[Equation]
to
[Equation]
using Lemma J.4.
K T HE LOWER BOUND
Theorem K.1 (Restatement of Theorem 4.2) .
Specifying the model to have linear representation f = ⟨ θ, ϕ ⟩ .
There exist a pair of universal constants c, c ′ > 0 such that given dimension d , horizon
H and sample size K > c ′ d 3 , one can always find a family of MDP instances such that for any algorithm (cid:98) π
[Equation]
[Equation]
where
[Equation]
⋆
[Equation]
Remark K.2.
Note Theorem 4.2 is a valid lower bound for comparison.
This is because the upper bound result holds true for all model f such that the corresponding F satisfies Assumption 2.1, 2.3.
Therefore, for the lower bound construction it suffices to find one model f such that the lower bound equation 34 holds.
Here we simply choose the linear function approximation.
K. R EGARDING THE PROOF OF LOWER BOUND
The proof of Theorem 4.2 can be done via a reduction to linear function approximation lower bound.
In fact, it can be directly obtained from Theorem 3.5 of Yin et al.
and the original proof comes from Theorem 2 of Zanette et al.
Concretely, all the proofs in Theorem 3.5 of Yin et al.
(2022) follows and the only modification is to replace
[Equation]
in Section E.5 by
[Equation]
and the final result holds with ϕ ( · , · ) = ∇ θ f ( θ ⋆h , ϕ ( · , · )) by the reduction f = ⟨ θ, ϕ ⟩ .
L A UXILIARY LEMMAS
Lemma L.1 ( k -th Order Mean Value Form of Taylor’s Expansion) .
Let k ≥ 1 be an integer and let function f : R d → R be k times differentiable and continuous over the compact domain Θ ⊂ R d .
Then for any x, θ ∈ Θ , there exists ξ in the line segment of x and θ , such that
[Equation]
Here ∇ k f ( θ ) denotes k -dimensional tensor and (cid:78) denotes tensor product.
Lemma L.2 (Vector Hoeffding’s Inequality) .
Let X =
( X 1 , . . .
, X d ) be d -dimensional vector
Random Variable with E [ X ] = 0 and ∥ X ∥ 2 ≤ R .
X (1) , . . .
, X ( n ) ’s are n samples.
Then with probability 1 − δ ,
[Equation]
Proof of Lemma L.2.
Since ∥ X ∥ 2 ≤ R implies | X j | ≤ R , by the univariate Hoeffding’s inequality, for a fixed j ∈ { 1 , ..., d } , denote Y j := 1 n (cid:80) ni
=1 X ( i ) j .
Then with probability 1 − δ (note | X
( i ) j | ≤ R ),
P [Equation] (cid:114) [Equation]
[Equation] ) (cid:33) ≤ δ.
By a union bound,
[Equation]
Lemma L.3 (Hoeffding inequality for self-normalized martingales (Abbasi-Yadkori et al., 2011)) .
Let { η t } ∞ t =1 be a real-valued stochastic process.
Let {F t } ∞ t
=
be a filtration, such that η t is F tmeasurable.
Assume η t also satisfies η t given F t − 1 is zero-mean and R -subgaussian, i.e.
[Equation]
Let { x t } ∞ t =1 be an R d -valued stochastic process where x t is F t − 1 measurable and ∥ x t ∥ ≤ L .
Let Λ t = λI d
+ (cid:80) ts =1 x s x ⊤ s .
Then for any δ > 0 , with probability 1 − δ , for all t > 0 ,
[Equation]
Lemma L.4 (Bernstein inequality for self-normalized martingales (Zhou et al., 2021)) .
Let { η t } ∞ t = be a real-valued stochastic process.
Let {F t } ∞ t
=
be a filtration, such that η t is F t -measurable.
Assume η t also satisfies
[Equation]
Let { x t } ∞ t =1 be an R d -valued stochastic process where x t is F t − 1 measurable and ∥ x t ∥ ≤ L .
Let Λ t = λI d
+ (cid:80) ts =1 x s x ⊤ s .
Then for any δ > 0 , with probability 1 − δ , for all t > 0 ,
[Equation]
(cid:13) t
[Equation]
Lemma L.5.
Let ∇ f ( θ, ϕ ( · , · )) :
S × A →
R d be a bounded function s.t. sup θ ∈ Θ ∥∇ f
( θ, ϕ ( · , · )) ∥ 2 ≤ κ 1 .
If K satisfies
[Equation]
Then with probability at least 1 − δ , for all ∥ u ∥ 2 ≤ B simultaneously, it holds that
[Equation]
where Σ h = (cid:80) Kk =1 ∇ f
( (cid:98) θ h , ϕ ( s kh , a kh )) · ∇ f ( (cid:98) θ h , ϕ ( s kh , a kh )) ⊤ + λI d .
Proof of Lemma L.5.
For a fixed θ , define ¯ G = (cid:80)
Kk =1 ∇ f ( θ, ϕ ( s kh , a kh )) · ∇ f ( θ, ϕ ( s kh , a kh ))
⊤ + λI d , and G = E µ [ ∇ f ( θ, ϕ
( s h , a h )) · ∇ f ( θ, ϕ
( s h , a h )) ⊤ ] , then by Lemma H.5.
of Min et al.
(2021), as long as
[Equation]
then with probability 1 − δ , for all u ∈ R d simultaneously, ∥ u ∥ ¯ [Equation] ∥
[Equation] ∥ G − 1 . As a corollary, if we constraint u to the subspace ∥ u ∥ 2 ≤ B , then we have: with probability 1 − δ , for all { u ∈ R d : ∥ u ∥ 2 ≤ B } simultaneously,
[Equation]
Next, for any θ , define h [Equation] [Equation] λI
[Equation]
(cid:33) −
[Equation]
u and ¯ G ( θ ) =
(cid:80)
Kk =1 ∇ f ( θ, ϕ ( s kh , a kh )) · ∇ f ( θ, ϕ ( s kh , a kh ))
⊤ + λI d , we have for any θ 1 , θ(cid:13) (cid:13) ¯ G ( θ 1 ) − ¯ G ( θ 2 ) (cid:13) (cid:13) 2 ≤ (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) K (cid:88) k = (cid:0) ∇ f ( θ 1 , ϕ ( s k h , a k h ))
[Equation]
f
[Equation] θ 2 , ϕ [Equation] s k h , a k h )) (cid:1)
[Equation]
f [Equation] θ 1 , ϕ ( s k h , a k h ))
⊤ (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) K (cid:88) k = ∇ f ( θ [Equation]
Use the basic
[Table]
for a, b
[Table]
0 ⇒ |√ a − √ b | ≤ (cid:112) |
[Table]
−
[Table]
| ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Therefore, the ϵ -covering net of { h ( θ ) : θ ∈ Θ } is implies by the λ
[Equation]
-covering net of { θ : θ ∈ Θ } , so by Lemma L.8, the covering number N ϵ satisfies
[Equation]
Select θ = (cid:98) θ h .
Choose ϵ = O (1 /K )
and by a union bound over equation 36 to get with probability 1 − δ , for all ∥ u ∥ 2 ≤ B (note By Assumption 2.3 (cid:13)(cid:13) G − 1 (cid:13)(cid:13) 2 ≤ 1 /κ ),
[Equation]
if (union bound over the condition equation 35)
[Equation]
where this condition is satisfied by the Lemma statement.
Lemma L.6.
let ϕ : S ×
A → R d satisfies ∥ ϕ ( s, a ) ∥ ≤ C for all s, a ∈ S × A .
For any
K > 0 , λ > 0 , define ¯ G K = (cid:80)
Kk =1 ϕ ( s k , a k ) ϕ ( s k , a k ) ⊤ + λI d where ( s k , a k ) ’s are i.i.d samples from some distribution ν .
Then with probability 1 − δ ,
[Equation]
Proof of Lemma L.6.
See Lemma H.5 of Yin et al.
(2022) or Lemma H.4 of Lemma Min et al.
(2021) for details.
Lemma L.7 (Lemma H.4 in Yin et al.
Let Λ 1 and Λ 2 ∈ R d
×
d are two positive semi-definite matrices.
Then:
[Equation]
2 ∥ and
[Equation]
for all ϕ ∈ R d .
L. C OVERING A RGUMENTS
Lemma L.8.
(Covering Number of Euclidean Ball)
For any ϵ > 0 , the ϵ -covering number of the
Euclidean ball in R d with radius R > 0 is upper bounded by (1 + 2 R/ϵ ) d .
Lemma L.9.
Define V to be the class mapping S to R with the parametric form
[Equation]
where the parameter spaces are { θ : ∥ θ ∥ 2 ≤ C Θ } and { A : ∥ A ∥ 2 ≤ B } .
Let N V ϵ be the covering number of ϵ -net with respect to l ∞ distance, then we have
[Equation]
Proof of Lemma L.
[Equation]
Here ∥·∥ F is Frobenius norm.
Let C θ be the
[Equation]
[-Let C θ be the (cid:15) 2 4( κ 1 √ C Θ +
 √ Bκ 1 κ 2 ) 2 -net of-] space { θ : ∥ θ ∥ 2 ≤ C Θ } and C w be the ϵ 2 4 κ 21 -net of the space { A : ∥ A ∥ F ≤ √ dB } , then by Lemma L.8,
[Equation]
Therefore, the covering number of space V satisfies log
[Equation]
Lemma L.10 (Covering of E µ ( X ( g, V, f )) ) .
Define
X [Equation] [Equation]
[Equation] f
[Equation]
, ϕ
[Equation]
s, a )
[Equation]
θ
[Equation]
s ′ ) [Equation] θ ′ ( s, a ) − r − V θ ′ ( s ′ )) 2 , where f V := P h V + δ V and V ( s ) has form V θ ( s ) that belongs to V (as defined in Lemma L.9).
Here
X ( θ, θ ′ ) is a function of s, a, r, s ′ as well, and we suppress the notation for conciseness only.
Then the function class H =
{ h ( θ, θ ′ ) :
= E µ [ X ( θ, θ ′ )] | ∥ θ ∥ 2 ≤ C Θ , V θ ∈ V} has the covering number of ( ϵ + 4 Hϵ F ) -net bounded by
[Equation]
1 + 288 H√ dBκϵ(cid:33) .
Proof of Lemma L.10.
First of all,
X ( θ, θ ′ ) = f ( θ, ϕ ( s, a )) 2 − f V θ ′
( s, a ) 2 − 2 f ( θ, ϕ ( s, a )) · ( r + V θ ′ ( s ′ )) + 2 f V θ ′ ( s, a ) · ( r + V θ ′ ( s ′ )) ,
[Equation]
where the second inequality comes from f V = P h V + δ V .
Note the above holds true for all s, a, r, s ′ , therefore it implies
[Equation]
Now let C 1 be the [-(cid:15) 
-]ϵ 12( H +1)
κ 1 -net of { θ : ∥ θ ∥ 2 ≤ C Θ } and C 2 be the ϵ/ 6 H -net of V , applying
Lemma L.8 and Lemma L.9 to obtain
[Equation]
which implies the covering number of H to be bounded by
[Equation]
Lemma L.11.
Denote σ
[Equation]
and define
[Equation]
where f V := P h V and V ( s ) has form V θ ( s ) that belongs to V (as defined in Lemma L.9).
Here ¯
X ( θ, θ ′ , u, v ) is a function of s, a, r, s ′ as well, and we suppress the notation for conciseness only.
Then the function class H =
{ h ( θ, θ ′ , u, v ) :
= E µ [ ¯ X ( θ, θ ′ , u, v )] | ∥ θ ∥ 2 ≤ C Θ , V θ ∈ V} has the covering number of ϵ -net bounded by
[Equation]
Proof of Lemma L.11.
Recall
[Equation]
f [Equation] ϕ [Equation]
[Equation]
H − [Equation] [Equation] 2 } , and since max , truncation are non-expansive operations, then we can achieve for any s, a
[Equation]
Hence
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] , r, s ′ , therefore it implies [Equation] µ [ ¯ X
[Equation]
θ 1 , θ
[Equation]
1 , u 1 , v 1 )] [Equation] [Equation] and similar to Lemma L.10, the covering number of ϵ -net will be bounded by
[Equation]
Comparing to Lemma L.10, the last two terms are incurred by covering u, v arguments.