P ESSIMISTIC M ODEL - BASED O FFLINE R EINFORCEMENT L EARNING UNDER P ARTIAL C OVERAGE
A BSTRACT
We study model-based ofﬂine Reinforcement Learning with general function approximation without a full coverage assumption on the ofﬂine data distribution.
We present an algorithm named Constrained Pessimistic Policy Optimization (CPPO) which leverages a general function class and uses a constraint over the model class to encode pessimism.
Under the assumption that the ground truth model belongs to our function class (i.e., realizability in the function class), CPPO has a PAC guarantee with ofﬂine data only providing partial coverage, i.e., it can learn a policy that competes against any policy that is covered by the ofﬂine data.
We then demonstrate the ﬂexibility of CPPO: it can be seamlessly applied to many specialized Markov Decision Processes where additional structural assumptions further reﬁne the concept of partial coverage.
Two notable examples are: (1) lowrank MDP with representation learning where the partial coverage condition is deﬁned using a relative condition number measured by the unknown ground truth feature representation; (2) factored MDP where the partial coverage condition is deﬁned using density ratios associated with individual factors.
I NTRODUCTION
Ofﬂine Reinforcement Learning (RL) is one of the important areas of RL where the learner is presented with a static dataset consisting of transition-related information (state, action, reward and next state) collected by some behavior policy and needs to learn purely from the ofﬂine data without any future online interaction with the environment.
Ofﬂine RL is used in a number of applications where online random experimentation is costly or dangerous such as health care (Kosorok & Laber, 2019), digital marketing (Chen et al., 2019) and robotics (Levine et al., 2020).
The performance guarantees of ofﬂine RL often rely on two quantities: the coverage of the ofﬂine data and the property of the function approximation used in the algorithms.
For instance, for the classic Fitted-Q-iteration (FQI) algorithm (Ernst et al., 2005; Munos & Szepesv ´ ari, 2008), it requires (a) full coverage in the ofﬂine data, i.e., max ( s,a ) d
π
( s, a ) /ρ
( s, a ) <
∞ for any stochastic policies π including history-dependent non-Markovian policies, where d π ( s, a ) is a state-action occupancy distribution of a policy π and ρ ( s, a ) is an ofﬂine distribution, (b) realizability in a Q function class, i.e., the optimal Q function belongs to the function class, and (c) Bellman completeness, i.e., applying the Bellman operator on any function in the function class results in a new function that also belongs to the function class (see the ﬁrst row in Table 1).
Among these three assumptions, the full coverage and the Bellman completeness are particularly strong.
The full coverage means that the behavior policy needs to be exploratory enough, although ﬁguring out an exploratory policy itself is an extremely hard problem for large-scale MDPs.
The Bellman completeness assumption does not have a monotonic property, i.e., even starting with a function class that originally permits
Bellman completeness, slightly increasing the capacity of the function class could result in a new class that does not have Bellman completeness anymore.
Thus, we aim to relax the assumptions on the ofﬂine data and the function class.
Particularly, we are interested in the following question:
Given a realizable function class and an ofﬂine distribution that only provides partial coverage, can we learn a policy that is able to compete with any policy that is covered by the ofﬂine distribution?
We study this question from a model-based learning perspective and provide an afﬁrmative answer to the question .
More speciﬁcally, different from FQI, we start with a realizable model class, i.e., the ground truth transition falls into the model class.
We further abandon the strong full coverage assumption, and instead, assume partial coverage which means the ofﬂine data distribution only covers a state-action distribution of some high-quality comparator policy π ∗ ( π ∗ is not necessarily
[Table]
although the analysis in Jin et al. (2020a) is done under the full coverage for linear MDPs, based on the argument (Zhang et al., 2021b), we can show the algorithm has the PAC guarantee under partial coverage in terms of the relative condition number for linear MDPs.
† † † means that we can reﬁne it to a more adaptive quantity using the model class (i.e., Deﬁnition 1).
All the methods in the table require realizability in the function class.
the optimal policy, and π ∗ could be non-Markovian), i.e., max s,a d π ∗ ( s, a ) /ρ
( s, a ) < ∞ , We design an algorithm — Constrained Pessimistic Policy Optimization (CPPO), which can learn a policy that is as good as any comparator policy π ∗ that is covered by the ofﬂine data.
The fact that CPPO can learn to compete against history-dependent policies is meaningful in ofﬂine RL when the ofﬂine data does not cover the optimal policy.
While one could assume density ratio based concentrability coefﬁcient ( max s,a d π ∗ ( s, a ) /ρ
( s, a ) ) to be under control for small size MDPs, in large-scale MDPs (e.g. continuous state space), the density ratio could quickly become an extremely large quantity which makes the performance guarantee vacuous.
When applying CPPO to MDPs with additional structural assumptions, we can seamlessly reﬁne the density ratio based concentrability coefﬁcient to more natural and tighter quantities.
Notably, we consider the ofﬂine representation learning setting where the underlying MDPs permit a low-rank structure (unlikely linear MDPs (Jin et al., 2020a; Yang & Wang, 2020), we do not assume the ground truth state-action feature representation φ (cid:63) is known, and instead we need to learn φ (cid:63) ) and we show that we can reﬁne the density ratio to a relative condition number that is deﬁned using the unknown true state-action feature representation φ (cid:63) .
Intuitively this means that as long as there exists a high-quality comparator policy that only visits the subspace (deﬁned using the true representation φ ) that is covered by the ofﬂine data, CPPO can compete against such a policy, even without knowing the true φ (cid:63) .
Such bounded relative condition number assumption is much weaker than the bounded density ratio assumption.
3 While the concept of relative condition number was originally introduced in the online RL setting (e.g., Agarwal et al.
(2020c;a) with a known linear feature φ ), and later was introduced in ofﬂine RL (Zhang et al.
(2021b);
Chang et al.
, these prior works all rely on the fact that the feature representation φ is known to the learner a priori (see Table 1 for the comparison).
Another interesting example is factored MDPs (Kearns & Koller, 1999) where we show CPPO reﬁnes the density ratios to be density ratio associated with individual factors, which leverages the factored structure and is provably tighter.
We also give examples on linear MDPs (Yang & Wang, 2020), kernelized nonlinear regulator (KNRs) (Kakade et al., 2020), where we again show that CPPO enjoys problem speciﬁc quantities for measuring the coverage.
Our contributions.
Our contributions are two folds, which we summarize below:
We show that in the model-based setting, realizability and partial coverage is enough to learn a high-quality comparator policy (Theorem 1).
Notably, (1) this result holds for any MDPs with realizable model classes , (2) we can compete against even history-dependent policies.
This is in sharp contrast to the state-of-art provable model-free ofﬂine RL results: see Table 1 on page 2 for detailed comparisons to prior works.
Under additional structural assumptions (e.g., KNRs, linear MDPs (Yang & Wang, 2020), linear mixture MDPs (Ayoub et al., 2020), low-rank MDPs, factored MDPs), we show that we can seamlessly reﬁne the density ratio based concentrability coefﬁcients to problem speciﬁc quantities.
This ﬂexibility to adapt to problem speciﬁc coverage measuring quantities is in sharp contrast to other model-free ofﬂine RL algorithms such as minimax based approaches (Uehara et al., 2020) which, to the best of our knowledge, cannot leverage MDP’s structures (e.g., linear MDPs) to reﬁne its density ratio based concentrability coefﬁcients. While we focus on the model-based setting and have demonstrated advantages of our approach over model-free ones (i.e., no more Bellman completeness assumption on function classes, being able to compete against a larger pool of policies, and the ability to seamlessly adapt to problem dependent structures), it is worth noting that realizability in the model-based setting is usually considered stronger than the one in the model-free setting. On the empirical side, model-based ofﬂine RL algorithms are the state-of-art (e.g., Yu et al.
; Kidambi et al. ; Matsushima et al. Cang et al.
; Chang et al.
Our theoretical results provide a sharp contrast between model-based and model-free approaches in ofﬂine RL.
R ELATED WORK
We discuss two families of related works of ofﬂine RL. In Appendix C, we discuss related works about representation learning in RL.
Insufﬁcient coverage of the dataset due to the lack of online exploration is known as the main challenge in ofﬂine RL (Wang et al., 2020).
To deal with this problem, a number of methods have been recently proposed from both model-free (Wu et al., 2019;
Touati et al., 2020; Kumar et al., 2020; Liu et al., 2020; Rezaeifar et al., 2021; Fujimoto et al., 2019; Fakoor et al., 2021; Ghasemipour et al., 2021; Buckman et al., 2020) and model-based perspectives (Yu et al., 2020; Kidambi et al., 2020; Matsushima et al., 2020; Yin et al., 2021).
More or less, their methods rely on the idea of pessimism and its variants in the sense that the learned policy can avoid uncertain regions not covered by ofﬂine data.
As a theoretical side, Munos & Szepesv ´ ari (2008); Duan et al.
Fan et al.
(2020) proved FQI has a PAC (probably approximately correct) guarantee under realizability, the global coverage, and Bellman completeness conditions.
Other ofﬂine model-free RL methods such as minimax ofﬂine RL methods also require realizability and the global coverage (Chen & Jiang, 2019; Antos et al., 2008;
Uehara et al., 2021; Duan et al., 2021; Zhang et al., 2020;
Nachum et al., 2019).
Recently, by leveraging the aforementioned the pessimism idea, Jin et al.
(2020a)
; Rajaraman et al.
(2020) showed that pessimistic FQI can be applied to partial coverage setting for linear and tabular MDPs.
Comparing to their works, our analysis focuses on modelbased approaches with general function approximation.
The ofﬂine model-based method is known to have a PAC guarantee under the realizability and the global coverage (Ross & Bagnell, 2012;
Chen & Jiang, 2019).
As the most closely related work, Chang et al. (2021) proved a model-based method with an additional penalty term can weaken the assumption from the global coverage to the partial coverage for structured MDPs such as KNRs and Gaussian Processes models (Deisenroth &
Rasmussen, 2011).
In this work, we consider arbitrary MDPs with a realizable model class and aim for PAC bounds under a partial coverage condition.
P RELIMINARIES
We consider a Markov Decision process (MDP) M = {S , A , P, γ, r, d 0 }
where P : S ×A → ∆( S ) is the transition, r : S ×
A → [0 , 1] is the reward function, γ ∈ [0 , 1) is the discount factor, and d 0 ∈ ∆( S ) is the initial state distribution.
A policy π maps from state (or history) to distribution over actions.
Given a policy π and a transition distribution P , V πP denotes the expected cumulative reward of π under P, d 0 and r .
Similarly, Q πP : S ×
A → R , A πP : S ×
A → R are a Qfunction and advantage-function under P and π .
Given a transition P , we denote π ( P ) as the optimal policy associated with model P under reward r .
We also denote d πP ⊂ ∆( S × A ) as the average state-action distribution of π under the transition model P , i.e, d π P =
(1 − γ ) (cid:80) ∞ t =
γ t d π P,t , where d πP,t ∈ ∆( S × A ) is the distribution of ( s t , a t ) under π and P at a time-step t .
We denote the true transition distribution as P (cid:63) , which we do not know in advance.
For simplicity, we suppose r is known.
The extension to the unknown reward is straightforward.
In the ofﬂine RL setting, we have an ofﬂine distribution ρ ∈ ∆( S × A ) , and an ofﬂine dataset
D = { s ( i ) , a ( i ) , r ( i ) , s (cid:48)
( i ) } ni =1 which is sampled in the following way: s, a ∼ ρ, r = r ( s, a ) , s (cid:48) ∼
P (cid:63) ( ·| s, a ) .
We hope to obtain π ( P (cid:63) ) =
arg max π V πP (cid:63) from this ofﬂine dataset without any further interaction with the environment.
We often denote E D
[ f ( s, a, s (cid:48) )]
= 1 /n (cid:80) ( s,a,s (cid:48) ) ∈D f
( s, a, s (cid:48) ) .
Our goal is to construct an ofﬂine RL algorithm Alg, which maps from D to π so that the suboptimality gap
[Equation]
for any comparator policy π ∗ ∈ Π is minimized, where Π in this work can be an unrestricted policy class (e.g., including non-Markovian policies).
Hereafter, c, c 1 , c 2 , · · · are always universal constants.
Partial coverage.
Throughout this work, we do not assume ρ has global coverage.
The global coverage in this work means that the density ratio based concentrability coefﬁcient d πP (cid:63) ( s, a ) /ρ
( s, a ) is upper-bounded by some constant C ∈ R + for all polices π ∈ Π , or the feature covariance matrix corresponding to the ofﬂine distribution E s,a ∼ ρ φ ( s, a ) φ ( s, a ) (cid:62) ( φ ∈ S ×
A → R is a feature representation) is full rank and has a non-zero minimum eigenvalue, which are commonly used assumptions in ofﬂine RL (Munos, 2005; Antos et al., 2008; Chen & Jiang, 2019; Duan et al., 2020).
Under the full coverage, they show the output policy can compete with the globally optimal policy π ( P (cid:63) ) .
However, this assumption may not be true in practice as computing an exploratory policy itself is a challenging task for large-scale RL problems.
Instead, we are interested in the partial coverage setting such as d π ∗ P (cid:63) ( s, a ) /ρ
( s, a ) ≤ C , which means the state-action occupancy measure under some comparator policy π ∗ is covered by the ofﬂine dataset.
We want to design an algorithm that can compete against any policy π ∗ that is covered by the ofﬂine data.
This assumption is much weaker than the global coverage.
P ESSIMISTIC M ODEL - BASED O FFLINE RL
We ﬁrst introduce a general model-based algorithm that has a PAC guarantee of the suboptimality gap under partial coverage deﬁned with a newly introduced concentrability coefﬁcient.
The algorithm takes a realizable model class as input and outputs a policy that is as good as any comparator policy that is covered by the ofﬂine data in the sense of the bounded concentrability coefﬁcient.
Our algorithm, Constrained Pessimistic Policy Optimization (CPPO) (Algorithm 1), takes a realizable hypothesis class M (with P (cid:63) ∈ M ) consisting of |M| candidate models as input, computes the maximum likelihood estimator (MLE) (cid:98) P MLE using the given ofﬂine data D
= { s, a, s (cid:48) } .
It then forms a min-max objective subject to a constraint.
The min-max objective introduces pessimism via searching for the least favorable model P (in terms of its policy’s value V πP ) that is feasible with respect to the constraint.
We can also express the constrained optimization procedure using a version space M D and a policy optimization procedure deﬁned below: [Equation] where M
[Equation]
where TV( P 1 , P 2 ) is a total variation (TV) distance between two distributions P 1 and P 2 .
The version space M D contains models that are not far away from (cid:98) P MLE in terms of the average TV distance under D .
The version space is constructed such that with high probability P (cid:63) ∈
M D .
Below we state the algorithm’s performance guarantee.
Assuming for now that P (cid:63)
∈
M D holds with high probability, then, ˆ V π :=
min P ∈M D V πP is a pessimistic policy evaluation estimator, which satisﬁes ˆ V π ≤ V πP (cid:63) for all π ∈ Π .
Using the idea of pessimism, we have the following observation:
[Equation]
where the ﬁrst inequality uses ˆ π = arg max π ∈ Π
ˆ V π
and the second inequality uses ˆ V π ≤ V πP (cid:63) for all π ∈ Π .
Thus, the ﬁnal error only incurs the policy evaluation error for the comparator policy π ∗ , which leads to the error only depending on the concentrability coefﬁcient for the comparator policy.
We deﬁne the following new concentrability coefﬁcient that uses the model class M : [Equation]
Deﬁnition 1 (Model-based Concentrability Coefﬁcient) .
For a comparator policy π ∗ , we deﬁne the concentrability coefﬁcient C † π ∗ as follows:
[Equation]
The following theorem shows CPPO learns a policy that competes against π ∗ when C † π ∗ < ∞ .
Theorem 1 (PAC Bound for CPPO with general function class) .
Assume P (cid:63) ∈ M .
We set ξ = c 1 ln( c 2 |M| /δ ) n .
Then, with probability 1 − δ , for any comparator policy π ∗ ∈ Π
( Π can be the unrestricted policy class containing non-Markovian policies),
[Equation]
To the best of our knowledge, this is the ﬁrst algorithm that achieves a PAC guarantee for any
MDPs under the partial coverage assumption C † π ∗ < ∞ with only a realizable hypothesis class.
We emphasize that the inequality in the above uniformly holds for all policies with probability 1 − δ including history-dependent non-Markovian policies (see Remark 2).
Note that the ability to compete against non-Markovian policies in ofﬂine RL is meaningful when the ofﬂine data does not cover the optimal policy π (cid:63) (i.e., there could be a high-quality history-dependent policy that is covered by the ofﬂine data against which we want to compete).
In model-free approaches, this type of result generally cannot be obtained.
Indeed, the model-free approach from Xie et al.
(2021) requires Π to be a restricted Markovian policy class, since their bound contains poly (ln( | Π | )) dependence.
For the detailed discussion, refer to Remark 1.
The quantity C † π ∗ adaptively captures the discrepancy between the ofﬂine data and the state-action occupancy measure under a comparator policy π ∗ depending on the model class M .
For example,
C † π ∗ can be reduced to a relative condition number in KNRs.
Besides, it is always upper bounded by the density ratio based concentrability coefﬁcient:
[Equation]
One extreme case is that functions in M are all the same, which implies C † π ∗ = 1 regardless.
Theorem 1 consider the case where the hypothesis class M is ﬁnite.
When the hypothesis class is inﬁnite, we can still obtain the PAC guarantee by utilizing the generalized result in Section A for any realizable model class with valid statistical complexity (e.g., localized Rademacher complexity).
Prior works that achieve PAC guarantees with only realizable model classes rely on much stronger global coverage sup π C π, ∞ < ∞ (Chen & Jiang, 2019).
Even when the comparator policy is the optimal policy π ( P (cid:63) ) , the partial coverage condition C π ( P (cid:63) ) , ∞ < ∞ is weaker.
Existing pessimistic model-based algorithms and their theoretical results (Chang et al., 2021) often assume that a point-wise model uncertainty measure is given as a by-product of model ﬁtting, which limits the applicability to special linear models such as KNRs/GPs.
CPPO can work for any MDPs with the realizable function class having a valid statistical complexity such that the MLE properly works.
Remark 1 (Comparison to the model-free approach from Xie et al.
; Zanette et al.
Xie et al.
(2021) study the model-free setting where the function class Q models Q functions assumed to be Bellman complete for any Markovian policy in Π .
While directly comparing modelbased approaches to model-free approaches is hard as they use different inductive biases in function classes, we can leverage the approach from Chen & Jiang (2019, Corollary 6) to convert a model class M to a pair of Q and Π class.
Speciﬁcally, we can convert a model class M to a pair of Q class and Π class such that Q will be realizable and also Bellman complete with respect to all π ∈ Π .
After such conversion from the model-based setting to the model-free setting, running the algorithm from Xie et al.
(2021) using Q and Π achieves V π ∗ P (cid:63) − V ˆ
πP (cid:63) =
(cid:112) C (cid:5) ln
( |M|| Π | /n ) , ∀ π ∗ ∈ Π , where
C (cid:5) is some concentrability coefﬁcient.
For the detailed derivation, we refer readers to Appendix
D. Since the suboptimality gap from such conversion incurs log | Π | , a policy class Π cannot be too large.
Especially, unlike our results, it cannot take the unrestricted policy class as Π .
This restriction cannot be ﬁxed even if we use natural policy gradient (NPG) algorithms unless models have special structures (Xie et al., 2021; Zanette et al., 2021).
The details are given in Section D.
Thus, our theorem indicates two advantages of model-based approaches: (1) realizability in function class is enough to ensure a PAC guarantee under a partial coverage condition, (2) it can compete against a larger pool of candidate policies including history-dependent non-Markovian policies, which is a meaningful property when the ofﬂine data does not cover the globally optimal policy.
Next, we demonstrate another key advantage of our approach which is its ﬂexibility to be seamlessly applied to MDPs with special structures.
E XAMPLES WITH R EFINED C ONCENTRABILITY C OEFFICIENTS
In the previous section, our results apply to any MDP as long as its true transition belongs to a function class M .
In this section, we consider several concrete MDPs with additional structural conditions.
We show that by leveraging the additional structural conditions, we can reﬁne the modelbased concentrability coefﬁcient to more natural quantities.
The examples that we discuss here are: (1) linear mixture MDPs which generalize linear MDPs from Yang & Wang (2020) and tabular
MDPs, (2) KNRs which generalize LQRs, (3) low-rank MDPs, and (4) factored MDPs.
Before proceeding, we clarify CPPO cannot capture linear MDPs in Jin et al.
(2020a) that is different from the one (Yang & Wang, 2020) we use, and linear Bellman-complete MDPs (Duan et al., 2020) without any modiﬁcation since MLE-based model learning is no longer applicable to them.
However, other objective functions for learning models could be applied to these models (e.g., see the nonparametric model-based learning approach from Lykouris et al.
; Neu & Pike-Burke (2020) in the online setting), which we leave it as a future work.
5. T ABULAR MDP S AND L INEAR M IXTURE MDP S
Tabular MDPs Tabular MDPs are MDPs where the state and action spaces are ﬁnite.
Although the corresponding hypothesis class for tabular MDPs is inﬁnite, we can still run MLE, that is, estimating P (cid:63) by the empirical distribution.
Then, Algorithm 1 has the following guarantee.
Corollary 1 (PAC bound for tabular MDP) .
We set ξ = c 1 |S| 2 |A| ln( n |S|A| c 2 /δ ) n .
Then with probability 1 − δ , for all π ∗ ∈ Π ,
[Equation]
Here, for tabular MDPs with M = { P : P ( ·| s, a ) ∈ ∆( S ) , ∀ s, a } , the model-based concentrability coefﬁcient in Deﬁnition 1 is equal to the density ratio based concentrability coefﬁcient C π ∗ , ∞ which is the right quantity for small-size tabular MDPs.
Linear mixture MDPs We deﬁne linear mixture MDPs (Ayoub et al., 2020;
Modi et al., 2020).
Deﬁnition 2 (Linear mixture MDPs) .
Given a feature vector ψ : ( S , A , S )
→ R d , a linear mixture
MDP is an MDP where the ground truth transition is P (cid:63) ( s (cid:48) | s, a ) :
= θ (cid:63) (cid:62) ψ ( s, a, s (cid:48) ) , θ (cid:63) ∈ R d .
By setting, ψ ( s, a, s (cid:48) ) =
µ ( s (cid:48) )
(cid:78) φ ( s, a ) ( ⊗ denotes the Kronecker product), linear mixture MDPs include the following linear MDPs (Yang & Wang, 2020):
Deﬁnition 3 (Linear MDPs) .
Linear MDP has P (cid:63) ( s (cid:48) | s, a ) :
= (cid:80) d 1 i =1 (cid:80) d 2 j =1 M (cid:63)ij µ i ( s (cid:48) )
φ j ( s, a ) with µ : S → R d 1 and φ : S ×
A → R d 2 are known features, and M (cid:63) ⊂
R d 1 × d 2 .
We use CPPO to learn on linear mixture MDPs.
The corresponding M is
[Equation]
Given a function V : S → R , deﬁne the state-action feature indexed by V as ψ V ( s, a ) :
= (cid:82) ψ ( s, a, s (cid:48) )
V ( s (cid:48) )
d ( s (cid:48) ) , we have the following PAC guarantee.
Corollary 2 (PAC bound for linear mixture MDPs) .
Suppose inf s,a,s (cid:48) P (cid:63) ( s (cid:48) | s, a ) ≥ c 3 > 0 , Θ
= { θ : (cid:107) θ (cid:107) 2 ≤ R } , (cid:107) ψ V
( s, a ) (cid:107) 2 ≤ 1 , ∀ V ∈ S
→ [0 , 1] and P (cid:63)
∈ M Mix .
We set ξ = c 1 d ln 2 ( c 2 nR/δ ) /n .
Then, with probability 1 − δ , for any π ∗ in Π (again Π can be the unrestricted policy class), CPPO outputs a policy ˆ π such that:
[Equation]
where the concentrability coefﬁcient ¯ C π ∗ , mix is deﬁned as:
[Equation]
with the localized class Z P
[Equation]
= { P : [Equation] [Equation] =
E
[Equation]
[Equation]
ψ V π ∗ P ( s, a ) (cid:62) ] , and Σ π ∗ ,ψ Vπ ∗ P
= E s,a ∼ d π ∗ P(cid:63)
[ ψ V π ∗ P ( s, a ) ψ V π ∗ P ( s, a ) (cid:62) ] .
When specializing to linear MDPs, the above bound still holds with ¯ C π ∗ , mix being replaced by the relative condition number ¯ C π ∗ : [Equation] x
[Equation]
, x, where Σ ρ = E [Equation] [Equation]
[Equation]
[Equation]
[Equation]
] , Σ π ∗ =
E
[Equation]
s,a
[Equation]
∼ d π ∗
[Equation]
[ φ (
[Equation]
, a
[Equation]
φ ( s, a [Equation] (cid:62) ] . This is the ﬁrst PAC-guarantee result in the ofﬂine setting under partial coverage ¯ C π ∗ , mix < ∞ for linear mixture MDPs.
¯ C π ∗ , mix is a newly-introduced concentrability coefﬁcient for linear mixture
MDPs.
This coefﬁcient is measured on the integrated feature vectors φ V ( s, a ) for V : S → [0 , 1] .
Note the class of V is localized, i.e., we consider state-value functions V π ∗ P ( s ) for all P centered around P (cid:63) under data distribution ρ (i.e., P ∈ Z P (cid:63) ).
Such localization property ensures that ¯ C π ∗ , mix ≤ C † π (cid:63) (see Lemma 10 in Section F).
Note that these relative condition number based quantiﬁers are always tighter than the density ratio based concentrability coefﬁcients (i.e., max { ¯ C π ∗ , ¯ C π ∗ , mix } ≤ C π ∗ , ∞ ).
For the special case where φ ( s, a ) is a one-hot encoding vector, then they are reduced to the density ratio based concentrability coefﬁcient.
In a non-tabular setting, even if when the density ratio is inﬁnite, the relative condition number can be still ﬁnite.
Intuitively, the bounded relative condition number implies that the ofﬂine data covers the subspace that the comparator policy π ∗ visits.
We remark P (cid:63) ( s (cid:48) | s, a ) ≥ c 3 > 0 in Corollary 2 is a technical condition that allows us to calculate the entropy integral of the hypothesis class easily.
It can be potentially discarded by a more careful argument following (van de Geer, 2000, Chapter 7).
The norm assumption (cid:107) ψ V ( s, a ) (cid:107) 2 < 1 is commonly assumed in the online setting (Zhou et al., 2021).
5. K ERNELIZED N OLINEAR R EGULATORS
We consider the example of KNRs in this section.
A kernelized Nonlinear Regulator (KNR) (Kakade et al., 2020) is a model where the ground truth transition P (cid:63) ( s (cid:48) | s, a ) is deﬁned as s (cid:48)
=
W (cid:63) φ ( s, a )+ (cid:15) , (cid:15) ∼ N (0 , ζ 2 I ) , with φ : S ×
A → R d being a possibly nonlinear feature mapping.
We denote the corresponding model on W by P ( W ) .
We can apply Algorithm 1 and obtain its guarantee.
Especially, since TV( P ( W )( · | s, a ) , P ( W (cid:63) )( · | s, a ))
= Θ( (cid:107) ( W − W (cid:63) )
φ ( s, a ) (cid:107) 22 ) (Devroye et al., 2018), C † π ∗ is upper-bounded by the relative condition number ¯ C π ∗ .
Then, we can also recover the result of Chang et al.
(2021) which proposes a reward penalty-based pessimistic ofﬂine RL algorithm.
The detail is given in Section B.
In summary, we can show
[Equation]
−
[Equation]
min
[Equation] √ [Equation] where ¯ R := rank[Σ ρ ] { rank[Σ ρ ] + ln( c 2 /δ ) } and d S is the dimension of the state.
This implies CPPO can learn a policy that can compete against π ∗ with partial coverage ¯ C π ∗ < ∞ .
Note that the condition ¯ C π ∗ < ∞ does not require Σ ρ to be full-rank.
Also the bound uses rank[Σ ρ ] instead of d , which means that our bound is distribution dependent and is still valid even when d = ∞ as long as the ofﬂine data only concentrate on a low-dimensional subspace.
5. L OW - RANK MDP S WITH R EPRESENTATION L EARNING
We consider the representation learning in ofﬂine RL.
Following FLAMBE (Agarwal et al., 2020b), we study low-rank MDPs but in the ofﬂine setting.
Note that low-rank MDPs here are a more generalized model of the aforementioned linear MDPs (Yang & Wang, 2020) since the true feature representation φ (cid:63) in a low-rank MDP is unknown.
Deﬁnition 4 (Low rank MDPs) .
The ground-truth model P (cid:63) admits a low rank decomposition with a dimension d
if there exists two embedding functions µ ∗ : S → R d , φ ∗ : S ×
A → R d s.t.
P (cid:63) ( s (cid:48) | s, a ) = µ ∗ ( s (cid:48) )
(cid:62) φ ∗ ( s, a ) .
Neither µ ∗ nor φ ∗ is known to the learner.
One interesting special case of a low-rank MDP is the following latent variable model (see Agarwal et al.
(2020b)
for more details).
Deﬁnition 5 (Latent variable models) .
There exists a latent space Z along with functions µ ∗ : Z
→ ∆( S ) and φ ∗ : S × A → ∆( Z ) s.t.
P (cid:63) ( · | s, a ) = (cid:80)
z ∈Z µ ∗ ( · | z ) φ ∗ ( z | s, a ) .
To tackle representation learning under partial coverage on low-rank MDPs, we setup function classes as follows: given two function classes Ψ ⊂ S → R d , Φ ⊂ S ×
A → R d (both are realizable in the sense that µ ∗ ∈ Ψ and φ ∗ ∈ Φ ), we consider a hypothesis class { µ ( s (cid:48) )
(cid:62) φ ( s, a ); µ ∈ Ψ , φ ∈ Φ } .
Then, CPPO (Algorithm 1) and Theorem 1 still work under this setting.
Note that this function class setup is exactly the same as the one from FLAMBE.
Here we show that by leveraging the low-rankness, we can reﬁne the concentrability coefﬁcient to a relative condition number deﬁned by the unknown true representation φ ∗ .
We emphasize that this does not depend on the other features.
Particularly, given a comparator policy π ∗ , we deﬁne ¯ C π ∗ ,φ (cid:63) :
[Equation]
We can show CPPO learns a policy that can compete against π ∗ as long as ¯ C π ∗ ,φ (cid:63) < ∞ .
Theorem 2 (PAC bound for low-rank MDP) .
We set ξ = c 1 ln( | Φ || Ψ | c 2 /δ ) n .
Suppose (a): (cid:107) φ ( s, a ) (cid:107) 2 ≤ 1 , ∀ ( s, a ) ∈ S ×A , ∀ φ ∈ Φ , (cid:82) µ ( s (cid:48) )
(cid:62) φ ( s, a ) d ( s (cid:48) )
= 1 and (cid:82) (cid:107) µ ( s ) (cid:107)
d s ≤ √ d, ∀ µ ∈ Ψ , φ ∈ Φ , (b) ρ ( s, a ) =
d π b P (cid:63) ( s, a ) , (c) P (cid:63)
[Equation]
a ) for some µ ∗ ∈ Ψ , φ ∗ ∈ Φ .
With probability at least 1 − δ , for all π ∗ ∈ Π (again Π can be an unrestricted policy class), CPPO (Algorithm 1) ﬁnds ˆ π such that:
[Equation]
To the best of our knowledge, this is the ﬁrst established PAC result under the partial coverage condition ¯ C π ∗ ,φ (cid:63) < ∞ , ω π ∗ < ∞ for low-rank MDPs in the ofﬂine setting.
We also emphasize that our bound in Theorem 2 is distribution dependent, i.e., it depends on rank (Σ ρ ) rather than the exact rank d .
Note that rank (Σ ρ ) ≤ d , and rank (Σ ρ ) could be much smaller than d when the ofﬂine distribution only concentrates on a low-dimensional subspace (deﬁned using φ ∗ ).
Note that the assumption that ω π ∗ < ∞ does not imply the state-action density ratio C π ∗ , ∞ is small.
Indeed, ω π ∗ < ∞ is much weaker than C π ∗ , ∞ < ∞ .
5. F ACTORED MDP S
The last example we include is the factored MDP (Kearns & Koller, 1999) deﬁned as follows:
Deﬁnition 6 (Factored MDPs) .
Let d ∈ N +
and O being a small ﬁnite set.
The state space S =
O d , and for each state s , we denote s
[ i ] ∈
O as the i -th variable of the state s .
For each i ∈ [1 , · · · , d ] , the parents of i , pa i ⊂ [1 , · · · , d ] , is the subset of state variables that directly inﬂuences i , i.e., the transition is deﬁned as follows:
[Equation]
We will denote S i
= O | pa i | , and given s ∈ S , we will have s [pa i ] ∈ S
i
Due to the factorization, the transition operator P (cid:63) can be described with L := (cid:80) di =1 |A||O| 1+ | pa i | many parameters.
In contrast, the non-factored transition will need O ( |O| d ) parameters.
When | pa i | (cid:28) d ∀ i , it is expected that we can learn this model with lower sample complexity by leveraging the factorization which has been demonstrated in the online setting (Kearns & Koller, 1999).
We remark a factored MDP is an example where model-based approaches are necessary as neither the optimal policy nor the Q functions are factored (Koller & Parr, 2000).
We will slightly modify Algorithm 1 to take the factorization into consideration.
First, we perform
MLE for model learning: each factor P (cid:63)i is independently learned via MLE:
[Equation]
Next, the constrained policy optimization procedure is deﬁned as
[Equation]
Note that in the above objective, there is no restriction on the policy, i.e., the arg max operator searches over all possible policies including non-Markovian ones.
To analyze the performance of the above modiﬁed CPPO, we introduce a specialized concentration coefﬁcient for factored MDPs that utilizes the factored structure.
We focus on density ratio based concentrability coefﬁcients since in a factored MDP with the function class
M :=
{ P = (cid:81) i P i : P i ∈ S
i × A → ∆( O ) } , the concentrability coefﬁcient associated with
M in Deﬁnition 1 will be reduced to the density ratio.
For any π ∗ , we deﬁne the concentrability coefﬁcients for the factored MDP as follows:
[Equation]
where for s j ∈ S j , we denote ν ( s j , a ) :=
(cid:80) s ∈S : s [pa j ]= s j ν ( s, a ) for any distribution ν ∈ ∆( S×A ) .
Comparing to C π ∗ , ∞ deﬁned on the original state space S , here ¨ C π ∗ , ∞ is deﬁned over each state space S j associated with each factor j .
Note that when | pa j | = Θ(1) , |S j | is exponentially smaller than |S| .
One can verify that ¨ C π ∗ , ∞ ≤ C π ∗ , ∞ (see Appendix E.7), where C π ∗ , ∞ ignores the factored structure and treat S as a whole single space.
This formally demonstrates the beneﬁt of the factored structure in terms of the coverage condition in ofﬂine RL .
With the new deﬁnition of the concentrability coefﬁcients, now we are ready to state the PAC bound of CPPO for factored MDPs.
Recall L := (cid:80) di
=1 L i , L i =
|A||O| 1+ | pa i | .
Theorem 3 (PAC bound for factored MDP) .
We set ξ i =
c 1 L i ln
( L i c 2 d/δ ) n .
Then with probability 1 − δ , CPPO ﬁnds a policy ˆ π such that for all comparator policy π ∗ ∈ Π
( Π can be unrestricted),
[Equation]
( nLc 4 d/δ ) n Note that our sub-optimality gap scales polynomially with respect to L , i.e., the complexity of the factored MDP, rather than |S| which can be Ω(exp( d )) .
C ONCLUSION
We study model-based ofﬂine RL with function approximation under partial coverage.
We show that for the model-based setting, realizability in function class and partial coverage together are enough to learn a policy that is comparable to any policies (including history-dependent policies) covered by the ofﬂine distribution.
Our result demonstrates a sharp contrast to model-free ofﬂine RL approaches which often require additional structural conditions in the function class (e.g., Bellman completion) and have restrictions on the pool of candidate policies that they can compete against.
Some readers might wonder whether CPPO is computationally efﬁcient.
The minimax optimization problem arg max π ∈ Π min P ∈ M V π P ﬁts into a framework of planning on robust MDPs (Nilim & El Ghaoui, 2005; Iyengar, 2005). By introducing a robust Bellman equation, they proposed value iteration and policy iteration algorithms, and showed that algorithms are practically tractable in the tabular setting. In the non-tabular setting, Lim & Autef (2019); Tamar et al. (2014) propose the extension using function approximation.
Thus, we can apply their methods to approximately solve the minimax optimization problem in a model-free fashion.
We leave the formal theoretical justiﬁcation when using these approximation planning algorithms as an important direction for future work.
A CKNOWLEDGEMENT
The authors would like to thank Nan Jiang, Tengyang Xie for valuable feedback.
Masatoshi Ueharra is partially supported by Masason foundation
A G ENERALIZATION OF T HEOREM
We present the generalized version of Theorem 1 when the hypothesis class is inﬁnite.
We deﬁne the modiﬁed function class of M :
[Equation]
Given a function class F
[Equation]
let N [] ( δ, F , d ) be the bracketing number of F w
[Equation]
r.t the metric d ( a, b ) given by
[Equation]
Then, the entropy integral of F is given by
[Equation]
We also deﬁne the localized class of H :
[Equation]
where h ( P ( · | s, a ) (cid:107) P (cid:63) ( · | s, a ))
denotes Hellinger distance deﬁned by
[Equation]
Based on Theorem 7.4 (van de Geer, 2000)
[Equation]
the MLE has the following guarantee.
Theorem 4 (MLE guarantee with general function approximation) .
We take a function G ( (cid:15) ) :
[0 , 1] → R s.t. G ( (cid:15) ) ≥ J B [ (cid:15), H ( (cid:15) ) , d ] and G ( (cid:15) ) /(cid:15) 2 is a non-increasing function w.r.t (cid:15) .
Then, letting ξ n be a solution to √ n(cid:15) 2 ≥ cG ( (cid:15) ) w.r.t (cid:15) .
With probability 1 − δ , we have
[Equation]
We remark that the original guarantee in (van de Geer, 2000) is given for the estimation of unconditional distributions.
The adaption to the conditional case is straightforward.
For more details, refer to Section A.1.
Besides, when we assume the convexity of the function class, the entropy integral with bracketing (4) can be replaced with the entropy integral with covering number (Wainwright, 2019, Chapter 14).
By using the above notation and the MLE guarantee, we can generalize Theorem 1.
Theorem 5 (Finite sample error bound of CPPO with an inﬁnite hypothesis class) .
Assume P (cid:63) ∈
M .
Let f ( P )( s, a ) =
TV( P ( · | s, a ) , P (cid:63) ( · | s, a [Equation]
Deﬁne
[Equation]
Here, in G ( M 1 ) and G ( M 2 ) , the expectation is taken over the data.
We set ξ = cG ( M 1 ) + cξ 2 n + c (cid:16) ln( c/δ ) n (cid:17) .
Then, for all π ∗ ∈ Π , we have
[Equation]
This theorem shows once we can calculate G M 1 , G M 2 and ξ n , we can obtain the tight rate.
Importantly, G M 1 and G M 2 are upper-bounded by the localized versions of Rademacher complexities based on symmetrization argument.
Hence, their rates are faster than the ones of the nonlocalized versions.
For example, when |M| is ﬁnite, we ﬁrst have ξ n = (cid:112) ln
( |M| c/δ ) /n .
Then, from Bernstein’s inequality (Wainwright, 2019, Exercise 2.8) and union bound, G ( M 1 ) is upper-bounded by
[Equation]
[Table]
[Equation]
Similarly, from empirical Bernstein’s inequality,
[Equation]
[Table]
[Equation]
Then, we can obtain the result of Theorem 1:
[Equation]
We stress if we use Hoeffeding’s inequality above, we would immediately get the slower rate
O ( n − 1 / 4 ) .
To calculate G ( M 1 ) and G ( M 2 ) in a tight manner, we need to leverage the knowledge that the variance of each element in M 1 and M 2 is controlled from the restriction P ∈ M 1 or
P ∈ M 2 .
A. R ATE OF C ONVERGENCE OF M AXIMUM L IKELIHOOD E STIMATION WITH I NFINITE H YPOTHESIS C LASS
We aim for obtaining a PAC guarantee of MLE following (van de Geer, 2000).
We explain how we should modify the proof of van de Geer (2000) for unconditional density estimation to conditional density estimation.
For simplicity, we assume P (cid:63) > 0 .
We ﬁrst introduce the notation:
[Equation]
Recall
[Equation]
Here, from Lemma 4.2 (van de Geer, 2000)
[Equation]
the following holds:
Lemma 1 (Some property of Hellinger distance) .
[Equation]
We also recall Hellinger distance is stronger than TV distance:
Lemma 2 (Relation of Hellinger distance and TV distance ) .
[Equation]
The following lemma is useful to connect the log-loss and the Hellinger distance.
Lemma 3 (Basic Inequality for MLE) .
[Equation]
This is proved by Lemma 4.1 (van de Geer, 2000).
To simplify the notation, we deﬁne
[Equation]
Here, our goal is showing with probability 1 − δ ,
[Equation]
This is proved by showing for x ≥ ξ n ,
[Equation]
This corresponds to the statement in Theorem 7.4 (van de Geer, 2000).
To prove the above, we ﬁrst use
[Equation]
from Lemma 1.
Then, from Lemma 3, this is upper-bounded by
[Equation]
where ν n = √ n ( E D − E ( s,a ) ∼ ρ,s (cid:48) ∼ P (cid:63) ( ·| s,a ) ) .
To prove the term 5 is less than c exp( − nδ 2 /c 2 ) , we use Theorem 5.11 (van de Geer, 2000), that is, some uniform inequality based on entropy with bracketing.
The rest of the proof is the same as Theorem 7.4 (van de Geer, 2000).
In summary, the only difference is we use the distance H ( a, b ) tailored to the conditional density estimation instead of unconditional density estimation.
B M ORE DETAILS FOR KNR S
We explain the algorithm and present the PAC guaranteed for KNRs.
Here, we denote the dimension of S by d S .
We tailor Algorithm 1 to KNRs as follows to obtain a tighter guarantee.
First, MLE procedure is replaced with ˆ W MLE by regularized MLE:
[Equation]
s, a ) − s [Equation] (cid:107) 22 ] + λ (cid:107) W (cid:107) 2 F , where (cid:107) · (cid:107) F is a Frobenius norm.
Then, the ﬁnal policy optimization procedure is
[Equation]
[Equation]
[Equation]
[Equation]
ˆ
[Equation]
≤ ξ } where Σ n = (cid:80) ni
=1 φ ( s i , a i ) φ (cid:62) ( s i , a i ) .
We state the theoretical guarantee for KNRs below.
Corollary 3 (PAC bound for KNRs) .
Assume (cid:107) φ ( s, a ) (cid:107) 2 ≤ 1 , ∀
( s
, a ) ∈ S × A .
We set
[Equation]
¯ I n = ln (det(Σ n ) / det( λ I )) .
Suppose the KNR model is well-speciﬁed.
By letting (cid:107) W (cid:63) (cid:107)
=
O (1) , ζ 2 =
O (1) , λ
= O (1) , with probability 1 − δ , for all π ∗ , we have
V
[Equation]
(cid:114)
[Equation]
[-ψ i ( x ) ψ j ( x ) dx = 
-]
n , where ¯ R := rank[Σ ρ ] { rank[Σ ρ ] + ln( c 2 /δ ) } .
The proof is deferred to Section E.5.
C M ORE R ELATED W ORKS
We discuss literature related to representation learning in RL.
Representation learning for low-rank MDPs (ground truth feature representation is unknown) in online learning is studied from a model-based perspective (Agarwal et al., 2020b) and
model-free perspective
(Modi et al., 2021).
In the online setting, Zhang et al.
(2021a) ; Papini et al. (2021) also study representation learning under different model assumptions. Comparing with these works, since our setting is ofﬂine, the algorithm and analysis are totally different.
In the ofﬂine setting, Ni et al.
(2021) study dimensionality reduction in a given kernel space, and Hao et al.
(2021) study feature selection in sparse linear MDPs.
Their focus is different as they do not study PAC guarantees under partial coverage.
Ni et al.
(2021) assumes the transition operator can be properly embedded into predeﬁned Reproducing Kernel Hilbert Spaces and learns low-dimensional state-action representations via kernelized embedding and low-rank tensor decomposition.
However, they did not study the errors for policy optimization after using these learned features.
Regarding ofﬂine distribution coverage, Ni et al.
(2021) assumes that the feature covariance matrix (feature associated with the pre-deﬁned kernel) of the ofﬂine distribution is full rank.
Hao et al.
(2021) studies an OPE problem on sparse linear Bellman complete MDPs in the ofﬂine learning setting where they assume all covariance matrices (covariance matrices that correspond to all possible subsets of features) under the ofﬂine distribution are full rank as well.
We study policy optimization in lowrank MDPs (with unknown feature representation), and we do not assume full coverage, i.e., we do not assume the feature covariance matrix is full rank, and indeed our result is distribution-dependent since it scales with respect to the rank of the covariance matrix that is deﬁned using the ground truth feature representation.
D C OMPARISON TO X IE ET AL . (2021)
We compare a result in (Xie et al., 2021) to our result in detail.
Let F be a function class for Qfunctions.
Here, we consider a more general version of their algorithm by replacing the original
E ( f, π ; D ) in their algorithm with
[Equation]
In their original algorithm, they set G = F .
Here, we consider the version such that a discriminator class G can be different from F .
They show the PAC result under partial coverage as follows.
Here, T πP (cid:63) is a Bellman operator under π and P (cid:63) :
[Equation]
Theorem 6 (Extension of Result in (Xie et al., 2021) ) .
Suppose realizaibility Q πP (cid:63) ∈ F , ∀ π ∈ Π and closeness
max f ∈F min g ∈G E s,a ∼ ρ
[( g − T πP (cid:63) f ) 2 ( s, a )]
= 0 , ∀ π ∈ Π .
Then, with 1 − δ , for any π ∗ ∈ Π , we have
[Equation]
By combining this result with the conversion from model-free results to model-based results in (Chen & Jiang, 2019, Corollary 6), we can obtain the following result under partial coverage.
Theorem 7 (PAC guarantee from the direct application of (Xie et al., 2021) to mode-based RL ) .
Assume P (cid:63) ∈ M .
Then, there exists an algorithm s.t.
with 1 − δ , for any policy π (cid:63) ∈ Π ,
[Equation]
As we mentioned, this is worse than our result since it includes | Π | .
Besides, the algorithm can only compete against policies restricted in Π , while our algorithm works for the unrestricted policy class Π which could even include history dependent policies.
For completeness, we give the proof as follows.
We remark that their results (Theorem 4.1) with NPG that can possibly compete with any stochastic policies, are not applicable here.
This is because they need an assumption that the comparator policy π ∗ needs to satisfy Q π ∗ P (cid:63) ∈ F and max f ∈F min g ∈G E s,a ∼ ρ
[( g − T π ∗ P (cid:63) f ) 2 ( s, a )] = 0 , which does not hold for the corresponding Q-function class F after the conversion.
As a notable exception, when the model is a linear Bellman-complete MDP (Zanette et al., 2021), any stochastic policies satisfy the Bellman completeness for the linear Q-function class; then, their algorithms can learn policies that can compete with any stochastic policies satisfying partial coverage.
Proof of Theorem 7.
Given a model class M , consider the following reduction.
We deﬁne a Qfunction class:
[Equation]
Then, we deﬁne a discriminator class G :
[Equation]
The above satisﬁes the realizability Q πP (cid:63) ∈ F , ∀ π ∈ Π and the closedness T πP (cid:63) F ⊂ G , ∀ π ∈ Π .
Thus, the assumptions in Theorem 6 are satisﬁed.
Then, we have
[Equation]
noting |F| =
| Π ||M| and |G| =
| Π | 2 |M| 2 .
E M ISSING P ROOFS
Below we use c, c 1 , c 2 , · · · to denote universal constants.
For a d -dimensional vector a and a matrix
A ∈ R d × d , we denote (cid:107) a (cid:107)
A = a (cid:62) Aa .
Here, a (cid:46) B means a ≤ cB for some universal constant.
c
E. P ROOFS FOR G ENERAL F UNCTION A PPROXIMATION
Proof of Theorem 1.
From Lemma 6, the MLE guarantee gives us the following generalization bound: with probability 1 − δ ,
[Equation]
Letting
[Equation]
s
[Equation]
with probability 1 − δ , from union bound and Bernstein’s inequality, we also have
[Equation]
Hereafter, we condition on the above two events.
Recall that we construct the version space using
D and (cid:98) P MLE as follows:
[Equation]
First Step: Show P (cid:63) ∈
M D in high-probability.
We set ξ = c ln( |M| /δ ) n .
Conditioning on the above two events equations (6) and (7), we prove P (cid:63) ∈
M D .
This is proved by
[Equation]
[Equation]
[Equation] a )) [Equation] ] −
[Equation]
a ) ∼ ρ [Equation] var [Equation]
n ln
[Equation]
| /δ ) . (Plug in MLE guarantee) Second Step: Show [Equation]
[Equation] in high probability. We show for any P ∈ M D , the distance between P (cid:63) is sufﬁciently controlled in terms of TV distance.
More concretely (conditioning on the above two events (6) and (7) ), we show
[Equation]
TV( P
[Equation]
In order to observe this
[Equation]
for any P ∈ M D
[Equation]
we have
[Equation]
Thus, we have:
[Equation]
Here, from (7), we have
[Equation]
[Equation]
P [Equation] [Equation] ]] ln
[Equation]
/δ )
n
[Equation]
Then, for any P ∈ M D , we have
A ( P ) ≤ (cid:114)
[Equation]
≤
[Equation]
≤ (cid:114)
[Equation]
c
[Equation]
( P )
[Equation] cξ ) ln( |M [Equation] ) n + c 2 ln( |M| /δ ) n .
From ( a + b ) 2 ≤ 2 a
+
b
A 2 ( P ) (cid:46) (cid:32)(cid:114) c
( A ( P ) + ξ ) ln( |M| /δ )
n + c ln(
[Equation]
M
[Equation]
/δ )
n (cid:33)(cid:46) ( A ( P ) + ξ ) ln
( |M| /δ )
n +
[Equation]
c ln( |M| /δ )
n (cid:27)(cid:46)
( A ( P ) + ξ ) ln ( |M| /δ ) n ( ξ includes ln( |M| /δ ) )
(cid:46) ( A ( P ) + 1 /n ln
( |M| /δ ))
ln( |M| /δ ) n .
Then, we have
[Equation]
This concludes
[Equation]
Thus, by using the above A ( P ) (cid:46) ξ ( P ∈ M D ) and (8)
[Equation]
with probability 1 − δ
[Equation]
we have:
[Equation]
Third Step:
Calculate the ﬁnal error bound taking the distribution shift into account
For any P ∈ M D , we prove
[Equation]
For any P ∈ M D , this is proved as follows:
[Equation]
[Equation]
on the consequence of the second step)
Combining all things together
[Equation]
with probability 1 − δ
[Equation]
for any π ∗ ∈ Π
[Equation]
we have
[Equation]
[Equation]
Remark 2 (To compete with all history-dependent polices) .
Consider the case where Π is all
Markovian polices.
We want to show we can compete with all history-dependent non-Markovian polices:
[Equation]
We take an element π ∗ from ¯Π .
Then, V π ∗ P (cid:63) and d π ∗ P (cid:63) are still well-deﬁned.
Then, every step in the proof still holds.
The only step we need to check carefully is this line:
[Equation]
This is proved by max π ∈ ¯ Π V πP = max π ∈ Π V πP for any P .
E. P ROOFS FOR G ENERAL FUNCTION APPROXIMATION WITH INFINITE HYPOTHESIS CLASS
Proof of Theorem 5.
From Theorem
[Equation]
the MLE guarantee gives us the following generalization bound: with probability 1 − δ ,
[Equation]
We deﬁne
[Equation]
for some large c .
From a functional Bernstein’s inequality (Lemma 12), by deﬁning
[Equation]
with probability 1 − δ , we have
[Equation]
Similarly, by deﬁning
[Equation]
from a functional Bernstein’s inequality [Equation] with probability 1 − δ
[Equation]
we have
[Equation]
Hereafter, we condition on the above three events:(10), (11) and (12).
First step: Show P ∗ ∈
M D in high probability.
From (10), we have
[Equation]
Thus, P (cid:63) ∈ M 1 .
Then, from (11) and (10),
[Equation]
Thus, P (cid:63) ∈
M D recalling the deﬁnition of M D (we set ξ =
G ( M 1 ) + ξ 2 n + ln( c/δ ) n ).
Second step: Show the upper bound of E ( s,a ) ∼ ρ [ f ( P )( s, a )] for any P ∈ M D .
For any
P ∈ M D , as the proof of Theorem 1, we can prove P ∈ M 2 .
Thus,
[Equation]
Thus, we will hereafter analyze Z n .
Here from (12), for any P ∈ M 2 , we have
E ( s,a ) ∼ ρ [ f ( P )( s, a )] ≤ sup P ∈M 2 (cid:0) | ( E
[Equation]
s,a
[Equation]
∼D
[Equation]
s
[Equation]
a
[Equation]
∼
[Equation] [ f ( P )( s [Equation] )] | (cid:1) + sup P ∈M
[Equation]
E
[Equation]
s
[Equation]
a ) ∼D [ f ( P [Equation]
[Equation]
Hence,
[Equation]
This shows
[Equation]
Third step: Calculate the ﬁnal error bound.
Following the proof of Theorem 1, we can prove
[Equation]
E. P ROOFS FOR T ABULAR MDP S
Proof of Corollary 1.
We prove in a similar way as Theorem 1.
First step We set ξ = c |S| 2 |A| ln( n |S|A| c 2 /δ )
} n .
Then, from Lemma 7, with probability 1 − δ , we can show P (cid:63) ∈
M D since
[Equation]
Hereafter, we condition on the above event.
Second step.
Following the second step in the proof of Theorem 1 based on (8), for any P ∈ M D , we have
[Equation]
13) where
[Equation]
Our goal here is showing with probability 1 − δ ,
[Equation]
To prove (14), consider an (cid:15) -net
{ P 1 ( s, a ) , · · · , P K ( s, a ) } covering a simplex in terms of (cid:107)·(cid:107) 1 4 for each ﬁxed pair ( s, a ) ∈ S × A .
We take (cid:15) = 1 /n .
Since the covering number K is upper-bounded by ( c/(cid:15) ) |S| (Wainwright, 2019, Lemma 5.7), we can obtain ¯ M = { P 1 , · · · , P K |S|×|A| } s.t.
for any possible P ⊂ S ×
A → ∆( S ) , there exists P i s.t.
[Equation]
This implies for any P ⊂ S ×
A → ∆( S ) , there exists P i ( · | s, a ) s.t. ∀ ( s, a ) , | TV( P ( · | s, a ) , P (cid:63) ( · | s, a ))
2 − TV
( P i ( · | s, a ) , P (cid:63) ( · | s, a )) 2 | ≤ 4 | TV( P ( · | s, a ) , P (cid:63) ( · | s, a )) − TV( P i ( · | s, a ) , P (cid:63) ( · | s, a )) | ( a 2 − b 2 = ( a
[Equation]
a
[Equation]
b ) ) ≤ 4TV( P · | s, a ) , P i ( · | s, a ))
( |(cid:107) a (cid:107) − (cid:107) b
(cid:107)| ≤ (cid:107) a − b (cid:107) ) ≤ 4 (cid:15).
We often use this property (15) hereafter.
Next, we deﬁne M (cid:48)
⊂ ¯ M so that it covers M D .
Concretely, we deﬁne M (cid:48) :
[Equation]
The construction is illustrated in Figure 1.
Here, from the deﬁnition, for any P ∈ M D , we can also ﬁnd P (cid:48) ∈ M (cid:48) [Equation] .t.
[Equation]
This is because from the deﬁnition of ¯ M , we can always ﬁnd P ∈ ¯ M satisfying the above.
Such P belongs to M (cid:48) from the deﬁnition of M (cid:48) .
We use this fact later.
Then, from (16) and recalling (13), we have
[Equation]
because
[Equation]
(Take some P (cid:48)(cid:48)
∈ M D noting (16)) ≤ cξ + A ( P ) .
[Equation]
From (13))
Then, with probability 1 − δ , from Bernstein’s inequality, we have A
[Equation]
Hereafter, we condition on the above event.
Based on (17), we can state var[TV
[Equation]
P
[Equation]
| s, a
[Equation]
P
[Equation]
a )
[Equation]
s
[Equation]
a ))
[Equation]
[Equation]
A
[Equation]
P [Equation] [Equation] , with probability 1 − δ .
Following the argument of Theorem 1, for P ∈ M (cid:48) , we have
[Equation]
Then, with probability 1 − δ , we have
[Equation]
This shows for any P ∈ M D , we have
[Equation]
[Equation]
s [Equation] a ) [Equation]
[Equation]
(We take P (cid:48)
∈ M (cid:48) such that (16))
[Equation]
(From the deﬁnition of M (cid:48)
[Equation]
[Equation] and P (cid:48) ∈ M (cid:48) )
Thus, (14) is proved.
Third step.
We follow the third step of Theorem 1:
[Equation]
E. P ROOFS FOR L INEAR M IXTURE MDP S
Proof of Corollary 2.
Here, letting P ( θ )
= θ (cid:62) ψ
( s, a, s (cid:48) ) , recall
M Mix = [Equation]
Upper-bounding E ( s,a ) ∼ ρ [TV( P ( [Equation] (cid:63) )( · | s [Equation]
a [Equation] , P (
[Equation] MLE
[Equation] [Equation] | s, a
[Equation]
By invoking Theorem 4, we ﬁrst show
[Equation]
To do that
[Equation]
we calculate the entropy integral with bracketing.
First
[Equation]
we have
[Equation]
19) where
[Equation]
[Equation]
Here, we use two observations.
The ﬁrst observation is
[Equation]
due to the mean-value theorem √
[Equation]
and assumption P (cid:63) ( s (cid:48) | s, a ) ≥ c 0 > 0 .
The second observation is when we have P (cid:48) <
g < P (cid:48)(cid:48) , we also have (cid:112) ( P (cid:48)
+ P (cid:63) ) / 2 < (cid:112) ( g + P (cid:63) ) /
2 < (cid:112) ( P (cid:48)(cid:48)
+ P (cid:63) ) / 2 .
Then, (19) is concluded.
Next, by letting θ (1) , · · · , θ ( K ) be an (cid:15) -cover of the d -dimensional ball with a radius R , i.e, B d ( R ) , we have the brackets { [ P ( θ ( i ) ) − (cid:15), P ( θ ( i ) )
+ (cid:15) ] } Ki =1 which cover M mix .
This is because for any
P ( θ ) ∈ M mix , we can take θ ( i ) s.t.
(cid:107) θ − θ ( i ) (cid:107) 2 ≤ (cid:15) , then,
P
[Equation]
<
P
[Equation]
[Equation]
noting
[Equation]
The last equality is from Lemma 10.
The brackets above are size of (cid:15) .
Therefore, we have
[Equation]
where N ( (cid:15), B d ( cR ) , (cid:107) · (cid:107) 2 ) is a covering number of B d ( cR ) w.r.t (cid:107) · (cid:107) 2 .
This is upper-bounded by ( cR/(cid:15) )
d (Wainwright, 2019, Lemma 5.7).
Thus, we can calculate the upper bound of the entropy integral J B ( δ, M mix , (cid:107) · (cid:107) 2 ) : [Equation] δ
[Equation]
By taking G ( x ) =
d 1 / 2 x ln( cR/x ) in Theorem 4, δ n = ( d/n ) 1 / 2 ln( nR ) satisﬁes the critical inequality
[Equation]
Finally, with probability 1 − δ
[Equation]
Upper bounding E D [TV( P ( θ (cid:63) )( · | s, a ) , P (ˆ θ MLE )( · | s, a )) 2 ] .
We take an (cid:15) -cover of the ball
B d ( R ) in terms of (cid:107)·(cid:107) 2 , i.e., ¯ M = { θ (1) , · · · , θ ( K ) } , where K = ( cR/(cid:15) ) d .
We take (cid:15) = 1 /n .
Then, for any θ ∈ B d ( R ) , there exists θ ( i )
s.t. ∀
( s, a ) ,
[Equation]
≤
[Equation]
(From Lemma 10.)
[Equation] [Equation]
Hereafter, we condition on the event:
[Equation]
This event holds with probability 1 − δ from Bernstein’s inequality.
Here, note for ˆ θ MLE
[Equation]
we have θ ( i ) s.t.
(cid:107) ˆ θ MLE − θ ( i ) (cid:107) 2 ≤ cid:15) .
Then, following the ﬁrst step of
Theorem
[Equation]
) ∼ ρ [TV( P ( θ (cid:63) )( ·
[Equation]
s, a ) , P
[Equation]
θ
[Equation]
i [Equation]
, a )
[Equation]
n
[Equation]
n + (cid:15) + E ( s,a ) ∼ ρ [TV( P ( θ (cid:63) )( · | s, a ) , P ( θ ( i ) )
( · | s, a )) 2 ] . ( TV( P ( θ (cid:63) )( · | s, a ) , P ( θ ( i ) ) ( · | s, a )) 2 ≤ 4 ) Then, we have
[Equation]
n [Equation] n + (cid:15) + E ( s
[Equation]
a )
[Equation]
TV( P (
[Equation]
(cid:63)
[Equation]
( · |
[Equation]
a
[Equation]
P
[Equation]
MLE
[Equation]
[Equation]
(cid:114) {
[Equation]
n +
[Equation]
n + (cid:15)
[Equation]
ξ (cid:48) .
(From (23))
In the end, by taking (cid:15) = 1 /n , we have with probability 1 − δ ,
[Equation]
This also implies with probability 1 − δ , P (cid:63) ∈ M D .
Show E ( s,a ) ∼ ρ TV( P ( θ (cid:63) )( · | s, a ) , P ( θ )( · | s, a [Equation] 2 (cid:46) ξ, ∀ P ( θ )
∈ M D .
We show for any P ∈ M D , the distance between P (cid:63) is controlled in terms of TV distance.
Our goal is showing
[Equation]
First, following the second step of Theorem 1 based on equation 8, we have
[Equation]
[Equation]
where
[Equation]
From now on, we again consider an (cid:15) -cover of the ball B d ( R ) in terms of (cid:107) · (cid:107) 2 , i.e., ¯ M = { θ (1) , · · · , θ ( K ) } , where K = ( c 1 R/(cid:15) )
d ( (cid:15) = 1 /n ).
This also covers the space M D .
We take
M (cid:48)
= { θ ( i 1 ) , θ ( i 2 ) · · · , } ⊂ M which covers M D , that is,
[Equation]
Recall Figure 1, which illustrates this deﬁnition.
Here, for any θ s.t. ∀ P ( θ ) ∈ M D , we can take θ (cid:48) ∈ M (cid:48)
s.t. (cid:107) θ − θ (cid:48)
(cid:107) 2 ≤ 1 /n .
This is because we can take θ ∈ ¯ M s.t.
(cid:107) θ − θ (cid:48)
(cid:107) 2 ≤ (cid:15) noting ¯ M is an (cid:15) -net, but such θ belongs to M (cid:48) from the deﬁnition of M (cid:48) .
Then, we have
[Equation]
This is because for any θ ( i ) ∈ M (cid:48) , we can take P ( θ ) ∈
M D such that
[Equation]
[Equation]
[Equation]
Here, from (25), we have
[Equation]
Based on the construction of M (cid:48) and (27) [Equation] we have
[Equation]
Then, following the second step of Theorem 1, A ( θ ) satisﬁes
[Equation]
Then, we have
[Equation]
We combine all steps.
Recall for any ∀ P ( θ ) ∈
M D , we can take θ (cid:48) ∈ M (cid:48)
s.t. (cid:107) θ − θ (cid:48)
(cid:107) 2 ≤ 1 /n .
Then, for any P ( θ ) ∈ M D , we have
A
[Equation]
≤
[Equation]
[Equation]
(cid:63)
[Equation]
[Equation]
|
[Equation]
a ))
[Equation]
( P (
[Equation]
)( · | s
[Equation]
a )
[Equation]
P ( θ (cid:63) )( · | s
[Equation]
a )) 2 ] +
[Equation]
E D
[Equation]
E ( s,a ) ∼ ρ )
[TV( P ( θ (cid:48) )( · | s, a ) , P (
[Equation]
(cid:63) )( · | s, a )) 2 ] ≤ 8 (cid:15) + | ( E D − E ( s,a ) ∼ ρ ) [TV( P ( θ (cid:48) )( · | s, a ) , P ( θ (cid:63) )
( · | s, a )) 2 ] (From equation 24) (cid:46) ξ.
(From equation 28 and θ (cid:48) ∈ M (cid:48) )
Then, we have with probability 1 − δ ,
A ( θ ) (cid:46) ξ, ∀ P ( θ )
∈
M D .
Finally, for any P ( θ ) ∈ M D , with probability 1 − δ , we have
E ( s,a ) ∼ ρ [TV( P ( θ (cid:63) )( · | s, a ) , P ( θ )( · | s, a )) 2 ] ≤ A ( θ ) + cξ (From equation 26) (cid:46) ξ.
(From equation 29)
Distribution shift part
Here, for P ∈ M D we prove
[Equation]
Following the third step of the proof of Theorem 5, this immediately concludes the bound
[Equation]
Since (31) is obvious from simulation lemma, we only prove (30).
To prove (30), we take a distribution P ( θ ) ∈
M D .
First, recall for P ( θ ) ∈
M D , we have
[Equation]
From the third statement of Lemma 10, for any V : S → [0 , 1] , we have
[Equation]
Thus, ∀ V : S → [0 , 1] ,
[Equation]
Here, we have
[Equation]
[Table]
Simulation lemma, Lemma 5)
[Table]
The ﬁrst term (a) is upper-bounded by (cid:112) { (1 − γ ) − 2 ξ + λR 2 } noting
≤ V π ∗ P ≤ (1 − γ ) − 1 .
The term (b) is upper-bounded by
[Equation]
By taking λ s.t. λR 2 (cid:46) (1 − γ ) − 2 ξ , (30) is proved.
For linear MDPs, from the fourth statement of Lemma 10, C π ∗ , mix ≤ ¯ C π ∗ .
Then, the statement is concluded.
E. P ROOFS FOR KNR S
Proof of Corollary 3.
We prove in a similar way as Theorem 1.
[Table]
[Equation]
Thus, from Lemma 8, with probability 1 − δ , we can show W ∗ ∈ W D since
[Equation]
Hereafter
[Table]
we
[Table]
this
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Then
[Table]
(deﬁnition of ˆ π )
[Equation]
Then, by setting W (cid:48) =
arg min W ∈M D V π ∗ P ( W ) , we have
[Equation]
Lemma 9)
[Equation]
CS inequality)
[Equation]
(Second step)
From Chang et al.
(2021, Theorem 20) [Equation] with probability 1 − δ , we have
[Equation]
In addition, from Chang et al.
(2021, Theorem 21), with probability 1 − δ , we also have
[Equation]
n .
Finally
[Equation]
by combining all things
[Equation]
we have
[Equation]
E. P ROOFS FOR L OW - RANK MDP S
Proof of Theorem 2.
Until the second step
[Equation]
we can perform the same analysis as Theorem 1.
More concretely
[Equation]
with probability 1 − δ
[Equation]
we have P (cid:63) ∈ M D and
[Equation]
Hereafter, we condition on the above event
[Equation]
Letting f ( s, a )
=
TV( P ( · | s, a ) , P (cid:63) ( · | s, a )) , we use Lemma 4.
Then,
[Equation]
where Σ = n E ρ [ φ (cid:63) φ (cid:63) (cid:62) ] + λI .
We consider how to bound E ( s,a ) ∼ d πP(cid:63) [ (cid:107) φ (cid:63) ( s, a ) (cid:107) Σ − 1 ρ,φ(cid:63) ] .
This is upper-bounded by
[Equation]
From Lemma
[Equation]
Here, in the last line, by letting the SVD of Σ ρ = E ρ [ φφ (cid:62) ] be U ˜Σ ρ U (cid:62) where ˜Σ ρ is a d × d diagonal matrix and U is a d × d orthogonal matrix , we use
[Equation]
Hence, when P ∈ M D , by setting λ s.t. λd (cid:46) nω π ξ , we have
[Equation]
We use (32) here.
Finally,
[Equation]
(Recall the proof of the third step in the proof of Theorem 1)
[Equation]
n
[Equation]
The following inequality is an important lemma to connect E ( s,a ) ∼ d πP(cid:63) { f ( s, a ) } with an elliptical potential E ( [Equation] s, [Equation]
φ
[Equation] s, [Equation] a
[Equation]
Lemma 4 (One-step back inequality) .
Take any f ⊂ S ×
A → R s.t.
(cid:107) f (cid:107) ∞ ≤ B and 0 < λ ∈ R .
Letting ω = max s,a ( π ( a | s ) /π
b ( a | s )) , for any policy π , we have
[Equation]
[ f
[Equation]
s, a
[Equation]
where Σ = n E ( s,a ) ∼ ρ
[ φ (cid:63) ( s, a ) φ (cid:63) (cid:62) ( s, a )]
+ λI .
Proof of Lemma 4.
First, we have an equality:
[Equation]
The second term in (33) is upper-bounded by
[Equation]
Next we consider the ﬁrst term in (33).
By CS inequality
[Equation]
we have
[Equation]
Then,
[Equation]
the
[Equation]
(Jensen)
Finally [Equation] the the ﬁrst term in (33) is upper-bounded by n
[Equation]
(Importance sampling
[Equation]
(Deﬁnition of ρ )
The ﬁnal statement is immediately concluded.
E. P ROOFS FOR F ACTORED MDP S
Proof of Theorem 3.
We denote the constrained set as M D :
[Equation]
Following the ﬁrst step in the proof of Corollary
[Equation]
with probability 1 − δ [Equation] the product (cid:81) i P (cid:63)i is in
M D , i.e.,
[Equation]
Note d comes from the union bound.
Besides, following the second step in the proof of Corollary 1, for any P ∈ M D , with probability 1 − δ ,
[Equation]
After conditioning on the above two events, then, for any P ∈ M D and π (cid:63) ∈ Π , we have
[Equation]
Here, recall
[Equation]
Following the third step in the proof of Corollary 1, the statement is concluded.
Next, we show that ¨ C
[Equation]
Proposition 1 (Comparison of L ∞ -density-ratio based concentrabiliity coefﬁcient between factored
MDPs and non-factored MDPs ) .
For any π ∗ , we have:
[Equation]
Proof.
From now on, for any i ∈ [1 , · · · , d ] , by deﬁning S (cid:48)
i s.t.
S =
S i × S (cid:48)
i , we prove
[Equation]
Then, (34) is easily proved
[Equation]
First, for any s i ∈ S i , a ∈ A , we have
[Equation]
Here, we use
[Equation]
which is proved by the contradiction argument
[Equation]
that is, if 1 >
[Equation]
i | s i ,a ) ρ ( s (cid:48) i | s i ,a ) , both ρ and d π ∗ P (cid:63) cannot be probability mass functions since we would get 1 = (cid:88)
[Equation]
Then, by taking the maximum over s i ∈ S i
[Equation]
a ∈ A for both sides on (35)
[Equation]
we have
[Equation]
i , a ) .
F A UXILIARY L EMMAS
Lemma 5 (Simulation Lemma) .
Consider any two transitions P and (cid:98) P , and any policy π : S → ∆( A ) .
We have:
[Equation]
Proof.
Such simulation lemma is standard in model-based RL literature and the derivation can be found, for instance, in the proof of Lemma 10 from Sun et al.
Lemma 6 (MLE guarantee) .
Given a set of models M =
{ P : S ×
A → ∆( S ) } with P (cid:63) ∈ M , and a dataset D =
{ s i , a i , s (cid:48) i } ni =1 with s i , a i ∼ ρ , and s (cid:48)
i ∼ P (cid:63) ( s i , a i ) , let (cid:98) P MLE be
[Equation]
With probability at least 1 − δ , we have:
[Equation]
Proof.
Refer to (Agarwal et al., 2020b, Section E)
Lemma 7 (MLE guarantee for tabular models)
[Equation]
Proof.
From Chang et al.
(2021, Lemma 12) [Equation] with probability 1 − δ
[Equation]
where N ( s, a ) is the number of visiting times for ( s, a ) .
Then,
[Equation]
Lemma 8 (MLE guarantee for KNRs)
[Equation]
Proof.
The proof directly follows the conﬁdence ball construction and proof from (Kakade et al., 2020).
Lemma 9 ( (cid:96) 1 Distance between two Gaussians) .
Consider two Gaussian distributions P 1 :=
N ( µ 1 , ζ 2 I ) and P 2 := N ( µ 2 , ζ 2 I ) .
We have:
[Equation]
Proof.
This lemma is proved by Pinsker’s inequality and the closed-form of the KL divergence between P 1 and P 2 .
Refer to (Kakade et al., 2020).
Lemma 10 (Property of linear mixture MDPs) .
Let P ( θ )
= θ (cid:62) ψ
( s, a, s (cid:48) ) .
Suppose P ( θ ) ∈ S ×
A → ∆( S ) .
For any function V ∈ S → [0 , 1] , letting ψ V ( s, a ) = (cid:82) ψ ( s, a, s (cid:48) )
V ( s (cid:48) )
d ( s (cid:48) ) , we suppose (cid:107) ψ V ( s, a ) (cid:107) 2 ≤ 1 .
The following theorems hold:
For any ( s, a, s (cid:48) ) , we have | P ( θ )( s, a, s (cid:48) )
− P ( θ (cid:48) )
( s, a, s (cid:48) )
| ≤ (cid:107) θ − θ (cid:48) (cid:107) 2 . 2.
For any ( s, a ) , we have TV( P ( θ )( s, a, · ) , P ( θ (cid:48) )
( s, a, · )) ≤ (cid:107) θ − θ (cid:48)
(cid:107) 2 .
Besides, for any V : S → [0 , 1] , we have | ( θ − θ (cid:48) )
ψ V ( s, a ) | ≤ TV( P ( θ )( s, a, · ) , P ( θ (cid:48) )
( s, a, · )) .
C
[Equation]
(cid:63) ,P (cid:63) =
sup x
[Equation]
[Equation]
[Equation]
[Equation]
( s, a )
[Equation]
(cid:62) [Equation] ( s,a,x ) ( s, a ) [Equation]
x
[Equation]
( s,a
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] In linear MDPs (i.e., ψ ( s, a, s (cid:48) ) =
φ ( s, a ) ⊗ µ ( s (cid:48) ) ), we have
[Equation]
Proof.
We prove the ﬁrst statement.
This is proved by
[Equation]
Here, we use (cid:107) ψ ( s, a, s (cid:48) )
(cid:107) 2 ≤ 1 which is proved by the assumption by setting V ( s ) =
I ( s (cid:48) =
s ) for any s (cid:48) .
Next, we prove the second statement.
For ﬁxed θ ∈ R d and ( s, a ) ∈ S × A , we have
[Equation]
In the third line, we deﬁne V ( s, a,
[Equation]
) | .
Then, from CS inequality,
[Equation]
We use the assumption (cid:107) ψ V ( s,a,θ ) ( s, a ) (cid:107) 2 ≤ 1 .
This concludes the second statement.
Besides, for any V : S → [0 , 1] , we have
[Equation]
The third statement is immediately concluded by
[Equation]
Finally, we prove the fourth statement.
Suppose ψ ( s, a, s (cid:48) )
=
φ ( s, a ) ⊗ µ ( s (cid:48) )
( ⊗ denotes kronerker product).
Then, φ V ( s, a, s (cid:48) )
=
φ ( s, a ) ⊗ (cid:82) µ ( s (cid:48) )
V ( s (cid:48) )
d ( s (cid:48) ) .
Then, by deﬁning a vector µ ( V ) =
(cid:82) µ [Equation] s
[Equation]
) V ( s (cid:48) )
d ( s (cid:48) ) , we immediately have
[Equation]
Here, we have
E
[Equation]
We notice
[Equation]
This is because the square root of a matrix is unique and we have ( A 1 / 2 ⊗ B 1 / 2 )( A 1 / 2 ⊗ B 1 / 2 ) =
AB for symmetric matrices A and B .
Then, by denoting F ρ = E ρ [ φ ( s, a ) φ ( s, a ) (cid:62) ] , F d πP(cid:63) =
E d πP(cid:63) [ φ ( s, a ) φ ( s, a ) (cid:62) ] and denoting the pseudo inverse of F as F + , we can see (37) is equal to { F 1 / 2 ρ
[Equation]
Here, I k is a diagonal matrix s.t. k ∈ N + values in the diagonal entries are 1 and the rest of values are 0 .
Then, the maximum singular value of { F − 1 / 2 ρ
F d πP(cid:63) F − 1 / 2 ρ } ⊗ I k is equal to the one of { F − 1 / 2 ρ
F d πP(cid:63) F − 1 / 2 ρ } .
This is equal to sup x x
[Equation]
Hence, the fourth statement is concluded.
Lemma 11 (Distribution shift lemma) .
Suppose A 1 , A 2 , A 3 are semipositive deﬁnite matrices:
[Equation]
Note
[Equation]
Proof.
[Equation]
In addition, for any semipositive deﬁnite matrices A, B we have
[Equation]
, where U Λ U (cid:62) is the SVD decomoposition of A .
This concludes that
[Equation]
The following lemma is useful to obtain the generalized result of Theorem 1.
The proof is given in (Wainwright, 2019, Theorem 3.27).
We ﬁrst deﬁne
[Equation]
Lemma 12 (Functional Bernstein’s inequality: Talagrand concentration inequality for empirical process) .
Suppose (cid:107) f (cid:107) ∞ ≤ B .
With probability 1 − δ ,
[Equation]
As an immediate corollary,
[Equation]