On Gap-dependent Bounds for Ofﬂine Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
This paper presents a systematic study on gap-dependent sample complexity inofﬂine reinforcement learning.
Prior work showed when the density ratio betweenan optimal policy and the behavior policy is upper bounded (the optimal policycoverage assumption), then the agent can achieve an O  1 ✏ rate, which is alsominimax optimal.
We show under the optimal policy coverage assumption, the ratecan be improved to O  1 ✏  when there is a positive sub-optimality gap in the optimal
Q -function.
Furthermore, we show when the visitation probabilities of the behaviorpolicy are uniformly lower bounded for states where an optimal policy’s visitationprobabilities are positive (the uniform optimal policy coverage assumption), thesample complexity of identifying an optimal policy is independent of 1 ✏ .
Lastly, wepresent nearly-matching lower bounds to complement our gap-dependent upperbounds.
Introduction
Reinforcement Learning (RL) aims to learn a policy that maximizes the long-term reward in unknownenvironments [Sutton and Barto, 2018].
The success of reinforcement learning often relies on beingable to deploy the algorithms that directly interact with the environment.
However, such direct interactions with real environments can be expensive or even impossible in many real-world applications,e.g., health and medicine [Murphy et al., 2001, Gottesman et al., 2019], education [Mandel et al.
,2014], conversational AI [Ghandeharioun et al., 2019] and recommendation systems [Chen et al.
Instead, we have access to a dataset generated from some past suboptimal policies.
Ofﬂinereinforcement learning (ofﬂine RL) aims to ﬁnd a near-optimal policy using the ofﬂine dataset, andhas achieved promising empirical successes [Lange et al., 2012, Levine et al., 2020].
Recently, a line of works showed that under the single policy coverage assumption (Assumption 3.2),one can obtain a near-optimal policy with polynomial number of samples [Rashidinejad et al., 2021,
Xie et al., 2021, Yin and Wang, 2021, Li et al., 2022].
In particular, for the tabular setting, recent workshave obtained minimax optimal sample complexity bounds e ⇥ ⇣
H
[Equation]
⌘ where H is the planninghorizon, S is the number of states, C ⇤ is the constant for the single policy coverage assumption, and✏ is the target accuracy [Xie et al., 2021, Li et al., 2022].
The O  1 / ✏ is for the worst case and inmany benign settings, one may use much fewer samples to learn a (near-)optimal policy.
How the benign problem structures help reduce the sample complexity has been extensively studied inthe online bandits and reinforcement learning [Simchowitz and Jamieson, 2019, Jonsson et al., 2020,
Wagenmaker et al., 2021b, Xu et al., 2021].
In particular, when there is a suboptimality gap betweenthe optimal policy and the rest, then one can obtain log T -type regret in contrast to p T -type regretbounds in the worst case where T is the number of interactions.
However, to our knowledge, howthe gap structure helps reduce sample complexity in ofﬂine RL has not been thoroughly investigated.
This paper presents a systematic study on gap-dependent bounds for ofﬂine RL in the canonicaltabular setting, with nearly-matching upper and lower bounds in different regimes.
1.1 Main Contributions
We present novel analyses for the standard VI-LCB algorithm (Algorithm 2).
Our main results aresummarized in Table 1.1.1.
We develop a novel technique, deﬁcit thresholding , to obtain gap-dependent bounds inofﬂine RL.
Different from the clip trick widely used in online RL [Simchowitz and Jamieson,2019, Xu et al., 2021] to obtain gap-dependent bounds, our deﬁcit thresholding techniqueis adaptive to the problem-instance.
As will be shown in Section 6.2, this technique helpsreduce the dependency on H by adapting to the variance of the estimate.2.
Using the deﬁcit thresholding technique, we obtain the ﬁrst gap-dependent bound underthe optimal policy coverage assumption.
Speciﬁcally, we obtain an e
[Equation]
⌘ boundwhere gap min is the minimum suboptimality gap between the optimal Q -value of the bestaction and that of the second-to-the-best action.
Notably, compared with the worst-casegap-indepdent, the rate improves from 1 / ✏ 2 to 1 / ✏ .3.
We also present the ﬁrst gap-dependent lower bound for ofﬂine RL, ⌦ ⇣
[Equation]
SC ⇤ ✏ gap min ⌘ to showthe O (1 / ✏ ) is unimprovable even with the gap condition, and our upper bound is tight up toan H 2 factor.
The main insight from our lower bound is that if there exists a state whosevisitation probability of the behavior policy and the optimal policy is O ( ✏ ) , then we cannotlearn much of the state using the ofﬂine dataset and will inevitably incur an O ( ✏ ) error.4.
We further study what condition permits even faster rate than O (1 / ✏ ) .
Leveraging our lowerbound mentioned above, we propose a new condition, uniform optimal policy coverage ,which posits that the visitation probabilities of the behavior
policy are uniformly lower
bounded by P for states where an optimal policy’s visitation probabilities are positive.
Under this assumption, we obtain an e
O
⇣ H 3 P gap 2min ⌘ bound.
Importantly, this bound isindependent of 1 / ✏ (not even log(1 / ✏ ) ), a.k.a., one can identify an exact optimal policy.
Wealso complement this upper bound with an ⌦ ⇣ H P gap 2min ⌘ lower bound to show our upperbound is tight up to an H 2 factor.
Lastly, we note that all of our bounds are obtained with the same algorithm, i.e., the algorithmautomatically exploits the benign problem structure without any prior knowledge.
Related Work
We focus on existing theoretical results on gap-dependent bounds and ofﬂine RL in the tabular setting.
Theoretical Results on Ofﬂine Tabular RL.
Theoretical analysis of ofﬂine RL can be traced backto Szepesvári and Munos [2005], under the uniform coverage assumption where every state-actionpairs are visted by the behavior policy with a positive probability.
Sharp sample complexity boundshave been obtained under this assumption [Xie and Jiang, 2021, Xie et al., 2019, Yin et al., 2020,2021b, Ren et al., 2021].
Recently, a line of works showed that under a much weaker assumption,single policy coverage, one can design sample efﬁcient algorithms with both model-based and modelfree methods based on the pessimism principle, [Rashidinejad et al., 2021, Yin and Wang, 2021, Xieand Jiang, 2021, Jin et al., 2021, Uehara and Sun, 2021, Uehara et al., 2021, Zanette et al., 2021].
Recently, Yin and Wang [2021] obtained a problem-dependent sample complexity in terms of thevariance.
, the agent can still learn a near-optimal policy (as long as the dataset covers an optimal action).
On the other hand, this dependency is needed in the online setting because the agent needs to explore the all actions.
Condition
Upper bound
Lower
[Table]
[Table]
Instance-dependent Sample Complexity in Online Learning and Generative Models.
In online
RL, a line of work studied the upper and lower bounds of gap-dependent sample complexity in both theregret and PAC settings [Simchowitz and Jamieson, 2019, Xu et al., 2021, Dann et al., 2021, Jonssonet al., 2020, Wagenmaker et al., 2021b,a, Tirinzoni et al., 2021].
Besides gap-dependent bounds, thereare other problem-dependent bounds such as ﬁrst-order and variance-dependent bounds Zanette and
Brunskill [2019].
For generative models, Zanette et al.
[2019] derived an upper bound depending on
Gap-dependent bounds in Ofﬂine Learning.
The most related work is by Hu et al.
[2021]who studied the convergence rate of Q -learning in the discounted MDPs under the uniform coverage assumption.
They proved that one can obtain an ✏ -optimal policy with
O
S 3 A 3 log(1 / ✏ ) / (1   ) 4 P 2 gap 2min  samples for tabular MDP and e
O (1 / ✏ ) (ignoring other parameters) for linear MDP.
In comparison, we show that under the weaker uniform optimal policy coverageassumption, we can identify an exact optimal policy with O
[Equation]
sample complexity,which has no dependency on 1 / ✏ .
Preliminaries
Notations.
We let [ n ] = { 1 , 2 · · · n } .
For two vectors a, b of the same length k , we use a  b to denote 93 the Hadamard product ( a 1 b 1 , a 2 b 2 · · · a k b k ) .
We use the standard deﬁnitions of O ( · ) , ⇥ ( · ) , ⌦ ( · ) tohide absolute constants, and tilded notations e
O ( · ) , e ⇥ ( · ) , e ⌦ ( · ) to hide absolute constants as well aspoly-logarithmic factors except for log 1  .
Var p ( V ) =
p > V  V  ( p > V ) 2 refers to the variance of V with respect to the weight p .
a .
b means a  Cb for some positive absolute constant C , andsimilarly a & b means a  Cb .
I { ⇠ } is the indicator function of the event ⇠ , which equals 1 when ⇠is true and {+
 otherwise.+}
( X ) is the probability simplex over X .
3.1 Markov Decision Processes
We consider tabular ﬁnite-horizon time-inhomogeneous MDPs described by the tuple M =( S , A , H, P , p 0 , r ) .
Here S is the state space with cardinality S , A is the action space with car- 102 dinality A ,
P = { p h,s,a } ( h,s,a )
[ H ] ⇥ S ⇥
A with p h,s,a 2  ( S ) is the transition kernel at timestep h , 103 state s and action a , p 0 2  ( S ) is the initial state distribution of s 1 , and r = { r 1 , r 2 , . .
, r H } with 104 r h : S ⇥ A !
[0 , 1] is the reward function.
For each episode, the player will generate a trajectory
{ ( s h , a h , r h ) } Hh =1 where s 1 ⇠ p 0 , s
h
+
⇠ p h,s h ,a h , and r h =
r h ( s h , a h ) , by controlling the actions{ a h } Hh =1 .
The target of the player is to maximize the total reward
P
Hh =1 r h .
Policies.
A policy ⇡ = { ⇡ h ( ·| s ) } ( h,s )
[ H ] ⇥ S refers to a set of distributions over A .
With a slightabuse of the notations, when a policy ⇡ is deterministic, we use ⇡ h ( s ) to denote the action takenat timestep h and state s .
We deﬁne E ⇡ , M [ · ] , E  ⇠ ( ⇡ , M ) [ · ] , where  = { ( s h , a h , r h ) }
h
[ H ] is a 110 trajectory sampled using policy ⇡ in the MDP
M and the expectation is over the randomness of boththe policy and the transitions.
M will be omitted when there is no confusion.
Value Functions, Q-Functions and Policy Distributions.
For a given policy ⇡ and an MDP M ,we deﬁne the state value function and the state-action value function to be
[Equation]
We deﬁne the optimal Q-function as Q ⇤ h
( s, a ) , sup ⇡ Q ⇡ h ( s, a ) , and similarly V ⇤ h ( s ) , sup ⇡ V ⇡ h ( s )for all ( h, s, a )
[ H ] ⇥ S ⇥ A .
It is well known that there exists a deterministic optimal policy ⇡ ⇤ 116 that can achieve the above maximum for all s 2 S , a 2 A and
h 2 [ H ] simultaneously.
We denote the value of a policy by V ⇡ 0 = E ⇡ hP
Hh =1 r h
[Equation]
s h , a h ) i , and the value of the optimal policy by V ⇤
= V ⇡ ⇤ 0 .
A policy ⇡ is ✏ -optimal if
Suboptimal( ⇡ ) , V ⇤
V ⇡
[Equation]
 ✏ .
We use d ⇡ h ( · ) to denote the probability of reaching a state s under policy ⇡ :
[Equation]
h
[Equation]
Sub-optimality Gap.
For a MDP instance M , we deﬁne the gap at ( h, s, a ) 2 [ H ] ⇥ S ⇥
A to 119 be gap h ( s, a ) =
V
⇤ h ( s )  Q
⇤ h
( s, a ) , which is always non-negative because of the deﬁnition of ⇡ ⇤ .
Our results will depend on the smallest positive gap: gap min ,
[Equation] [Equation]
|
[Equation]
0 gap h ( s, a ) , whichquantiﬁes the difﬁculty of learning the optimal policy in the MDP instance.
3.2 Ofﬂine Reinforcement Learning
Ofﬂine Learning.
For ofﬂine reinforcement learning, we want to solve an MDP M =( S , A , H, P , r ) with unknown transitions by utilizing a given dataset collected by some unknownbehavior policy µ .
Note that the algorithm is not allowed to perform any kind of additional sampling.
The dataset is D
= {{ ( s h,i , a h,i , r h,i ) }
h
[ H ] } i 2 [ N ] , which contains N trajectories collected by thebehavior policy µ independently.
A ( ✏ ,  ) -PAC ofﬂine reinforcement learning algorithm is deﬁned tooutput an ✏ -optimal policy ⇡ with probability at least 1   .
Assumptions.
We introduce two dataset assumptions that will be used in this paper.
Assumption 3.1 (Uniform optimal policy coverage) .
We deﬁne the uniform optimal policy coveragecoefﬁcient to be
[Equation]
where ⇡ ⇤ is an optimal policy.
We assume that P > 0 .
Assumption 3.1 states that the behavior policy µ covers all the state-action pairs that some ⇡ ⇤ willchoose with positive probability.
This is a natural assumption if we want to recover the optimalpolicy.
A closely related assumption is the uniform coverage assumption, i.e., all ( h, s, a ) tuples are coveredby the dataset [Yin and Wang, 2020, Ren et al., 2021, Yin et al., 2021a, Hu et al., 2021].
Ourassumption is signiﬁcantly weaker as it only assumes covering the optimal policy.
We will prove thatunder this assumption, we can identify the optimal policy with ﬁnite samples.
input : Dataset D 0 , reward function r 1 Set Q H +
( s, a )
=2 Set V H +
( s, a ) =3 for h
H to 1 do
Compute the empirical transition kernel
ˆ P h
ˆ P h,s,a ( s 0 )
=
N h ( s,a,s 0 ) N h
( s,a ) with 0 / 0 =
for s 2 S , a 2 A do b h ( s, a )
C b r
Var ˆ Ph,s,a ( V h +1 ) ◆ N 0 h ( s,a ) + C b
H ◆ N 0 h ( s,a ) , where N 0 h ( s, a ) =
N h ( s, a ) _ ◆
Q h ( s, a ) max { 0 , r h ( s, a ) + ˆ P > h,s,a V h +1  b h ( s, a ) } for s 2 S do
V h ( s ) max a 2 A Q h ( s, a ) ⇡ h ( s ) arg max a 2 A
Q h ( s, a ) output : policy ⇡
Algorithm 2: input : Dataset D , reward function r 1 Split D into 2 halves containing same number of sample trajectories, D main and D aux 2 D 0 = {} 3 for ( h, s )
[ H ] ⇥ S do
N trim h ( s ) max { 0 , N aux h ( s ) q
N aux h ( s ) log HS  }
Randomly subsample min { N trim h ( s ) , N main h ( s ) } samples of transition from ( h, s ) from
D main to add to D6 ⇡
VI-LCB ( D 0 , r ) output : policy ⇡
Assumption 3.2 (Optimal policy coverage) .
We deﬁne the relative optimal policy coverage coefﬁcientto be
[Equation]
⇤ h
( s, a )
d µh ( s, a ) with convention that 0 / 0 = 0 , where ⇡ ⇤ is an optimal policy.
We assume that C ⇤ < 1 .
Similar coverage assumption has been widely adopted in Li et al.
[2022], Shi et al.
[2022], Yan et al.[2022], Jin et al.
[2021], Rashidinejad et al.
Researchers have designed algorithms based onthe pessimism principle to efﬁciently solve ofﬂine RL problems under this assumption.
Assumption3.2 is usually weaker than Assumption 3.1.
For example, if there exists one unique optimal policy⇡ ⇤ and µ = ⇡ ⇤ , we have C ⇤ =
while P can still be arbitrarily small.
In addition, we always have
C ⇤  1 P .
3.3 Subsampled VI-LCB
We brieﬂy introduce the algorithm (Algorithm 2) that will be used in the analysis.
Value Iteration with Lower Conﬁdence Bound (VI-LCB) was ﬁrst introduced by Rashidinejad et al.
[2021] and improvedby Li et al.
The main idea is to maintain a pessimistic estimate on the value functions sothat the suboptimality of the output policy only depends on the uncertainty of the optimal policy.
Byutilizing the subsampling technique and Bernstein-style bonus, subsampled VI-LCB achieves theminimax sample complexity e
O [Equation] H [Equation]
SC [Equation] In the algorithm, N h ( s, a ) refers to the number of sample transitions starting from state s , takingaction a at time step h of some given dataset, and N h ( s ) =
P a 2 A N h ( s, a ) .
Superscripts stand for 158 the dataset.
See Li et al.
[2022] for a more detailed description of the algorithm.
Finding an Exact Optimal Policy with Assumption 3.1
In this section, we show that we can identify the exact best policy with ﬁnite samples by utilizing the gap structure under Assumption 3.1.
On the other hand, for the minimax sample complexity e
O ( H 3 SC ⇤ ✏  2 ) , it will become inﬁnite when ✏ approaches 0.
Note that directly setting ✏ < gap min does not imply that the output policy ⇡ is optimal.
Instead, we only have
[Equation]
while ⇡ can still be suboptimal at states visited with low probability.
Theorem 4.1.
For an MDP M and a behavior policy µ with uniform optimal coverage coefﬁcient P ,if the number of sample trajectories satisﬁes
[Equation]
then with probability at least 1   , Algorithm 2 returns an optimal policy.
Sketch of Proof.
First, if for all ( h, s, a ) 2 [ H ] ⇥ S ⇥
A satisfying d ⇤ h ( s, a ) > 0 , we have
[Equation]
then we have ⇡ is an optimal policy.
This is because as Q is a pessimistic estimate of Q ⇤ , we have
[Equation]
for any action a 0 that is sub-optimal.
As a result, ⇡ chooses the optimal action for all ( h, s, a )
[ H ] ⇥ S ⇥
A covered by the optimal policy, which implies ⇡ is an optimal policy.
Second, we show that (1) is satisﬁed with N  e
O ⇣
H 4 P gap 2min ⌘ .
This is because if the number ofsamples at ( h, s, a ) exceeds e
O ( H 4 / gap 2min ) , we can guarantee that the estimation error at that stepis smaller than gap min /H by Hoeffding’s inequality.
Then the accumulated estimation error at
Q h ( s, a ) can be bounded by ( H  h )gap min /H <
gap min , which implies (1).
To further improvethe dependence on H , we will use Bernstein’s inequality and the proof is deferred to Appendix
B.4.
Gap-dependent Upper Bounds with Assumption 3.
In the previous section, we show that the optimal policy can be identiﬁed if Assumption 3.1 issatisﬁed.
However, the uniform optimal coverage coefﬁcient P can be very small, which makes thebound N
[Equation]
⌘ useless.
In this section, we present two results on learning an ✏ -optimalpolicy with assumption (Assumption 3.2) and we provide the proof sketch in the next section.
Thefull proof can be found in Appendix B.
Theorem 5.1.
For an MDP M and behavior policy µ with relative optimal policy coverage coefﬁcient
C ⇤ , if the number of sample trajectories satisﬁes
[Equation]
Algorithm 2 returns an ✏ -suboptimal policy with probability at least 1   .
Theorem 5.1 shows that to learn an ✏ -optimal policy, we only need e
O (1 / ✏ gap min ) samples, whichsigniﬁcantly improves the minimax sample complexity e
O (1 / ✏ 2 ) when ✏ ⌧ gap min .
Now we show that we can further improve the bound if both Assumption 3.1 and Assumption 3.2 aresatisﬁed.
Theorem 5.2.
For an MDP M and behavior policy µ with uniform optimal policy coverage coefﬁcient
P and relative optimal policy coverage coefﬁcient C ⇤ , if the number of sample trajectories satisﬁes
[Equation]
Algorithm 2 returns an ✏ -suboptimal policy with probability at least 1   .
Theorem 5.2 improves an H factor compared with Theorem 5.1, with the cost of an extra H/Pterm.
As the additional term is a constant with respect to ✏ and gap min , the bound is improved when✏ gap min is small.
We believe this H/P can be removed and we leave it to future works.
Main Proof Techniques
In this section, we will provide a proof sketch for Theorem 5.1 and Theorem 5.2.
6.1 Pessimistic Algorithms
First, we deﬁne pessimistic algorithms and imaginary MDPs M =
( S , A , H, P , r ) determined by 196 the pessimistic algorithms.
Our analysis will generally hold for all pessimistic algorithms deﬁnedby Deﬁnition A.
and we will show that VI-UCB (Algorithm 2) is a pessimistic algorithm.
We use
VI-LCB only to derive the ﬁnal sample complexity guarantees.
Deﬁnition 6.1 (Pessimistic algorithms) .
An ofﬂine learning algorithm with output policy ⇡ ispessimistic if with probability at least 1   , the following arguments hold,
It maintains a pessimistic estimate Q of the true Q ⇤ .2.
Q is the optimal Q function of an imaginary MDP M =
( S , A , H, P , p 0 , r ) , wherer h ( s, a )  r h ( s, a ) for all ( h, s, a )
[ H ] ⇥ S ⇥ A . 43.
⇡ is the greedy policy with respect to Q .
Most of the existing ofﬂine RL algorithms are pessimistic algorithms and we will also prove it for
VI-LCB [Li et al., 2022] in Appendix B.
Deﬁnition 6.2 (Deﬁcit) .
For any pessimistic algorithm and the corresponding imaginary MDP
M = ( S , A , H, P , p 0 , r ) , we deﬁne the deﬁcit to be
[Equation]
From the deﬁnition of pessimistic algorithms, we have E h
( s, a )  0 immediately.
Intuitively, deﬁcitstands for how pessimistic the estimates are.
Note that the deﬁcit is related to the algorithm itself andusually we can bound it by utilizing concentration inequalities.
6.2 Deﬁcit Thresholding for Analysising LCB-style Algorithms
By deﬁning V ⇡ to be the value function of policy ⇡ in M and recalling that ⇡ is the optimal policyin M , we have V ⇤
 V ⇡ 0  V ⇡ 0  V ⇤ 0 and thus
[Equation]
can upper bound the suboptimality of ⇡ .
Surprisingly, we will show that even if we threshold the deﬁcit, a similar upper bound still holds.
Deﬁne the thresholded deﬁcits and the corresponding reward functions as ¨
[Equation]
( s,
[Equation]
[Equation]
for any non-negative threshold function ✏ h ( s, a ) .
Then we can deﬁne a thresholded MDP ¨ M =( S , A , H, P , ¨ r ) and we use ¨ V to denote the value function in ¨ M .
Now we present the key lemmashowing that if the thresholding function satisﬁes certain conditions related to gap min , then thesuboptimality of ⇡ can still be bounded by the thresholded deﬁcit.
Lemma 6.1.
Suppose for a thresholding function ✏ h ( s, a ) , for all ( h, s )
[ H ] ⇥ S , we have
[Equation]
Then we can bound the suboptimality of ⇡ by the thresholded deﬁcit:
[Equation]
See Appendix A.2 for the rigorous proof, where this lemma is decomposed and restated as Theorem
A.1 and Deﬁnition A.5.
One way to satisfy (2) is to set ✏ h ( s, a ) =
gap min 2 H for all ( h, s, a ) 2[ H ] ⇥ S ⇥ A .
As a result, the reward at each timestep h is increased by at most gap min 2 H afterthresholding, so the overall increase of the value function is bounded by gap min / 2 .
This kind of✏ h ( s, a ) leads to the following corollary.
Corollary 6.1.
For any pessimistic algorithm, we have
[Equation]
In addition, note that E h ( s h , a h ) depends on variance of the transition p h,s h ,a h as if the transitionvariance is small, the estimate will be accurate and the deﬁcit will be small.
This inspires us tothreshold adaptively based on the variance.
We design the following adaptive threshold function:
[Equation]
It turns out that e ⌦ ( HP ) sample complexity is enough to guarantee this kind of threshold functionsatisﬁes (2).
(See Lemma B.8 in Appendix B.5.1)
Corollary 6.2.
For any pessimistic algorithm and number of samples N  e
O ( HP ) , we have
[Equation]
To proceed from Lemma 6.1, we will utilize the following inequality
[Equation]
Then with different choices of ✏ h ( s, a ) used in Corollary 6.1 and Corollary 6.2 and the bonus functionused in the VI-LCB algorithm, we can derive Theorem 5.1 and Theorem 5.2.
Here we brieﬂy explainthe e
O (1 / ✏ ) dependence.
We have E h
( s, a )  O ( b h ( s, a )) , where b h ( s, a ) is the pessimistic bonusscaling as O (1 / p n ) (ignoring other dependences).
Then by Lemma 6.1 and the above inequality, wecan achieve an e
O (1 / ✏ ) bound immediately.
A more detailed proof is provided in Appendix B.
Our technique is similar to the clip techique [Simchowitz and Jamieson, 2019] in online MDP as bothof them threshold the estimates (deﬁcit here and surplus in their work) by some ⇥ (gap min ) terms.
Simchowitz and Jamieson [2019] further clip another gap h ( s, ¯ a ) H term, and we can actually achievethat as well, but it does not improve the sample complexity in the ofﬂine setting as we do not need toexplore all the actions.
In addition, we develop a new thresholding function based on the empiricalvariance, which can improve the H factor by utilizing the Bernstein’s inequality and the total variancetechnique.
We believe this new technique can also be applied to the online setting and improve thesample complexity there.
In this section, we provide two lower bounds for uniform optimal coverage assumption (Assumption3.1) and optimal policy coverage assumption (Assumption 3.2) respectively.
Our lower bounds showthat Theorem 4.1 and Theorem 5.1 are optimal up to H factors and logarithm terms.
We begin witha general lower bound.
Here the ofﬂine learning algorithm , ALG , is deﬁned as the algorithm thattakes a dataset D as input and output a policy ˆ ⇡ .
Note that ALG can be stochastic.
Theorem 7.1.
There exists some absolute constant C , such that for any A  3 , S  2 , H  2 , ⌧ <12 ,   13 ,  1  2 and ofﬂine RL algorithm ALG , if the number of sample trajectories satisﬁes
[Equation]
there exists a MDP instance M and a behavior policy µ with gap min = ⌧ , P  [Equation] [Equation]
⇤   1 suchthat the output policy ˆ ⇡ of ALG has expected suboptimality
[Equation]
By choosing  = 1 / 3 ,  1 = 1 3 ePS and ⌧ = gap min , Theorem 7.1 indicates the following corollary.
Corollary 7.1.
For any given instance coefﬁcients ( H, gap min , P  16 eS ) , target suboptimality✏ .
H gap min and any ofﬂine reinforcement learning algorithm ALG , there exists an instance( M , µ ) such that if the number of trajectory samples satisﬁes
[Equation]
the output policy ˆ ⇡ would have expected suboptimality more than ✏ , i.e.,
[Equation]
By choosing  = 12 ✏ H gap min ,  1 = C ⇤ , ⌧ = gap min , Theorem 7.1 indicates the following corollary.
Corollary 7.2.
For any given instance coefﬁcients ( H, S, gap min , C ⇤  2) , target suboptimality✏ .
H gap min and any ofﬂine learning algorithm ALG , there exists an instance ( M , µ ) such that ifthe number of trajectory samples satisﬁes
[Equation]
the output policy ˆ ⇡ would have expected suboptimality more than ✏ , i.e.,
[Equation]
The proof of the corollaries can be found in Appendix C.1.
We construct a family of MDPs thatinvolve solving HS independent bandit problems, where each bandit requires e ⌦ ( HA ⌧ 2 ) visits so thatthe estimation error can be bounded by ⌧ .
Similar constructions have been made in Yin et al.
[2021a],
Dann et al.
[2017], Xie et al.
Our key observation is that the initial state distribution canlinearly determine the uniform optimal policy coverage coefﬁcient P and the ﬁnal suboptimality.
Also, our proof can lead to another version of lower bound, where a ⌦ ( ✏ ) suboptimality occurs with aconstant probability.
See Appendix C.2.3 for details.
Conclusion
We presented a systematic study on gap-dependent upper and lower bounds for ofﬂine reinforcementlearning.
Depending on different assumptions, the rates can be improved from e
O (1 / ✏ 2 ) in the worstto e
O (1 / ✏ ) or even independent of 1 / ✏ .
One open question is that there still a gap between the upper and lower bounds in terms of H .
Wenote that this gap also exists in the online setting [Simchowitz and Jamieson, 2019, Xu et al., 2021].
Another direction to generalize our results to the function approximation setting [He et al., 2021].