Ofﬂine Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps
Abstract
We consider a challenging theoretical problem in ofﬂine reinforcement learning (RL): obtaining sample-efﬁciency guarantees with a dataset lacking sufﬁcient coverage, under only realizability-type assumptions for the function approximators.
While the existing theory has addressed learning under realizability and under non-exploratory data separately, no work has been able to address both simultaneously (except for a concurrent work which we compare in detail).
Under an additional gap assumption, we provide guarantees to a simple pessimistic algorithm based on a version space formed by marginalized importance sampling (MIS), and the guarantee only requires the data to cover the optimal policy and the function classes to realize the optimal value and density-ratio functions.
While similar gap assumptions have been used in other areas of RL theory, our work is the ﬁrst to identify the utility and the novel mechanism of gap assumptions in ofﬂine RL with weak function approximation.
INTRODUCTION AND RELATED WORKS
In ofﬂine reinforcement learning (RL) the learner searches for a good policy purely from historical (or ofﬂine) data, without direct interactions with the real environment.
The lack of intervention with the system makes ofﬂine RL a promising paradigm for learning sequential decision-making strategies in many important real-world applications.
Early research in ofﬂine RL focused on analyzing approximate value and policy iteration algorithms and had signiﬁcant overlap with the approximate dynamic programming literature (Munos, 2003, 2007; Munos and Szepesvári, 2008; Antos et al., 2008; Farahmand et al., 2010).
These algorithms and their guarantees typically require relatively strong assumptions on both the expressivity of the function class and the exploratoriness of the dataset.
For example, the analyses of Fitted Q-Iteration (Ernst et al., 2005; Antos et al., 2007) require the function class to be closed under Bellman updates (also known as Bellman-completeness), and the ofﬂine data distribution to provide coverage (in some technical sense) over all candidate policies (Chen and Jiang, 2019).
The former requirement is non-monotone in the function class, which shatters the standard machine-learning intuition that a richer function class should always have better (or at least no worse) approximation power; it is also closely related to the instability of RL training and the infamous “deadly triad” (Sutton and Barto, 2018;
Wang et al., 2021a).
The latter requirement is very likely violated in practice since we have no control over how the historical data is collected (Fujimoto et al., 2019).
Given these considerations, it is desirable to come up with novel algorithms and/or analyses to relax these assumptions.
In particular, the ideal assumption on the function class is realizability , that there is a target function of interest (such as the optimal value function) and we only require the function class to (approximately) capture such a function.
The ideal assumption on the data distribution is single-policy coverage, that it is ok for the data to not cover all policies, as long as an optimal (or sufﬁciently good) policy is covered.
In recent years, signiﬁcant progress has been made towards providing provable guarantees in ofﬂine RL under these relaxed assumptions.
In particular, the principle of pessimism in face of uncertainty proves to be useful in designing algorithms that work under single-policy coverage (Liu et al., 2020; Jin et al., 2021; Xie et al., 2021a;
Yin et al., 2021;
Rashidinejad et al., 2021), but most of the existing pessimistic algorithms require Bellman completeness on the function class.
On the other hand, relaxing Bellman completeness to realizability has been difﬁcult: there is merely one existing result that requires only the realizability of optimal value function (Xie and Jiang, 2021), yet their data assumption is even stronger than all-policy coverage.
In fact, a recent information-theoretic lower bound by Foster et al.
(2021) conﬁrms that even with a strong notion of all-policy coverage called (all-policy) concentrability, plus the realizability of value functions for all policies, ofﬂine RL is still fundamentally intractable.
Despite the lower bound, not all hope is lost.
A promising way of breaking the lower bound is to assume the realizability of other functions beyond value functions.
Indeed, positive results that are analogues to what we want are established in off-policy evaluation (OPE)—where the goal is to estimate the performance of a target policy from ofﬂine data—when additional realizability of density-ratio functions is assumed.
In particular, Liu et al.
; Uehara et al.
(2020) show that, as long as the data covers the target policy, and we are given function classes that can represent both the value function and the density-ratio function (or marginalized importance weights) of the target policy, it is possible to estimate the performance of the target policy in a sample-efﬁcient manner.
One way of using such results for policy learning is to use OPE as a subroutine and optimize a policy using OPE’s assessment of the policy’s performance.
Unfortunately, such a direct application introduces prohibitive expressivity assumptions we wanted to avoid beginning with, such as the realizability of value functions for all candidate policies (Jiang and Huang, 2020).
In this paper, we provide sample-efﬁciency guarantees for ofﬂine RL under the desired assumptions, that data is only guaranteed to cover the optimal policy, and the function classes only represent the optimal value function and density ratio, respectively.
Our algorithms have a simple procedure that combines marginalized importance sampling (MIS) with pessimism in a novel fashion.
The key enabler of our guarantees is an additional gap assumption, that there is a nontrivial gap between the values of the (unique) greedy action and the second-best for every state.
Similar gap assumptions are common in RL theory to characterize easy problems in which stronger-than-usual guarantees can be obtained.
They are often used to achieve logarithmic or constant regret in bandits and tabular online RL (Bubeck and
Cesa-Bianchi, 2012;
Ok et al., 2018; Slivkins et al., 2019;
Lattimore and Szepesvári, 2020; He et al., 2021; Papini et al., 2021), and similar guarantees in ofﬂine RL under Bellmancompleteness and additional structural assumptions on the value-function class (Hu et al., 2021).
They are also used in online RL with function approximation to block exponential error ampliﬁcation (Du et al., 2019).
To our knowledge, our work is the ﬁrst one to identify the utility of gap assumptions in ofﬂine RL with weak function approximation and offer interesting insights into novel aspects and mechanisms of gaps (see Section 5).
Paper Organization The rest of the paper is organized as follows: Section 2 introduces preliminary concepts and the problem setting.
Section 3 describes the algorithm.
Sectionprovides the core analysis.
We further extend our results to the setting where the function classes are misspeciﬁed (Section 5), and when the gap parameter is unknown but we have access to a small amount of online interactions (Section 6).
We conclude the paper with further discussions in Section 7, including a detailed comparison to the concurrent work of
Zhan et al.
on the same problem.
PRELIMINARIES
Markov Decision Processes (MDPs)
We consider ﬁnite horizon episodic MDPs deﬁned in the form of M =
( X , A , P, R, H, x 0 ) , where X = X 0 (cid:83) . . .
(cid:83) X H − 1 is the layered state space with X h denoting the state space at timestep h , A is the action space, P =
( P 0 , . .
, P H − 1 ) is the transition function with P h :
X h ×
A → ∆( X h +1 ) ,
R =
( R 0 , . . .
, R H − 1 ) is the reward function with R h :
X h
×
A → [0 , 1] , H is the length of horizon, and xis the ﬁxed initial distribution.
1 We assume the state and action spaces are ﬁnite but can be arbitrarily large, and ∆( · ) denotes the probability simplex over a ﬁnite set.
We deﬁne a policy π = { π 0 , . . .
, π H − 1 } , where for each h ∈ [ H ] , π h : X h → ∆( A ) is the policy at timestep h
and we use [ H ] to denote { 0 , . .
, H − 1 } .
With a slight abuse of notation, when π h ( · ) is a deterministic policy, we assume π h ( · ) :
X h → A .
Policy π induces a distribution over trajectories from the initial state distribution, which we denote as Pr π ( · ) and can be described as starting with x 0 and a h ∼ π ( ·| x h ) , r h
= R h
( x h , a h ) , x h +1 ∼
P h ( ·| x h , a h ) , ∀ h ∈
[ H ] .
As a convention, we will use x h , a h , r h to refer the state, action, and reward at timestep h (thus x h ∈ X h ).
The performance of a policy is measured by its expected return, deﬁned as v π :=
E π [ (cid:80) H − 1 h =
r h ] , where the expectation is taken with respect to Pr π ( · ) .
For any f h ∈ R X h ×A , we use π f h ( x h ) :
= argmax a h ∈A f h ( x h , a h ) to denote its greedy policy at timestep h .
Among all policies, there always exists a policy, denoted as π ∗ , that maximizes the return from all starting states simultaneously.
This policy is the greedy policy of the optimal action-value (or
Q-) function, Q ∗ =
( Q ∗ 0 , . . .
, Q ∗ H − 1 ) , i.e., π ∗ =
π Q ∗ :=
( π Q ∗ 0 , . . .
, π Q ∗ H − 1 ) .
Q ∗ is the unique solution to the Bellman optimality equations Q ∗ h
= T h Q ∗
h
+1 , where T h :
R X
h +
×A → R X h ×A is the Bellman optimality operator: ∀ f h +1 ∈ R X
[Equation]
+1 ×A , ( T h f h +1 )( x h , a h ) :
=
R h ( x h , a h ) + E x h +1 ∼ P h
·| x h ,a h ) [max a h +1 f h
+1 ( x h +1 , a h +1 )] .
We can similarly deﬁne policy-speciﬁc Q-functions Q π and their state-value function counterparts, namely V ∗ and V π .
Another useful concept is the notion of state-action occupancy of a policy π , d πh ( x (cid:48) h , a (cid:48) h ) :
=
Pr π ( x h = x
(cid:48) h , a h = a (cid:48) h ) .
As a shorthand, we deﬁne d ∗ h :=
d π ∗ h and use a i : j to refer actions a i , . .
. , a j .
Ofﬂine RL We consider a standard theoretical setup for ofﬂine RL, where we are given a dataset D =
D 0 (cid:83) . . .
(cid:83) D H − 1 with the form D h
= { x ( i ) h , a ( i ) h , r ( i ) h , x ( i ) h +1 } ni =
and D h consists of { x h , a h , r h , x h +1 } tuples sampled i.i.d.
from the following generative process:
[Equation]
Note that r h and x h +1 are generated according to the MDP reward and transition functions, and d Dh fully determines the quality and coverage of the data distribution.
For a given policy π , w πh ( x h , a h ) :
= d πh ( x h , a h ) /d
Dh ( x h , a h ) measures how well d Dh covers the occupancy induced by π at timestep h and is often known as the density-ratio function or the marginalized importance weight.
It plays an important role in ofﬂine RL algorithms and analyses.
As another shorthand, we use notation w π
=
( w π 0 , . .
w πh ) to denote the densityratio function over all timesteps and notation w ∗ :=
w π ∗ to denote the density ratio of the optimal policy.
Function Approximation We consider the function approximation setting, where we are given a function class
F = F 0 × . . .
×F H − 1 with F h ⊆ ( X h ×A → R ) , ∀ h ∈
[ H ] and a weight function class W = W 0 × . . .
× W H − 1 with
W h ⊆ ( X h ×
A → R ) ,
∀ h ∈
[ H ] .
We assume these are ﬁnite classes and use log( |F| ) and log( |W| ) to measure their statistical capacities.
The extension to continuous or inﬁnite classes with a covering argument is standard.
By default, for any f ∈ F , we assume f H = 0 for technical simplicity and use V f to denote its induces state-value function, i.e.,
V f ( x h ) = max a h ∈A f h ( x h , a h ) .
We will also use π f ( x h ) instead of π f h ( x h ) for simplicity since only f h operates on x h ∈ X h and there is no confusion.
ALGORITHM
In this section, we introduce our algorithm PABC (Pessimism under Average Bellman error Constraints), whose pseudo-code is given in Algorithm 1.
The algorithm takes two steps: a prescreening step (line 1), followed by the main step (line 2).
We ﬁrst give an intuition for the main step, deferring the explanation of the prescreening step and the related gap deﬁnitions to the later part of this section.
The main step (line 2) runs a constrained optimization to select a function ˆ f ∈ F , whose greedy policy is the output.
The objective of the optimization minimizes the value at the initial state, which is a form of initial-state pessimism (Xie et al., 2021a; Zanette et al., 2021) and proved to be useful in handling insufﬁcient data coverage.
The constraints eliminate functions with large average Bellman errors (Jiang et al., 2017; Xie and Jiang, 2020).
Average Bellman Error Constraints To provide intuition, we know that Q ∗ has 0 average Bellman errors for all state-action pairs, that is, ∀ h ∈
[ H ] , x h ∈ X h , a h ∈ A , ( Q ∗ h − T h Q ∗ h +1 )
( x h , a h ) = 0 .
Thus it also has 0 average to average Bellman error constraints
[Equation]
where the empirical loss L D ( f, w, h ) is deﬁned as
[Equation]
Output: policy π ˆ f and return estimation ˆ f 0 ( x 0 , π ˆ f ( x
[Equation]
Bellman errors under any distribution ν h at timestep h :
[Equation]
This holds even if ν h is an unnormalized distribution.
Therefore, we can safely eliminate any candidate function f ∈ F , if it has a large average Bellman error E
ν h
[ f h − T h f h
+1 ] under any (possibly unnormalized) distribution ν h .
Unlike the more standard versions of Bellman errors such as E ν h
[( f h − T h f h +1 ) 2 ] , which squares the Bellman error in each state before taking expectation and cannot be directly estimated due to the infamous double-sampling difﬁculty (Baird, 1995; Farahmand and Szepesvári, 2011), the average
Bellman error can be easily estimated.
In the algorithm, we consider a variety of (possibly unnormalized) distributions ν h
= w h · d Dh for w ∈ W , h ∈
[ H ] .
Since the average
Bellman error can only be empirically approximated (see
Eq.
, we relax the constraints and allow a threshold α to incorporate the statistical errors.
We note that the constraint alone is similar to the MABO algorithm by Xie and Jiang (2020), but they do not use pessimism and cannot handle insufﬁcient data coverage.
They also assume that W is sufﬁciently rich to approximate w π f for all f ∈ F , and a main goal of our work is to avoid such “for all” assumptions.
Gap and Prescreening As mentioned in the introduction, a key assumption that enables our results is a gap assumption on value functions.
To prepare for the discussion, we deﬁne the gap of a function as follows:
Deﬁnition 1 (Gap) .
For any f = ( f 0 , . . .
, f H − 1 ) , we deﬁne its gap at timestep h ∈
[ H ] and state x h ∈ X h as follows:
If argmax a h ∈A f h ( x h , a h ) is unique, then we deﬁne gap( f ; h, x h ) :
= min a h (cid:54) =
π f ( x h )
f h ( x h , π f ( x h ))
− f ( x h , a h ) .
Otherwise, we deﬁne gap( f ; h, x h ) :
= 0 .
The gap of f is then deﬁned as
[Equation]
As we see, this deﬁnition of the gap is similar to the one used in prior works (Simchowitz and Jamieson, 2019;
Mou et al., 2020; Du et al., 2019; He et al., 2021; Yang et al., 2021;
Hu et al., 2021; Wang et al., 2021b; Papini et al., 2021;
Wu et al., 2021), except that we require a unique optimal action for the gap to be non-zero.
A motivating example of similar gap assumptions in other areas of RL theory can be found in Wu et al.
With Deﬁnition 1, we can now deﬁne the minimum gap of a function class:
Deﬁnition 2 (Gap of a function class) .
Given a function class G = G 0 × . . .
× G H − 1 , where G h ⊆ ( X h ×
A → R ) ,
∀ h ∈
[ H ] , we deﬁne its gap as
[Equation]
g ) .
Prior theoretical results relying on similar gap assumptions often make such assumptions on the true optimal value function Q ∗ (Simchowitz and Jamieson, 2019;
Yang et al., 2021).
As we will see in our analyses, however, what is really important for us is that the learned function ˆ f has a large gap, not the true Q ∗ .
Since we have no control over which f in the function class will be ﬁnally chosen, we perform the prescreening step in line 1 to eliminate functions with the gap lower than a pre-deﬁned threshold C gap ≥ 0 .
It is immediate to see that gap( F ( C gap )) ≥ C gap .
Of course, this runs into the risk of eliminating Q ∗ , and if we do not want any misspeciﬁcation, we need to ensure Q ∗ ∈ F ( C gap ) , which requires that C gap ≤ gap( Q ∗ ) .
For clarity, in Section 4.
we will assume that we have the knowledge of gap( Q ∗ ) and can set C gap accordingly, while later in Section 6 we show how to handle unknown gap( Q ∗ ) .
Moreover, as we will see in Section 5, when we allow misspeciﬁcation errors in the analysis, gap( Q ∗ ) and C gap become disentangled, which leads to some interesting implications.
MAIN GUARANTEES
In this section, we present the main sample complexity results of our algorithms.
We start with a weak version of guarantee by showing that our algorithm can identify v ∗ , the optimal expected return at the initial state, with polynomial samples under realizability and single-policy coverage assumptions, even without any gap assumption (Section 4.1).
Such a result will also be useful when we handle the unknown gap setting later in Section 6.
Then,
Section 4.2 provides an algorithm-speciﬁc counterexample to show that our algorithm fails to ﬁnd a near-optimal policy under these assumptions, motivating the necessity of the gap assumption.
Finally, Section 4.3 provides the main result of this paper under the additional gap assumption.
4.1 ESTIMATING THE OPTIMAL EXPECTED RETURN
We ﬁrst show how to identify v ∗ , the optimal expected return of the problem, without needing the gap assumption.
In this case, we will run Algorithm 1 with C gap = 0 , that is, without the prescreening step.
To our knowledge, there is no prior work that can achieve this goal under our weak assumptions.
Despite not producing a near-optimal policy, this procedure and guarantee allows us to check whether any given policy is close to optimal, assuming we can evaluate the policy’s performance by off-policy evaluation or a small amount of online interactions.
This capability can be very useful especially in certain model selection scenarios (see e.g., Modi et al., 2020, Section 5).
Indeed, we will reuse this result later in Section 6 to handle the unknown gap setting.
We start by introducing the assumptions.
The ﬁrst two are the standard realizability assumptions.
Assumption 1 (Realizability of F ) .
We assume Q ∗ =
( Q ∗ 0 , . . .
, Q ∗ H − 1 ) ∈ F .
Assumption 2 (Realizability of W ) .
We assume w ∗ =
( w ∗ 0 , . . .
, w ∗ H − 1 ) ∈ W .
We make these assumptions exact for now to allow for a clean presentation of the main results and core proof ideas, and defer the handling of misspeciﬁcation errors to Section 5.
Also, following the arguments in Uehara et al.
Xie and Jiang (2020), Assumption 2 can be further relaxed such that w ∗ only needs to lie in the convex hull of W .
Next, we introduce the standard boundedness assumptions.
Assumption 3 (Boundness of F ) .
For any f ∈ F , we assume f h ∈
( X h
×
A → [0 , H − h ]) , ∀ h ∈
[ H ] .
Assumption 4 (Boundness of W ) .
For any w ∈ W , we assume (cid:107) w h (cid:107) ∞ ≤ C, ∀ h ∈
[ H ] .
Assumption 2 and Assumption 4 together immediately imply that our data covers π ∗ :
[Equation]
This version of coverage is often called π ∗ -concentrability
(Scherrer, 2014; Xie et al., 2021b; Rashidinejad et al., 2021; xx A x B x C x D + π ∗ , Lπ f , Rπ ∗ , π f , L
R
Zhan et al., 2022).
As we will see when we consider misspeciﬁcation errors in Section 5, we do not really need our data to satisfy π ∗ -concentrability, and the deﬁnition of coverage can be relaxed using the structure and generalization effects of F similarly to Jin et al.
; Xie et al. (2021b).
With all the above assumptions, we are ready to state our ﬁrst result formally below.
The proof is deferred to Appendix A.2.
Theorem 1 (Sample complexity of estimating v ∗ ) .
Suppose Assumptions 1, 2, 3, 4 hold and the total samples nH satisﬁes
[Equation]
Then with probability at least ≥ 1 − δ , running Algorithmwith C gap = 0 and α = ε/ (2 H ) guarantees
[Equation]
4.2 ALGORITHM-SPECIFIC COUNTEREXAMPLE
Despite being able to identify v ∗ , we show that Algorithmcannot be guaranteed to learn a near-optimal policy without further assumptions, even with inﬁnite data.
As we will see, a key aspect of the construction is a tie between the values of different actions, so such counterexamples can be effectively excluded by assuming a unique optimal action.
The counterexample is given in Figure 1.
Circles denote states and arrows denote actions with deterministic transitions, and states without arrows have a default null action.
There is only a +1 reward at state x C , while the rewards are 0 everywhere else.
Taking L 1 at x 0 deterministically transits to x A and we omit the remaining speciﬁcations as they are clearly indicated in the ﬁgure.
It is easy to see that the optimal policy π ∗ takes action
L 1 at state x 0 and action L 2 at state x A .
We construct a bad function f , which only differs from Q ∗ at ( x 0 , R 1 ) and ( x B , null) by setting f 0 ( x 0 , R 1 )
= 1 and f 1 ( x B , null) = 1 .
By adversarial tie breaking, we assume π f ( x 0 )
=
R 1 .
We immediately have a realizable class F = { Q ∗ , f } .
It is easy to see that π f is not ε -optimal for any ε < 1 because it deviates from the optimal branch at x 0 .
In addition, we let data d D covers ( x 0 , L 1 ) , ( x A , L 2 ) , ( x C , null) .
For the weight function, we deﬁne an invalid weight function w bad that puts all weight on ( x 0 , R 1 ) , ( x A , L 2 ) , ( x C , null) in each level respectively.
Then we also have a realizable class W = { w ∗ , w bad } .
Since both f and Q ∗ have zero population average Bellman error under W , and f 0 ( x 0 , π f ( x 0 ))
=
Q ∗ 0 ( x 0 , π Q ∗ ( x 0 ))
= 1 , either of them can be the ˆ f learned in Algorithm 1, but returning π f leads to failure of learning.
We note that the reason for failure is that no data covers the state π f visits.
Additional Consistency Constraints
In our setting, there are additional constraints that one can add to ensure some form of consistency.
For example, for any f ∈ F , we can additionally require that there exists w ∈ W that is consistent with π f , since w ∗ ∈ W should only give non-zero weight to actions chosen by π f (i.e., ∀ h ∈
[ H ] , w h ( x h , a h ) =
if a h (cid:54) =
π f ( x h ) ).
In addition, as we can estimate v ∗ with the assumptions in Theorem 1 and we know E d D [ w ∗ · R ] = v ∗ , we can eliminate any w ∈ W that violates this condition.
While these constraints are reasonable (or at least harmless) and may be of independent interest, we can verify that they do not help with this counterexample, which implies our algorithm fails even under these additional consistency checks.
4.3 LEARNING A NEAR-OPTIMAL POLICY
As mentioned above, a key aspect of the counterexample is a tie between the values of actions.
In this section, we show that a positive gap assumption not only excludes the counterexample, but enables a general guarantee for learning near-optimal policies with our Algorithm 1.
Assumption 5 (Gap of Q ∗ ) .
The gap of Q ∗ satisﬁes gap( Q ∗ ) > 0 .
Here the implicit assumption is that we want gap( Q ∗ ) to be sufﬁciently large, as our later sample complexity guarantees will scale inversely with gap( Q ∗ ) .
Note that Assumptionis stronger than the standard gap assumption in the literature (Simchowitz and Jamieson, 2019; Yang et al., 2021;
Hu et al., 2021).
Compared with their deﬁnition, we additionally assume the optimal action is unique at each state.
On the other hand, these papers require additional strong assumptions (e.g., linear MDPs, Bellman-completeness, or pointwise convergence) or focus on the tabular setting, whereas we handle general function approximation in ofﬂine RL under weak realizability-type assumptions.
Plus, the technical mechanisms under which the gap plays a role in the analyses are very different, so the assumptions are not really comparable.
For now, we assume gap( Q ∗ ) is known and will later handle the case of an unknown gap Section 6.
Plus, as a side effect of handling misspeciﬁcation errors, Section 5 will lift the stringent gap assumption in a novel and interesting manner.
We now state the guarantee of learning a near-optimal policy under the gap assumption.
A sketch of proof is provided after the theorem statement, while the complete proof is deferred to Appendix A.3.
Theorem 2 (Sample complexity of learning a near-optimal policy) .
Suppose Assumptions 1, 2, 3, 4, 5 hold and the total number of samples nH satisﬁes
[Equation]
Then with probability at least ≥ 1 − δ , running Algorithmwith α = ε gap( Q ∗ ) / (2 H 2 ) and C gap =
gap( Q ∗ ) guarantees
[Equation]
Proof sketch of Theorem 2 As standard, all our results depend on a high-probability concentration event, that |L D ( f, w, h ) − E [ L D ( f, w, h )] | ≤ ε stat ,n holds for all f ∈ F , w ∈ W ,
h ∈
[ H ] with high probability; the detailed expression of ε stat ,n is given in Lemma 1.
From Lemma 1, for any f ∈ F that satisﬁes all constraints in Algorithm 1, we can guarantee the population loss to be small, that is, | E
[ L D ( f, w, h )] | ≤ ε stat ,n + α.
The central step of our proof is to use a telescoping argument and the gap assumption to establish the following inequality:
[Equation]
This implies the policy deviation can be bounded as
[Equation]
The LHS of this inequality is the probability that the learned policy π ˆ f disagrees with the optimal policy π ∗ , along the distribution induced by π ∗ .
From here, we can apply the RLto-supervised-learning (SL) reduction in imitation learning (e.g., Theorem 2.1 in Ross and Bagnell (2010)) to translate it to the ﬁnal performance difference bound between π ˆ f and π ∗ .
We also provide a different proof in Appendix A.3, which itself may be of independent interest.
ROBUSTNESS TO MISSPECIFICATION
We now consider the case when Q ∗ and w ∗ may not exactly belong to F and W , but can be reasonably approximated up to small errors.
More often than not, such robustness results in RL theory are nothing but routine exercises where the proofs are largely straightforward extensions of those for the exact case.
In our case, however, the misspeciﬁcation analyses reveal an interesting phenomenon of disentangling the true gap of Q ∗ and that of F , and how our gap and coverage assumptions can be relaxed in nontrivial ways.
We start with deﬁning the approximation errors of our function classes.
Inspired by Xie and Jiang (2020), we deﬁne the approximation error of W as
[Equation]
and use ˜ w ∗ to denote the best approximator in W that obtains the minimum.
The expression inside the min-maxmax measures the difference between d Dh · w h and d ∗ h , using f h − T h f h +1 for f ∈ F as discriminators.
When w ∗ ∈ W (Assumption 2), d Dh · w ∗ h =
d ∗ h (because w ∗ h is deﬁned as d ∗ h
/d Dh ), so we have ε W = 0 .
However, the opposite direction is not always true: it is entirely possible to achieve ε W = 0 when w ∗ / ∈ W (or even when d ∗ h ( x h , a h ) /d
Dh ( x h , a h ) =
∞ for some ( x h , a h ) and w ∗ does not exist), as long as d Dh · ˜ w ∗ h and d ∗ h can not be distinguished by f h −
T h f h +1 for f ∈ F as discriminators .
We also provide an example in Appendix E. Note that since our data coverage assumption is implicitly made through the realizability and boundedness of W (see the discussion below Assumption 4), this means that our data coverage assumption is also relaxed using the information of F , which is a common characteristics of recent results in ofﬂine RL (e.g., Xie et al.
(2021a) also use the Bellman error class induced by the value function class as discriminators, which is similar to our deﬁnition at a high level), but not enjoyed by the concurrent work of Zhan et al.
For function class F , we deﬁne the approximation error in a way that uses W as discriminators, plus a term that measures the difference under the initial state x 0 :
[Equation]
[Equation] and use ˜ Q ∗F to denote the best approximator that achieves the minimum value.
Under mild regularity assumptions on
W , 4 it is straightforward to show that ε F is weaker than (cid:96) ∞[-
 The idea of using-] discriminators has also been explored in Farahmand et al.
(2017); Sun et al.
; Modi et al.
but the application is different here.
Namely, w ≥ 0 and E d D [ w ] = 1 .
The former is trivial and the latter can be easily veriﬁed approximately on data.
error up to multiplicative constants:
[Equation]
and a more detailed discussion can be found in Lemma 4.
Similarly, we can deﬁne the function class F ( C gap ) related approximation error ε F ( C gap ) and its best approximator ˜ Q ∗F ( C gap ) by replacing F with F ( C gap ) in Eq.
5.1 ESTIMATING THE OPTIMAL EXPECTED
RETURN
With the approximation error deﬁned above, we now extend Theorem 1 to the approximate case.
Assuming the approximation error ε F (or a reasonably tight upper bound of it) is known, we can relax the constraint to ensure that ˜
Q ∗ F ( C gap ) is not eliminated and obtain the sample complexity guarantee as in Theorem 3.
The full proof is deferred to
Appendix B.2.
As before, we do not need the gap assumption to identify v ∗ approximately and can run the algorithm with C gap = 0 .
Theorem 3 (Robust version of Theorem 1) .
Suppose Assumptions 3, 4 hold and the total number of samples nH satisﬁes
[Equation]
Then with probability 1 − δ , running Algorithm 1 with α = ε/ (2 H ) + ε F and C gap = 0 guarantees
[Equation]
While we need the knowledge of ε F to set α , we do not need to know ε W , which shows a difference between the behaviors of F and W .
This is also the case in the next subsection where we try to learn a near-optimal policy.
5.2 LEARNING A NEAR-OPTIMAL POLICY
Similarly, we can also extend Theorem 2 to the misspeciﬁed case.
Our guarantee is established with a user-speciﬁed
C gap parameter and the approximation error related to prescreened class F ( C gap ) .
We provide the sample complexity guarantee in Theorem 4 and the complete proof in Appendix B.3.
Theorem 4 (Robust version of Theorem 2) .
Suppose Assumptions 3, 4 hold and the total number of samples nH satisﬁes
[Equation]
Then with probability 1 − δ , running Algorithm 1 with a user-speciﬁed C gap and α = εC gap / (2 H 2 )+ ε F ( C gap ) guarantees
[Equation]
Theorem 4 gives us a convenient way to set the gap parameter C gap .
We also provide a sample complexity guarantee (Corollary 5) in Section B.4 for the case that gap( Q ∗ ) and the (cid:96) ∞ approximation error of F are known.
Unknown Approximation Errors Notice that in the robustness results (Theorem 3 and Theorem 4) we require the knowledge of approximation errors ε F or ε F ( C gap ) to set the threshold α in PABC (Algorithm 1).
In Appendix D we show a variant of PABC based on Lagrangians (PABC-L;
Algorithm 1) does not require such knowledge, and still enjoys the same sample complexity guarantees.
In PABC-L, the original constraints in Eq.
(2) are moved to the objective, thus the threshold α is no longer needed as the input.
We refer the reader to Appendix D for the formal description of
PABC-L and its results and proofs.
Relaxed Gap Assumption An outstanding characteristic of Theorem 4 is that it no longer depends on gap( Q ∗ ) explicitly, and only depends on C gap , a parameter of our choice.
Therefore, it may seem to have lifted Assumptionthat gap( Q ∗ ) > 0 , as we can choose C gap to be sufﬁciently large.
However, below we show that this issue is more complicated than it may seem, and while our result does relax
Assumption 5 in signiﬁcant ways, it does so in a very nuanced manner.
First of all, in the worst-case scenario, Assumption 5 is still needed to provide non-vacuous guarantees.
This is because, if Q ∗ has no gap, yet we artiﬁcially create a large C gap in our prescreened function class F ( C gap ) , we could eliminate all the good approximations of Q ∗ .
Among the remaining functions, the best (cid:96) ∞ approximation of Q ∗ must have an (cid:96) ∞ error no less than C gap , and if we plug that into the ε F ( C gap ) term in the approximation guarantee, the C gap on the numerator and the denominator will cancel out, leaving a constant suboptimality gap which makes the guarantee vacuous.
Having said that, the nuance here is that we do not use the most stringent (cid:96) ∞ norm to deﬁne ε F ( C gap ) , but rather use an average notion of error (Eq.
(5)), which is possibly much smaller than the (cid:96)
∞ error (Eq.
Therefore, there are still cases where gap( Q ∗ ) =
yet our result yields nontrivial guarantees.
As a concrete example, imagine a
Q ∗ that has large gaps in most states, but the gap is 0 in a few “bad” states.
In this case, gap( Q ∗ ) is 0 .
However, there can still exist ˜ Q ∗F ( C gap ) that approximates Q ∗ well everywhere except on those bad states, and as long as no w ∈ W puts signiﬁcant probabilities on the bad states, we have ε F ( C gap ) (cid:28) C gap and hence Theorem 4 will provide meaningful guarantees.
HANDLING THE UNKNOWN GAP PARAMETER WITH ONLINE ACCESS
In this section, we extend the main algorithm and analyses in Section 4 in a different direction than Section 5.
In particular, we are concerned about the fact that Theoremassumes the knowledge of gap( Q ∗ ) .
While it is common for ofﬂine RL algorithms to have hyperparameters that need to be tuned separately (and this is particularly the case for version-space-based algorithms (Jiang et al., 2017; Xie et al., 2021a)), here we show that we can address the unknown gap( Q ∗ ) issue by a small amount of additional online interactions for Monte-Carlo policy evaluation.
This is particularly interesting as our result provides an example of how one can use a small amount of online interactions to mitigate limitations in purely ofﬂine learning, a practically relevant problem that is also of great interest to the RL theory community (Xie et al., 2021b).
Algorithm
1: Input class weight function class and dataset D do
Use n and gap t to calculate ε t = (cid:112) 8 C 2 H 6 ι ( t ) /
( n (gap guess t ) 2 ) , where ι ( t ) =
log(24 |F||W| H · 2 t /δ ) .
Run Algorithm 1 with α = ε t / (2 H ) and get scalar estimation ˆ v ∗ t .
Run Algorithm 1 with α = ε t gap guess t / (2 H 2 ) and
C gap =
gap guess t , and get policy ˆ π t .
Estimate v ˆ π t by running Monte Carlo algorithm with ˜
O ( H 3 log(1 /δ )
/ε
2 t ) online samples and denote the estimate as ˆ v ˆ π t .
8: if ˆ v ˆ π t ≥ ˆ v ∗ t − 3 ε t then 9:
Output ˆ π t and terminate.
10: end if 11: end for
As shown in Algorithm 2, the algorithm PABC-OA (PABC with Online Access) proceeds iteration by iteration.
We start with the maximum possible value of the unknown gap( Q ∗ ) .
For simplicity, we choose H here, and alternatively we can also use max f ∈F gap( f ) which is tighter.
In iteration t , we use gap guess t =
H/ 2 t as the guess of gap( Q ∗ ) and calculate the desired α according to Theorem 1 to estimate v ∗ (line 5), or calculate the desired α and C gap according to
Theorem 2 to ﬁnd a near-optimal policy (line 6).
Finally we conduct Monte-Carlo policy evaluation with online samples (line 7).
If the stopping condition (line 8) is satisﬁed, we are guaranteed to learn a near-optimal policy and can terminate (line 9).
Otherwise, we proceed to the next iteration, shrink our guessed value of gap guess t , and continue the routine.
We can observe an interesting connection between Theoremand Theorem 2, and identifying v ∗ is indeed useful.
It can be shown that Algorithm 2 will terminate once the guessed value gap guess t =
H/ 2 t drops below the true value gap( Q ∗ ) , which leads to the sample complexity result in
Theorem 5.
The formal proof can be found in Appendix C.
Theorem 5 (Sample complexity of learning a near-optimal policy with unknown gap( Q ∗ ) ) .
Suppose Assumptions 1, 2, 3, 4, 5 hold but gap( Q ∗ ) is unknown.
Assume we have a dataset D with size n for each D h and additional online access to collect
[Equation]
samples.
Then with probability at least 1 − δ , the output policy ˆ π from Algorithm 2 satisﬁes
[Equation]
7) where ι ( t ) = log(24 |F||W| H · 2 t /δ ) .
The suboptimality in Eq.
(7) has the same order (up to polylog terms) as that of running Algorithm 1 with known gap( Q ∗ ) in Theorem 2.
If we set this value to be ε (cid:48) , i.e.,
[Equation]
Q
[Equation]
), then the number of required online samples is ˜ O (cid:16)
[Equation]
5 log
[Equation]
1 /δ
[Equation]
( ε
[Equation]
gap( Q ∗ )) 2 (cid:17) , which does not depend on the complexity of the function classes F and W .
DISCUSSION AND CONCLUSION
We conclude the paper with a detailed discussion of how our work compares to the closely related concurrent work of Zhan et al.
(2022), which also provides a good summary of our contributions and promising future directions.
The very recent work of Zhan et al.
(2022) aims at solving the same problem: 5 ofﬂine RL under only single-policy coverage and realizability assumptions.
Similar to our counterexample in Section 4.2, they also realize the difﬁculties in the setting where the optimal weight and value functions are realizable in a straightforward manner.
Instead of making a gap assumption like we do, they attack the problem from a different angle by introducing regularization into the
Lagrangian of the linear program for MDPs.
Despite that the two approaches have some fundamental differences (which we will elaborate further below), it is still worth comparing the nature of the two results.
To this end, our approach has several advantages:
Regularization changes the deﬁnition of the value function in Zhan et al.
In fact, the function they need to realize does not obey any form of Bellman equations, and probably should not be called value functions anymore.
This makes their realizability
assumption somewhat difﬁcult to interpret and connect to the existing literature.
In contrast, we work with the most
standard notion
of Q ∗ .
Due to regularization, the policy learned by Zhan
et al
(2022) is generally suboptimal even with inﬁnite data, so the strength of regularization needs to be carefully controlled for the bias-variance trade-off.
As a result, when competing with π ∗ , their sample complexity rate is O (1 /ε 6 ) , which is much slower than our O (1 /ε 2 ) .
Our coverage assumption can be signiﬁcantly relaxed using the structure of F ; see discussion in Section 5.
While this is standard in recent ofﬂine RL works based on Bellman-completeness assumptions (Jin et al., 2021;
Xie et al., 2021a), Zhan et al.
(2022)’s guarantee relies on the boundedness of the raw density ratios and does not enjoy such a relaxation.
That said, Zhan et al.
(2022)’s result is also attractive in several aspects:
They do not require gap assumptions.
While similar gap assumptions are standard in RL theory literature, it is unclear how prevalent it is in real problems and how algorithms that depend on gap assumptions perform in problems when the assumption is violated.
Our guarantees only hold if the data covers π ∗ (though the notion of coverage can be relaxed using a structure of F , as mentioned above).
In comparison, Zhan et al.
(2022) can still provide meaningful guarantees even when π ∗ is not covered by data, in which case they compete with the best policy under data coverage.
Regarding computation, their algorithm is a convexconcave minimax optimization problem when the function
classes are convex.
In comparison, the computational characteristics of our method are less clear, though we note that a Lagrangian form of our main step (line 2) (see
Appendix D for details) is similar to the kind of minimax optimization commonly found in the MIS literature (Nachum et al., 2019;
Uehara et al., 2020; Yang et al., 2020; Jiang and Huang, 2020).
We reiterate that these comparisons are made only on the results themselves.
The two works take fundamentally different approaches and are of independent interests.
For example, despite that both works use density-ratio functions, Zhan et al.
(2022)’s method is based on the linear programming (LP)-formulation of MDPs where the optimal state-value function V ∗ is modeled, whereas we model the optimal Q-function Q ∗ .
This difference is more signiﬁcant than it may seem, as the LP formulation and the Bellman optimality equations for Q ∗ are very different foundations for designing learning algorithms, and the gap assumption only makes sense for Q-functions and cannot be used in state-value functions.
That said, it will be interesting to investigate if the two works can borrow each other’s ideas to address their own weaknesses, which we leave to future investigation.
Acknowledgements
NJ acknowledges funding support from ARL Cooperative
Agreement W911NF-17-2-0196, NSF IIS-2112471, NSF
CAREER award, and Adobe Data Science Research Award.