Oracle Inequalities for Model Selection in Ofﬂine Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Ofﬂine reinforcement learning (RL) is a promising paradigm where a learnerleverages prior data to learn a good policy without interacting with the environment.
A major challenge in applying such methods in practice is the lack of boththeoretically principled and practical tools for model selection and evaluation.
Toaddress this, we study the problem of model selection in ofﬂine RL with valuefunction approximation where the learner is given a nested sequence of model classesto minimize squared Bellman error and must select among these to achieve theoptimal balance of approximation and estimation error of the classes.
We propose,to our knowledge, the ﬁrst model selection algorithm for ofﬂine RL that achievesminimax rate-optimal oracle inequalities up to logarithmic factors.
The algorithm,
M OD BE, takes as input the model classes and a base ofﬂine RL algorithm designedto minimize squared Bellman error.
It successively eliminates model classes usinga novel one-sided generalization test, ﬁnally returning a policy that competes withthe performance of the best model class.
In addition to its theoretical guarantees,it is conceptually simple and computationally efﬁcient, amounting to calculatingand comparing relative squared errors between classes.
Finally, we demonstrateit is capable of reliably selecting a good model class in small simulated experiments.
Introduction
Model selection is a fundamental task in supervised learning and statistical learning theory.
Given asequence of model class, the goal is to optimally balance the approximation error (bias) and estimationerror (variance) offered by the potential model class choices.
Model selection algorithms are extremelywell-studied in learning theory [Mas07, LN99, BBL02, Bar08] and methods like cross-validation areessential steps for practitioners.
In recent years, interest has turned to model selection in online banditsand reinforcement learning [ALNS17, FKL19, PPAY + 20, LPM + 21, MJTS20, CMB20, MK21].
Despite this, the current understanding of model selection in the context of ofﬂine (or batch) reinforcement learning (RL) is comparatively nascent.
The ofﬂine RL setting describes the paradigmwhere the learner leverages prior datasets of interactions with the environment [LGR12, LKTF20].
The learner is tasked with returning a good policy without further environment interaction.
As has been acknowledged in several recent papers [XJ21, MXW + 21, KST + 21], one of the major challenges preventing widespread deployment of ofﬂine RL algorithms in the real world is the lack of algorithmic tools for model selection, evaluation, and hyperparameter tuning.
In experimental settings, researchers typically evaluate candidate learned models by using online rollouts of the policies after learning withofﬂine data.
Such approaches are not feasible in many real world settings, where due to logistics, safety or performance requirements, the entire process of ofﬂine RL through to producing a single policy,must be conducted only on the ofﬂine dataset
In recent years, this problem has been recognized as a major deﬁciency in the ﬁeld and a numberof efforts have been made to remedy it.
On the empirical side, several researchers have proposedworkﬂows and general heuristics speciﬁcally addressing this problem [KST + 21, TW21, PPM + 20].
However, all have noted that solutions designed to evaluate or select models typically have theirown hyperparameters and modeling choices.
Thus, applying off-the-shelf methods, such as thosefrom the ofﬂine policy evaluation (OPE) literature [Pre00, TB16], does not solve the model selectionproblem.
Similarly, recent efforts to solve model selection in online bandits and RL are inapplicableas they almost universally require interaction with the environment [FKL19, PPAY + 20, LPM + 21].
The solution to the ofﬂine problem seems to require new ideas.
On the theoretical side, [LTND21] formalized the problem of model selection in the ofﬂine setting withthe intent of addressing the above recursive issue.
It was shown that full model selection (competitivewith an oracle that has knowledge of the best model class) is surprisingly impossible in general inofﬂine reinforcement learning.
However, they proposed several relaxations to accomplish partialmodel selection.
Namely, if one is only concerned with optimally balancing approximation error withestimation error (which is our focus), model selection is indeed possible, but they were only able toshow this for contextual bandits (single-step horizon).
1.1 Contributions
Theoretical Guarantees In this paper, we build on the theoretical foundations of [LTND21] and give,to our knowledge, the ﬁrst rate-optimal model selection algorithm for ofﬂine reinforcement learning.
We consider the model selection problem where we are given an ofﬂine dataset of n samples and anested sequence of M model classes F 1 ✓ ... ✓ F M .
For any individual class F k , the gold-standardregret bound is ˜
O
[Equation] [Equation]
] where A PPROX ( F k ) denotes the approximation error (i.e., completeness error) of F k , C OMP ( F k ) denotes the statistical complexity of F k ,and n is the number of ofﬂine samples.
This is achieved, for example, by Fitted Q-Iteration (FQI)
[CJ19].
We propose a novel and conceptually simple algorithm, MODel selection via Bellman Error (M OD BE), for model selection and prove that it achieves the following oracle inequalitiessimultaneously: (1) If k ⇤ is the index of the class of minimal complexity satisfying completeness (thatis, A PPROX ( F k ⇤ )
=0 ), it achieves regret ˜
O
[Equation]
; (2) If no such class exists, it achieves
[Equation]
O
[Equation]
, where ⇠ k  A PPROX ( F k ) is a global completenesserror deﬁned in Section 3.1.
Crucially, both guarantees are rate-optimal in C OMP ( · ) and n , and thisis achieved without prior knowledge of the optimal model class, the approximation errors, or onlineinteraction with the environment.
Technical Highlights
The key to achieving the near optimal regret rate is to achieve the nearoptimal excess risk rate of the squared Bellman error (which is on the order of ˜ O
(1 /n ) ).
To do this,
M OD BE iteratively compares the relative effectiveness of a two candidate model class by employinga hypothesis test that compares the difference of their estimated risks to a one-sided generalizationbound.
The fact that the test leverages only the one-sided generalization bound is crucial: deferringto the easier case of two-sided bounds (e.g. from uniform deviation bounds on risk estimators) leadsto a squared Bellman error rate of ˜ O
(1 / p n ) , which translates to a slow ˜ O (1 /n 1 / 4 ) regret rate.
Weuse the one-sided generalization error to create a conceptually simple hypothesis test that enablesa risk rate of ˜ O
(1 /n ) , which translates to the optimal ˜ O
(1 / p n ) regret rate.
Practical Results In practice, M OD BE can be instantiated with any base ofﬂine RL algorithm thatattempts to minimize squared Bellman error, including but not limited to FQI.
M OD BE is also computationallyefﬁcient, requiring O ( Hk ⇤ M ) callstoanempiricalsquaredlossminimizationoracleand O ( k ⇤ )
79 calls to the instantiated ofﬂine RL algorithm.
In Section 5, we demonstrate the effectiveness of M OD BEon several simulated experimental domains.
We use neural network-based ofﬂine RL algorithms asbaselines and show that M OD BE is able to reliably select the good model class compared to baselines.
1.2 Additional Related Work
Several prior works have speciﬁcally set out to address the model selection problem from a theoreticalperspective, as we do here.
[LTND21] formalized the end-to-end model selection problem for ofﬂine
RL where, given nested model classes, the goal is to produce a regret bound competitive with an oraclethat has knowledge of the optimal model class.
While the focus was a negative result, their positiveresults were limited only to linear model classes for contextual bandits.
Model selection guaranteesof this type for reinforcement learning with general model classes has since remained an open problem.
An earlier work by [FS11] had partially addressed this problem but made several restrictive assumptionsthat are incompatible with that of [LTND21], such as a known generalization bound that underestimatesthe approximation error (which is generally unknown).
Another notable work is the BVFT algorithm of[XJ21].
While initially designed for general policy optimization, BVFT can be applied to model selection [ZJ21] but it incurs a slow 1 /n 1 / 4 regret rate in theory and and requires a stronger data coverage assumption.
One advantage of BVFT over our algorithm is that it can be used more generally to tune hyperparametersbeyondtheselectionofmodelclasses.
However, thespecializationofouralgorithmtomodelselection enables the stronger guarantees.
As a result, we view the two algorithms as complementary.
On the empirical side, several authors have also proposed general workﬂows and best practices to makemodel selection and hyperparameter tuning useful in practice [KST + 21, TW21, PPM + 20], but they didnot address the theoretical aspects of the problem, which still present a signiﬁcant technical challenge.
Preliminaries
Notation For any n 2 N , we let [ n ] = { 1 ,...,n } .
The notation a .
b implies that a  Cb for someabsolute constant C > 0 .
We will use C,C 1 ,C 2 ... > 0 to denote absolute constants (independent ofproblem parameters).
For a set A ,  ( A ) denotes the set of distributions over A .
We consider the ﬁnite-horizon Markov decision process M ( X , A ,H, P ,r, ⇢ ) where X is the (potentiallyinﬁnite) state-space, A is the action space, H is the length of the horizon, P : X ⇥ A !
( X ) is the 106 transition kernel, r :
X ⇥ A !
[0 , 1] is a deterministic reward function, and ⇢ 2  ( X ) is an initial state 107 distribution.
A learner interacts with the MDP by proposing an H -step policy ⇡ =( ⇡ h )
h
[ H ] whereeach ⇡ h : x 7! ⇡ h ( ·| x ) maps x 2 X to a distribution over actions in  ( A ) 2 .
At step h
=1 , x 1 is drawn 109 according to ⇢ .
Then at step h 2 [ H ] , the agent observes x h , draws a h according to ⇡ h ( ·| x h ) observes 110 reward r ( x h ,a h ) and the MDP transitions to x h +1 according to P ( ·| x h ,a h ) .
For a policy ⇡ , we let 111 P ⇡ h ( x,a ) and P ⇡ h ( x ) denote the marginal state-action and state of ⇡ densities respectively at step h .
Note that P ⇡ 1 ( x )= ⇢ ( x ) for all ⇡ .
Following standard deﬁnitions, we let V ⇡ h : X !
R denote the value function of ⇡ at step h
[ H ]which is given by V ⇡ h ( x ) = E ⇡
[Equation]
i .
Here, the expectation E ⇡ is over trajectories under ⇡ with a h ⇠ ⇡ h ( ·| x h ) .
Similarly, the action-value function Q ⇡ h : X ⇥ A !
R isdeﬁned as Q ⇡ h ( x, a ) =
E ⇡
[Equation]
[Equation] [Equation]
a i .
The optimal policy (which exists under mild conditions when H is ﬁnite [SB18]) is denoted by ⇡ ⇤ and this maximizes V ⇡ h ( x ) for allx and h .
The average value of a policy ⇡ is given by v ( ⇡ ) :=
E x ⇠ ⇢ [ V ⇡ 1 ( x )] .
Finally, we deﬁnethe Bellman operators: T ⇡ h Q ( x,a ) =
r ( x,a )+ E x 0 ⇠ P ( ·| x,a ) ,a 0 ⇠ ⇡ h +1 ( ·| x 0 )
[ Q ( x 0 ,a 0 )] and T ⇤ h Q ( x,a )
=
r ( x,a )+ E x 0 ⇠ P ( ·| x,a ) [max a 0 2 A Q ( x 0 ,a 0 )] .
Notethatthevaluesof v ( ⇡ ) , V ⇡ h , and Q ⇡ h arealwaysin [0 ,H ]due to the constraint on r .
For convenience, we denote the Q function of the optimal policy as Q ⇤ =
Q ⇡ ⇤ .
To deal with potentially large state and action spaces and enable generalization, we consider thesetting where the learner is provided with a model class F ✓ ( X ⇥ A !
[0 ,H ]) to estimate actionvalue functions at each step.
For exposition, we assume this model class is ﬁnite , however, it isstraightforward to extend to inﬁnite settings with appropriate complexity measures.
For notationalsimplicity, we will assume that the learner uses the same F for each timestep h 2 [ H ] but this canbe trivially extended to the time-varying case.
We assume that 0 2 F , and by convention, we willalways write f H +1 = 0 .
For any function f 2 X ⇥ A !
[0 ,H ] , we deﬁne the argmax policy at h as 129 ⇡ f ( x )=argmax a 2 A f ( x,a ) .
We will also write f ( x )=max a 2 A f ( x,a ) .
2.1 Ofﬂine Reinforcement Learning
The distinguishing feature of the ofﬂine (or batch) RL is that we assume that the learner is providedwith a dataset D of example transitions in the MDP.
This prior data might be random interactionswith the environment, data collected with an existing policy, or even demonstrations from an expert.
Furthermore, the learner itself is not permitted to interact in the environment.
The objective is toproduce a good policy ˆ ⇡ using only data from the dataset D .
Formally, the dataset decomposes as D =
( D h ) h
[ H ] for each timestep where D h
= { ( x,a,r,x 0 ) }consists of tuples of transitions and incurred rewards.
We assume D h contains n datapoints that aresampled i.i.d from a ﬁxed marginal distribution µ h 2  ( X ⇥ A ) and the data are independent across 139 timesteps h .
That is, there are Hn datapoints total.
For example, the data could be generated fromh -step state-action distribution of a behavior policy ⇡ b so that µ h ( x,a )=
P ⇡ b h ( x,a )=
⇡ bh ( a | x ) P ⇡ b h ( x )where ⌫ ⇡ b h denotes the h -step marginal state distribution of policy ⇡ b .
However, in general, we donot assume the data was collected by rolling out a policy.
For f,g 2 ( X ⇥ A ! R ) , we use the notation k f  g k 2 µ h =
E µ h ⇥ ( f ( x,a )  g ( x,a ))
⇤ .
The averagesquared Bellman error under µ at state h with respect to f,g is k f  T
⇤ h g k 2 µ h .
Following classical conventions [MS08, DJL21], we make the assumption that the data distributionµ has good coverage over the MDP for all reachable state-actions.
Assumption 1.
There exists a constant C ( µ ) > 0 such that sup h,x,a, ⇡ P ⇡ h ( x,a ) µ h ( x,a )  C ( µ ) .
The constant C ( µ ) is typically unknown.
Such assumptions are common in the ofﬂine literature andcan sometimes be weakened depending on the function class, e.g. linear classes.
Recent work hasendeavored to relax this assumption so that, for example, only good coverage for a comparison policyis required as opposed to every policy via pessimistic methods [JYW21, XCJ + 21, US21].
Despite this,
Theorem 2 of [LTND21] shows that in general model selection bounds of this type are not possible (eventhough the single model class bounds are possible), so we will not be concerned with this reﬁnement.
In this ofﬂine setting, the learner aims to use D and F to produce a policy ˆ ⇡ so as to minimize theregret, which measures the difference in average value between the optimal policy and ˆ ⇡ :
[Equation]
Throughout the paper, we will make use of the following variant of the performance difference lemma,which shows that for value function approximation, it is sufﬁcient to control the squared Bellman errorto bound regret.
Lemma 1 ([DJL21]) .
For any f =( f h ) h
[ H ] , let ⇡ =( ⇡ f h )
h
[ H ] .
Then,
[Equation]
Model Selection Objectives and Main Result
In this section, we state our primary model selection objectives and introduce our main contributions,namely the model selection algorithm, M OD BE, and its theoretical guarantee showing that it is ableto compete in a rate-optimal manner with an oracle that knows the optimal model class.
3.1 The Model Selection Problem
For a ﬁnite function class F that we consider here, the gold-standard regret guarantee for ofﬂinealgorithms with value function approximation is
[Equation]
where A PPROX ( F ) = max h 2 [ H ] ,f 0 2 F min f 2 F k f  T ⇤ h
f 0 k 2 µ is the completeness error of the class
F [CJ19].
This is achieved, for example, by the Fitted Q-Iteration (FQI) algorithm.
If we wereusing inﬁnite classes, we would replace log |F| with another suitable notion of complexity such as 170 pseudodimension.
Such bounds naturally exhibit a trade-off: larger function classes may have a better chance of keeping A PPROX ( F ) close to zero 3 but require more data to minimize the estimation error.
Meanwhile small classes face the opposite problem.
The objective of model selection is to achieve reﬁned regret bounds that balance approximation errorand estimation error.
To this end, we assume that the learner is presented with, not just a single modelclass F , but rather a nested sequence of M classes F 1 ✓ ... ✓ F M .
Solving a problem with nested 176 model classes is an extremely common practice in both supervised learning and RL.
For example,one often starts with an extremely large class F and the considers restrictions of F to an increasingsequence F 1 ,..., F M = F .
In a linear setting, this could correspond to trying to ﬁnd a subset ofcandidate features that are sufﬁcient to solve the problem.
Since it is unknown a priori which class has the best balance (as the approximation error is unknown),we aim to design an algorithm capable of selecting a good class in data-dependent manner.
Inparticular, we would like to achieve oracle inequalities that reﬂect we can compete with theperformance of an oracle that has this knowledge.
Model selection asks whether it is possible toachieve
[Equation]
A slightly weaker objective is to achieve the following two types of oracle inequalities, which weconsider in this paper:
Let k
⇤ =
min { k : A PPROX ( F k )=0 } .
Find ˆ ⇡ so that Reg (ˆ ⇡ )=
˜
O
[Equation]
k ⇤
[Equation]
/n
[Equation]
Deﬁne the global completeness error as ⇠ k :=
max h 2 [ H ] ,f 0 2 F M min f 2 F k k f  T
⇤ h f 0 k 2 µ h .
Findˆ ⇡ so that Reg (ˆ ⇡ )=
˜
O ⇣
[Equation]
{+k 
+} [ M ] [Equation] F
[Equation]
| [Equation] o⌘ .
The ﬁrst oracle inequality asks whether it is possible to compete against the minimally complex classthat has no approximation error.
That is F k ⇤ is smallest class that satisﬁes completeness on the data distribution.
Such oracle inequalities are extremely common in model selection for online bandits and RL –albeit they are generally not rate-optimal in that literature unlike the stronger objective we consider here[ALNS17].
We would also like to be robust to the case where k ⇤ does not exist and not have to rely on theassumption that some (albeit unknown) model class satisﬁes completeness.
The second oracle inequality handles this by deﬁning a global completeness error ⇠ k to replace the approximation error.
Note that⇠ k  A PPROX ( F k ) by deﬁnition.
Our proposed algorithm will be able to handle both simultaneously .
A crucial aspect of both oracle inequalities is that their estimation errors are rate-optimal in the sensethat both n and log |F k | match the rate of the single model class case in (2).
That is, we do not tolerateany worse dependence on either quantity such as O (1 /n 1 / 4 ) rates and other lower order terms.
3. M OD BE Algorithm
Having introduced the model selection objectives, we now present our main result, a novel modelselection algorithm for ofﬂine RL that provably achieves the aforementioned oracle inequalities.
Wewill focus speciﬁcally on the implications of the result and defer the intuition of the method to Section 4.
The algorithm, M OD BE (Model Selection via Bellman Error), is presented in Algorithm 1.
It takesas input the base ofﬂine RL algorithm, the model classes F 1 ✓ ... ✓ F M , and the ofﬂine dataset D 207 of n i.i.d samples for each [-time-step h 
-]timestep
h 2 [ H ] .
The dataset is split randomly into a training set D train and avalidation set D valid .
The algorithm begins optimistically, starting with the candidate model class
k =1and running the base algorithm with F k on the training dataset to generate the candidate action-valuefunctions f k .
We then use the larger model classes k 0 >k to evaluate this function, by using the targetvalues r + f kh +
( x 0 ) as regression targets for model class F k 0 .
Varying over h , this amounts to solvinga sequence of H least squares regression problems using class k 0 , yielding the functions ( f k 0 h )
h
[ H ] .
Since f kh and f k 0 h are attempting to solve the same regression problem (with the same target values),we can use the validation set to compare their performance on this shared squared loss objective˜ L .
This comparison takes the form of a generalization error test in Line 11.
If the test fails and it isdiscovered that the larger model class F k 0 is able to achieve substantially smaller loss than f k (relativeto a tolerance), then we have reason to believe performance can be much better by moving to a larger
[Equation] [Equation]
[Equation] [Equation]
(4) 11: if ˜ L ( g k 0 h ,f kh +1 ) <
˜ L ( f kh ,f kh +1 )
T OL n train ( F k , F k 0 ) for any h 2 [ H ] then 12: k k + 13: goto Line 6.
14: end if 15: end for 16: goto Line17: end while 18: return ˆ ⇡ = ⇣
[Equation]
f
[Equation]
h 2 [ H ] model class k k +1 .
The process is repeated until we exhaust all our model classes or ﬁnd that nolarger model class k 0 offers a big enough improvement over k to cause the test to fail.
3.3 Rate-Optimal Oracle Inequalities
We show that this simple procedure is able to achieve the oracle inequalities of the previous section.
We start with a generic version of the theorem that is stated in terms of an assumed performance boundon the base algorithm for value function approximation.
We then instantiate the base algorithm with
FQI, showing that this version precisely achieves the desired oracle inequalities with the correct rate.
Deﬁnition 1.
Let B be a base ofﬂine RL algorithm for value function approximation that takes asinput a model class F , an ofﬂine dataset D of n samples for each h 2 [ H ] , and a failure probability .
For L> 0 and a function !
, we say that B is ( L, ! )
-regular if (1) !
is a known function and satisﬁes!
n,  ( F k )  !
n,  ( F 0 k ) for all k 0  k ; (2) B ( D, F k ,  ) returns f = ( f h )
h
[ H ] ✓ F k such that f h +1 is 229 independent of D h and
[Equation] [Equation]
Theorem 1.
Let B be an ( L, ! )
-regular algorithm.
Then Algorithm 1 with inputs D , B , F 1 ✓ ... ✓ F M , 231 and   1 /e outputs ˆ ⇡ such that, with probability at least 1   ,
[Equation]
for some absolute constant C 0 > 0 and ◆ = log( M 2 H/  ) .
Furthermore if k ⇤ =
min { k {+
 [ M ]+} :
A PPROX ( F k )=0 } exists, then, under the same event, ˆ ⇡ also satisﬁes
[Equation]
for some absolute constant C 1 > 0 .
For concreteness, consider standard ﬁnite-horizon FQI [DJW20], which is a base algorithm satisfying
Deﬁnition 1 with ! n ( F ) =
O (log |F| /n ) .
This in turn translates to the desired rate-optimal oracle 237 inequalities.
Lemma 2. Consider the FQI algorithm (stated in Appendix B for completeness).
For a model class
F , FQI is a (3 , ! )
-regular base algorithm with !
n,  ( F )= 200 H 2 log(16 H |F| /  ) n .
Armed with this guarantee, it is now immediate that Algorithm 1 instantiated with FQI achieves thedesired oracle inequalities.
Corollary 1.
Let B be instantiated with FQI (Algorithm 2 in Appendix B).
Deﬁne ◆ =log( M 2 H/  )
Then, under the same conditions as Theorem 1, there are absolute constants C 0 ,C 1 > 0 such that, withprobability at least 1   , Algorithm 1 outputs ˆ ⇡ satisfying
[Equation]
and, if k ⇤ exists,
[Equation]
Computational Complexity
M OD BE is computationally efﬁcient given a squared loss regressionoracle.
The procedure itself requires an inner and outer loop over the model classes.
Within theseloops, a squared loss minimizer is computed on the training dataset and then functions are evaluatedon the validation set.
Given access to squared loss regression oracle [SLX21], M OD BE requiresonly O
( Hk ⇤ M ) calls to the computational oracle when k ⇤ exists (a consequence of Theorem 1) or
O ( HM 2 ) in the worst case.
Note that algorithms for optimizing squared loss regression problemsare ubiquitous in machine learning.
Technical Challenges and Overview of the Method
In this section, we discuss the intuition for Algorithm 1.
We ﬁrst provide a description of challengesfaced by seemingly natural approaches to model selection that are actually unable to resolve theproblem satisfactorily.
We then provide further intuition for the algorithm and proof.
4.1 Challenges
Adaptive ofﬂine policy evaluation The most natural approach, to which we have alluded in theintroduction, is to apply an off-the-shelf ofﬂine policy evaluation approach such as ﬁtted Q -evaluation[MS08, DJW20], DICE methods [NCDL19, DNC + 20, ZHH + 22], marginalized importance estimators [XMW19], or doubly robust estimators [TB16].
One could then attempt to estimate v (ˆ ⇡ k ) for eachk 2 [ M ] generated by the base algorithm using function class F k .
The main drawback of this approachis that nearly all of the above methods require selecting a model class to perform the estimation 4 , and itis unclear how to balance the estimation and approximation error optimally to compete with the oracle.
One possible solution is to employ the adaptive estimator of [SSK20], which takes as inputs a sequenceof ofﬂine estimators and known upper bounds on their deviations and returns an estimator that competeswith the best one.
This is precisely the approach taken by [LTND21] for linear contextual bandits.
However, for general function classes in RL, there is no obvious way to compute the analogous deviationbounds, which oftentimes depend on the unknown quantity C ( µ ) .
Since these bounds are requiredby the adaptive estimator as inputs, we are yet again left with unknown hyperparameters to tune
Bellman error estimators Another very natural approach arises from the fact that we are consideringbase algorithms that attempt to minimize the squared Bellman error of objective.
One might askwhether it is possible to estimate the Bellman errors (e.g. with the validation dataset) and comparethe model classes using the Bellman error as a proxy.
The main issue with this approach is the classicdouble-sampling problem [Bai95, DJL21] where the standard estimator of the Bellman error turnsout to be biased, as a result of using an empirical version of the Bellman operator T ⇤ .
By selectingbased on the validation error alone, we will end up favoring model classes that also induce low variancesince the expectation of ˜ L h ( f,g ) (deﬁned in (4)) is given by:
[Equation]
In the same vein, another approach we might consider is recent BVFT algorithm of [XJ21] to selectamong the f k learned by the base algorithm.
However the guarantee of BVFT unfortunately exhibitsa slow O (1 /n 1 / 4 ) and thus does not achieve either oracle inequality.
It also, in theory, requires thata discretization parameter is set based on a concentrability coefﬁcient stronger than C ( µ ) , which istypically unknown.
Representation Learning Readers familiar with work in representation learning for RL [AKKS20]might observe that our selection procedure vaguely resembles algorithms designed to select featurerepresentations for linear MDPs such as [MCK + 21].
They are similar in the sense that they both useone model class (or feature representation) to identify a gap in another model class.
Unfortunately,the problem settings are quite different, and we cannot simply adapt such representation learningalgorithms to the model selection problem since they are either insensitive to the model classcomplexities, thus making them incapable of satisfying an oracle inequality, or they require strongrealizability assumptions.
However, it would be interesting to better understand the relationshipbetween these two problems and their methods in the future.
4.2 A Test for Generalization Error
We now outline the intuition of the algorithm and the proof of Theorem 1.
For more technical depth,we include both a proof sketch and the full proof in Appendix A.
M OD BE can be viewed as a reﬁnement of the Bellman error estimator mentioned in the previoussection.
Recall that the problem with the naive method is that L h ( f kh +1 ,f kh +1 ) produces a biasedestimate of the Bellman error due to the double-sampling problem.
Inherently, the issue withcomparing two classes F k and F k 0 by using L h ( f kh +1 ,f kh +1 ) and L h ( f k 0 h +1 ,f k 0 h +1 ) is that f kh and f k 0 hare solving two different regression problems.
For class k , the regression target is T
⇤ f
kh +1 .
Meanwhile,for class k 0 , the target is T ⇤ f k
h
+1 .
However, we can compare model classes based on their relative performance on the same regressionproblem.
For any h 2 [ H ] , Deﬁnition 1 provides us with a bound on the Bellman error; i.e., this is the generalization error that we should expect when the regression target is f kh +1 .
We can also use class F k 0 totry to solve the same regression problem, generating g k 0 h which minimizes the training loss ˆ L h ( · ,f k h +1 ) .
On the validation set, if we ﬁnd that ˜ L h ( f kh ,f kh +1 )  ˜ L h ( g k 0 h ,f kh +1 ) , then we have no evidence toabandon class F k because it was able to solve its own regression problem better than F k 0 .
On the otherhand, if ˜ L h ( g k 0 h ,f kh +1 )  ˜ L h ( f kh ,f kh
+
it is plausible that we might beneﬁt from moving to a largermodel class since F k 0 was able to achieve better generalization error.
However, while F k 0 may have 310 good generalization error when T ⇤ h
f kh +1 is the target, its true performance will still be measured byits generalization error when T ⇤ h
f k
h
+1 is the target, which could end up being much larger.
To dealwith this, we propose a one-sided generalization error test: a switch will occur if
[Equation]
That is, a switch will occur not when F k 0 performs better than F k when the target is T
⇤ h
f kh +1 , but when 314 it performs substantially better as measured by the estimation error that we see for both classes onthis regression problem.
If (11) holds, then there is reason to believe that the approximation error of F k is large enough to make F k 0 a better class.
Crucially, the test only checks for generalization error,so the tolerance is in terms of !
n train ( F k ) and  n train ( F k
[Equation]
= ˜
O ⇣
[Equation]
⌘ , which is the correct rate for thisproblem.
Thus, if the test turns out to be wrong, we will only lose additive factors of the correct rate.
Empirical Results
[Figure]
The previous sections outlined the strong theoretical properties of M OD BE.
In this section, we ask: whatpractical insights can be gleaned from M OD BE and its theoretical guarantees?
In particular, we wouldlike to understand if the core selection method of M OD BE can be applied out-of-the-box on existingofﬂine RL algorithms with minimal effort.
We evaluated M OD BE in three simulated environmentswith discrete actions: (1) synthetic contextual bandits (CB), (2) Gym CartPole, (3) Gym MountainCar.
See Appendix C for speciﬁc details about the setups.
All training and validation sets were split 80/20.
Contextual Bandit
As a basic validation experiment, we started with the CB setting of [LTND21]which considers a nested sequence of linear model classes with increasing dimension d .
Without anytuning, we simply set the tolerance of M OD BE to T OL ( F k , F k 0 ) =
d k 0 n .
Figure 1 shows the resultsin terms of the log -regret as a function of the dataset size.
We observe that both M OD BE and Hold-Out(choosing the model class with the smallest error) are able to easily match the performance of the bestmodel class while SLOPE [LTND21] ends up being fooled by nearby classes.
RL Discrete Control Our experimental setup for the RL problems in Gym [BCP + 16] builds ontop of the d3rlpy framework [SI21], which contains open-source implementations of ofﬂine RLalgorithms.
We used DQN [MKS + 15] (which is closest to FQI).
In both CartPole and MountainCar,we considered model classes that were two-layer neural networks with ReLU activations and d nodesin the hidden layer and varied the parameter d .
Again, we simply set the tolerance of M OD BE to d k /nmotivated by pseudodimension bounds [BHLM19].
For simplicity, we modiﬁed M OD BE to work inthe discounted inﬁnite horizon setting, which can trivially be done (see Appendix C for details on thismodiﬁcation).
The neural network classes considered had d 2 { 10 , 50 , 1000 , 5000 , 25000 , 50000 } .
Inboth settings, we compared M OD BE to Hold-Out, which is a seemingly sensible baseline that choosesthe model class with lowest estimated Bellman error on a validation set.
For deterministic settings only,this is theoretically justiﬁed.
Figure 1 shows the reward as a function of the dataset size (in episodes).
On CartPole, M OD BE and Hold-Out are both able to compete with the best classes and are roughlyat parity.
However, on MountainCar, we ﬁnd that Hold-Out does surprisingly poorly while M OD BE issuccessfully able to reject the poor model classes.
We conjecture that the empirical failure of Hold-out(which is not predicted in theory since the environment is deterministic) is possibly due to sensitivityto optimization error that makes the inherent Bellman error misleading.
In contrast, the generalizationtest of M OD BE seems to be more robust to this.
Discussion
In this paper, we introduced a new algorithm, M OD BE, for model selection in ofﬂine RL.
To our knowledge, this is the ﬁrst model selection algorithm in this setting to achieve rate-optimal oracle inequalitiesin n and C OMP ( F k ⇤ ) .
A number of interesting open questions remain.
(1) The global completeness 353 ⇠ is potentially much worse than A PPROX (
F ) .
Is it possible to achieve a robust oracle inequality ofthe form O (min k p
A PPROX ( F k )+ p log |F
k | /n ) when k ⇤ does not exist?
(2) Are there rate-optimal 355 procedures that can be used to select hyperparameters beyond model complexity such as learning rates,batch sizes, et cetera ?
(3) Can the ideas of M OD BE be extended to more general algorithms that do notrely on Bellman error minimization?
We believe these questions are of great practical and theoreticalimportance for understanding how to effectively evaluate and select models in ofﬂine RL.