P ESSIMISTIC M ODEL S ELECTION FOR O FFLINE D EEP R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep Reinforcement Learning (DRL) has demonstrated great potentials in solving sequential decision making problems in many applications.
Despite its promising performance, practical gaps exist when deploying DRL in real-world scenarios.
One main barrier is the over-ﬁtting issue that leads to poor generalizability of the policy learned by DRL.
In particular, for ofﬂine DRL with observational data, model selection is a challenging task as there is no ground truth available for performance demonstration, in contrast with the online setting with simulated environments.
In this work, we propose a pessimistic model selection (PMS) approach for ofﬂine DRL with a theoretical guarantee, which features a provably effective framework for ﬁnding the best policy among a set of candidate models.
Two reﬁned approaches are also proposed to address the potential bias of DRL model in identifying the optimal policy.
Numerical studies demonstrated the superior performance of our approach over existing methods.
I NTRODUCTION
The success of deep reinforcement learning (Mnih et al., 2013; Henderson et al., 2018)
(DRL) often leverages upon executive training data with considerable efforts to select effective neural architectures.
Deploying online simulation to learn useful representations for DRL is not always realistic and possible especially in some high-stake environments, such as automatic navigation (Kahn et al., 2018;
Hase et al., 2020), dialogue learning (Jaques et al., 2020), and clinical applications (Tang et al., 2020a).
Ofﬂine reinforcement learning (Singh & Sutton, 1996; Levine et al., 2020; Agarwal et al., 2020) (OffRL) has prompted strong interests (Paine et al., 2020; Kidambi et al., 2020) to empower DRL toward training tasks associated with severe potential cost and risks.
The idea of
OffRL is to train DRL models with only logged data and recorded trajectories.
However, with given observational data, designing a successful neural architecture in OffRL is often expensive (Levine et al., 2020), requiring intensive experiments, time, and computing resources.
Unlike most aforementioned applications with online interaction, Ofﬂine tasks for reinforcement learning often suffer the challenges from insufﬁcient observational data from ofﬂine collection to construct a universal approximated model for fully capturing the temporal dynamics.
Therefore, relatively few attempts in the literature have been presented for provide a pipeline to automate the development process for model selection and neural architecture search in OffRL settings.
Here, model selection refers to selecting the best model (e.g., the policy learned by a trained neural network) among a set of candidate models (e.g. different neural network hyperparameters).
In this work, we propose a novel model selection approach to automate OffRL development process, which provides an evaluation mechanism to identify a well-performed DRL model given ofﬂine data.
Our method utilizes statistical inference to provide uncertainty quantiﬁcation on value functions trained by different DRL models, based on which a pessimistic idea is incorporated to select the best model/policy.
In addition, two reﬁned approaches are further proposed to address the possible biases of DRL models in identifying the optimal policy.
In this work, we mainly focus on deep Q-network (Mnih et al., 2013; 2015) (DQN) based architectures, while our proposed methods can be extended to other settings.
Figure 1 demonstrates the superior performance of the proposed pessimistic model selection (PMS) method in identifying the best model among 70 DRL models of different algorithms on one navigation task (See Appendix C for details), compared with the model selection method by (Tang & Wiens, 2021) which uses three ofﬂine policy evaluation (OPE)
[Figure]
estimates for validation.
Speciﬁcally, based on the derived conﬁdence interval of the OPE value for each candidate model, the ﬁnal selected model by our PMS method is the one that has the largest lower conﬁdence limit, which exactly has the largest true OPE value among all candidate models.
In contrast, none of three OPE estimates used for model selection by Tang & Wiens (2021) can identify the best model due to the inevitable overﬁtting issue during the validation procedure.
To close this section, we summarize the contributions of this work as follows:
• We propose a novel PMS framework, which targets ﬁnding the best policy from given candidate models (e
g., neural architecture, hyperparameters, etc) with ofﬂine data for DQN learning.
Unlike many existing methods, our approach essentially does not involve additional hyperparameter tuning except for two interpretable parameters.
• Leveraging asymptotic analysis in statistical inference, we provide uncertainty quantiﬁcation on each candidate model, based on which our method can guarantee that the worst performance of ﬁnally selected model is the best among all candidate models.
See Corollary 1 for more details.
• To address potential biases of candidate models in identifying the optimal policy, two reﬁned approaches are proposed, one of which can be shown to have regret bounded by the smallest error bound among all candidate models under some technical conditions (See Corollary 2).
To the best of our knowledge, this is the ﬁrst model-selection method in ofﬂine DRL with such a guarantee.
• The numerical results demonstrate that the proposed PMS shows superior performance in different DQN benchmark environments.
R ELATED W ORK
Model Selection for Reinforcement Learning: Model selection has been studied in online decision-making environments (Fard & Pineau, 2010; Lee & Taylor, 2014).
Searching nearly optimal online model is a critical topic for online bandits problems with limited information feed-backs.
For linear contextual bandits, Abbasi-Yadkori et al.
(2011); Chu et al.
(2011) are aiming to ﬁnd the best worst-case bound when the optimal model class is given.
For model-based reinforcement learning, Pacchiano et al.
(2020) introduces advantages of using noise augmented Markov Decision
Processes (MDP) to archive a competitive regret bound to select an individual model with constraints for ensemble training.
Recently, Lee et al.
(2021) utilized an online algorithm to select a low-complexity model based on a statistical test.
However, most of the previous model selection approaches are focused on the online reinforcement learning setting.
Very few works including Farahmand & Szepesv´ari (2011); Paine et al.
Su et al.
; Yang et al. Kuzborskij et al. ; Tang & Wiens (2021); Xie & Jiang (2021) are focused on the ofﬂine setting.
In particular, (Su et al., 2020; Yang et al., 2020; Kuzborskij et al., 2021) focus on model selection for OPE problem.
(Farahmand & Szepesv´ari, 2011; Xie & Jiang, 2021) select the best model/policy based on minimizing the Bellman error, while the ﬁrst approach requires an additional tuning and latter does not.
(Paine et al., 2020; Tang & Wiens, 2021) proposed several criteria to perform model selection in OffRL and mainly focused on numerical studies.
In this work, we provide one of the ﬁrst model selection approaches based on statistical inference for RL tasks with ofﬂine data collection.
Ofﬂine-Policy Learning: Training a DRL agent with ofﬂine data collection often relies on batchwise optimization.
Batch-Constrained deep Q-learning (Fujimoto et al., 2019)
(BCQ) is considered one OffRL benchmark that uses a generative model to minimize the distance of selected actions to the batch-wise data with a perturbation model to maximize its value function.
Other popular OffRL approaches, such as behavior regularized actor-critic (BRAC) (Wu et al., 2019), and random ensemble mixture (Agarwal et al., 2020) (REM) (as an optimistic perspective on large dataset), have also been studied in RL Unplugged (RLU)
(Gulcehre et al., 2020) benchmark together with behavior cloning (Bain & Sammut, 1995; Ross & Bagnell, 2010) (BC), DQN, and DQN with quantile regression (Dabney et al., 2018)
(QR-DQN).
RLU suggests a naive approach based on human experience for ofﬂine policy selection, which requires independent modiﬁcation with shared domain expertise (e.g., Atari environments) for tuning each baseline.
Meanwhile, how to design a model selection algorithm for OffRL remains an open question.
Motivated by the beneﬁts and the challenges as mentioned earlier of the model selection for ofﬂine DRL, we aim to develop a uniﬁed approach for model selection in ofﬂine DRL with theoretical guarantee and interpretable tuning parameters.
B ACKGROUND AND N OTATIONS
Consider a time-homogeneous Markov decision process (MDP) characterized by a tuple M = ( S , A , p, r, γ ) , where S is the state space, A is the action space, p is the transition kernel, i.e., p ( s (cid:48) | s, a ) is the probability mass (density) of transiting to s (cid:48) given current state-action ( s, a ) , r is the reward function, i.e., E ( R t | S t
= s, A t = a ) =
r ( s, a ) for t ≥ 0 , and 0 ≤ γ < 1 is a discount factor.
For simplifying presentation, we assume A and S are both ﬁnite.
But our method can also be applied in continuous cases.
Under this MDP setting, it is sufﬁcient to consider stationary Markovian policies for optimizing discounted sum of rewards (Puterman, 1994).
Denote π as a stationary Markovian policy mapping from the state space S into a probability distribution over the action space.
For example, π ( a | s ) denotes the probability of choosing action a given the state value s .
One essential goal of RL is to learn an optimal policy that maximizes the value function.
Deﬁne V π ( s ) =
(cid:80) + ∞ t =
γ t E π
[ R t | S
= s ] and then the optimal policy is deﬁned as π ∗ ∈ argmax π {V ( π ) (cid:44) (1 − γ ) (cid:80) s ∈S V π ( s )
ν ( s ) } , where ν denotes some reference distribution function over S .
In addition, we denote Q-function as Q π ( s, a ) =
(cid:80) + ∞ t =
γ t E π ( R t
| A
= a, S
= s ) for s ∈ S and a ∈ A .
In this work, we consider the OffRL setting.
The observed data consist of N trajectories, corresponding to N independent and identically distributed copies of { ( S t , A t , R t ) } t ≥ 0 .
For any i ∈ { 1 , · · · , n } , data collected from the i th trajectory can be summarized by { ( S i,t , A i,t , R i,t , S i,t +1 ) } 0 ≤ t<T , where T denotes the termination time.
We assume that the data are generated by some ﬁxed stationary policy denoted by b .
Among many RL algorithms, we focus on Q-learning type of methods.
The foundation is the optimal Bellman equation given below.
[Equation]
where Q ∗ is called optimal Q-function, i.e., Q-function under π ∗ .
Among others, ﬁtted q-iteration (FQI) is one of the most popular RL algorithms (Ernst et al., 2005).
FQI leverages supervised learning techniques to iteratively solve the optimal Bellman equation (1) and shows competitive performance in OffRL.
To facilitate our model-selection algorithm, we introduce the discounted visitation probability, motivated by the marginal importance sampling estimator in (Liu et al., 2018).
For any t ≥ 0 , let p πt ( s, a ) denote the t -step visitation probability Pr π ( S t = s, A t = a ) assuming the actions are selected according to π at time 1 , · · · , t .
We deﬁne the discounted visitation probability function as d π ( s, a ) =
(1 − γ ) (cid:80) t ≥ 0 γ t p πt ( s, a ) .
To adjust the distribution from behavior policy to any target policy π , we use the discounted probability ratio function deﬁned as
[Equation]
where p bt ( s, a ) is the t -step visitation probability under the behavior policy b , i.e., Pr b ( S t = s, A t = a ) .
The ratio function ω π,ν ( s, a ) is always assumed well deﬁned.
The estimation of ratio function is motivated by the observation that for every measurable function f deﬁned over S × A ,
[Equation]
based on which several min-max estimation methods has been proposed such as (Liu et al., 2018; Nachum et al., 2019; Uehara & Jiang, 2019); We refer to (Uehara & Jiang, 2019, Lemma 1) for a formal proof of equation (3).
Finally, because our proposed model selection algorithm relies on an efﬁcient evaluation of any target policy using batch data, we introduce three types of ofﬂine policy evaluation estimators in the existing RL literature.
The ﬁrst type is called direct method via estimating Q-function, based on the relationship that V ( π ) =
(1 − γ ) (cid:80) s ∈S ,a ∈A π ( a | s ) Q ( s, a ) ν ( s ) .
The second type is motivated by the importance sampling (Precup, 2000).
Based on the deﬁnition of ratio function, we can see V ( π ) =
E [ 1 T (cid:80) T − 1 t =0 ω π,ν ( S t , A t ) R t ] , from which a plugin estimator can be constructed.
The last type of OPE methods combines the ﬁrst two types of methods and construct a so-called doubly robust estimator (Kallus & Uehara, 2019; Tang et al., 2020b).
This estimator is motivated by the efﬁcient inﬂuence function of V ( π ) under a transition-sampling setting and the model that consists of the set of all observed data distributions given by arbitrarily varying the initial, transition, reward, and behavior policy distributions, subject to certain minimal regularity and identiﬁability conditions (Kallus & Uehara, 2019), i.e.,
[Equation]
A nice property of doubly robust estimators is that as long as either the Q-function Q π ( s, a ) or the ratio function ω π,ν ( s, a ) can be consistently estimated, the ﬁnal estimator of V ( π ) is consistent (Robins et al., 1994; Jiang & Li, 2015; Kallus & Uehara, 2019;
Tang et al., 2020b).
Furthermore, a doubly robust estimator based on (4) can achieve semiparametric efﬁciency under the conditions proposed by (Kallus & Uehara, 2019), even if nuisance parameters are estimated via black box models such as deep neural networks. Therefore such an estimator is particularly suitable under the framework of DRL.
Our proposed algorithm will rely on this doubly robust type of OPE estimator.
P ESSIMISTIC M ODEL S ELECTION (PMS) FOR B EST P OLICY
In this section, we discuss our pessimistic model selection approach.
For the ease of presentation, we focus on the framework of (deep) Q-learning, where policy optimization is performed via estimating the optimal Q-function.
While this covers a wide range of state-of-the-art RL algorithms such as FQI (Ernst et al., 2005), DQN (Mnih et al., 2013) and QR-DQN (Dabney et al., 2018), we remark that our method is not restricted to this class of algorithms.
Suppose we have total number of L candidate models for policy optimization, where each candidate model will output an estimated policy, say ˆ π l for 1 ≤ l ≤ L .
Our goal is to select the best policy among L policies during our training procedure.
Note that these L models can be different deep neural network architectures, hyper-parameters, and various classes of functions for approximating the optimal Q-function or policy class, etc.
4. D IFFICULTIES AND C HALLENGES
Given a candidate l among L models, we can apply for example FQI using the whole batch data D n to learn an estimate of Q ∗ as (cid:98) Q l and an estimated optimal policy ˆ π l deﬁned as ˆ π l ( a | s ) ∈ argmax a ∈A (cid:98) Q l ( s, a ) , for every s ∈ S .
In order to select the ﬁnal policy, one may use a naive greedy approach to choose some ˜ l such that ˜ l ∈ argmax l E S 0 ∼ ν
[ (cid:80) a ∈A ˆ π l ( a | s ) (cid:98) Q l ( S 0 , a )] , as our goal is to maximize V ( π ) .
However, using this criterion will lead to over-ﬁtting.
Speciﬁcally, due to the distributional mismatch between the behavior policy and target policies, which is regarded as a fundamental challenge in OffRL (Levine et al., 2020), we may easily overestimate Q-function, especially when some state-action pairs are not sufﬁciently visited in the batch data.
This issue becomes more serious when we apply max-operator during our policy optimization procedure.
Such observations have already been noticed in recent works, such as (Kumar et al., 2019; 2020; Paine et al., 2020; Yu et al., 2020; Tang & Wiens, 2021; Jin et al., 2021).
Therefore, it may be inappropriate to use this criterion for selecting the best policy among L models.
One may also use cross-validation procedure to address the issue of over-ﬁtting or overestimating Q-function for model selection.
For example, one can use OPE approaches on the validate dataset to evaluate the performance of estimated policies from the training data set (see Tang & Wiens (2021) for more details).
However, since there is no ground truth for the value function of any policies, the OPE procedure on the validation dataset cannot avoid involving additional tuning on hyperparameters.
Therefore such a procedure may still incur a large variability due to the overﬁtting issue.
In addition, arbitrarily splitting the dataset for cross-validation and ignoring the Markov dependent structure will cause additional errors, which should be seriously taken care of.
4. S EQUENTIAL M ODEL S ELECTION
In the following, we propose a pessimistic model selection algorithm for ﬁnding an optimal policy among L candidate models.
Our goal is to develop an approach to estimate the value function under each candidate model during our policy optimization procedure with theoretical guarantee.
The proposed algorithm is motivated by recent development in statistical inference of sequential decision making (Luedtke & Van Der Laan, 2016; Shi et al., 2020).
The idea is to ﬁrst estimate optimal Q-function Q ∗ , optimal policy π ∗ and the resulting ratio function based on a chunk of data, and evaluate the performance of the estimated policy on the next chunk of data using previously estimated nuisance functions.
Then we combine the ﬁrst two chunks of data, perform the same estimation procedure and evaluation on the next chunk of data.
The framework of MDP provides a nature way of splitting the data.
Speciﬁcally, denote the index of our batch dataset D n as J
= { ( i, t ) : 1 ≤ i ≤ n, 0 ≤ t < T } .
We divide J 0 into O number of non-overlapping subsets, denoted by J 1 , · · · , J O and the corresponding data subsets are denoted by D 1 , · · · , D O .
Without loss of generality, we assume these data subsets have equal size.
We require that for any 1 ≤ o 1 < o 2 ≤ O , any ( i 1 , t 1 ) ∈
J o 1 and ( i 2 , t 2 ) ∈
J o 2 , either i 2 (cid:54) =
i 1 or t 1 < t 2 .
For 1 ≤ o ≤ O , denote the aggregate chunks of data as ¯ D o = (cid:8) ( S i,t , A
i,t , R i,t , S i,t +1 ) , ( i,
[Equation]
We focus on FQI algorithm for illustrative purpose and it should be noticed that our algorithm can be applied to other RL algorithms.
Starting from the ﬁrst chunk of our batch data, at the o -th step ( o = 1 , · · · , O − 1) , for each candidate model l = 1 , · · · , L , we apply FQI on ¯ D o to compute (cid:98) Q ( o ) l as an estimate of optimal Q-function and obtain ˆ π ( o ) l correspondingly such that ˆ π ( o ) l ( a | s ) ∈ argmax s ∈A (cid:98) Q ( o ) l ( s, a ) for every s ∈ S .
Additionally, we compute an estimate of ratio function ω
ˆ π ( o ) l ,ν using ¯ D o by many existing algorithms such as Nachum et al.
Denote the resulting estimator as (cid:98) ω
ˆ π ( o ) l ,ν .
The purpose of estimating this ratio function is to improve the efﬁciency and robustness of our value function estimation for each candidate model.
Then we compute the estimated value function of ˆ π ( o ) l on D o +1 as
[Equation]
where E D o +1 denotes the empirical average over the ( o + 1) chunk of dataset and ( S, A, R, S (cid:48) ) is one transition tuple in D o +1 .
While one can aggregate ˆ V D
o +
(ˆ π ( o ) l ) for 1 ≤ o ≤ ( O − 1) to evaluate the performance of L models, the uncertainty of these estimates due to the ﬁnite sample estima- tion should not be ignored.
Therefore, in the following, we derive an uncertainty quantiﬁcation of our estimated value function for each candidate model, for performing model selection.
Based on equation (4), (conditioning on ¯ D o ), the variance of ˆ V D
o +
(ˆ π ( o ) l ) is
[Equation]
where ( S, A, S (cid:48) ) is a transition tuple with ( S, A ) follows some stationary distribution.
See Assumption 1.
Correspondingly we have an estimate deﬁned as
[Equation]
The estimation procedure stops once we have used all our ofﬂine data and denote the ﬁnal estimated policy as ˆ π l for each l = 1 , · · · , L .
Notice that ˆ π l = ˆ π ( O ) l .
Finally, we compute the weighted average of all the intermediate value functions as our ﬁnal evaluation of the estimated policy ˆ π l , i.e.,
[Equation]
In Section 5, we show that under some technical conditions, the following asymptotic result holds:
[Equation]
where ˆ σ ( l ) =
( O − 1)( (cid:80) O − 1 o =1 { σ o +
(ˆ π ( o ) l ) } − 1 ) − 1 , = ⇒ refers to weak convergence when either n or T goes to inﬁnity, and N (0 , 1) refers to the standard normal distribution.
Based on the asymptotic result in (10), we can construct a conﬁdence interval for the value function of each policy ˆ π l .
Given a conﬁdence level α , for each l , we can compute U ( l ) =
ˆ V (ˆ π l ) − z α/ 2 (cid:112) O/
nT ( O − 1)ˆ σ ( l ) , where z α/ 2 is (1 − α 2 ) -quantile of the standard normal distribution.
Our ﬁnal selected one is ˆ l ∈ argmax 1 ≤ l ≤ L U ( l ) .
The use of U ( l ) is motivated by the recent proposed pessimistic idea to address the overestimation issue of value (or Q) function in the OffRL setting.
See Kumar et al.
(2019; 2020); Jin et al.
Xie et al. ; Uehara & Sun (2021); Zanette et al.
(2021) for details.
The ﬁnal output of our algorithm is ˆ π ˆ l and an outline of the proposed algorithm can be found in Algorithm 1.
As we can see, our algorithm is nearly tuning free, which provides great ﬂexibility in real-world applications.
The only two adjustable parameters is O and α .
The size of O balances the computational cost and the ﬁnite-sample accuracy of evaluating each candidate model.
In speciﬁc, we can indeed show that the variance of the estimated value function by our algorithm can achieve the semi-parametric efﬁciency bound, which is best one can hope for.
So in the asymptotic sense, the effect of O is negligible.
In the ﬁnite-sample setting, we believe the performance will be discounted by a factor (cid:112) O − 1 /O .
Therefore, if O is large enough, (cid:112) O − 1 /O will have a mere effect on the performance. See Theorem 1. However, using large O will result in a large computational cost.
As a sacriﬁce for the nearly tuning free algorithm, we need to apply OffRL algorithms O times for each candidate model.
The parameter α determines how worst the performance of each policy we should use to evaluate each policy.
See Corollary 1 for more insights.
Algorithm Input: Dataset D n and L candidate models for estimating optimal Q-function and policy; We divide D n into non-overlapping subsets denoted by D 1 , · · · , D O .
We require that for any 1 ≤ o 1 < o 2 ≤ O , any ( i 1 , t 1 ) ∈
J o 1 and ( i 2 , t 2 )
∈
J
o 2 , either i 2 (cid:54) =
i 1 or t 1 ≤ t 2 .
1 for l ∈ L do for o = 1 to O − 1 do
For l ∈ L models, construct the optimal (cid:98) Q ( o ) l and ˆ π ( o ) l using ¯ D o data subset
Compute (cid:98) ω
o l using ¯ D o by Nachum et al.
and min-max solver
Compute ˆ V D
o +
(ˆ π ( o ) l ) and
ˆ σ 2 o +1 ( l ) using D o +1 given in (5) (8) respectively
For l l
ˆ V (ˆ π l − z
/ 2 (cid:112) nT ( O − 1) /O
ˆ σ ( l ) ˆ V (ˆ π l ) and ˆ σ ( l ) are
Pick ˆ l = arg max l U ( l ) as the model and run the on dataset obtain ˆ π ˆ l 8 Return ˆ π ˆ l .
T HEORETICAL R ESULTS
In this section, we justify our asymptotic result given in (10).
We use O p to denote the stochastic boundedness.
Before that, we make several technical assumptions:
Assumption 1 The stochastic process { A t , S t } t ≥ 0 is stationary with stationary distribution p ∞ .
Assumption 2 For every 1 ≤ l ≤ L and 1 ≤ o ≤ O , we have E |V (ˆ π ( o ) l ) −V ( π ∗ ) | ≤ C 0 ( nT/O ) − κ , for some constant C 0 and κ > 1 / 2 .
Assumption 1 is standard in the existing literature such as (Kallus & Uehara, 2019).
Assumption 2 is key to our developed asymptotic results developed.
This assumption essentially states that all candidate models are good enough so that eventually their value functions will converge to that of the optimal one.
This implies that there is no asymptotic bias in identifying the optimal policy.
While this is reasonable thanks to the capability of deep neutral networks, which has demonstrated their empirical success in many RL applications, such an assumption could still be strong.
In Section 6, we try to relax this assumption and provide two remedies for addressing possibly biased estimated policies.
In addition, Assumption 1 also requires that the convergence rates of value functions under estimated policies are fast enough.
This has been shown to hold under the margin condition on π ∗ , see e.g., (Hu et al., 2021) for more details.
Assumption 3 For every 1 ≤ l ≤ L and 1 ≤ o ≤ O − 1 , suppose E ( S,A ) ∼ p ∞ | (cid:98) Q ( o ) l ( S, A ) − Q ˆ π ( o ) l ( S, A ) | {+
+}
= O p { ( nT/O ) − 2 κ 1 } for some constant κ 1 ≥ 0 .
In addition, (cid:98) Q ( o ) l is uniformly bounded almost surely.
Assumption 4 For every 1 ≤ l ≤ L and 1 ≤ o ≤ O − 1 , suppose E ( S,A ) ∼ p ∞ | (cid:98) ω
ˆ π ( o ) l ,ν ( S, A ) − ω
ˆ π ( o ) l ,ν ( S, A ) | {+
+}
= O p { ( nT/O ) − 2 κ 2 } for some constant κ 2 ≥ 0 .
In addition, both ω ˆ
π
( o ) l ,ν and (cid:98) ω
ˆ π ( o ) l ,ν are uniformly bounded above and below away from 0 almost surely.
Assumption [-
-]5 For every 1 ≤ l ≤ L and 1 ≤ o ≤ O − 1 , σ 2 (ˆ π ( o ) l ) and ˆ σ 2 o +
(ˆ π ( o ) l ) are bounded above and below from 0 almost surely.
Assumptions 3 and 4 impose high-level conditions on two nuisance functions.
Our theoretical results only require κ 1 + κ 2 > 1 / 2 , which is a mild assumption.
For example, if considered parametric models for both Q-function and ratio function, then κ 1 = κ 2 = 1 / 2 .
If considered nonparametric models for these two nuisance functions such as deep neural networks, then 1 / 4 < κ 1 , κ 2 < 1 / 2 can be obtained under some regularity conditions.
See Fan et al.
(2020) and Liao et al.
; Uehara et al.
(2021) for the convergence rates of Q-function and ratio function by non-parametric models respectively.
In addition, Assumption 5 is a mild assumption, mainly for theoretical justiﬁcation.
Then we have the following main theorem as a foundation of our proposed algorithm.
Theorem 1 Under Assumptions 1-5, we have
[Equation]
Theorem 1 provides an uncertainty quantiﬁcation of each candidate model used in policy optimization.
Such uncertainty quantiﬁcation is essential in OffRL as data are often limited.
We highlight the importance of such results in Appendix A. A consequent result following Theorem 1 validates the proposed Algorithm 1: Corollary
[Equation]
under Assumptions 1-5. As can be seen clearly from Corollary 1 and the proposed PMS, with a large probability (by letting α small), we consider the worst performance of each candidate model
ˆ π l in the sense of the lower conﬁdence limit of the value function, and then select the best one among all models.
T WO R EFINED A PPROACHES
In this section, we relax Assumption 2 by allowing possibly non-negligible bias in estimating the optimal policy and introduce two reﬁned approaches for addressing this issue.
Instead of imposing Assumption 2, we make an alternative assumption below.
Assumption 6 For every 1 ≤ l ≤ L , there exists B ( l ) such that max 1 ≤ o ≤ ( O − 1)
|V
(ˆ π ( o ) l ) − V ( π ∗ ) | ≤ B ( l ) almost surely.
Assumption 6 is a very mild assumption.
It essentially states that the biases for all our intermediate value function estimates are bounded by some constant, which is much weaker than Assumption 2.
In this case, the asymptotic results in (11) may not hold in general.
Correspondingly, we have the following result.
Theorem 2 Under Assumptions 1, 3-6, for every 1 ≤ l ≤ L , the following inequality holds.
[Equation]
Motivated by Lepski’s principle (Lepski & Spokoiny, 1997) from nonparametric statistics and (Su et al., 2020) studying the model selection of OPE, we consider the following reﬁned model-selection procedure to ﬁnd the best policy.
We ﬁrst rank L candidate models in an non-increasing order based on the value of ˆ σ ( l ) , i.e., for 1 ≤ i < j ≤ L , ˆ σ ( i ) ≥ ˆ σ ( j ) .
Then for i -th model, we construct an interval as I ( l ) =
[ˆ V (ˆ π l ) − 2 z α/ (2 L ) (cid:112) O/ nT ( O − 1)ˆ σ ( l ) , ˆ V (ˆ π l ) + 2 z α/ (2 L ) (cid:112) O/ nT ( O − 1)ˆ σ ( l )] .
Finally the optimal model/policy we choose is ˆ π ˆ i
such that
ˆ
i = max { i : 1 ≤ i ≤ L, ∩ 1 ≤ j ≤ i I ( j ) (cid:54) = ∅} .
To show this procedure is valid, we need to make one additional assumption.
Assumption 7 There exists a ζ < 1 such that for 1 ≤ i ≤ L , B ( i ) ≤ B ( i + 1) and ζ ˆ σ ( i ) ≤ ˆ σ ( i + 1) ≤ ˆ σ ( i ) almost surely.
This assumption is borrowed from Su et al.
It typically assumes that after sorting our model based on ˆ σ ( l ) , the bias of estimated policy is monotonically increasing and the standard deviation is monotonically deceasing but not too quickly.
This is commonly seen when all candidate estimators exhibit some bias-variance trade-off phenomena.
Deﬁne the following event
[Equation]
[Equation]
Then we have the following theoretical guarantee for our reﬁned procedure.
Corollary 2 Under Assumptions 1, 3-7, we have lim inf nT →∞ Pr( E ) ≥ 1 − α .
If we further assume that for any δ > 0 , with probability at least 1 − δ , for every 1 ≤ i ≤ L , |V (ˆ π i ) − ˆ V (ˆ π i ) | ≤ c ( δ ) log( L )ˆ σ ( i ) / √ NT for some constant c ( δ ) , then lim inf nT →∞ Pr( ¯ E ) ≥ 1 − α − δ , where
[Equation]
The additional assumption (i.e., the high probability bound) in Corollary 2 can be shown to hold by the empirical process theory under some technical conditions (Van de Geer, 2000).
Hence Corollary 2 provides a strong guarantee that the regret of the ﬁnal selected policy is bounded by the smallest error bound among all L candidate policies.
Note that Assumption 3 imposed here could be strong.
Another reﬁned approach: Notice that the above reﬁned approach indeed focuses on OPE estimates to select the best policy with regret warranty.
The rough motivation behind is to ﬁnd a policy that has the smallest estimation error to the optimal one.
However, such procedure may not directly match the goal of maximizing the value function in OffRL.
To relieve this issue , we can alternatively choose the ﬁnal policy as ˆ π ˆˆ i such that ˆˆ i = argmax 1 ≤ i ≤
ˆ i ˆ V (ˆ π i ) − 2 z α/ 2 (cid:112) nT ( O − 1) /O
ˆ σ ( i ) , where the argmax is taken over ˆ i models.
This approach can be viewed as a combination of PMS and the above reﬁned approach.
By adopting this approach, candidate models with large biases are ﬁrtly removed by the truncation on ˆ i .
Then, we use the idea of PMS to select the best model having the best worst performance among the remaining candidates.
Unfortunately, we do not have theoretical guarantee for this combined approach.
E XPERIMENTAL R ESULTS
We select six DQN environments ( E 1 to E 6 ) from open-source benchmarks (Brockman et al., 2016; Juliani et al., 2018) to conduct numerical experiments, as shown in Fig. 5 of Appendix C.
These tasks of deployed environments cover different domains that include tabular learning (Fig 5(a)); automatic navigation in a geometry environment with a physical ray-tracker (Fig 5(b));
Atari digital gaming (Fig 5(c) and (d)), and continuous control (Fig 5(e) and (f)).
We provide detailed task description and targeted reward for each environment in Appendix C.
Experiment setups.
To evaluate the performance of PMS with DQN models in offRL, we choose different neural network architectures under ﬁve competitive DRL algorithms including DQN by (Mnih et al., 2013; 2015), BCQ by (Fujimoto et al., 2019), BC by (Bain & Sammut, 1995; Ross & Bagnell, 2010), BRAC by (Wu et al., 2019) from RLU benchmarks, and REM by (Agarwal et al., 2020). Within each architecture, 70 candidate models are created by assigning different hyperparameters and training setups.
See Appendix C.1 for details.
We then conduct performance evaluation of different OffRL model selection methods on these generated candidate models. Evaluation procedure. We utilize validation scores from OPE for each model selection algorithm, which picks the best (or a good) policy from the candidate set of size L based on its own criterion. [Figure]
1k 2k 4k Num episodes 0.0 0.
R e g r e t
PMS
WIS
AM
FQE
Regret is used as the evaluation metric for each candidate.
The regret for model l is deﬁned as V ( π l ∗ ) − V (ˆ π l ) , where l ∗ = arg max l (cid:48) =
L V ( π l (cid:48) ) corresponds to the candidate policy with the best OPE validation performance.
In our implementation, we treat π l ∗ as π ∗ , the oracle but unknown best possible policy.
A small regret is desirable after model selection.
Note the optimal regret is not zero since we can only use data to obtain ˆ π l instead of π l for each model.
We provide additional top-k regret and precision results from Figure 6 to 16 in Appendix C.
Performance comparison.
As highlighted in Fig.
1 in the introduction, we report estimated OPE values by different model selection approaches, i.e. PMS and three methods by (Tang & Wiens, 2021), versus the true OPE values.
In this experiment, we consider 70 DQN models under the above mentioned ﬁve DRL algorithms, i.e., 14 models are considered for each architecture.
We use fewer models for each DRL algorithm mainly for clear presentation.
By using the conﬁdence interval constructed by our PMS procedure, our method is able to correctly select the top models, while the other three methods fail.
To further investigate the performance of PMS, we implement model selection among 70 models within each DRL algorithm separately.
Fig.
2 shows the box plots of averaged regret over six environments after OPE per neural network architecture.
Each subﬁgure contains results from one particular DRL algorithm with different hyperparameters or training setups.
The left box plot refers to the regrets of all 70 models and the right one represents the regrets of top 10% models selected by the proposed PMS method.
Note that the right box plot is a subset of the left one.
The results show that our proposed PMS successfully helps to select models with the best policies and improve the average regret by a signiﬁcant margin.
In particular, PMSREM-based models attain the lowest regrets, due to the beneﬁt from its ensemble process.
Detailed results for each environment is given in Appendix C.
Sensitivity analysis.
Fig.
3 compares different selection algorithms with varying training data size.
PMS outperforms others across all scales, and larger number of episodes gives smaller variation and lower sensitivity.
PMS algorithm with reﬁnements.
We replicate our experiments on in the ofﬂine navigation task in E 2 ( Banana Collector ) fortimes and report regrets of top 10% models selected by PMS and two reﬁnements in Fig.
As we can see, while the overall performances of the proposed three model selection methods are similar, two reﬁned approaches have better regrets than PMS in terms of median, demonstrating their promising performances in identifying the best model.
OPE results have been also evaluated also in DRL tasks with E 1 and E 3 to E 6 , where the reﬁnement algorithms (PMS R1/R2) have only a small relative ± 0.423 % performance difference compared to its original PMS setups.
C ONCLUSION
We propose a new theory-driven model selection framework (PMS) for ofﬂine deep reinforcement learning based on statistical inference.
The proposed pessimistic mechanism is warrants that the worst performance of the selected model is the best among all candidate models.
Two reﬁned approaches are further proposed to address the biases of DRL models.
Extensive experimental results on six DQN environments with various network architectures and training hyperparameters demonstrate that our proposed PMS method consistently yields improved model selection performance over existing baselines.
The results suggest the effectiveness of PMS as a powerful tool toward automating model selection in ofﬂine DRL.
A C OMMENTS ON A SYMPTOTIC R ESULTS
We remark here that all theoretical justiﬁcation in this paper is based on asymptotics. It might be possible to investigate ﬁnite sample regimes when one has an exact conﬁdence interval instead of (11) or a non-asymptotic bound. However, having an exact conﬁdence interval might require some model speciﬁcation of the value function, and using non-asymptotic bounds might require additional tuning steps (e.g., constants in many concentration inequalities), which is beyond the scope of this paper. In addition, as seen from our empirical evaluations below, with a relatively large sample size, the proposed model selection approach performs well.
B T ECHNICAL P ROOFS
Notations : The notation ξ ( N ) (cid:46) θ ( N ) (resp.
ξ ( N ) (cid:38) θ ( N ) ) means that there exists a sufﬁciently large (resp.
small) constant c 1 > 0 (resp.
c 2 > 0 )
such that ξ ( N ) ≤ c 1 θ ( N ) (resp.
ξ ( N ) ≥ c 2 θ ( N ) ) for some sequences θ ( N ) and ξ ( N ) related to N .
In the following proofs, N often refers to some quantity related to n and T .
Lemma 1 and its proof : Let J denotes some index of our batch data D n .
Deﬁne φ ( J, Q π , ω π,ν , π ) = 1 | J | (cid:88) ( i,t ) ∈ J ω π,ν ( S i,t , A t ) (cid:32) [Equation] (cid:88) a (cid:48) ∈A
[Equation]
[Equation]
(cid:48)
[Equation]
t +
[Equation]
[Equation]
[Equation]
Q π ( S
[Equation] A [Equation] ,t ) (cid:33) , where | J | is the cardinality of the index set J , e.g., | J o | = nTO for every 1 ≤ o ≤ O .
Then we have the following Lemma 1 as an intermediate result to Theorem 1.
Lemma 1 Under Assumptions 1 and 3-5, for every 1 ≤ l ≤ L and 1 ≤ o ≤ O − 1 , the following asymptotic equivalence holds.
[Equation]
where o p (1) refers to a quantity that converges to 0 as n or T goes to inﬁnity [Equation]
The proof is similar to that of Theorem 7 in Kallus & Uehara (2019).
First, notice that
[Equation]
Then it sufﬁces to show the term in the ﬁrst bracket converges to 0 faster than √ nT .
Notice that
[Equation]
where
[Equation]
and
[Equation]
Next, we bound each of the above three terms.
For term E 1 , it can be seen that
[Equation]
In addition, by Assumptions 3 and 4, we can show
[Equation]
where the inequality is based on that each item in E 3 is uncorrelated with others.
Then by Markov’s inequality, we can show
[Equation]
Similarly, we can show
[Equation]
For term ( E 3 ) , by Cauchy Schwarz inequality and similar arguments as before, we can show
[Equation]
Therefore, as long as ( κ 2 + κ 1 ) > 1 / 2 , we have E 1 + E 2 + E 3 =
o ( (cid:112) O/nT ) , which concludes our proof.
Proof of Theorem 1 We aim to show that
[Equation]
It can be seen that
[Equation]
{+ˆ σ o +
+}
(ˆ π ( o ) l ) (cid:33) + (cid:115) nT
O ( O − 1) (cid:32) O − 1 (cid:88) o =
{+V (ˆ π ( o ) l ) − V (ˆ π l ) ˆ σ o +
+}
(ˆ π ( o ) l ) (cid:33) .
Deﬁne
[Equation]
, where | J | is the cardinality of the index
set J , i
[Equation]
e., | J | = nTO .
Then by Lemma 1, we show that
[Equation]
If we can show that
[Equation]
which will be shown later, then by Slutsky theorem, we can show that
[Equation]
For ( I ) , we can see that
[Equation]
By the sequential structure of our proposed algorithm
[Equation]
( I ) forms a mean zero martingale
[Equation] Then we use Corollary 2.8 of (McLeish, 1974) to show its asymptotic distribution.
First of all, by the uniformly bounded assumption on Q-function, ratio function and the variance [Equation] we can show that
[Equation]
Next
[Equation]
we aim to show that O
[Equation]
Notice that the left hand side of the above is bounded above by
O nT
[Equation]
[Equation]
Because, for each 1 ≤ o ≤ ( O − 1) ,
[Equation]
forms a mean zero martingale, we apply Freedman’s inequality in (Freedman, 1975) with Assumptions 3-5 to show it is bounded by O p ( (cid:113) OnT ) .
Applying union bound shows (19) is o p (1) and furthermore consistency of ˆ σ (ˆ π l ) in (16) holds.
Then we apply the martingale central limit theorem to show
[Equation]
The remaining is to show (cid:115)
[Equation]
is asymptotically negligible.
Consider
[Equation]
where we use Assumption 2 for the last inequality.
Summarizing together [Equation] we can show that
[Equation]
o (1) , where we obtain the second inequality by that (cid:80) O − 1 o =1 o − κ ≤ 1 + (cid:82) O 1 o − κ do (cid:46)
O 1 − κ .
In the last inequality, we use κ > 1 in Assumption 2.
Then Markov inequality gives that
[Equation]
Moreover, by Assumption 5 that inf 1 ≤ o ≤ O − 1 ˆ σ o +
(ˆ π ( o ) l ) ≥ c for some constant c > 0 , we can further show that
[Equation]
which completes our proof.
Proof of Corollary 1 Denote the sets E l
= {|V (ˆ π l ) − ˆ V (ˆ π l ) | ≤ ˆ u ( l ) } , l = 1 , . . .
, L , where ˆ u ( l ) =
z α/ 2 (cid:112) nT ( O − 1) /O
ˆ σ ( l ) .
Note that lim inf
nT →∞ Pr( ∩ Lj =1 E j ) ≥ 1 − Lα and
[Equation]
where the last inequality holds because given the event
∩
Lj =1 E j , one has − ˆ u
(ˆ l ) ≤ V (ˆ π ˆ l ) − ˆ V (ˆ π ˆ l ) and V (ˆ π l ) − ˆ V (ˆ π l ) ≤ ˆ u ( l ) for any l .
This completes the proof by taking lim inf on both sides.
Proof of Theorem 2 To show the results in Theorem 2, it can be seen that
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12) [Equation]
nT
[Equation]
Then by results in the proof of Theorem 1, we can show that
[Equation]
This implies that
[Equation]
which concludes our proof.
Proof of Corollary 2 : We mainly show the proof of the second claim in the corollary, based on which the ﬁrst claim can be readily seen.
Deﬁne an event E such that 1 ≤ l ≤ L , |V
(ˆ π l ) − ˆ V
(ˆ π l ) | ≤ c ( δ ) log( L )ˆ σ ( i ) / √ NT and |V ( π ∗ )
− ˆ V (ˆ π l ) | ≤ z α/ (2 L ) (cid:112) O/
nT ( O − 1)ˆ σ ( l ) + B ( l ) .
Based on the assumption given in Corollary 2 and Theorem 2, we have lim inf nT →∞ P ( E ) ≥ 1 − δ − α .
In the following, we suppose event E holds.
Inspired by the proofs of Corollary 1 in (Math´e, 2006) and Theorem 3 of (Su et al., 2020), we deﬁne ˜
l = max { l : B ( l ) ≤ u 1 ( l ) + u 2 ( l ) } , where u 1 ( l ) =
z α/ (2 L ) (cid:112) O/nT
( O − 1)ˆ σ ( l ) .
Let u 2 ( l )
=
c ( δ ) log( L )ˆ σ ( i ) / √ NT .
By Assumption 7, for l ≤ ˜ l ,
B ( l ) ≤ B (˜ l ) ≤ u 1 (˜ l ) ≤ u 1 ( l ) , which further implies that for any l ≤ ˜ l ,
| ˆ V (ˆ π l ) − V ( π ∗ ) | ≤ B ( l ) + u 1 ( l ) ≤ 2 u 1 ( l ) .
Then V ( π ∗ ) ∈
I ( l ) based on the construction of I ( l ) for all l ≤ ˜ l .
In addition, we have for l ≤ ˜ l
[Equation]
32) by triangle inequality and event E .
Since I ( l ) share at least one common element for 1 ≤ l ≤ ˜ l , we have ˆ i ≥ ˜ l .
Moreover, there must exist an element x such that x ∈ I (˜ l ) ∩ I (ˆ i ) , where | ˆ V (ˆ π ˜ l ) − x | ≤ u 1 (˜ l ) and | ˆ V
(ˆ π ˆ i ) − x | ≤ u 1 (ˆ i ) .
This indicates that
[Equation] [Equation]
by again triangle inequality and Assumption 7, and [Equation]
[Equation]
by event E and Assumption 7.
Deﬁne l ∗ =
min { l : B [Equation] l ) + u 1 ( l ) + u 2 ( l ) } .
Then following the similar proof of (Su et al., 2020), we consider two cases:
Case 1: If l ∗ ≤ ˜ l , then we have
[Equation]
where we use Assumption 7.
Case 2: If l ∗ > ˜ l , then we have
[Equation]
where we use Assumption 7.
This implies that
[Equation]
Combining two cases, we can show that
[Equation]
as ζ < 1 .
Together with (33), we have
[Equation]
which concludes our proof.
C M ORE D ETAILS ON DQN E NVIRONMENTS
[Figure] We introduce our deployed DQN environments in this section, which included four environments with discrete action ( E 1 to E 4 ) and two environments ( E 5 to E 6 ) with continuous action.
These environments cover wide applications, including tabular learning ( E 1 ), navigation to a target object in a geometrical space ( E 2 ), digital gaming ( E 3 to E 4 ), and continuous control ( E 5 to E 6 ).
E 1 : Frozen Lake: The Frozen Lake is a maze environment that manipulates an agent to walk from a starting point (S) to a goal point without failing into the hole (H).
We use FrozenLake-v0 from OpenAI Gym (Brockman et al., 2016) as shown in Fig 5(a).
We provide top-5 regret and precision results shown in Figure 6 and 7.
E 2 : Banana Collector: The Banana collector is one popular 3D-graphical navigation environment that compresses discrete actions and states as an open source DQN benchmark from Unity ML-Agents v0.3.(Juliani et al., 2018).
The DRL agent controls an automatic vehicle with 37 dimensions of state observations including velocity and a ray-based perceptional information from objects around the agent.
The targeted reward is 12 . 0 points by accessing correct yellow bananas ( +1 ) and avoiding purple bananas ( − 1 ) in ﬁrst-person point of view as shown in Fig 5(b).
We provide the related top-5 regret and precision results shown in Figure 8 and 9.
E 3 :
Pong: Pong is one Atari game environment from OpenAI Gym (Brockman et al., 2016) as shown in Fig 5(c).
We provide its top-5 regret and precision results shown in Figure 10 and 11.
E 4 :
Breakout: Breakout is one Atari game environment from OpenAI Gym (Brockman et al., 2016) as shown in Fig 12(d).
We provide the related top-5 regret and precision results shown in Figureand 13.
E 5 : HalfCheetah-v1: Halfcheetah is a continuous action and state environment to control agent with monuments made by MuJoCo simulators as shown in Fig 5(e).
We provide the related topregret and precision results shown in Figure 14 and 15.
E 6 : Walker2d-v1: Walker2d-v1 is a continuous action and state environment to control agent with monuments made by MuJoCo simulators as shown in Fig 5(f).
We provide the related top-5 regret and precision results shown in Figure 16 and 17.
C. H YPER -P ARAMETERS I NFORMATION
We select a total of 70 DQN based models for each environment.
We will open source the model and implementation for future studies.
Table 1, Table 2, and Table 3 summarize their hyper-parameter and setups.
In addition, Figure 18 and Figure 19 provide ablation studies on different scales of α and O selection in PMS experiments for the deployed DRL navigation task ( E 2 ).
From the experimental results, a more pessimistic α (e.g., 0.001) is associated with a slightly better attained top-5 regret. Meanwhile, the selection of O does not produce much different performance on selected policies but slightly affects the range of the selected policies.
[Table] [Table]
I MPACT
There are also some limitations of the proposed PMS as one of the preliminary attempts on model selection for ofﬂine reinforcement learning.
When the benchmarks environments (excluded Atari games) are based on simulated environments to collect the true policy (Barth-Maron et al., 2018; Siegel et al., 2019), more real-world-based environments could be customized and studied in future works.
For example, one experimental setup needs to be carefully controlled in clinical settings (Tang & Wiens, 2021) or resilience-oriented (Yang et al., 2021) reinforcement learning. [Table] [Table] [Table]