COMBO: C ONSERVATIVE O FFLINE M ODEL -B ASED P OLICY O PTIMIZATION
A BSTRACT
Model-based algorithms, which learn a dynamics model from logged experience and perform some sort of pessimistic planning under the learned model, have emerged as a promising paradigm for ofﬂine reinforcement learning (ofﬂine RL).
However, practical variants of such model-based algorithms rely on explicit uncertainty quantiﬁcation for incorporating pessimism.
Uncertainty estimation with complex models, such as deep neural networks, can be difﬁcult and unreliable.
We overcome this limitation by developing a new model-based ofﬂine RL algorithm,
COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model.
This results in a conservative estimate of the value function for out-of-support state-action tuples, without requiring explicit uncertainty estimation.
We theoretically show that our method optimizes a lower bound on the true policy value, that this bound is tighter than that of prior methods, and our approach satisﬁes a policy improvement guarantee in the ofﬂine setting.
Through experiments, we ﬁnd that COMBO consistently performs as well or better as compared to prior ofﬂine model-free and model-based methods on widely studied ofﬂine RL benchmarks, including image-based tasks.
I NTRODUCTION
Ofﬂine reinforcement learning (ofﬂine RL) (Lange et al., 2012; Levine et al., 2020) refers to the setting where policies are trained using static, previously collected datasets.
This presents an attractive paradigm for data reuse and safe policy learning in many applications, such as healthcare Wang et al.
(2018), autonomous driving Yu et al.
(2020a),
robotics Kalashnikov et al.
Rafailov et al.
(2020), and personalized recommendation systems Swaminathan & Joachims (2015).
Recent studies have observed that direct use of RL algorithms originally developed for the online or interactive paradigm leads to poor results in the ofﬂine RL setting Fujimoto et al.
(2018a)
; Kumar et al.
Kidambi et al.
This is primarily attributed to the distribution shift that arises over the course of learning between the ofﬂine dataset and the learned policy.
Thus, development of algorithms specialized for ofﬂine RL is of paramount importance to beneﬁt from the ofﬂine datasets available in the aforementioned application domains.
In this work, we develop a new model-based ofﬂine
RL algorithm that enjoys strong theoretical guarantees, while also matching or improving over state-of-the-art methods in ofﬂine RL benchmarks.
One paradigm for algorithm design in ofﬂine RL is to incorporate conservatism or regularization to the learning algorithm.
Model-free ofﬂine RL algorithms Fujimoto et al.
(2018b)
; Kumar et al.
Wu et al.
; Jaques et al.
; Kumar et al.
(2020) directly incorporate conservatism into the policy or value function training and do not require learning a dynamics model.
However, model-free algorithms learn only on the states in the ofﬂine dataset, which can lead to overly conservative algorithms.
In contrast, model-based algorithms Kidambi et al.
Yu et al.
(2020c) learn a pessimistic dynamics model, which in turn induces a conservative estimate of the value function.
By generating data, model-based algorithms can achieve better generalization and e.g. have demonstrated the ability to solve new tasks using the ofﬂine dataset Yu et al.
(2020c).
However, such algorithms rely crucially on uncertainty quantiﬁcation of the learned dynamics model to incorporate conservatism, which can be difﬁcult or unreliable for complex datasets or deep network models.
Furthermore, these methods do not adapt the uncertainty estimates as the policy and value function change over the course of learning.
In this work, our goal is to develop a new algorithm that retains the beneﬁts of model-based algorithms while removing the reliance on uncertainty estimation, which we argue is not necessary for ofﬂine RL.
Our main contribution is the development of conservative ofﬂine model-based policy optimization (COMBO), a new model-based algorithm for ofﬂine RL.
COMBO learns a dynamics model using the ofﬂine dataset.
Subsequently, it employs an actor-critic method where the value function is learned using both the ofﬂine dataset as well as synthetically generated data from the model, similar to
Dyna Sutton (1991).
However, in contrast to Dyna, COMBO learns a conservative critic function by penalizing the value function in state-action tuples that are not in the support of the ofﬂine dataset, obtained by simulating the learned model.
We theoretically show that for any policy, the Q-function learned with COMBO is a lower bound of the true Q-function, making it a good surrogate for policy optimization.
While the approach of optimizing a performance lower-bound is similar in spirit to prior model-based algorithms Kidambi et al.
Yu et al.
(2020c)
, COMBO crucially does not require uncertainty quantiﬁcation.
In addition, we show theoretically that the Q-function learned by COMBO represents a tighter lower bound of the true Q-function when the model bias is low compared to prior model-free algorithms like CQL Kumar et al.
Thus, as a consequence of optimizing a tighter lower bound, COMBO has the potential to learn higher rewarding policies compared to prior model-free algorithms.
This is illustrated through an example in Figure 1.
Finally, in our experiments, we ﬁnd that COMBO matches or exceeds the state-of-the-art results in commonly studied benchmark tasks for ofﬂine RL.
Speciﬁcally, COMBO achieves the highest score in 9 out of 12 continuous control domains we consider from the D4RL Fu et al.
(2020) benchmark suite, while the next best algorithm achieves the highest score in only 3 out of the 12 domains.
We also show that COMBO achieves the best performance in tasks that require out-of-distribution generalization and outperforms previous latent-space ofﬂine model-based RL methods in the image-based robotic manipulation task.
P RELIMINARIES 2. M ARKOV D ECISION P ROCESSES AND O FFLINE RL
We study RL in the framework of Markov decision processes (MDPs) speciﬁed by the tuple M =
( S , A , T, r, µ 0 , γ ) .
S , A denote the state and action spaces.
T ( s (cid:48) | s , a ) and r ( s , a ) ∈ [ − R max , R max ] represent the dynamics and reward function respectively.
µ 0 ( s ) denotes the initial state distribution, and γ ∈ (0 , 1) denotes the discount factor.
We denote the discounted state visitation distribution of a policy π using d π M ( s ) :
= (1 − γ ) (cid:80) ∞
t =0 γ t P ( s t = s | π ) , where P ( s t = s | π ) is the probability of reaching state s at time t by rolling out π in M .
Similarly, we denote the state-action visitation distribution with d π M ( s , a ) :
= d π M ( s ) π ( a | s ) .
The goal of RL is to learn a policy that maximizes the return, or long term cumulative rewards:
[Equation]
Ofﬂine RL is the setting where we have access only to a ﬁxed dataset D
= { ( s , a , r, s (cid:48) ) } , which consists of transition tuples from trajectories collected using a behavior policy π β .
In other words, the dataset D is sampled from d π β ( s , a ) :
= d π β ( s ) π β ( a | s ) .
We deﬁne M as the empirical MDP induced by the dataset D and d ( s , a ) as sampled-based version of d π β ( s , a ) .
In the ofﬂine setting, even in the limit of an inﬁnite size dataset, it may not be possible to ﬁnd the optimal policy for the underlying
MDP Chen & Jiang (2019); Kidambi et al.
Thus, we typically forgo the goal of ﬁnding the optimal policy, and instead aim to ﬁnd the best possible policy using the ﬁxed ofﬂine dataset.
2. M ODEL -F REE O FFLINE RL A LGORITHMS
One class of approaches for solving MDPs involves the use of dynamic programming and actor-critic schemes Sutton & Barto (1998); Bertsekas & Tsitsiklis (1996), which do not explicitly require the learning of a dynamics model.
To capture the long term behavior of a policy without a model, we deﬁne the action value function as
[Equation]
where future actions are sampled from π ( ·| s ) and state transitions happen according to the MDP dynamics.
Consider the following Bellman operator:
[Equation]
and its sample based counterpart:
[Equation]
associated with a single transition ( s , a , s (cid:48) ) and a (cid:48) ∼ π ( ·| s (cid:48) ) .
The action-value function satisﬁes the Bellman consistency criterion given by B π Q π ( s , a ) =
Q π ( s , a ) ∀ ( s , a ) .
When given an ofﬂine dataset D , standard approximate dynamic programming (ADP) and actor-critic methods use this criterion to alternate between policy evaluation Munos & Szepesvari (2008) and policy improvement.
A number of prior works have observed that such a direct extension of ADP and actor-critic schemes to ofﬂine RL leads to poor results due to distribution shift over the course of learning and over-estimation bias in the Q function Fujimoto et al.
(2018a)
; Kumar et al.
; Wu et al.
To address these drawbacks, prior works have proposed a number of modiﬁcations aimed towards regularizing the policy or value function (see Section 5).
In this work, we primarily focus on CQL Kumar et al.
(2020), which alternates between the two following steps.
Policy Evaluation: The Q function associated with the current policy π is approximated conservatively by repeating the following optimization:
[Equation]
(3) where µ ( ·| s ) is a wide sampling distribution such as the uniform distribution over action bounds.
CQL effectively penalizes the Q function at states in the dataset for actions not observed in the dataset.
This enables a conservative estimation of the value function for any policy Kumar et al.
(2020), mitigating the challenges of over-estimation bias and distribution shift.
Policy Improvement: After approximating the Q function as ˆ Q π , the policy is improved as
[Equation]
Actor-critic schemes with parameterized policies and Q functions approximate the arg max and arg min in above equations with a few steps of gradient descent.
2. M ODEL -B ASED O FFLINE RL A LGORITHMS
A second class of algorithms for solving MDPs involve the learning of the dynamics function, and using the learned model to aid policy search.
Using the given dataset D , a dynamics model (cid:98) T is typically trained using maximum likelihood estimation as: min (cid:98) T E ( s , a , s (cid:48) )
∼D (cid:104) log (cid:98) T ( s (cid:48) | s , a ) (cid:105) .
A reward model ˆ r ( s , a ) can also be learned similarly if it is unknown.
Once a model has been learned, we can construct the learned MDP (cid:99) M
=
( S , A , (cid:98) T, ˆ r, µ 0 , γ ) , which has the same state and action spaces, but uses the learned dynamics and reward function.
Subsequently, any policy learning or planning algorithm can be used to recover the optimal policy in the model as ˆ π = arg max π J ( (cid:99) M , π ) .
This straightforward approach is known to fail in the ofﬂine RL setting, both in theory and practice, due to distribution shift and model-bias Ross & Bagnell (2012);
Kidambi et al.
In order to overcome these challenges, ofﬂine model-based algorithms like MOReL Kidambi et al.
and
MOPO Yu et al.
(2020c) use uncertainty quantiﬁcation to construct a lower bound for policy performance and optimize this lower bound.
By using an uncertainty estimation algorithm like bootstrap ensembles Osband et al.
; Azizzadenesheli et al.
; Lowrey et al.
(2019), we can obtain u ( s , a ) , an estimate of uncertainty in dynamics model prediction.
In the case of
MOPO, an uncertainty penalized MDP is constructed where the reward is given by (cid:101) r ( s , a )
=
ˆ r
( s , a ) − λu ( s , a ) , and the learned dynamics model is used without modiﬁcation.
MOPO learns a policy in this “uncertainty-penalized” MDP (cid:102)
M = ( S , A , (cid:98) T, (cid:101) r, µ 0 , γ ) which has the property that
J ( (cid:102) M , π ) ≤ J ( M , π ) ∀ π .
By constructing and optimizing such a lower bound, ofﬂine model-based
RL algorithms avoid the aforementioned pitfalls like model-bias and distribution shift.
While any
RL or planning algorithm can be used to learn the optimal policy for (cid:102) M , we focus speciﬁcally on
MBPO Janner et al.
; Sutton (1991) which was used in MOPO.
MBPO follows the standard structure of actor-critic algorithms, but in each iteration uses an augmented dataset D ∪ D model for policy evaluation.
Here, D is the ofﬂine dataset and D model is a dataset obtained by simulating the current policy using the learned dynamics model.
Speciﬁcally, at each iteration, MBPO performs k -step rollouts using (cid:98) T starting from state s ∈ D with a particular rollout policy µ ( a | s ) , adds the model-generated data to D model , and optimizes the policy with a batch of data sampled from
D ∪ D model where each datapoint in the batch is drawn from D with probability f ∈ [0 , 1] and D model with probability 1 − f .
C ONSERVATIVE O FFLINE M ODEL -B ASED P OLICY O PTIMIZATION
The principal challenge in practice with prior ofﬂine model-based algorithms (discussed in Section 2) is the strong reliance on uncertainty quantiﬁcation, which can be challenging for complex datasets or deep neural network models Ovadia et al.
We also empirically verify this in the ofﬂine dynamics model learning setting in Appendix
E. Our goal is to develop a model-based ofﬂine RL algorithm that enables optimizing a lower bound on the policy performance, but without requiring uncertainty quantiﬁcation.
We achieve this by extending conservative Q-learning Kumar et al.
(2020), which does not require explicit uncertainty quantiﬁcation, into the model-based setting.
Our algorithm, summarized in Algorithm 1 in Appendix A, alternates between a conservative policy evaluation step and a policy improvement step.
We also provide theoretical analysis of COMBO in Appendix B.
Conservative Policy Evaluation: Given a policy π , an ofﬂine dataset D , and a learned model of the
MDP ˆ M , the goal in this step is to obtain a conservative estimate of Q π .
To achieve this, we penalize the Q-values evaluated on data drawn from a particular state-action distribution that is more likely to be out-of-support while pushing up the Q-values on state-action pairs that are trustworthy, which is implemented by repeating the following recursion:
[Equation]
Here, ρ ( s , a ) and d f are sampling distributions that we can choose.
Model-based algorithms allow ample ﬂexibility for these choices while providing the ability to control the bias introduced by these choices.
For ρ ( s , a ) , we make the following choice:
[Equation]
where d π (cid:99) M ( s ) is the discounted marginal state distribution when executing π in the learned model (cid:99) M .
Samples from d π (cid:99) M ( s ) can be obtained by rolling out π in (cid:99) M .
Similarly, d f is an f − interpolation between the ofﬂine dataset and synthetic rollouts from the model:
[Equation]
where f ∈ [0 , 1] is the ratio of the datapoints drawn from the ofﬂine dataset as deﬁned in Section 2. and µ ( ·| s ) is the rollout distribution used with the model, which can be modeled as π or a uniform distribution.
To avoid notation clutter, we also denote d f := d µf .
Under such choices of ρ and d f , we push down (or conservatively estimate) Q-values on state-action tuples from model rollouts and push up Q-values on the real state-action pairs from the ofﬂine dataset.
When updating Q-values with the Bellman backup, we use a mixture of both the model-generated data and the real data, similar to Dyna Sutton (1991).
Note that in comparison to CQL and other model-free algorithms, COMBO learns the Q-function over a richer set of states beyond the states in the ofﬂine dataset.
This is made possible by performing rollouts under the learned dynamics model, denoted by d µ (cid:99) M
( s , a ) .
We will show in Appendix B that the Q function learned by repeating the recursion in Equation 4 provides a lower bound on the true Q function, without the need for explicit uncertainty estimation.
Furthermore, we will theoretically study the advantages of using synthetic data from the learned model, and characterize the impacts of model bias.
Policy Improvement Using a Conservative Critic: After learning a conservative critic ˆ Q π , we improve the policy as:
[Equation]
where ρ ( s ) is the state marginal of ρ ( s , a ) .
When policies are parameterized with neural networks, we approximate the arg max with a few steps of gradient descent.
In addition, entropy regularization can also be used to prevent the policy from becoming degenerate if required Haarnoja et al.
In Appendix B.2, we show that the resulting policy is guaranteed to improve over the behavior policy.
Practical Implementation Details.
Our practical implementation largely follows MOPO, with the key exception that we perform conservative policy evaluation as outlined in this section, rather than using uncertainty-based reward penalties.
Following MOPO, we represent the probabilistic dynamics model using a neural network, with parameters θ , that produces a Gaussian distribution over the next state and reward: (cid:98) T θ ( s t +1 , r | s , a ) =
N ( µ θ ( s t , a t ) , Σ θ ( s t , a t )) .
The model is trained via maximum likelihood.
For training the conservative critic, which is the major distinction between COMBO and
MOPO, the ﬁxed constant β is tuned with an ofﬂine cross-validation scheme for all low-dimensional continuous control tasks and is decided with a limited number of rollouts in the actual environment in the vision-based environments.
We set the ratio f = 0 .
5 to have an equal split between model rollouts and data from the ofﬂine dataset.
For conservative policy evaluation (eq. 4) and policy improvement (eq. 5), we augment ρ with states sampled from the ofﬂine dataset, which shows more stable improvement in practice.
Additional details about practical implementation are provided in
Appendix D.1.
E XPERIMENTS
In our experiments, we aim to answer the follow questions: (1) How does COMBO compare to prior ofﬂine model-free and model-based methods in standard ofﬂine RL benchmarks?
(2) Can COMBO generalize better than previous approaches in a setting that requires generalization to tasks that are different from what the behavior policy solves?
(3) How does COMBO compare with prior work in tasks with high-dimensional image observations?
To answer those questions, we compare COMBO to several prior methods.
In the domains with compact state spaces, we compare with recent model-free algorithms like BEAR (Kumar et al., 2019),
BRAC (Wu et al., 2019), and CQL (Kumar et al., 2020); as well as MOPO (Yu et al., 2020c) which is a model-based algorithm.
In addition, we also compare with an ofﬂine version of SAC (Haarnoja et al., 2018)
(denoted as SAC-off), and behavioral cloning (BC).
In high-dimensional image-based domains, which we use to answer question (3), we compare to LOMPO (Rafailov et al., 2020), which is a latent space ofﬂine model-based RL method that handles image inputs, latent space MBPO (denoted
[Table]
LMBPO), similar to Janner et al.
(2019) which uses the model to generate additional synthetic data, the fully ofﬂine version of SLAC (Lee et al., 2020) (denoted SLAC-off), which only uses a variational model for state representation purposes, and CQL from image inputs.
To our knowledge, CQL,
MOPO, and LOMPO are representative of state-of-the-art model-free and model-based ofﬂine RL methods.
Hence we choose them as comparisons to COMBO.
For more details of our experimental set-up, comparisons, and hyperparameters, see Appendix D.
4. R ESULTS ON THE D4RL BENCHMARK
To answer the question (1), we evaluate COMBO on the OpenAI Gym (Brockman et al., 2016) domains in the D4RL benchmark (Fu et al., 2020), which contains three environments (halfcheetah, hopper, and walker2d) and four dataset types (random, medium, medium-replay, and medium-expert).
We include the results in Table 1.
The numbers of BC, SAC-off, BEAR, BRAC-P and BRAC-v are taken from the D4RL paper, while the results for MOPO and CQL are based on their respective papers (Yu et al., 2020c; Kumar et al., 2020).
COMBO achieves the best performance in 9 out of 12 settings while attaining similar performance to the best-performing method in the remainingsettings.
As noted by Yu et al.
(2020c) and Rafailov et al.
(2020), model-based ofﬂine methods are generally more performant on datasets that are collected by a wide range of policies and have diverse state-action distributions (random, medium-replay datasets) while model-free approaches do better on datasets with narrow distributions (medium, medium-expert datasets).
However, in these results, COMBO outperforms or performs comparably to the best method among existing model-free and model-based approaches, suggesting that COMBO is robust to different dataset types.
Such results can be explained by COMBO being less conservative compared to prior model-free ofﬂine methods and enjoying lower worst-case suboptimality when the learned model is inaccurate compared to previous model-based ofﬂine approaches as shown in Appendix B. COMBO also does not rely on the heuristics of uncertainty estimation as in prior model-based ofﬂine RL methods, which also potentially leads to COMBO’s superior performance in various dataset types since uncertainty estimation is particularly challenging in settings where the learned model is not precise.
We also empirically show that the heuristics of uncertainty estimation used in prior model-based ofﬂine RL methods are inaccurate on the medium datasets in D4RL in Appendix E and might be the major reason of the poor results of prior model-based approaches on those datasets, which further corroborates the importance of removing uncertainty estimation in model-based ofﬂine RL.
4. R ESULTS ON TASKS THAT REQUIRE GENERALIZATION
To answer question (2), we use the two environments halfcheetah-jump and ant-angle constructed in Yu et al.
(2020c), which requires the agent to solve a task that is different from what the behavior policy solved.
In both environments, the ofﬂine dataset is collected by policies trained with the original reward functions of halfcheetah and ant , which reward the halfcheetah and the ant to run as fast as possible.
The behavior policies are trained with SAC with 1M steps and we take the full replay buffer as the ofﬂine dataset.
Following Yu et al.
(2020c)
, we relabel the rewards in the ofﬂine datasets to reward the halfcheetah to jump as high as possible and the ant to run to the top corner with a 30 degree angle as fast as possible.
Following the same manner, we construct a third
[Table]
task sawyer-door-close based on the environment in Yu et al.
(2020b);
Rafailov et al.
In this task, we collect the ofﬂine data with SAC policies trained with a sparse reward function that only gives a reward of 1 when the door is opened by the sawyer robot and 0 otherwise.
The ofﬂine dataset is similar to the “medium-expert“ dataset in the D4RL benchmark since we mix equal amounts of data collected by a fully-trained SAC policy and a partially-trained SAC policy.
We relabel the reward such that it is 1 when the door is closed and 0 otherwise.
Therefore, in these datasets, the ofﬂine RL methods must generalize beyond behaviors in the ofﬂine data in order to learn the intended behaviors.
We visualize the sawyer-door-close environment in the right image in Figure 2.
We present the results on the three tasks in Table 2.
COMBO outperforms MOPO and CQL, two representative model-based and model-free methods respectively, in the halfcheetah-jump and sawyer-door-close tasks, and achieves an approximately 8% and 12% improvement over
MOPO and CQL respectively on the ant-angle task.
These results validate that COMBO achieves better generalization by behaving less conservatively than prior model-free ofﬂine methods and more robustly than prior model-based ofﬂine methods, as shown theoretically in Appendix B.
4. R ESULTS ON IMAGE - BASED TASKS
To answer question (3), we evaluate COMBO on two image-based environments: the standard walker ( walker-walk ) task from the the DeepMind Control suite Tassa et al.
(2018) and a visual door opening environment with a Sawyer robotic arm ( sawyer-door ) as used in Section 4.2.
For the walker task we constructdatasets: medium-replay (M-R), medium (M), mediumexpert (M-E), and expert, similar to Fu et al.
(2020), each consisting of 200 trajectories.
For sawyer-door task we use only the medium-expert and the expert datasets, due to the sparse reward – the agent is rewarded only when it successfully opens the door.
Both environments are visulized in Figure 2.
To extend COMBO to the imagebased setting, we follow Rafailov et al.
and train a recurrent variational model using the ofﬂine data and use train COMBO in the latent space of this model.
We present results in Table 3.
On the walker-walk task, COMBO performs in line with LOMPO and previous methods.
On the more challenging Sawyer task, COMBO matches LOMPO and achieves 100% success rate on the medium-expert dataset, and substantially outperforms all other methods on the narrow expert dataset, achieving an average success rate of 96.7%, when all other model-based and model-free methods fail.
R ELATED W ORK
Ofﬂine RL (Ernst et al., 2005; Riedmiller, 2005; Lange et al., 2012; Levine et al., 2020) is the task of learning policies from a static dataset of past interactions with the environment.
It has found applications in domains including robotic manipulation (Kalashnikov et al., 2018; Mandlekar et al., 2020; Rafailov et al., 2020; Singh et al., 2020), NLP (Jaques et al., 2019; 2020) and healthcare (Shortreed et al., 2011;
Wang et al., 2018).
Similar to interactive RL, both model-free and model-based
[Table]
algorithms have been studied for ofﬂine RL, with explicit or implicit regularization of the learning algorithm playing a major role.
Model-free ofﬂine RL.
Prior model-free ofﬂine RL algorithms have been designed to regularize the learned policy to be “close“ to the behavioral policy either implicitly via regularized variants of importance sampling based algorithms (Precup et al., 2001;
Sutton et al., 2016; Liu et al., 2019;
Swaminathan & Joachims, 2015; Nachum et al., 2019), ofﬂine actor-critic methods (Siegel et al., 2020;
Peng et al., 2019), applying uncertainty quantiﬁcation to the predictions of the Q-values (Agarwal et al., 2020; Kumar et al., 2019; Wu et al., 2019; Levine et al., 2020), and learning conservative
Q-values (Kumar et al., 2020) or explicitly measured by direct state or action constraints Fujimoto et al.
(2018a)
; Liu et al.
, KL divergence (Jaques et al., 2019;
Wu et al., 2019;
Zhou et al., 2020), Wasserstein distance, and MMD (Kumar et al., 2019).
Different from these works, COMBO uses both the ofﬂine dataset as well as simulated data from a learned dynamics model.
This allows
COMBO to behave less conservatively by optimizing a tighter lower bound of policy performance, as well as gain broader generalization when the model bias is small, as demonstrated through our theoretical analysis and experiments.
Model-based ofﬂine RL.
Model-based ofﬂine RL methods (Finn & Levine, 2017; Ebert et al., 2018;
Kahn et al., 2018; Kidambi et al., 2020;
Yu et al., 2020c; Matsushima et al., 2020; Argenson & DulacArnold, 2020; Swazinna et al., 2020; Rafailov et al., 2020; Lee et al., 2021) provide an alternative approach to policy learning that involves the learning of a dynamics model using techniques from supervised learning and generative modeling.
Such methods however rely either on uncertainty quantiﬁcation of the learned dynamics model which can be difﬁcult for deep network models Ovadia et al.
(2019), or on directly constraining the policy towards the behavioral policy similar to model-free algorithms Matsushima et al.
In contrast, COMBO conservatively estimates the value function by penalizing it in out-of-support states generated through model rollouts.
This allows COMBO to retain all beneﬁts of model-based algorithms such as broad generalization, without the constraints of explicit policy regularization or uncertainty quantiﬁcation.
C ONCLUSION
In the paper, we present conservative ofﬂine model-based policy optimization (COMBO), a modelbased ofﬂine RL algorithm that penalizes the Q-values evaluated on out-of-support state-action pairs.
In particular, COMBO removes the need of uncertainty quantiﬁcation as widely used in previous model-based ofﬂine RL works (Kidambi et al., 2020; Yu et al., 2020c), which can be challenging and unreliable with deep neural networks (Ovadia et al., 2019).
Theoretically, we show that COMBO achieves a tighter lower-bound on the true policy value compared to prior model-free ofﬂine RL methods (Kumar et al., 2020) and guarantees a safe policy improvement.
In our empirical study,
COMBO outperforms prior model-based and model-free ofﬂine RL methods in 9 out of 12 datasets in the standard D4RL benchmark and attains comparable performance in rest of 3 datasets.
COMBO also achieves the best generalization performances in 3 tasks that require adaptation to unseen behaviors.
Finally, COMBO is able scale to vision-based tasks and outperforms or obtain comparable results in vision-based locomotion and robotic manipulation tasks.
Despite the advantages of COMBO, there are few challenges left such as the lack of a fully ofﬂine hyperparameter tuning mechanism and an automatically tuned f that takes the model error into account.
We leave them for future work.
Appendices A A LGORITHM O VERVIEW
We present the COMBO algorithm in Algorithm 1.
B T HEORETICAL A NALYSIS OF COMBO
In this section, we theoretically analyze our method and show that it optimizes a lower-bound on the expected return of the learned policy.
This lower bound is close to the actual policy performance (modulo sampling error) when the policy’s state-action marginal distribution is in support of the state-action marginal of the behavior policy and conservatively estimates the performance of a policy otherwise.
By optimizing the policy against this lower bound, COMBO guarantees policy improvement beyond the behavior policy.
Furthermore, the lower-bound of the expected return in the case of COMBO is a tighter lower bound compared to model-free counterparts.
B. COMBO O PTIMIZES A L OWER -B OUND
We ﬁrst show that training the Q-function using Equation 4 obtains a Q-function such that the expected off-policy policy improvement objective (Degris et al., 2012) computed using this learned Q-function lower-bounds its actual value.
We will reuse notation for d f and d from Section 2 and Section 3.
Assuming that the Q-function is represented as a table, the Q-function found by approximate dynamic programming in iteration k , can be obtained by differentiating Equation 4 with respect to Q k (see
Appendix C for details):
[Equation]
Equation 6 effectively applies a penalty that depends on the three distributions appearing in the
COMBO critic training objective (Equation 4), of which ρ and d f are free variables that we choose in practice as discussed in Section 3.
Before stating our main result, we will ﬁrst show that the penalty term in equation 6 is positive in expectation.
Such a positive penalty is important to combat any overestimation that may arise as a result of using (cid:98) B .
Lemma 1 (Interpolation Lemma) .
For any f ∈ [0 , 1] , and any given ρ ( s , a ) ∈ ∆ |S||A| , let d f be an f-interpolation of ρ and D , i.e., d f ( s , a ) :
=
fd ( s , a ) + (1 − f ) ρ ( s , a ) .
For a given iteration k of
Equation 6, deﬁne the expected penalty under ρ ( s , a ) as:
[Equation]
Then ν ( ρ, f ) satisﬁes, (1) ν
( ρ, f ) ≥ 0 , ∀ ρ, f , (2) ν
( ρ, f ) is monotonically increasing in f for a ﬁxed ρ , and (3) ν
( ρ, f )
=
iff ∀ s , a , ρ ( s , a ) =
d ( s , a ) or f = 0 .
The proof can be found in Appendix C.1.
Lemma 1 characterizes ρ ( s , a ) for which COMBO (Equation 4) induces a conservative penalty.
COMBO sets ρ ( s ) =
d π (cid:99) M
( s )
and uses ρ ( a | s )
=
π ( a | s ) , and hence each step of update (Equation 6) penalizes the Q-function making it more conservative.
The total amount of conservatism induced in the Q-function, given by ν ( ρ, f ) , is controlled by the choice of f .
Based on result (2) in Lemma 1, we note that by controlling the amount of real data, f , we can control the amount of conservatism: f = 1 induces the maximum conservatism, and f =induces no conservatism at all.
Next, we will show that the asymptotic Q-function learned by COMBO lower-bounds the actual
Q-function of any policy π with high probability for a large enough β ≥ 0 .
Let M represent the empirical MDP which uses the empirical transition model based on raw data counts.
The Bellman backups over the dataset distribution d f in equation 4 can be interpreted as an f − interpolation of the backup operator in the empirical MDP (denoted by B π M ) and the backup operator under the learned model (cid:99) M (denoted by B π (cid:99) M ).
The empirical backup operator suffers from sampling error, but is unbiased in expectation, whereas the model backup operator induces bias but no sampling error.
We assume that all of these backups enjoy concentration properties with concentration coefﬁcient
C r,T,δ , dependent on the desired conﬁdence value δ (details in Appendix C.2).
This is a standard assumption in literature (Laroche et al., 2019).
Now, we state our main results below.
Theorem 2 (Asymptotic lower-bound) .
Let P π denote the Hadamard product of the dynamics P and a given policy π in the actual MDP and let S π :=
( I − γP π ) − 1 c S , where c S is a suitably chosen positive constant.
Let D denote the total-variation divergence between two probability distributions.
For any π ( a | s ) , the Q-function obtained by recursively applying Equation 6, with ˆ B π = f B π M + (1 − f ) B π (cid:99) M , with probability at least 1 − δ , results in ˆ Q π that satisﬁes:
[Equation]
where, (cid:15) s , (cid:15) c and (cid:15) m are given by:
[Equation]
The proof for Theorem 2 can be found in Appendix C.2.
Corollary 3.
For a sufﬁciently large β , we have that E s ∼ µ 0 , a ∼ π ( ·| s )
[ ˆ Q π ( s , a )] ≤
E s ∼ µ 0 , a ∼ π ( ·| s )
[ Q π ( s , a )] , where µ 0 ( s ) is the initial state distribution.
Furthermore, when (cid:15) s is small, such as in the large sample regime; or when the model bias (cid:15) m is small, a small β is sufﬁcient along with an appropriate choice of f .
Corollary 3 directly appeals to Theorem 2 and Lemma 1.
Theorem 2 further implies that for large β , deviating away from the behavior policy is expected to lead to smaller values.
Finally, while
Kumar et al.
(2020) also analyze how regularized value function training can provide lower bounds on the value function at each state in the dataset (Kumar et al., 2020)
(Theorem 3.1-3.2), our result goes further, showing that we can handle unseen states that were not seen in the dataset by virtue of using a learned dynamics model, and more importantly, attains a tighter lower bound: COMBO does not underestimate the value function at every state in the dataset like Kumar et al.
(2020), but only lower-bounds the expected value function under the initial state distribution as discussed in
Corollary 3.
We elaborate on how COMBO is less conservative as it attains tighter lower bounds in
Remark 8 (Appendix C.2).
B. S AFE P OLICY I MPROVEMENT G UARANTEES
Now that we have shown that learned Q-function penalizes deviation from the behavior policy, we provide policy improvement guarantees for the COMBO algorithm.
Formally, Theorem 4 discuss safe improvement guarantees over the behavior policy.
building on prior work (Petrik et al., 2016;
Laroche et al., 2019; Kumar et al., 2020).
Theorem 4 ( ζ -safe policy improvement) .
Let ˆ π out ( a | s ) be the policy obtained by COMBO.
Then, the policy ˆ π out ( a | s ) is a ζ -safe policy improvement over π β in the actual MDP M , i.e., J (ˆ π out , M ) ≥
J ( π β , M ) − ζ , with probability at least 1 − δ , where ζ is given by,
[Equation]
The complete statement (with constants and terms that grow smaller than quadratic in the horizon) and proof for Theorem 4 is provided in Appendix C.3.
D CQL denotes a notion of probabilistic distance between policies (Kumar et al., 2020) which we discuss further in Appendix C.3.
The expression for ζ in Theorem 4 consists of three terms: term (1) captures the decrease in the policy performance due to limited data, and decays as the size of D increases.
The second term (2) captures the suboptimality induced by the bias in the learned model.
Finally, as we show in Appendix C.3, the third term (3) is equivalent to the improvement in policy performance as a result of running COMBO in the empirical and model MDPs.
Since the learned model is trained on the dataset D with transitions generated from the behavior policy π β , the marginal distribution ρ β ( s , a ) is expected to be closer to d ( s , a ) for π β as compared to the counterpart for the learned policy, ρ π .
Thus, term (3) is expected to be positive in practice, and in such cases, an appropriate (large) choice of β will make term (3) large enough to counteract terms (1) and (2) that reduce policy performance.
We discuss this elaborately in
Appendix C.3 (Remark 11).
Further note that in contrast to Theorem 3.6 in Kumar et al.
(2020), note that our result indicates the sampling error (term (1)) is signiﬁcantly reduced (multiplied by a fraction f ) when a near-accurate model is used to augment data for training the Q-function, and in contrast to prior model-based ofﬂine RL results Kidambi et al.
Yu et al.
(2020c), our bound implies that COMBO will incur lower worst-case suboptimality in cases when the learned dynamics model is inaccurate by biasing the backups towards the empirical MDP via f .
To summarize, through an appropriate choice of f ,
Theorem 4 guarantees safe improvement over the behavior policy without requiring access to an oracle uncertainty estimation algorithm.
C P ROOFS FROM A PPENDIX B
In this section, we provide proofs for theoretical results in Appendix B. Before the proofs, we note that all statements are proven in the case of ﬁnite state space (i.e., |S| < ∞ ) and ﬁnite action space (i.e., |A| < ∞ )
we deﬁne some commonly appearing notation symbols appearing in the proof:
• P M and r M (or P and r with no subscript for notational simplicity) denote the dynamics and reward function of the actual MDP M • P M and r M denote the dynamics and reward of the empirical MDP M generated from the transitions in the dataset • P (cid:99) M and r (cid:99)
M denote the dynamics and reward of the MDP induced by the learned model (cid:99)
M
We also assume that whenever the cardinality of a particular state or state-action pair in the ofﬂine dataset D , denoted by |D ( s , a ) | , appears in the denominator, we assume it is non-zero.
For any non-existent ( s , a ) / ∈ D , we can simply set |D ( s , a ) | to be a small value < 1 , which prevents any bound from producing trivially ∞ values.
C. P ROOF OF L EMMA
Lemma 5 (Lemma 1 restated) .
For any f ∈ [0 , 1] , and any given ρ ( s , a ) ∈ ∆ |S||A| , let d f be an f-interpolation of ρ and D , i.e., d f ( s , a ) :
= fd ( s , a ) + (1 − f ) ρ ( s , a ) .
For a given iteration k of
Equation 6, deﬁne the expected penalty under ρ ( s , a ) as:
[Equation]
Then ν ( ρ, f ) satisﬁes, (1) ν
( ρ, f ) ≥ 0 , ∀ ρ, f , (2) ν
( ρ, f ) is monotonically increasing in f for a ﬁxed ρ , and (3) ν
( ρ, f )
=
iff ∀ s , a , ρ ( s , a ) =
d ( s , a ) or f = 0 .
Proof.
To prove this lemma, we use algebraic manipulation on the expression for quantity ν ( ρ, f ) and show that it is indeed positive and monotonically increasing in f ∈ [0 , 1] .
[Equation]
Since the derivative of ν ( ρ, f ) with respect to f is always positive, it is an increasing function of f for a ﬁxed ρ , and this proves the second part (2) of the Lemma.
Using this property, we can show the part (1) of the Lemma as follows:
[Equation]
Finally, to prove the third part (3) of this Lemma, note that when f = 0 , ν ( ρ, f ) = 0 (as shown above), and similarly by setting ρ ( s , a ) =
d ( s , a ) note that we obtain ν ( ρ, f )
= 0 .
To prove the only if side of (3), assume that f (cid:54) =
and ρ ( s , a ) (cid:54) =
d ( s , a ) and we will show that in this case ν ( ρ, f ) (cid:54)
= 0 .
When d ( s , a ) (cid:54) =
ρ ( s , a ) , the derivative dν ( ρ,f ) df > 0 (i.e., strictly positive) and hence the function ν ( ρ, f ) is a strictly increasing function of f .
Thus, in this case, ν ( ρ, f ) >
= ν ( ρ, 0) ∀ f > 0 .
Thus we have shown that if ρ ( s , a ) (cid:54) = d ( s , a ) and f > 0 , ν ( ρ, f ) (cid:54) = 0 , which completes our proof for the only if side of (3).
C. P ROOF OF T HEOREM 2 AND C OROLLARY
Before proving this theorem, we provide a bound on the Bellman backup in the empirical MDP,
B M .
To do so, we formally deﬁne the standard concentration properties of the reward and transition dynamics in the empirical MDP, M , that we assume so as to prove Theorem 1.
Following prior work (Osband & Van Roy, 2017; Jaksch et al., 2010; Kumar et al., 2020), we assume:
Assumption A1.
∀ s , a ∈ M , the following relationships hold with high probability, ≥ 1 − δ | r M ( s , a
[Equation]
( s ,
[Equation]
≤
[Equation]
Under this assumption and assuming that the reward function in the MDP, r ( s , a ) is bounded, as | r ( s , a ) | ≤ R max , we can bound the difference between the empirical Bellman operator, B M and the actual MDP, B M ,
[Equation]
Thus the overestimation due to sampling error in the empirical MDP, M is bounded as a function of a bigger constant, C r,P,δ that can be expressed as a function of C r,δ and C P,δ , and depends on δ via a
(cid:112) log(1 /δ ) dependency.
For the purposes of proving Theorem 2, we assume that:
[Equation]
Next, we provide a bound on the error between the bellman backup induced by the learned dynamics model and the learned reward, B (cid:99) M , and the actual Bellman backup, B M .
To do so, we note that:
[Equation]
where D ( P, P (cid:99) M ) is the total-variation divergence between the learned dynamics model and the actual MDP.
Now, we will use Equations 10 and 11 to prove Theorem 2.
We restate Theorem 2 for convenience.
Theorem 6 (Theorem 2 restated) .
Let P π denote the Hadamard product of the dynamics P and a given policy π in the actual MDP and let S π :=
( I − γP π ) − 1 .
Let D denote the total-variation divergence between two probability distributions.
For any π ( a | s ) , the Q-function obtained by recursively applying Equation 6, with ˆ B π = f B π M + (1 − f ) B π (cid:99) M , with probability at least 1 − δ , results in ˆ Q π that satisﬁes:
[Equation]
Proof.
We ﬁrst note that the Bellman backup ˆ B π induces the following Q-function iterates as per
Equation 6,
[Equation]
Since the RHS upper bounds the Q-function pointwise for each ( s , a ) [Equation] the ﬁxed point of the Bellman iteration process will be pointwise smaller than the ﬁxed point of the Q-function found by solving for the RHS via equality.
Thus [Equation] we get that
[Equation]
which completes the proof of this theorem.
Next, we use the result and proof technique from Theorem 2 to prove Corollary 3, that in expectation under the initial state-distribution, the expected Q-value is indeed a lower-bound.
Corollary 7 (Corollary 3 restated) .
For a sufﬁciently large β , we have a lower-bound that
E s ∼ µ 0 , a ∼ π ( ·| s )
[ ˆ Q π ( s , a )] ≤ E s ∼ µ 0 , a ∼ π ( ·| s )
[ Q π ( s , a )] , where µ 0 ( s ) is the initial state distribution.
Furthermore, when (cid:15) s is small, such as in the large sample regime; or when the model bias (cid:15) m is small, a small β is sufﬁcient along with an appropriate choice of f .
Proof.
To prove this corollary, we note a slightly different variant of Theorem 2.
To observe this, we will deviate from the proof of Theorem 2 slightly and will aim to express the inequality using B (cid:99) M , the Bellman operator deﬁned by the learned model and the reward function.
Denoting ( I − γP (cid:99) M ) −as S π (cid:99) M , doing this will intuitively allow us to obtain β ( µ ( s ) π ( a | s ))
T (cid:16) S π (cid:99) M (cid:104)
ρ − d d f (cid:105)(cid:17) ( s , a ) as the conservative penalty which can be controlled by choosing β appropriately so as to nullify the potential overestimation caused due to other terms.
Formally,
[Equation]
By controlling ∆( s , a ) using the pointwise triangle inequality:
[Equation]
and then iterating the backup B π (cid:99) M to its ﬁxed point and ﬁnally noting that ρ ( s , a ) = (cid:16) ( µ · π ) T S π (cid:99) M (cid:17) ( s , a ) , we obtain:
[Equation]
The terms marked as “terms independent of β ” correspond to the additional positive error terms obtained by iterating (cid:12)(cid:12)(cid:12) B π ˆ Q k − B π (cid:99) M ˆ Q k (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12) B π M ˆ Q k − B π ˆ Q k (cid:12)(cid:12)(cid:12) , which can be bounded similar to the proof of Theorem 2 above.
Now by replacing the model Q-function, E µ,π [ Q π (cid:99) M ( s , a )] with the actual
Q-function, E µ,π
[ Q π ( s , a )] and adding an error term corresponding to model error to the bound, we obtain that:
[Equation]
Hence, by choosing β large enough, we obtain the desired lower bound guarantee.
Remark 8 ( COMBO attains a tighter lower-bound than CQL ) .
Before concluding this section, we discuss how the bound obtained by COMBO (Equation 14) is tighter than CQL.
CQL learns a Q-function such that the value of the policy under the resulting Q-function lower-bounds the true value function at each state s ∈ D individually (in the absence of no sampling error), i.e., ∀ s ∈ D , ˆ V π CQL ( s ) ≤ V π ( s ) , whereas the bound in COMBO is only valid in expectation of the value function over the initial state distribution, i.e., E s ∼ µ 0 ( s )
[ ˆ V π COMBO ( s )] ≤ E s ∼ µ 0 ( s )
[ V π ( s )] , and the value function at a given state may not be a lower-bound.
For instance, COMBO can overestimate the value of a state more frequent in the dataset distribution d ( s , a ) but not so frequent in the ρ ( s , a ) marginal distribution of the policy under the learned model (cid:99) M .
To see this more formally, note that the expected penalty added in the effective Bellman backup performed by COMBO (Equation 6), in expectation under the dataset distribution d ( s , a ) , (cid:101)
ν ( ρ, d, f ) is actually negative :
[Equation]
where the ﬁnal inequality follows via a direct application of the proof of Lemma 1.
Thus, COMBO actually overestimates the values at atleast some states (in the dataset) unlike CQL, making it a tighter lower bound.
C. P ROOF OF T HEOREM
To prove the policy improvement result in Theorem 4, we ﬁrst observe that using Equation 6 for
Bellman backups amounts to ﬁnding a policy that maximizes the return of the policy in the a modiﬁed “f-interpolant” MDP which admits the Bellman backup (cid:98) B π , and is induced by a linear interpolation of backups in the empirical MDP M and the MDP induced by a dynamics model (cid:99) M and the return of a policy π in this effective f-interpolant MDP is denoted by J ( M , (cid:99) M , f, π ) .
Alongside this, the return is penalized by the conservative penalty where ρ π denotes the marginal state-action distribution of policy π in the learned model (cid:99) M .
[Equation]
We will require bounds on the return of a policy π in this f-interpolant MDP, J ( M , (cid:99) M , f, π ) , which we ﬁrst prove separately as Lemma 9 below and then move to the proof of Theorem 4.
Lemma 9 (Bound on return in f-interpolant MDP) .
For any two MDPs, M 1 and M 2 , with the same state-space, action-space and discount factor, and for a given fraction f ∈ [0 , 1] , deﬁne the f-interpolant MDP M f as the MDP on the same state-space, action-space and with the same discount as the MDP with dynamics: P M f :=
fP M 1 + (1 − f ) P M 2 and reward function: r M f :
= fr M 1 + (1 − f ) r M 2 .
Then, given any auxiliary MDP, M , the return of any policy π in M f ,
J ( π, M f ) , also denoted by J ( M 1 , M 2 , f, π ) , lies in the interval:
[Equation]
where α is given by:
[Equation]
Proof.
To prove this lemma, we note two general inequalities.
First, note that for a ﬁxed transition dynamics, say P , the return decomposes linearly in the components of the reward as the expected return is linear in the reward function:
[Equation]
As a result, we can bound J ( P, r M f ) using J ( P, r ) for a new reward function r of the auxiliary
MDP, M , as follows
[Equation]
Second, note that for a given reward function, r , but a linear combination of dynamics, the following bound holds:
[Equation]
To observe the third equality, we utilize the result on the difference between returns of a policy π on two different MDPs, P M 1 and P M f from Agarwal et al.
(Chapter 2, Lemma 2.2,
Simulation Lemma), and additionally incorporate the auxiliary MDP M in the expression via addition and subtraction in the previous (second) step.
In the fourth step, we ﬁnally bound one term that corresponds to the learned model via the total-variation divergence D ( P M 2 , P M ) and the other term corresponding to the empirical MDP M is left in its expectation form to be bounded later.
Using the above bounds on return for reward-mixtures and dynamics-mixtures, proving this lemma is straightforward:
[Equation]
where the second step holds via linear decomposition of the return of π in M f with respect to the reward interpolation, and bounding the terms that appear in the reward difference.
For convenience, we refer to these offset terms due to the reward as ∆ R .
For the ﬁnal part of this proof, we bound
J ( P M f , r M ) in terms of the return on the actual MDP, J ( P M , r M ) , using the inequality proved above that provides intervals for mixture dynamics but a ﬁxed reward function.
Thus, the overall bound is given by J ( π, M
[Equation]
, J ( π, M ) + α ] , where α is given by:
[Equation]
This concludes the proof of this lemma.
Finally, we prove Theorem 4 that shows how policy optimization with respect to ˆ J ( f, π ) affects the performance in the actual MD by using Equation 15 and building on the analysis of pure model-free algorithms from Kumar et al.
We restate a more complete statement of the theorem below and present the constants at the end of the proof.
Theorem 10 (Formal version of Theorem 4) .
Let ˆ π out ( a | s ) be the policy obtained by COMBO.
Then, the policy π out ( a | s ) is a ζ -safe policy improvement over π β in the actual MDP M , i.e., J ( π out , M ) ≥
J ( π β , M ) − ζ , with probability at least 1 − δ , where ζ is given by (where ρ β ( s , a ) :=
d π β (cid:99) M ( s , a ) ):
[Equation]
Proof.
We ﬁrst note that since policy improvement is not being performed in the same MDP, M as the f-interpolant MDP, M f , we need to upper and lower bound the amount of improvement occurring in the actual MDP due to the f-interpolant MDP.
As a result our ﬁrst is to relate J ( π, M ) and J ( π, M f ) :
=
J ( M , (cid:99) M , f, π ) for any given policy π .
Step 1:
Bounding the return in the actual MDP due to optimization in the f-interpolant MDP.
By directly applying Lemma 9 stated and proved previously, we obtain the following upper and lower-bounds on the return of a policy π :
[Equation]
where α is shown in Equation 16.
As a result, we just need to bound the terms appearing the expression of α to obtain a bound on the return differences.
We ﬁrst note that the terms in the expression for α are of two types: (1) terms that depend only on the reward function differences (captured in ∆ R in Equation 17), and (2) terms that depend on the dynamics (the other two terms in
Equation 17).
To bound ∆ R , we sipmply appeal to concentration inequalities on reward (Assumption A1), and bound ∆ R as:
[Equation]
Note that both of these terms are of the order of O (1 / (1 − γ )) and hence they don’t ﬁgure in the informal bound in Theorem 4 in the main text, as these are dominated by terms that grow quadratically with the horizon.
To bound the remaining terms in the expression for α , we utilize a result directly from Kumar et al.
(2020) for the empirical MDP, M , which holds for any policy π ( a | s ) , as shown below.
[Equation]
Step 2: Incorporate policy improvement in the f-inrerpolant MDP.
Now we incorporate the improvement of policy π out over the policy π β on a weighted mixture of (cid:99) M and M .
In what follows, we derive a lower-bound on this improvement by using the fact that policy π out is obtained by maximizing ˆ J ( f, π ) from Equation 15.
As a direct consequence of Equation 15, we note that
[Equation]
Following Step
[Equation]
we will use the upper bound on J ( M , (cid:99) M , f, π ) for policy π = π out and a lower-bound on J ( M , (cid:99) M , f, π ) for policy π = π β and obtain the following inequality:
[Equation]
The term marked by ( ∗ ) in the above expression can be upper bounded by the concentration properties of the dynamics as done in Step 1 in this proof:
[Equation]
Finally
[Equation]
using Equation
[Equation]
we can lower-bound the policy return difference as:
[Equation]
Plugging the bounds for terms (a), (b) and (c) in the expression for ζ where J ( π out , M ) − J ( π β , M ) ≥ ζ , we obtain:
[Equation]
[Equation]
Remark 11 ( Interpretation of Theorem 4 ) .
Now we will interpret the theoretical expression for ζ in Equation 20, and discuss the scenarios when it is negative .
When the expression for ζ is negative, the policy π out is an improvement over π β in the original MDP, M
[Equation]
• First note that we have never used the fact that the learned model P (cid:99) M is close to the actual MDP, P M on the states visited by the behavior policy π β in our analysis.
We will use this fact now: in practical scenarios, ν ( ρ β , f ) is expected to be smaller than ν ( ρ π , f ) , since ν ( ρ β , f ) is directly controlled by the difference and density ratio of ρ β ( s , a ) and d ( s , a ) : ν ( ρ β , f ) ≤ [Equation] [Equation]
) =
[Equation]
2 by Lemma 1 which is expected to be small for the behavior policy π β in cases when the behavior policy marginal in the empirical MDP, d π β M ( s , a ) , is broad.
This is a direct consequence of the fact that the learned dynamics integrated with the policy under the learned model: P π β (cid:99) M is closer to its counterpart in the empirical MDP: P π β M for π β .
Note that this is not true for any other policy besides the behavior policy that performs several counterfactual actions in a rollout and deviates from the data.
For such a learned policy π , we incur an extra error which depends on the importance ratio of policy densities, compounded over the horizon and manifests as the D CQL term (similar to Equation 19, or Lemma D.4.1 in Kumar et al.
Thus, in practice, we argue that we are interested in situations where ν ( ρ π , f ) >
ν ( ρ β , f ) , in which case by increasing β , we can make the expression for ζ in Equation 20 negative, allowing for policy improvement
• In addition, note that when f is close to 1, the bound reverts to a standard model-free policy improvement bound and when f is close to 0, the bound reverts to a typical model-based policy improvement bound.
In scenarios with high sampling error (i.e. smaller |D ( s ) | ), if we can learn a good model, i.e., D ( P M , P (cid:99) M ) is small, we can attain policy improvement better than model-free methods by relying on the learned model by setting f closer to 0.
A similar argument can be made in reverse for handling cases when learning an accurate dynamics model is hard.
D E XPERIMENTAL DETAILS
In this section, we include all details of our empirical evaluations of COMBO.
D. P RACTICAL ALGORITHM IMPLEMENTATION DETAILS
Model training.
In the setting where the observation space is low-dimensional, as mentioned in Section 3, we represent the model as a probabilistic neural network that outputs a Gaussian distribution over the next state and reward given the current state and action:
[Equation]
We train an ensemble of 7 such dynamics models following Janner et al.
and pick the bestmodels based on the validation prediction error on a held-out set that contains 1000 transitions in the ofﬂine dataset D .
During model rollouts, we randomly pick one dynamics model from the bestmodels.
Each model in the ensemble is represented as a 4-layer feedforward neural network with 200 hidden units.
For the generalization experiments in Section 4.2, we additionally use a two-head architecture to output the mean and variance after the last hidden layer following Yu et al.
(2020c).
In the image-based setting, we follow Rafailov et al.
and use a variational model with the following components:
[Equation]
Latent
[Equation]
We train the model using the evidence lower bound:
[Equation]
At each step τ we sample a latent forward model (cid:98) T θ τ from a ﬁxed set of K models [ (cid:98) T θ 1 , . . .
, (cid:98) T θ K ] .
For the encoder E θ we use a convolutional neural network with kernel size 4 and stride 2.
For the Walker environment we use 4 layers, while the Door Opening task has 5 layers.
The D θ is a transposed convolutional network with stride 2 and kernel sizes [5 , 5 , 6 , 6] and [5 , 5 , 5 , 6 , 6] respectively.
The inference network has a two-level structure similar to Hafner et al.
with a deterministic path using a GRU cell with 256 units and a stochastic path implemented as a conditional diagonal Gaussian with 128 units.
We only train an ensemble of stochastic forward models, which are also implemented as conditional diagonal Gaussians.
Policy Optimization.
We sample a batch size of 256 transitions for the critic and policy learning.
We set f = 0 . 5 , which means we sample 50% of the batch of transitions from D and another 50% from D model .
The equal split between the ofﬂine data and the model rollouts strikes the balance between conservatism and generalization in our experiments as shown in our experimental results in
Section 4.
We represent the Q-networks and policy as 3-layer feedforward neural networks withhidden units.
For the choice of ρ ( s , a ) in Equation 4, we can obtain the Q-values that lower-bound the true value of the learned policy π by setting ρ ( s , a ) =
d π (cid:99) M
( s ) π
( a | s ) .
However, as discussed in Kumar et al.
(2020), computing π by alternating the full off-policy evaluation for the policy ˆ π k at each iteration k and one step of policy improvement is computationally expensive.
Instead, following Kumar et al.
(2020), we pick a particular distribution ψ ( a | s ) that approximates the the policy that maximizes the
Q-function at the current iteration and set ρ ( s , a ) =
d π (cid:99) M ( s ) ψ ( a | s ) .
We formulate the new objective as follows:
[Equation]
where R ( ψ ) is a regularizer on ψ .
In practice, we pick R ( ψ ) to be the − D KL ( ψ ( a | s ) (cid:107) Unif ( a )) and under such a regularization, the ﬁrst term in Equation 22 corresponds to computing softmax of the Q-values at any state s as follows:
[Equation]
We estimate the log-sum-exp term in Equation 23 by sampling 10 actions at every state s in the batch from a uniform policy Unif ( a ) and the current learned policy π ( a | s ) with importance sampling following Kumar et al.
D. W ORKFLOW WITH COMBO AND H YPERPARAMETER S ELECTION
In this section, we discuss the hyperparameters associated with COMBO along with guidelines to choose hyperparameters for ofﬂine RL tasks with a lot of variety in terms of dataset-diversity, input modality, etc.
In the D4RL and generalization experiments, our method are built upon the implementation of MOPO provided at: https://github.com/tianheyu927/mopo .
The hyperparameters used in COMBO that relates to the backbone RL algorithm SAC such as twin
Q-functions and number of gradient steps follow from those used in MOPO with the exception of smaller critic and policy learning rates, which we will discuss below.
In the image-based domains,
COMBO is built upon LOMPO without any changes to the parameters used there.
For the evaluation of COMBO, we follow the evaluation protocol in D4RL (Fu et al., 2020) and report the normalized score of the smooth undiscounted averaged return over 3 random seeds for all environments except sawyer-door-close and sawyer-door where we report the average success rate overrandom seeds.
We now list the additional hyperparameters and our decisions behind these (which can serve as guidelines/rules for hyperparameter selection when COMBO is used on a new task) as follows.
• Rollout length h .
We perform a short-horizon model rollouts in COMBO similar to Yu et al.
(2020c) and Rafailov et al.
For the D4RL experiments and generalization experiments, we followed the defaults used in MOPO and used h = 1 for walker2d and sawyer-door-close , h = 5 for hopper, halfcheetah and halfcheetah-jump , and h = 25 for ant-angle .
In the image-based domain we used rollout length of h = 5 for both the the walker-walk and sawyer-door-open environments following the same hyperparameters used in Rafailov et al.
• Q-function and policy learning rates.
According to Kumar et al.
(2020), we used smaller learning rates for the policy as compared to the Q-function and we found that 3 e − 4 for the Q-function learning rate (also used previously in Kumar et al.
and 1 e − 4 for the policy learning rate (also recommended previously in Kumar et al.
(2020) for gym domains) work well for almost all domains except that on walker2d where a smaller Q-function learning rate of 1 e − 4 and a correspondingly smaller policy learning rate of 1 e − 5 works the best.
In the image-based domains, we followed the defaults from prior work (Rafailov et al., 2020) and used 3 e − 4 for both the policy and Q-function.
• Conservative coefﬁcient β .
As noted in our theoretical results in Lemma 1, the amount of conservatism depends on the choice of fraction f
and ρ ( s , a ) .
In principle, we only need to control one of these factors, ρ , f , β to obtain the right degree of conservatism.
Since we do not alter f and ρ ( s , a ) for different quality datasets (see Appendix D.1 for our choice of f ; ρ was chosen based on model-prediction error as discussed next)
we instead choose values of β for different dataset types.
We ﬁxed β to take one of three values from a default set { 0 . 5 , 1 . 0 , 5 . 0 } , which correspond to low conservatism, medium conservatism and high conservatism.
A larger β would be desirable in more narrow dataset distributions with lower-coverage of the state-action space that propagates error in a backup whereas a smaller β is desirable with diverse dataset distributions.
On the D4RL experiments, we found that β
= 0 .
5 works well for halfcheetah agnostic of dataset quality, while on hopper and walker2d, we found that the more “narrow” dataset distributions: medium and mediumexpert datasets work best with larger β = 5 .
whereas more “diverse” dataset distributions: random and medium-replay datasets work best with smaller β ( β = 0 .
5 for walker2d and β = 1 . 0 for hopper) which is consistent with the intuition.
On generalization experiments, β = 1 . 0 works best for all environments.
In the image-domains we used β = 0 .
5 for the medium-replay walker-walk task and and β = 1 .
for all other domains, which again is in accordance with the impact of β on performance.
• Choice of ρ ( s , a ) .
We ﬁrst decouple ρ ( s , a )
=
ρ ( s ) ρ ( a | s ) for convenience.
As discussed in Appendix D.1, we use ρ ( a | s ) as the soft-maximum of the Q-values and estimated with log-sum-exp .
For ρ ( s ) , we use d π (cid:99) M for the hopper task in D4RL and d f for the rest of the environments.
As consistent with Lemma 1, we found that COMBO works well with ρ ( s ) =
d π (cid:99) M in settings where the learned model is accurate (as measured by model prediction error on a held-out validation dataset), which is the case in hopper.
For the remaining domains, we used ρ ( s ) =
d f . • Choice of µ ( a | s ) .
For the rollout policy µ , we use µ ( a | s ) =
Unif ( a ) for the hopper task in D4RL since the dynamics model is most accurate on this task (as measured by modelprediction error on a held-out validation subset of the data) which enables augmenting the dataset with uniform-at-random actions without introducing much model bias into learning, and also in the ant-angle generalization experiment.
For the remaining state-based environments, we use the default µ ( a | s ) =
π ( a | s ) which is consistent with our theory (Lemma 1).
In the image-based domain, we used µ ( a | s ) =
Unif ( a ) in the walker-walk domain and µ ( a | s ) =
π ( a | s ) for the sawyer-door environment.
We found that µ ( a | s ) =
Unif ( a ) behaves less conservatively and is suitable to tasks where dynamics models can be learned fairly precisely.
• Choice of Backup.
Following CQL (Kumar et al., 2020), we use the standard deterministic backup for COMBO.
D. D ETAILS OF GENERALIZATION ENVIRONMENTS
For halfcheetah-jump and ant-angle , we follow the same environment used in MOPO.
For sawyer-door-close , we train the sawyer-door environment in https://github. com/rlworkgroup/metaworld with dense rewards for opening the door until convergence.
We collect 50000 transitions with half of the data collected by the ﬁnal expert policy and a policy that reaches the performance of about half the expert level performance.
We relabel the reward such that the reward is 1 when the door is fully closed and 0 otherwise.
Hence, the ofﬂine RL agent is required to learn the behavior that is different from the behavior policy in a sparse reward setting.
D. D ETAILS OF IMAGE - BASED ENVIRONMENTS
We use the standard walker-walk environment from Tassa et al.
(2018) with 64 × 64 pixel observations and an action repeat of 2.
Datasets were constructed the same way as Fu et al.
(2020) with 200 trajectories each.
For the sawyer-door we use 128 × 128 pixel observations.
The medium-expert dataset contains 1000 rollouts (with a rollout length of 50 steps) covering the state distribution from grasping the door handle to opening the door.
The expert dataset containstrajectories samples from a fully trained (stochastic) policy.
The data was obtained from the training process of a stochastic SAC policy using dense reward function as deﬁned in Yu et al.
(2020b).
However, we relabel the rewards, so an agent receives a reward of 1 when the door is fully open and 0 otherwise.
This aims to evaluate ofﬂine-RL performance in a sparse-reward setting.
D. C OMPUTATION C OMPLEXITY
For the D4RL and generalization experiments, COMBO is trained on a single NVIDIA GeForce
RTX 2080 Ti for one day.
For the image-based experiments, we utilized a single NVIDIA GeForce
RTX 2070.
We trained the walker-walk tasks for a day and the sawyer-door-open tasks for about two days.
E E MPIRICAL E VIDENCE ON C HALLENGES OF U NCERTAINTY Q UANTIFICATION
In this section, we perform empirical evaluations to show that uncertainty quantiﬁcation with deep neural networks, especially in the setting of dynamics model learning, is challenging and could cause problems with uncertainty-based model-based ofﬂine RL methods such as MOReL (Kidambi et al., 2020) and MOPO (Yu et al., 2020c).
In our evaluations, we consider two uncertainty quantiﬁcation methods, maximum learned variance over the ensemble (denoted as Max Var ) max i =1 ,...,N (cid:107) Σ iθ ( s , a ) (cid:107) F (used in MOPO) and the variance of the model prediction over the ensemble (denoted as Ens.
Var ) max i =1 ,...,N (cid:107) µ iθ ( s , a ) − 1 N (cid:80) Nj =1 µ jθ ( s , a ) (cid:107) 2 (used in MOPO and MOReL) where we use an ensemble of N probabilistic dynamics models { (cid:98) T iθ ( s t +1 , r | s , a ) =
N ( µ iθ ( s t , a t ) , Σ iθ ( s t , a t )) }
Ni =1 .
As shown in Table 1, MOPO performs underwhelmingly on medium datasets in the D4RL datasets where the dataset is collected with a single policy and hence with relatively narrow data coverage of the whole state space.
To empirically analyze the poor performance of MOPO on those datasets, we visualize the correlation between the true model error and two uncertainty quantiﬁcation methods
Max Var and Ens.
Var .
We normalize both the model error and the uncertainty estimates to be within scale [0 , 1] .
As shown in Figure 3, on all three medium datasets, Max Var tends to be overly conservative and Ens.
Var behaves too optimistic to correctly quantify the true model error, suggesting that uncertainty estimation used by MOPO is not accurate and might be the major factor that results in its poor performance.
Meanwhile, COMBO circumvents challenging uncertainty quantiﬁcation problem and achieves much better performances on those medium datasets, indicating the effectiveness and the robustness of the method.