R EVISITING D ESIGN C HOICES IN O FFLINE M ODEL -B ASED R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Ofﬂine reinforcement learning enables agents to leverage large pre-collected datasets of environment transitions to learn control policies, circumventing the need for potentially expensive or unsafe online data collection.
Signiﬁcant progress has been made recently in ofﬂine model-based reinforcement learning, approaches which leverage a learned dynamics model.
This typically involves constructing a probabilistic model, and using the model uncertainty to penalize rewards where there is insufﬁcient data, solving for a pessimistic MDP that lower bounds the true MDP.
Existing methods, however, exhibit a breakdown between theory and practice, whereby pessimistic return ought to be bounded by the total variation distance of the model from the true dynamics, but is instead implemented through a penalty based on estimated model uncertainty .
This has spawned a variety of uncertainty heuristics, with little to no comparison between differing approaches.
In this paper, we compare these heuristics, and design novel protocols to investigate their interaction with other hyperparameters, such as the number of models, or imaginary rollout horizon.
Using these insights, we show that selecting these key hyperparameters using Bayesian Optimization produces superior conﬁgurations that are vastly different to those currently used in existing hand-tuned state-of-the-art methods, and result in drastically stronger performance.
I NTRODUCTION
In ofﬂine (or batch) reinforcement learning (RL) (Ernst et al., 2005; Levine et al., 2020), the goal is to leverage ofﬂine datasets of transitions in an environment to train a policy that transfers to an online task.
This could have vast implications for using RL in real-world settings, as agents can make use of ever-increasing amounts of data without the need for an accurate simulator, while also avoiding expensive and potentially even unsafe exploration in the environment.
Model-based reinforcement learning (MBRL) has recently shown promise in this paradigm, obtaining state-of-the-art performance on ofﬂine RL benchmarks (Kidambi et al., 2020;
Yu et al., 2021), improving upon powerful model-free approaches (e.g. Kumar et al.
MBRL works by training a dynamics model from the ofﬂine data, then optimizing a policy using imaginary rollouts from the model.
This allows the agent to learn from on-policy experience, as the model is agnostic to the policy used to generate data, making it possible to achieve high returns using data collected from even a random policy.
Furthermore, recent work has demonstrated the utility of world models beyond maximizing return, such as generalizing to unseen variations in an environment (Ball et al., 2021), transferring to new tasks (Yu et al., 2020), and learning with safety constraints (Argenson &
Dulac-Arnold, 2021).
Therefore, the case for MBRL in ofﬂine RL is clear: not only does it represent state-of-the-art in terms of performance, but it also provides the opportunity to maximize the signal in the ofﬂine data to generalize onto tasks beyond those encoded by the behavior policy.
This is crucial for ofﬂine RL to be useful for real-world tasks (Dulac-Arnold et al., 2021), where there will inevitably be differences between the data and desired task.
However, a common failure mode of MBRL is when policies exploit the model in parts of the state-action space where the model is inaccurate.
Thus, naïve application of MBRL to ofﬂine data can result in suboptimal performance.
To prevent this, concurrent works (Yu et al., 2020; Kidambi et al., 2020) have approached the problem by training a policy in a pessimistic MDP (P-MDP).
The
P-MDP lower bounds the true MDP, and discourages the policy from regions where there is large
[Figure]
True [Figure] discrepancy between the true and learned dynamics; this often provides a theoretical guarantee of improvement over cloning the behavior policy that generated the ofﬂine data.
This is made practically possible by adding a penalty correlated with the uncertainty in the dynamics model.
However, while these recent successes are similar in principle, in practice they differ in a series of design choices.
First and foremost, they make use of different heuristics to measure model uncertainty, in some cases deviating from simpler metrics which are more consistent with the theory.
In this paper we conduct a rigorous investigation into a series of these design choices.
We begin by focusing on the choice of uncertainty metric, comparing both recent state-of-the-art ofﬂine approaches (Kidambi et al., 2020; Yu et al., 2020; Rafailov et al., 2020) with additional metrics used in the online setting (Ball et al., 2020; Pan et al., 2020;
Cowen-Rivers et al., 2020).
We also explore the interaction with a series of other hyperparameters, such as the number of models and imaginary rollout length.
Interestingly, the relationship between these variables and model uncertainty varies signiﬁcantly depending on the choice of uncertainty penalty.
Furthermore, we compare these uncertainty heuristics under new evaluation protocols that, for the ﬁrst time, capture the speciﬁc covariate shift induced by model-based RL.
This allows us to assess calibration to model exploitation in MBRL, observing that some existing penalties are surprisingly successful at capturing the errors in predicted dynamics, as seen in Fig.
1a (see App.
D for details).
Then, using the insights gained from sections 4 and 5, we then achieve a 43% gain over a previously grid-searched method by using a single hyperparameter value across all environments.
To further test these insights, we then ﬁne-tune over our identiﬁed key variables, modeled jointly using a powerful Bayesian Optimization algorithm (Wan et al., 2021).
We ﬁnd that, when properly tuned, the simpler uncertainty measures can provide state-of-the-art results in continuous control ofﬂine benchmarks, and that the chosen optimal hyperparameters continue to align with our analysis.
Finally, we rigorously conﬁrm the aggregate statistical signiﬁcance of our results using the RLiable framework (Agarwal et al., 2021) in Fig.
1b, showing that the improvements over existing methods are signiﬁcant (see App.
H for details).
This work is intended to beneﬁt both researchers and practitioners in ofﬂine RL.
Our main ﬁndings include:
• Longer horizon rollouts with larger penalties can improve existing methods.
Contrary to common wisdom, conducting signiﬁcantly longer rollouts inside the model, coupled with larger uncertainty penalties, typically improves performance.
• Penalties that use canonical forms of uncertainty estimation achieve better correlation with OOD measures.
The deep ensembles approach of Lakshminarayanan et al.
(2017) often outperforms the penalty from state-of-the-art methods (Yu et al., 2020; Kidambi et al., 2020).
We observe that the ensemble standard deviation is statistically strikingly similar to that used in Kidambi et al.
but has improved correlation and scaling behavior.
• Uncertainty is more correlated with dynamics error than distribution shift .
We ﬁnd that successful penalties measure the discrepancy in dynamics, and can in fact assign high certainty to data
far away from the ofﬂine data.
R ELATED W ORK
Two recent works concurrently demonstrated the effectiveness of model-based reinforcement learning (MBRL) in the ofﬂine setting.
MOPO (Yu et al., 2020) follows the successful online RL algorithm
MBPO (Janner et al., 2019) but trains inside a conservative MDP , penalizing the reward based on the maximum aleatoric uncertainty over the ensemble members.
MOReL (Kidambi et al., 2020) achieves even stronger performance, penalizing the rewards by a penalty based on the maximum pair-wise difference in ensemble member predictions.
For pixel-based tasks, LOMPO (Rafailov et al., 2020) also proposed a novel penalty, using the variance of ensemble log-likelihoods.
Outside the ofﬂine setting, probabilistic dynamics models leveraging uncertainty have underpinned a series of successes (Chua et al., 2018;
Kurutach et al., 2018; Buckman et al., 2018; Pan et al., 2020; Pacchiano et al., 2021).
Uncertainty can also be measured in MBRL without the use of neural networks (Deisenroth &
Rasmussen, 2011), although these methods tend to be harder to scale and thus lack widespread use.
Effective hyperparameter selection in RL has been shown to be crucial to the success of popular algorithms (Engstrom et al., 2020; Andrychowicz et al., 2021).
This becomes even more challenging in MBRL with additional hyperparameters/design-choices for the dynamics model.
Recent work has shown that carefully optimizing these hyperparameters for online MBRL can signiﬁcantly improve performance, with the tuned agent breaking the MuJoCo simulator (Zhang et al., 2021).
In contrast, we focus on the ofﬂine setting, and investigate parameters speciﬁcally related to uncertainty estimation.
Previous work studied the impact of hyperparameters in ofﬂine RL (Paine et al., 2020), ﬁnding ofﬂine
RL algorithms to be brittle to hyperparameter choices.
However, unlike our work they only consider model-free approaches, whereas we speciﬁcally investigate model-based ofﬂine algorithms.
Abbas et al.
(2020) investigates the impact of different uncertainty estimation methods in online MBRL; they too ﬁnd penalizing with combined aleatoric and epistemic uncertainty improves performance.
Our work also relates to the rich literature on deep ensembles (Lakshminarayanan et al., 2017), which train multiple deep neural networks with different initializations and dataset orderings, and generally outperform variational Bayes methods (Mackay, 1992; Blundell et al., 2015).
Achieving effective calibration with neural networks is notoriously difﬁcult (Guo et al., 2017; Kuleshov et al., 2018;
Maddox et al., 2019), and furthermore we require calibration under co-variate shift (Ovadia et al., 2019), as the policy learned in the model will likely deviate from the behavior policy that generated the ofﬂine data.
Recent work has highlighted this issue in ofﬂine RL (Kumar et al., 2020; Yu et al., 2021) and has reported superior performance when eschewing model uncertainty entirely, and instead performing “conservative" Q-updates.
However, it is unclear if this improvement is due to poor uncertainty calibration, implementation details, or a limitation in the pessimistic-MDP formulation.
B ACKGROUND
All of the methods we investigate in this paper model the environment as a Markov Decision Process (MDP), deﬁned as a tuple M =
( S , A , P, R, ρ 0 , γ ) , where S and A denote the state and action spaces respectively, P ( s (cid:48) | s, a ) the transition dynamics, R ( s, a ) the reward function, ρ 0 the initial state distribution, and γ ∈ (0 , 1) the discount factor.
The goal is to optimize a policy π ( a | s ) that maximizes the expected discounted return E π,P,ρ
[ (cid:80) ∞ t =
γ t R ( s t , a t )] .
In ofﬂine RL , the policy is not deployed in the environment until test time.
Instead, the algorithm only has access to a static dataset D env = { ( s j , a j , r j , s j +1 ) } Jj =1 , collected by one or more behavioral policies π b .
Following the notation in Yu et al.
we refer to the distribution from which D env was sampled as the behavioral distribution .
The canonical approach in ofﬂine MBRL is to train an ensemble of N probabilistic dynamics models (Nix & Weigend, 1994).
These usually learn to predict both the next state s t +1 and reward r t from a state-action pair, and are trained on D env using supervised learning.
Concretely, each of the N models output a Gaussian (cid:98) P iφ ( s t +1 , r t | s t , a t ) =
N ( µ iφ ( s t , a t ) , Σ iφ ( s t , a t )) parameterized by φ .
The resulting learned dynamics model (cid:98) P and reward model (cid:98) R deﬁne a model MDP (cid:99)
M = ( S , A , (cid:98) P, (cid:98) R, ρ 0 , γ ) .
To train the policy, we use k -step rollouts inside (cid:99) M to generate trajectories (Sutton, 1991).
To prevent policy exploitation in a model, a pessimistic MDP (P-MDP) is constructed by lower bounding the true-expected return, η M ( π ) , using some error between the true and estimated models.
For instance, in Yu et al.
, the authors show that a lower bound on the return can be established by penalizing the reward by a measure that corresponds to estimated model error:
[Equation]
1) where ρ π ˆ P represents transitioning under the dynamics model ˆ P and policy π .
Several potential choices for | G π ˆ M ( s, a ) | are proposed, including an upper bound based on the total variation distance between the learned and true dynamics.
However, for their practical algorithm, the authors elect to use a heuristic based on impressive empirical results.
Concurrent to MOPO, MOReL (Kidambi et al., 2020) in theory constructs a P-MDP by augmenting a standard MDP with a negative valued absorbing state that is transitioned to when total variation distance between true and learned dynamics is exceeded.
They show that a policy learned in this P-MDP exceeds simple behavior cloning.
However, while dynamics-based total variation distance has desirable theoretical properties, the practical algorithm relies on another heuristic to approximate this quantity.
This motivates the study of penalties used, as well as other under-used candidates, and their overall effectiveness.
U NCERTAINTY P ENALTY
The key idea underpinning recent success in ofﬂine MBRL is the introduction of a P-MDP, penalized by some uncertainty penalty.
The theory dictates this should be some distance measure between the true and predicted dynamics.
Of course, this cannot be truly estimated without access to an oracle, so a proxy for this quantity is constructed instead based on uncertainty heuristics.
In this paper, we compare the following uncertainty heuristics, from recent works in both ofﬂine and online MBRL:
Max Aleatoric (Yu et al., 2020): max i =1 ,...,N || Σ iφ ( s, a ) || F , which corresponds to the maximum aleatoric error, computed over the variance heads of the model ensemble.
Max Pairwise Diff (Kidambi et al., 2020): max i,j || µ iφ ( s, a ) − µ jφ ( s, a ) || 2 , which corresponds to the pairwise maximum difference of the ensemble predictions.
LL Var (Log-Likelihood Variance) (Rafailov et al., 2020)):
Var ( { log (cid:98) P iφ ( s (cid:48) | s, a ) , i = 1 , . . .
, N } ) , where s (cid:48) is a next state sampled from a single ensemble member.
We evaluate its log-likelihood under each ensemble member and take the variance.
LOO KL (Leave-One-Out KL Divergence (Pan et al., 2020): D KL [ (cid:98) P φ i ( ·| s, a ) || (cid:98) P φ − i ( ·| s, a )] , which corresponds to the KL divergence between the Gaussian parameterized by a randomly selected ensemble member, and the aggregated Gaussian of the remaining ensemble members.
Ensemble Standard Deviation/Variance (Lakshminarayanan et al., 2017):
The variance is given as: Σ ∗ ( s, a ) =
1 N (cid:80) Ni ((Σ iφ ( s, a ))
+ ( µ iφ ( s, a )) 2 ) − ( µ ∗ ( s, a )) 2 where µ ∗ is the mean of the means ( µ ∗ ( s, a )
= 1 N (cid:80) Ni µ iφ ( s, a ) ).
This corresponds to a combination of epistemic and aleatoric model uncertainty.
This is surprisingly under-utilized in ofﬂine MBRL, and is a canonical method of uncertainty estimation used in the Bayesian inference literature (Ovadia et al., 2019; Filos et al., 2019; Scalia et al., 2020).
We choose to evaluate both standard deviation (the square root of the above) and variance, as this will provide intuition about the importance of penalty distribution shape .
These can all be computed using the output from an ensemble of probabilistic dynamics models (Lakshminarayanan et al., 2017), so we are able to compare them in a controlled manner.
4. H OW W ELL D O E NSEMBLE P ENALTIES D ETECT O UT OF D ISTRIBUTION E RRORS ?
We begin by assessing how well uncertainty penalties correlate with next state MSE.
This is crucial in penalizing the policy from visiting parts of the state-action space where the model is inaccurate, and therefore exploitable.
We use the datasets from D4RL (Fu et al., 2021a), train models on each dataset, then evaluate them on other datasets from the same environment, but collected under different policies.
These form our “Transfer” experiments as they directly measure the ability of uncertainty penalties to detect errors on unseen data.
We compare the penalties against true MSE for a variety of settings in App.
A.2, and show a summary in the “Transfer” column of Table 1.
We measure
Spearman rank ( ρ ) and Pearson bivariate ( r ) correlations, and justify their use in App. A.1.
Full details of all experiments and hyperparameters are given in App.
G. We will analyze these results in detail in the next section, after introducing a novel protocol for assessing our penalties under the out-of-distribution (OOD) data induced by model exploitation.
4. H OW D O T HESE P ERFORM D URING AN I MAGINARY R OLLOUT ?
We additionally design an experiment aimed at capturing the OOD data generated by the actual ofﬂine MBRL process , which we call our “True Model-Based” experiments.
First, we train a set of policies with 4 different starting seeds without a penalty inside the model for 500 iterations.
We then measure the difference between the return predicted by the model over a rollout, and the true return in the real environment.
We deﬁne a policy to be “exploitative” if the model signiﬁcantly [Table] over-estimates
the
[Table]
compared to the true return. It is these exploitative policies that induce the types of extrapolation errors which cause MBRL methods to fail in the ofﬂine setting.
It is therefore important that the penalty is able to accurately determine when the model is being exploited in this way.
We use a subset of the 5 most exploitative policies to generate trajectories in the model, and record the uncertainty predicted by each penalty at every time step.
To generate the True Model-Based data, we then “replay” these trajectories in the true environment, loading the state and action taken in the model into the environment, and record the “true” next state according to the MuJoCo simulator (Todorov et al., 2012).
True Model-Based therefore calculates the MSE between the predicted and actual next states.
Table 1 summarizes the results from both the Transfer and True Model-Based experiments.
Additional details are provided in App.
D along with full correlation plots in App.
A.2.
We are now in a position to analyze the results in Table 1.
It is immediately obvious that the LOO KL and LL Var penalties have very weak correlation with MSE.
We believe this is because LL Var relies on likelihood statistics, which are notoriously sensitive; it was designed for use with a KL-regularized latent state space model which has well-behaved dynamics.
Regarding LOO KL, we note that this penalty was designed for the online setting with signiﬁcantly less data, and becomes quite uncorrelated in this larger data setting.
This advocates penalties that are less reliant on distributional information concerning the separate Gaussians in the ensemble, as such penalties appear sensitive to the quality of their estimated distributions.
We observe that Max Aleatoric, Max Pairwise Diff and the Ensemble penalties perform broadly similarly despite their different analytical forms; Ensemble measures do however exhibit noticeably higher rank correlation.
We also observe a signiﬁcant performance loss between the Transfer and True Model-Based HalfCheetah settings, with the latter being relatively poor.
This implies further work is needed to develop penalties that can successfully detect the type of dynamics discrepancies that actually arise in ofﬂine MBRL.
Finally, we observe that despite the similar rank correlations ρ , the bivariate correlations r can vary considerably, and observe from the scatter plots that Max Aleatoric exhibits low kurtosis, having large penalty values “bunched” at its extreme; we provide 3rd and 4th order moment statistics to facilitate shape comparisons in App. C.
K EY H YPERPARAMETERS IN O FFLINE MBRL 5. H OW M ANY M ODELS D O W E N EED ?
At present the number of models used has not been discussed since MBPO, which trains seven probabilistic dynamics models of the same architecture (with different initializations), using only the top ﬁve models based on validation accuracy (referred to as “Elites” in the Evolutionary community, e.g. Mouret & Clune (2015)).
The reason or justiﬁcation for this is not discussed, but it has seemingly been adopted in the wider MBRL setting (Shen et al., 2020;
Omer et al., 2021; Pineda et al., 2021).
However, ofﬂine RL is a totally different paradigm, where it is possible that access to compute is less of a bottleneck and it may be preferable to use more models to extract the most signal possible from the static dataset.
Inevitably, many of the ensemble penalties are dependent on the number of models; for example, it is easy to see that the Max Aleatoric value could scale poorly with more models.
How Does Penalty Distribution Change with Model Count?
We now vary the number of models used in the calculation of the penalties and plot their respective distributions; an illustrative example is shown in Fig.
2 with full results in App.
B.
The scaling of penalties relying on max over sets is most affected with increasing the number of models due to admitting more extreme values, and we observe that the distribution shape of Max Aleatoric changes signiﬁcantly as we admit more models, which we validate in App. C.
This impacts the tuning of this hyperparameter, as we have to contend with a changing distribution along with calibration quality (which we explore in the next section).
Finally,
[Figure]
LL
[Figure]
[Figure]
N u m b e r o f
M o d e l s
[Figure]
we observe that the Ensemble penalties change the least with differing model count, highlighting their ease of tuning; this is clearly a desirable property for designing such metrics going forward
[Figure]
How does Penalty Performance Scale with Model
Count?
Empirically, there exists an optimal number of models to use in an ensemble for model-based RL (Kurutach et al., 2018; Matsushima et al., 2021).
Up to now, heuristics have been used to select how many models we use for uncertainty estimation, despite it being possible to use a different number of models for dynamics prediction and uncertainty estimation.
For instance, in MOPO, transitions are generated with ﬁve Elite models, but all seven models are used to calculate the penalty.
In MOReL, four models are used for both transitions and penalty prediction.
Therefore, we wish to understand if there is merit to using a larger number of models for uncertainty estimation compared with next state prediction.
We provide a snapshot in Fig.
3, showing the aggregated results on the True Model-Based data in Hopper, with full results in App.
B. We see there is no clear consensus, and that the optimal number of models is highly dependent on environment, the behavior data, and penalty type, with some settings showing improved calibration with model count and vice-versa.
This clearly justiﬁes treating the number of models as a hyperparameter that is important to tune, especially on transfer tasks.
Interestingly, we observe that it is possible to simultaneously improve rank ( ρ ) correlation, but reduce bivariate ( r ) correlation, especially with the MOPO penalty.
This again suggests that the number of models not only affects the quality of the estimation, but also its distributional shape .
5. T HE W EIGHT OF U NCERTAINTY λ
To weight penalty against reward, MOPO introduces a parameter λ that trades off between the two terms.
In their paper, the authors sweep over λ ∈ { 1 , 5 } for each environment.
However, the optimal values may lie outside this region.
Furthermore, we have shown this value will need to drastically change to account for using a different penalty or even number of models.
5. T HE R OLLOUT H ORIZON h
The horizon h of the rollouts plays a crucial role in ofﬂine RL.
Longer horizon rollouts increase the likelihood of errors in the transitions (we verify this intuition in App. D), but conversely can improve performance when errors are properly managed (Janner et al., 2019; Pan et al., 2020).
Furthermore, as highlighted in Fig.
1a, the model can generalize, and dynamics error does not necessarily increase with drift away from the ofﬂine dataset.
Instead, we observe spikes, and note it is possible to recover from these to valid states and transitions.
It is therefore imperative that a penalty captures these spikes over the course of an entire model rollout with horizon h , and down-weights the reward accordingly.
Using this observation, we design a novel experiment that treats these spikes as “positive” labels, and normalize each penalty to [0 , 1] .
This converts the penalties into a probabilistic classiﬁer, and we evaluate how well they classify these events that occur increasingly under longer h .
This is precisely the intuition behind the LOO KL and LL Var approaches, whereby the penalty acts as an anomaly detector, removing detrimental transitions that lie above a threshold. This is the regime we
[Table]
focus on here, where binary detection is more important than correlation.
Finally, we assess two “True Model-Based” errors: the dynamics error as before, and introduce the distance from the ofﬂine distribution trained on, which we calculate as the 2-norm between a state-action tuple and its nearest point in the ofﬂine data; these are called “Dynamics” and “Distribution” respectively.
We provide precision-recall curves and more details on this experiment in App.
D and E.
We observe in Table 2 that the penalties are powerful at identifying dynamics discrepancy, but not as accurate at identifying when the world-model data is out-of-distribution with respect to the ofﬂine data.
This is a well-known phenomenon in deep neural networks and has been recently investigated in terms of feature collapse (Van Amersfoort et al., 2020), where latent representations of points far away in the input space get mapped close together.
On the other hand, this shows an important distinction between the regularization induced by MBRL uncertainty and explicit state-action regularization in model-free approaches, such as Kumar et al.
; Wu et al.
In the latter approaches, policies are penalized for taking out of distribution actions w.r.t.
the ofﬂine dataset, but this is not always the case with policies trained under MBRL and uncertainty penalties.
The success of
MBRL methods in RL may therefore lie in the generation of state-action samples that are OOD but represent accurate dynamics , thus facilitating dynamics generalization in policies; recent work has shown that augmenting dynamics improves ofﬂine RL policy generalization (Ball et al., 2021).
We believe future work understanding the implications of this property is vitally important.
T ESTING THE L IMITS OF C URRENT A PPROACHES
Given our previous analysis, in this section we seek to answer the following question: how well can existing methods perform with a more optimal selection of the discussed hyperparameters?
To answer this, we consider, ﬁrst, a naïve selection of one hyperparameter set across all environments (based on our previous analysis), and then more deﬁnitively, tuning the conﬁguration for each individual
D4RL MuJoCo environment using a state-of-the-art Bayesian Optimization (BO) algorithm (Wan et al., 2021).
Our ﬁrst set of results show that following our analysis can provide signiﬁcant gains over existing baselines, whilst the second beats the current SoTA.
Note previous analysis focused on
HalfCheetah and Hopper environments, so we extend our evaluation to Walker2d as a held-out test.
General applicability of our insights.
Two of our main takeaways in Sections 4 and 5 are that we should favor the canonical Ensemble penalties and longer rollout horizons.
To test these claims, we design an experiment where we take a naïve choice of h = 20 for the horizon, and use Ensemble Std.
as our penalty (see App.
G for details).
Since tuning the penalty weight λ per environment can be laborious, we employ an automatic penalty tuning scheme, analogous to the automatic entropy tuning used in Haarnoja et al.
We tune the penalty weight on-the-ﬂy to a constraint value of Λ = 1 , meaning we use only a single hyperparameter across all environments .
We get an average reward of 49.0 , an increase of 43% over MOPO, which was grid-searched per environment.
This clearly shows the ﬁndings from our analysis provides large performance gains more generally.
This result is the best we know for a single hyperparameter, and is particularly signiﬁcant as other ofﬂine MBRL algorithms tune many hyperparameters individually per environment.
This restriction also displays the most realistic application of ofﬂine RL to real world problems.
If we were to allow ourselves to take the maximum over just 2 hyperparameter setups (the second setup being h = 10 , Λ = 0 . 5 ), we are able to obtain an average reward of 57.8 , an increase of 69% over MOPO.
We show the full results of our evaluation in Table 8 in App.
H with signiﬁcance testing.
Testing the limits of current approaches.
Next, we wish to further validate that our earlier theoretical analysis can correspond to strong empirical performance gains by performing BO over the key
[Table]
hyperparameters.
Details on the BO algorithm are listed in App
[Table]
G. We deﬁne our search space over hyperparameters most related to uncertainty quantiﬁcation :
• Penalty type (categorical): taking values over { Max Aleatoric, Max Pairwise Diff, LOO KL, LL Var, Ensemble Std, Ensemble Variance } .
• Penalty scale λ (continuous): taking values over [1 , 100] .
• h (integer): taking values over { 1 , 2 , . . .
• Models N (integer): taking values over { 1 , 2 , . .
Table 3 shows the optimal hyperparameters under BO.
We note that Ensemble penalties are mainly selected, corroborating the ﬁndings in our analysis that these are most correlated with model error.
We observe that Max Pairwise Diff is not chosen, likely because ensemble penalties are better correlated with true dynamics error, and are more stable under tuning since their scaling changes less with model number; we know that Max Pairwise Diff has very similar shape statistics to Ensemble Std.
(App. C).
Finally, we also observe these solutions have lower performance variance than MOPO.
The selection of Max Aleatoric is also explainable; we observe it displays signiﬁcantly lower skew and kurtosis than all other metrics (App. C), while still maintaining strong rank correlation.
We also found that in all Hopper experiments, Ensemble Var.
never achieved high performance, despite the only difference with Ensemble Std.
being its distributional shape.
Interestingly, in HalfCheetah, the opposite is true, with Ensemble Var.
delivering signiﬁcant performance gains.
This implies that distributional shape may play as important a role as calibration, and advocates the learning of meta-parameters that control for this.
Finally in Walker2d, the well-grounded ensemble penalties win in all cases.
We note that values of the rollout horizon h and penalty weight λ differ greatly from those chosen in the original MOPO paper, which chooses both from { 1 , 5 } .
Notably, the Hopper and Walker2d environments can prefer a much longer rollout length and higher penalty weight, even accounting for penalty magnitudes.
Again this is backed up by our analysis; along a single rollout, dynamics errors do not necessarily accumulate, they simply become more likely to occur.
Therefore, as long as we penalize errors appropriately, we can handle longer rollouts and, as a result, generate more on-policy data.
The number of models used to compute the uncertainty estimates can also differ greatly from the standard 7.
This again aligns with our ﬁndings that using more models for uncertainty estimation can be beneﬁcial, but is dependent on environment, data, and penalty.
Table 3 also demonstrates how these unconventional hyperparameter choices fare against state-of-theart ofﬂine model-based RL algorithms.
We spent considerable effort ensuring that our implementation of MOPO matched the performance of the authors’ results using the same hyperparameters.
We note the two are very similar 1 , thereby allowing us to make a faithful comparison when modifying hyperparameters.
Our approach, labeled “Optimized † ”, achieves statistically signiﬁcant improvements over our implementation of MOPO
[Figure] [Figure]
over the key design choices in ofﬂine MBRL.
As an additional bonus, and it is not the stated aim of this work, our approach manages to achieve state-of-the-art performance on ﬁve HalfCheetah and
Walker2d environments by a considerable margin.
Further notable results include the Hopper mixed and Hopper medium-expert environments, in which we show we are able to tune the MOPO-like method up to the performance of COMBO (Yu et al., 2021) and MOReL.
The importance of good uncertainty quantiﬁcation and hyperparameter selection for MOPO is shown visually in Fig.
4 where we improve MOPO performance by over 5 × whilst obtaining a stable solution.
As aforementioned, we found our policies are more stable than previous works and consequently do not need to cherry-pick high performing checkpoints 2 .
Instead, we report the average performance over our ﬁnal 10 policy-improvement iterations.
It should be noted that stability during training (Chan et al., 2020) is paramount for successful policy deployment in ofﬂine RL, and we should therefore prioritize hyperparameters that ensure this.
We further conﬁrm the reliability of our evaluation using the RLiable (Agarwal et al., 2021) framework in Fig.
1b, showing that the improvement over MOPO (with 95% bootstrap CIs shaded) is strongly statistically signiﬁcant for both MuJoCo and Adroit.
Results on Adroit dexterous hand manipulation tasks.
We present results in Table 4 on the Adroit
Pen and Hammer environments which, as far as we are aware, have not previously been used in ofﬂine MBRL , and present very different challenges to the locomotion tasks.
These tasks feature sparse rewards, real human demonstrations and narrow data distributions.
We compare against the current state-of-the-art model-free algorithm (CQL, Kumar et al.
and ﬁnd that ofﬂine MBRL can learn useful policies in the Adroit domains, providing the best performance seen so far on the hammer-cloned setting.
Best found penalties and hyperparameters are listed in App.
I, and mirror the ﬁndings in the locomotion experiments.
We believe issues with the world model not accurately capturing sparse rewards may account for any major performance difference.
Our work is therefore an important step towards bridging the gap between model-based and model-free methods for sparse reward tasks, especially in the ofﬂine setting where exploration is not possible.
We deﬁne MOPO to be the best performance with the Max Aleatoric penalty, searching λ , h in { 1 , 5 } 2 .
C ONCLUSION
In this paper, we rigorously evaluated the impact of various key design choices on ofﬂine MBRL, comparing for the ﬁrst time a number of different uncertainty penalties used in the literature.
By proposing novel evaluation protocols, we have also gained key insights into the nature of uncertainty in ofﬂine MBRL that we believe beneﬁts the RL community.
We demonstrated the impact of this analysis by signiﬁcantly improving upon existing ofﬂine MBRL by using vastly different key hyperparameters, obtaining statistically signiﬁcant performance improvements in almost all benchmarks.
Going forward, we are excited by developments in ofﬂine evaluation (Chen et al., 2021; Fu et al., 2021b) to accurately assess agent performance without querying the environment.
This would open the door for population-based training methods (Jaderberg et al., 2017; Parker-Holder et al., 2020), which have shown great success in online MBRL (Zhang et al., 2021).
Furthermore, throughout the paper we have highlighted potential areas of interest, from better understanding the generalization provided by world models, through to the development of meta-parameters controlling penalty distribution shape.
We also highlight key issues in implementation in App.
F as we strongly believe this is a vital frontier for disentangling the effect of algorithmic innovations from code-level details.
Finally, Ofﬂine MBRL so far has only focused on determinstic environments; the ensemble penalties we investigate support stochastic dynamics, and the novel tools for analysis we develop here could be readily applied to those new settings.
A C ALIBRATION A. C HOICE OF CALIBRATION METRICS
We consider both the Spearman rank ( ρ ) correlation and Pearson bivariate ( r ) correlation.
We believe that the former better represents the actual statistical power of the metric compared to the true distributional shift value, as it is robust to outliers and isn’t impacted by distributional shape (i.e., skewness, kurtosis).
After all, we do not know if some ‘true’ | G π ˆ M ( s, a ) | is even linearly correlated with the MSE values that we report, so naïvely comparing based on bivariate correlation may result in incorrect assessment of penalty efﬁcacy.
However, we do also include the Pearson bivariate correlation to gain insight into how the penalty distribution shape changes with design choices.
For instance, consider two metrics that have identical Spearman coefﬁcients, but vastly different Pearson coefﬁcients–this implies they have signiﬁcantly different distributional shapes whilst having the same statistical ranking power.
The two correlation coefﬁcients have the further advantage that they are unaffected by the scale of the uncertainty penalty, which can vary widely.
Furthermore, algorithms such as MOPO and MOReL will often scale the penalty by some coefﬁcient λ and thus the raw unscaled value is hard to interpret.
A. O FFLINE D ATASET T RANSFER C ALIBRATION A.2. H ALF C HEETAH
[Figure]
[Figure]
n a l t y
[Figure] [Figure]
A.
[Figure]
T RUE M ODEL -B ASED E RROR C ALIBRATION
[Figure]
[Figure] [Figure]
A. A DDITIONAL L OG P ROBABILITY C ORRELATION A NALYSIS
Table 5 shows correlation between reward penalties and the negative log-likelihood of the true data under the model in the Transfer experiments.
[Figure]
[Table]
B F ULL R ESULTS I NCREASING M ODELS B. P ENALTY D ISTRIBUTION B.1. O FFLINE D ATASET T RANSFER D ISTRIBUTION
[Figure]
[Figure]
Random transferred
[Figure] [Figure]
Medium transferred to
[Figure]
(c) Medium transferred to Random
[Figure] [Figure]
N u m b e r o f
M o d e l s
[Figure]
B.1.
T RUE M ODEL -B ASED
E
RROR
D
ISTRIBUTION
[Figure]
[Figure] [Figure]
[Figure] [Figure]
: T RANSFER
[Figure] [Figure]
[Figure]
H ALF C HEETAH D4RL
[Figure]
T RUE M ODEL
[Figure]
B ASED E RROR
[Figure] [Figure]
[Figure] [Figure]
B.2.
A LL A GGREGATED
[Figure]
Sp e a r m a n :   20 Number of Models 0.
P e a r s o n : r   20 Number of Models 0.
Sp e a r m a n :   20 Number of Models 0.
P e a r s o n : r
HalfCheetah
Hopper
[Figure]
Ensemble Std
[Figure]
Ensemble Var
[Figure]
LL Var
[Figure]
KL LOO
C S KEWNESS AND K URTOSIS C OMPARISONS C. S KEWNESS AND K URTOSIS O VERALL
[Table]
[Table] [Table]
S KEW AND K URTOSIS S CALING WITH M ODEL C OUNT
We omit LL Var.
and LOO KL due to the fact that their changes were so signiﬁcant as to obfuscate the changes of the more performant penalties.
We choose 7 models to act as our ’baseline’ (following the default MOPO setting), and we measure the change in the skew and kurtosis relative to this, hence 7 models always has a 0% change in our graphs.
For brevity, in the transfer experiments, we average over all ‘transferred to’ environments, e.g., Random, Medium, etc.; the graph title refers to the data that the model was trained on.
Again, we observe the environment and setting dependency of these metrics, sometimes having increasing skewness and kurtosis with model count, and other times decreasing.
This further justiﬁes using a ranking metric to compare penalties, as the overall penalty shape can vary hugely and unpredictably w.r.t.
co-dependent hyperparameters.
We do observe however in the True Model-Based experiments that ensemble standard deviation appears to be most robust to scaling with models.
We also observe that the Max Aleatoric penalty can change shape signiﬁcantly w.r.t.
model count, and all penalties are not fully immune to this.
This further advocates the use of shape meta-parameters to control for changing distribution properties when adjusting the number of models as a hyperparameter, as well as selecting penalties that are relatively invariant to model count to make tuning easier.
[Figure] [Figure]
Transfer
[Figure] [Figure]
D F URTHER DETAILS ON T RUE M ODEL -B ASED EXPERIMENTS D. M ETHODOLOGICAL D ETAILS
We leverage the MuJoCo (Todorov et al., 2012) simulator to provide us with ground truth dynamics that we can use to compare against our model predictions and penalties.
This is done by providing the state and action inputs given to the model to the simulator through the set_state method in the simulator API.
It must be noted that this method also requires an addition ‘displacement’ value which is not modelled by the world models (nor is it provided in the D4RL data), however we found in practice this did not affect the dynamics predicted by the simulator, and simply setting this towas sufﬁcient to generate ground truth predictions.
This makes it possible to provide the simulator the hallucinated model states, and provide a true proxy to the dynamics discrepancy.
We note that since the states are ‘hallucinated’ by the model, it might be the case that they may not be admissible under the true environment, but in reality the simulator was able to process almost any combination of state and action, barring settings that featured anomalously large magnitudes.
To handle such cases we found it necessary to clip the model states to the range [ − 10 , 10] .
In order to assess the permissibility of states, as well as measure the accuracy of the penalties as
OOD input detectors, we provide an alternative distance measure based on the distance away from the training set.
We use this measure for our analysis in Section 5.3, and is calculated as the distance from the ofﬂine training dataset, which we deﬁne to be the 2-norm between a given state-action tuple and its nearest point in the ofﬂine data.
We describe this quantity henceforth as ‘Distribution Error’.
D. O N THE NATURE OF OOD DATA ALONG HALLUCINATED TRAJECTORIES
Here we discuss the nature of OOD data along a single hallucinated trajectory (in the model) in ofﬂine MBRL, analyzing the inductive bias that some ‘error’ increases with increasing rollout length in the model.
We ﬁnd that there is merit to this assumption, and show this in Fig.
26 for all
HalfCheetah and Hopper environments in D4RL.
Here, we plot the median error at each time-step across 30 , 000 aggregated trajectories in the model.
Note that for all plots, we re-normalize all penalties by subtracting their mean and dividing by their standard deviation to facilitate comparison; this normalization was also applied in the analysis performed in Fig.
1a.
Concretely, each time step corresponds to the normalized median value of 30 , 000 data-points.
[Figure]
Median True Model-Based Errors as a function of rollout timestep
We observe indeed that both median dynamics and distribution errors increase with increasing time step in the model.
The only real exception is HalfCheetah Medium-Expert, which we believe to be due to our trained policy not being able to successfully exploit this environment.
The above analysis captures overall trends in the error over a large number of trajectories.
However, the way errors manifest during an individual rollout is not so straightforward.
To illustrate this, observe Fig. 27, where we plot a random subset of 5 individual rollouts from the Hopper MediumExpert data we generated.
[Figure]
We observe that errors along any single trajectory tend to manifest as ‘spikes’, and that it is entirely possible to recover from these, returning to either admissible dynamics, or parts of the state-action space that have been seen in the data.
This speaks to the nature of how we ought to penalize policies for accessing regions of inaccuracy/uncertainty, and may justify a hybrid MOPO/MOReL approach, whereby we penalize individual transitions along a trajectory, but do not stop rollouts early.
Indeed, this is similar to the approach taken in M2AC (non-stop), albeit they choose to ‘mask’ uncertain transitions, not penalize them.
We leave the design of such an algorithm to future work.
Finally, we address the issue of comparing OOD dynamics and inputs.
As already observed in Fig.
27, these two errors are not necessarily always the same, and oftentimes it is possible that one quantity is large, whilst the other is small.
We revisit Fig.
1a to explore this, now also plotting the Distribution
Error in Fig.
[Figure]
We ﬁrst speak to the inset annotated ‘ 1 ’.
Here we observe that the transitions generated in fact closely resemble the data that our model was trained on, however the predicted dynamics are incorrect, and cause an aforementioned ‘spike’.
This is the opposite of what is observed in the inset annotated ‘ 2 ’; where we actually predict accurate dynamics, however the resultant state-action tuples do not closely resemble the data that our model was trained on.
We generally observe that regions of high
Distribution Error tend to be preceded by ‘spikes’ pertaining to high Dynamics Error, and this present an exciting avenue for future work understanding how these quantities are related.
D. O N THE DIVERSITY OF EXPLOITATIVE POLICIES
It is possible that training policies purely to exploit the world models may result in generating stateaction tuples that are low in diversity, as the policy could discover "pockets" in the model that provide consistently high return.
To prevent this, we train multiple policies inside the model from different seeds, with the aim of inducing different modes of exploitation.
To validate this induces diverse trajectories in the world models, we visualize the state-action manifold using a t-SNE projection (van der Maaten & Hinton, 2008) of: 1) the D4RL Hopper Mixed-Replay (which contains diverse samples); 2) the imagined rollouts inside the model from the exploitative policies in Fig.
The policies were trained to exploit a model that itself was trained on the Hopper Mixed-Replay data.
[Figure]
We observe that the induced policies inside the model displays some overlap with the D4RL data, but also resides in parts of the manifold where there is little coverage from the D4RL data, likely representing regions of exploitation.
Importantly, the exploitative WM trajectories display strong state-action tuple diversity, comparable to that of the ofﬂine data it was trained on.
E U SING METRICS AS OOD EVENT DETECTORS
E.
M EASURING S TATISTICS
[Figure]
As noted previously, different penalties have varying scales and distribution proﬁles, so we need a way of standardizing the method of assessment.
Using our observation that errors manifest as ‘spikes’ during a rollout, we propose treating each penalty as a classiﬁer.
Concretely, our test set consists of the ground truth data labeled by whether or not they exceed a certain percentile at a particular time step.
Each penalty may be then be treated as a ‘classiﬁer’ by normalizing its range to lie in [0 , 1] .
We can then use standard classiﬁcation quality measures, such as AUC, to determine the effectiveness of these penalties at capturing these spikes, whilst sidestepping the issue of the different distributional proﬁles identiﬁed previously.
Fig.
30 shows how our proposed method may be used to compare the effectiveness of each metric at capturing OOD events.
In the ﬁgure, we plot a single rollout in the model, and the resultant ground truth MSE between the predicted next state and the true next state in black.
We then superimpose the 90th, 95th and 99th percentile MSEs across the entire imagined trajectories onto the ﬁgure in gray dashed lines.
To construct our OOD labels, we label any point below the percentile line as being ‘False’, and any point above that line as being ‘True’.
Finally, we normalize the uncertainty metrics as previously described into values in the range [0 , 1] , allowing us to construct precision-recall graphs and calculate classiﬁer statistics.
[Figure]
P
RECISION R ECALL C URVES
[Figure]
F I MPLEMENTATION D ETAILS
This paper extensively discusses key hyperparameters speciﬁc to current ofﬂine MBRL algorithms.
However, there are signiﬁcant code-level implementation details which are often critical for strong performance and make it hard to disambiguate between algorithmic and implementation improvements.
Worryingly, many of these details are not mentioned in their respective papers, or are different between the authors’ code and paper.
We detail clear examples of this below.
We believe further investigation of these code-level implementation details represents important future work, as has already been done for policy gradients (Engstrom et al., 2020; Andrychowicz et al., 2021).
Indeed – it is unclear if the improvement of MOReL over MOPO is due to its different P-MDP formulation, or if it is successful in spite of this formulation, due to a superior policy optimization strategy or dynamics model design.
We believe that this paper takes a signiﬁcant ﬁrst step in tackling this issue by directly comparing a number of key design choices, and understanding their individual impact.
Now we summarize key differences between the paper and code for the MOPO and MOReL algorithms which we compare against that are crucial to achieve the same reported performance.
In MOPO,
• Each layer in the model neural network has a different level of weight-decay • The authors’ code uses different objectives for training (log-likelihood) and validation (MSE).
• The authors use elites, but only for next state prediction (as discussed previously).
In MOReL,
• There is a difference in the authors’ code about how the penalty threshold is calculated and tuned, and isn’t provided as a hyperparameter in the appendix.
• The absorbing HALT state does not appear in the authors’ code.
• The negative halt penalty appears signiﬁcantly different between code and paper.
• There is a minimum trajectory steps parameter (hard-coded to 4) not mentioned in the paper.
• The reward function appears to be hard-coded in the authors’ implementation, not learned as stated in the paper.
• The policy architecture is different in the authors’ code (64,64 hidden layers) and the paper (32, hidden layers) •
It is not clear when the optional behavior cloning initialization step is applied.
G H YPERPARAMETERS AND E XPERIMENT D ETAILS
The D4RL (Fu et al., 2021a) codebase and datasets used for the empirical evaluation is available under the CC BY 4.0 Licence.
As stated in the main text, we choose to use ‘v0‘ experiments as these are more challenging for Hopper due to having low return trajectories (Kostrikov et al., 2021), and we clearly state when other benchmarks use the ‘v2‘ experiments, which have ofﬂine trajectories with higher returns on Hopper.
The remaining hyperparameters for the MOPO algorithm that we do not vary by Bayesian Optimization were taken from the original MOPO paper (Yu et al., 2020), apart from we ﬁx the number of policy epochs/iterations to 1,000 for all experiments.
This means our implementation uses the same probabilistic dynamics models (with unchanged hyperparameters) and policy optimizer (SAC, Haarnoja et al.
as MOPO, differing from MOReL, which uses Natural Policy Gradient (Kakade, 2002).
The hyperparameters used for the BO algorithm, CASMOPOLITAN , are listed in Table 7.
We use the batch-mode of CASMOPOLITAN , where multiple hyperparameters settings are proposed and evaluated concurrently.
[Table]
Each BO iteration is run for 300 epochs on a single seed, and single run over a single ofﬂine dataset took ~200 hours on a NVIDIA GeForce GTX 1080 Ti GPU taken up predominantly by MOPO training.
Unless speciﬁed otherwise, plots and reported statistics are completed with 7 models in the ensemble, as this is the number chosen in the original MOPO paper used with the Max Aleatoric penalty.
H RL IABLE FRAMEWORK FOR SHOWING STATISTICAL SIGNIFICANCE
Throughout this work we choose to adopt the RLiable framework introduced in Agarwal et al.
(2021) to assess the statistical signiﬁcance of our results.
RLiable advocates for computing aggregate performance statistics and probability of improvement across many tasks in a benchmark suite; indeed we take this approach when reporting the values in the analysis performed in Sections 4 and 5.
This is important when the number of tasks become large
[Table]
and also prevents outliers from dominating mean statistics.
For normalisation, we use the standard D4RL return scaling
[Table] [Table]
I B EST FOUND A DROIT HYPERPARAMETERS
[Table]