A UTOREGRESSIVE D YNAMICS M ODELS FOR O FFLINE P OLICY E VALUATION AND O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Standard dynamics models for continuous control make use of feedforward computation to predict the conditional distribution of next state and reward given current state and action using a multivariate Gaussian with a diagonal covariance structure.
This modeling choice assumes that different dimensions of the next state and reward are conditionally independent given the current state and action and may be driven by the fact that fully observable physics-based simulation environments entail deterministic transition dynamics.
In this paper, we challenge this conditional independence assumption and propose a family of expressive autoregressive dynamics models that generate different dimensions of the next state and reward sequentially conditioned on previous dimensions.
We demonstrate that autoregressive dynamics models indeed outperform standard feedforward models in log-likelihood on heldout transitions.
Furthermore, we compare different model-based and model-free off-policy evaluation (OPE) methods on RL Unplugged, a suite of ofﬂine MuJoCo datasets, and ﬁnd that autoregressive dynamics models consistently outperform all baselines, achieving a new state-of-the-art.
Finally, we show that autoregressive dynamics models are useful for ofﬂine policy optimization by serving as a way to enrich the replay buffer through data augmentation and improving performance using model-based planning.
I NTRODUCTION
Model-based Reinforcement Learning (RL) aims to learn an approximate model of the environment’s dynamics from existing logged interactions to facilitate efﬁcient policy evaluation and optimization.
Early work on Model-based RL uses simple tabular (Sutton, 1990; Moore and Atkeson, 1993; Peng and Williams, 1993) and locally linear (Atkeson et al., 1997) dynamics models, which often result in a large degree of model bias (Deisenroth and Rasmussen, 2011).
Recent work adopts feedforward neural networks to model complex transition dynamics and improve generalization to unseen states and actions, achieving a high level of performance on standard RL benchmarks (Chua et al., 2018;
Wang et al., 2019).
However, standard feedforward dynamics models assume that different dimensions of the next state and reward are conditionally independent given the current state and action, which may lead to a poor estimation of uncertainty and unclear effects on RL applications.
In this work, we propose a new family of autoregressive dynamics models and study their effectiveness for off-policy evaluation (OPE) and ofﬂine policy optimization on continuous control.
Autoregressive dynamics models generate each dimension of the next state conditioned on previous dimensions of the next state, in addition to the current state and action (see Figure 1).
This means that to sample the next state from an autoregressive dynamics model, one needs n sequential steps, where n is the number of state dimensions, and one more step to generate the reward.
By contrast, standard feedforward dynamics models take current state and action as input and predict the distribution of the next state and reward as a multivariate Gaussian with a diagonal covariance structure ( e.g., Chua et al. ; Janner et al.
This modeling choice assumes that different state dimensions are conditionally independent.
Autoregressive generative models have seen success in generating natural images (Parmar et al., 2018)
, text (Brown et al., 2020), and speech (Oord et al., 2016), but they have not seen use in Model-based RL for continuous control.
We ﬁnd that autoregressive dynamics models achieve higher log-likelihood compared to their feedforward counterparts on heldout validation transitions of all DM continuous control tasks (Tassa et al., 2018) from the RL Unplugged dataset (Gulcehre et al., 2020).
To determine the impact of improved transition dynamics models, we primarily focus on OPE because it allows us to isolate contributions of the dynamics model in value estimation vs. the many other factors of variation in policy optimization and data collection.
We ﬁnd that autoregressive dynamics models consistently outperform existing Model-based and Model-free OPE baselines on continuous control in both ranking and value estimation metrics.
We expect that our advances in model-based OPE will improve ofﬂine policy selection for ofﬂine RL (Paine et al., 2020).
Finally, we show that our autoregressive dynamics models can help improve ofﬂine policy optimization by model predictive control, achieving a new state-of-the-art on cheetah-run and ﬁsh-swim from RL Unplugged (Gulcehre et al., 2020).
Key contributions of this paper include:
• We propose autoregressive dynamics models to capture dependencies between state dimensions in forward prediction.
We show that autoregressive models improve log-likelihood over nonautogressive models for continuous control tasks from the DM Control Suite (Tassa et al., 2018).
• We apply autoregressive dynamics models to Off-Policy Evaluation (OPE), surpassing the performance of state-of-the art baselines in median absolute error, rank correlation, and normalized top-5 regret across 9 control tasks.
• We show that autoregressive dynamics models are more useful than feedforward models for ofﬂine policy optimization, serving as a way to enrich experience replay by data augmentation and improving performance via model-based planning.
P RELIMINARIES
Here we introduce relevant notation and discuss off-policy (ofﬂine) policy evaluation (OPE).
We refer the reader to Lange et al.
(2012) and Levine et al.
(2020) for background on ofﬂine RL, which is also known as batch RL in the literature.
A ﬁnite-horizon Markov Decision Process (MDP) is deﬁned by a tuple M =
( S , A , T , d 0 , r, γ ) , where S is a set of states s ∈ S ,
A is a set of actions a ∈ A , T deﬁnes transition probability distributions p ( s t +1 | s t , a t ) , d 0 deﬁnes the initial state distribution d 0 ≡ p ( s 0 ) , r deﬁnes a reward function r : S ×
A → R , and γ is a scalar discount factor.
A policy π ( a | s ) deﬁnes a conditional distribution over actions conditioned on states.
A trajectory consists of a sequence of states and actions τ = ( s 0 , a 0 , s 1 , a 1 , . . .
, s H ) of horizon length H .
We use s t,i to denote the i -th dimension of the state at time step t (and similarly for actions).
In reinforcement learning, the objective is to maximize the expected sum of discounted rewards over the trajectory distribution induced by the policy:
[Equation]
The trajectory distribution is characterized by the initial state distribution, policy, and transition probability distribution:
[Equation]
In ofﬂine RL, we are given access to a dataset of transitions D
= { ( s it , a it , r it +1 , s it +1 ) }
Ni =1 and a set of initial states S 0 .
Ofﬂine RL is inherently a data-driven approach since the agent needs to optimize the same objective as in Eq.
(1) but is not allowed additional interactions with the environment.
Even though ofﬂine RL offers the promise of leveraging existing logged datasets, current ofﬂine RL algorithms (Fujimoto et al., 2019; Agarwal et al., 2020; Kumar et al., 2019) are typically evaluated using online interaction, which limits their applicability in the real world.
The problem of off-policy (ofﬂine) policy evaluation (OPE) entails estimating V γ ( π ) , the value of a target policy π , based on a ﬁxed dataset of transitions denoted D , without access to the environment’s dynamics.
Some OPE methods assume that D is generated from a known behavior (logging) policy µ and assume access to µ in addition to D .
In practice, the logged dataset D may be the result of following some existing system that does not have a probabilistic form.
Hence, in our work, we will assume no access to the original behavior policy µ for OPE.
That said, for methods that require access to µ , we train a behavior cloning policy on D .
[Figure]
P ROBABILISTIC D YNAMICS M ODELS
Feedforward dynamics model.
In the context of our paper, we use the term “model” to jointly refer to the forward dynamics model p s ( s t +1 | s t , a t ) and reward model p r ( r t +1 | s t , a t ) .
We use neural nets to parameterize both distributions since they are powerful function approximators that have been effective for model-based RL (Chua et al., 2018; Nagabandi et al., 2018; Janner et al., 2019).
Let θ denote the parameters of a fully connected network used to model p θ ( s t +1 , r t +1 | s t , a t ) .
We expect joint modeling of the next state and reward to beneﬁt from sharing intermediate network features.
Similar to prior work (Janner et al., 2019), our baseline feedforward model outputs the mean and log variance of all state dimensions and reward simultaneously, as follows:
[Equation]
where µ ( s t , a t ) ∈ R n +1 denotes the mean for the concatenation of the next state and reward, l ( s t , a t ) ∈ R n +1 denotes the log variance, and Diag( v ) is an operator that creates a diagonal matrix with the main diagonal speciﬁed by the vector v .
During training, we seek to minimize the negative log likelihood of the parameters given observed transitions in the dataset D :
[Equation]
While it is possible to place different weights on the loss for next state and reward prediction, we did not apply any special weighting and treated the reward as an additional state dimension in all of our experiments.
This is straightforward to implement and does not require tuning an additional hyperparameter, which is challenging for OPE.
Note that the input has | s | + | a | dimensions.
Autoregressive dynamics model.
We now describe our autoregressive model.
We seek to demonstrate the utility of predicting state dimensions in an autoregressive way.
Therefore, rather than using a complex neural network architecture, where improvements in log-likelihood and policy evaluation are confounded by architectural differences, we opt to make simple modiﬁcations to the feedforward model described above.
This allows us to isolate the source of performance improvements.
The autoregressive model we use is a fully connected model that predicts the mean and log variance of a single state dimension.
We augment the input space of the baseline with the previous predicted state dimensions and a one-hot encoding to indicate which dimension to predict.
This is illustrated in Figure 1.
The autoregressive model therefore has 3 | s | + | a | input dimensions.
Hence, the autoregressive model has a small number of additional weights in the ﬁrst fully connected layer, but as will be shown in our experiments, these extra parameters are not the reason for a performance gain.
At training time, the autoregressive model has a similar computational cost to the fully connected model as we can mask ground truth states and use data parallelism to compute all state dimensions simultaneously.
At inference, the autoregressive model requires additional forward passes, on the order of the number of state dimensions in a given environment.
We use the default ordering for the state dimensions in a given environment, though it is interesting to explore different orderings in future works.
The negative log-likelihood for an autoregressive model takes the form of:
[Equation]
where we use chain rule to factorize the joint probability of p ( s (cid:48) , r (cid:48) | s, a ) .
The main advantage of the autoregressive model is that it makes no conditional independence assumption between next state dimensions.
This class of models can therefore capture non-unimodal dependencies, e.g., between different joint angles of a robot.
Paduraru (2007) demonstrates this increased expressivity in the tabular setting, constructing an example on which a model assuming conditional independence fails.
While the expressive power of autoregressive models have been shown in various generative models (Parmar et al., 2018; Oord et al., 2016), autoregressive dynamics models have not seen much use in Model-based RL for continuous control before this work.
Algorithm
Require: Number of rollouts n , discount factor γ , horizon length H , policy π , dynamics model p , set of initial states Sfor i = 1 , 2 , . . .
n do
R i ←sample initial state s 0 ∼ Sfor t = 0 , 1 , 2 , . . .
, H − 1 do sample from policy: a t ∼ π ( · | s t ) sample from the dynamics model: s t +1 , r t +1 ∼ p ( · , · | s t , a t )
R i ← R i + γ t r t + end for end for return 1 n (cid:80) ni
=1 R i
Model-based OPE.
Once a dynamics model is trained from ofﬂine data, OPE can be performed in a direct and primitive way.
We let the policy and model interact—the policy generates the next action, the model plays the role of the environment and generates the next state and reward.
Due to the stochasticity in the model and the policy, we estimate the return for a policy with Monte-Carlo sampling and monitor standard error.
See Algorithm 1 for pseudocode.
R ELATED W ORK
Our work follows a long line of OPE research, which is especially relevant to many practical domains such as medicine (Murphy et al., 2001), recommendation systems (Li et al., 2011), and education (Mandel et al., 2014) in order to avoid the costs and risks associated with online evaluation.
There exists a large body of work on OPE, including methods based on importance weighting (Precup, 2000; Li et al., 2014) and Lagrangian duality (Nachum et al., 2019;
Yang et al., 2020; Uehara and Jiang, 2019).
The model-based approach that we focus on in this paper lies within the class of algorithms referred to as the direct method (Kostrikov and Nachum, 2020;
Dudík et al., 2011; Voloshin et al., 2019), which approximate the value of a new policy by either explicitly or implicitly estimating the transition and reward functions of the environment.
While model-based policy evaluation has been considered by previous works (Paduraru, 2007; Thomas and
Brunskill, 2016a; Hanna et al., 2017), it has largely been conﬁned to simple domains with ﬁnite state and action spaces where function approximation is not necessary.
By contrast, our work provides an extensive demonstration of model-based OPE in challenging continuous control benchmark domains.
Previous instances of the use of function approximation for model-based OPE (Hallak et al., 2015) impose strong assumptions on the probabilistic dynamics models, such as factorability of the MDP.
Our results indicate that even seemingly benign assumptions about the independence of different state dimensions can have detrimental consequences for the effectiveness of a model-based OPE estimate.
While the use of model-based principles in OPE has been relatively rare, it has been more commonly used for policy optimization.
The ﬁeld of model-based RL has matured in recent years to yield impressive results for both online
(Nagabandi et al., 2018;
Chua et al., 2018; Kurutach et al., 2018;
Janner et al., 2019) and ofﬂine (Matsushima et al., 2020; Kidambi et al., 2020; Yu et al., 2020;
Argenson and Dulac-Arnold, 2020) policy optimization.
Several of the techniques we employ, such as the normalization of the observation space, are borrowed from this previous literature (Nagabandi et al., 2018;
Chua et al., 2018).
Conversely, we present strong empirical evidence that the beneﬁts of our introduced autoregressive generative models of state observations do carry over to model-based policy optimization, at least in the ofﬂine setting, and this is an interesting avenue for future work. [Table] [Table]
R ESULTS
We conduct our experiments on the DeepMind control suite (Tassa et al., 2018), a set of control tasks implemented in MuJoCo (Todorov et al., 2012).
We use the ofﬂine datasets from RL Unplugged (Gulcehre et al., 2020), the details of which are provided in Table 1.
These environments capture a wide range of complexity, from 40K transitions in a 5-dimensional cartpole environment to 1.5 million transitions on complex manipulation tasks.
We follow the evaluation protocol in the Deep OPE (Anonymous, 2021) benchmark and use policies generated by four different algorithms: behavioral cloning (Bain, 1995), D4PG (Barth-Maron et al., 2018), Critic Regularized Regression (Wang et al., 2020), and ABM (Siegel et al., 2019).
With varied hyperparameters, these form a diverse set of policies of varying quality.
We perform a thorough hyperparameter sweep in the experiments and use standard practice from generative modeling to improve the quality of the models.
We allocate 80% of the data for training and 20% of the data for model selection.
We vary the depth and width of the neural networks (number of layers ∈ { 3 , 4 } , layer size ∈ { 512 , 1024 } ), add different amounts of noise to input states and actions, and consider two levels of weight decay for regularization (input noise ∈ { 0 , 1e − 6 , 1e − 7 } , weight decay ∈ { 0 , 1e − 6 } ).
For the choice of optimizer, we consider both Adam (Kingma and Ba, 2014) and SGD with momentum and ﬁnd Adam to be more effective at maximizing log-likelihood across all tasks in preliminary experiments.
We thus use Adam in all of our experiments with two learning rates ∈ { 1e − 3 , 3e − 4 } .
We decay the optimizer’s learning rate linearly to zero throughout training, ﬁnding this choice to outperform a constant learning rate.
Lastly, we ﬁnd that longer training often improves log-likelihood results.
We use 500 epochs for training ﬁnal models.
For each task we consider in total 48 hyperparameter combinations (listed above) for both models and pick the best model in each model family based on validation log-likelihood.
This model is then used for model-based OPE and policy optimization.
Note that, in our experiments, 20% of the transitions are used only for validation, but we believe one can re-train the models with the best hyperparameter conﬁguration on the full transition datasets to improve the results even further.
5. A UTOREGRESSIVE D YNAMICS M ODELS O UTPERFORM F EEDFORWARD M ODELS IN NLL
To evaluate the effectiveness of autoregressive dynamics models compared to feedforward counterparts, Table 2 reports negative log-likelihood (NLL) on the heldout validation set for the best performing models from our hyperparameter sweep.
For each environment, we report the NLL for the best-performing model (Top-1) and the average NLL across the Top-5 models.
The autoregressive model has lower NLL on all environments, indicating that it generalizes better to unseen data.
To study the impact of model size on NLL, Figure 2 shows validation NLL as a function of parameter count.
We ﬁnd that on small datasets large models hurt, but more importantly autoregressive
[Figure] [Figure]
cheetah run 5.2 5.0 4.8 4.6 4.4 4.2 4.0 Negative log-likelihood 0.
C o rr e l a t i o n c o e ff .
Feedforward Autoregressive fish swim 4.2 4.0 3.8 3.6 3.4 3.2 Negative log-likelihood 0.
C o rr e l a t i o n c o e ff .
Feedforward Autoregressive walker walk models outperform feedforward models regardless of the parameter count regime, i.e., even small autoregressive models attain a lower validation NLL compared to big feedforward models.
This indicates that autoregressive models have a better inductive bias in modeling the transition dynamics than feedforward models that make a conditional independence assumption.
5. A RE D YNAMICS M ODELS WITH L OWER NLL BETTER FOR M ODEL - BASED OPE?
We ultimately care not just about the log-likelihood numbers, but also whether or not the dynamics models are useful in policy evaluation and optimization.
To study the relationship of NLL and OPE performance for model-based methods, we compute OPE estimates via Algorithm 1 and compute the Pearson correlation between the OPE estimates and the true discounted returns.
This serves as a measure of the effectiveness of the model for OPE.
We repeat this for all 96 dynamics models we trained on a given environment and plot the correlation coefﬁcients against validation NLL in Figure 3.
Models with low NLL are generally more accurate in OPE.
Lambert et al.
(2020) have previously demonstrated that in Model-based RL, “training cost does not hold a strong correlation to maximization of episode reward.
" We use validation NLL instead, and our results on policy evaluation decouple the model from policy optimization, suggesting a more nuanced picture: low validation NLL numbers generally correspond to accurate policy evaluation, while higher NLL numbers are generally less meaningful.
In other words, if the dynamics model does not capture the transition dynamics accurately enough, then it is very hard to predict its performance on OPE.
However, once the model starts to capture the dynamics faithfully, we conjecture that NLL starts to become a reasonable metric for model selection.
For instance, validation NLL does not seem to be a great metric for ranking feedforward models, whereas it is more reasonable for autoregressive models.
5. C OMPARISON WITH O THER OPE M ETHODS
We adopt a recently proposed benchmark for OPE (Anonymous, 2021) and compare our model-based approaches with state-of-the-art OPE baselines therein.
Figures 4 and B.4 compare OPE estimates from two Fitted-Q Evaluation (FQE) baselines (Le et al., 2019; Kostrikov and Nachum, 2020; Paine et al., 2020), our feedforward models, and the autoregressive approach.
Each plot reports the Pearson [Figure] correlation between the OPE estimates and the true returns.
The autoregressive model consistently outperforms the feedforward model and FQE methods on most environments.
We report ensembling results in the appendix, but compare single models for fairness in the rest of the paper.
We compute summary statistics for OPE methods in Table 3, Table A.1, and Table A.2.
These tables report the Spearman’s rank correlation, regret, and absolute error, respectively.
These metrics capture different desirable properties of OPE methods (Anonymous, 2021); more details about how they are computed are in the appendix.
In all three metrics, the autoregressive model achieves the best median performance across nine environments, whereas the baseline model is not as good as FQE.
The only environment in which the autoregressive model has negative rank correlation is manipulator insert ball.
In addition, a major advantage of our model-based approach over FQE is that the model only needs to be trained once per environment—we do not need to perform additional policy-speciﬁc optimization, whereas FQE needs to optimize a separate Q-function approximator per policy.
5. A UTOREGRESSIVE D YNAMICS M ODELS FOR O FFLINE P OLICY O PTIMIZATION
Policy evaluation is an integral part of reinforcement learning.
Improvement in policy evaluation can therefore be adapted for policy optimization.
In this section, we explore two possibilities of using models to improve ofﬂine reinforcement learning.
In all experiments, we use Critic Regularized
Regression (CRR) as a base ofﬂine reinforcement learning algorithm (Wang et al., 2020). [Table] [Table]
Also see Table A.1 and
2 for and
[Figure] [Figure] U n d i s c o un t e d
E p
i s o d
i c
[Figure]
First we utilize the model during test time for planning by using a modiﬁed version of Model
Predictive Path Integral (MPPI) (Williams et al., 2015).
Unlike MPPI, we truncate the planning process after 10 steps of rollout and use the CRR critic to evaluate future discounted returns.
We provide additional details in the appendix.
Secondly, we use the model to augment the transition dataset to learn a better critic for CRR.
More precisely, given s it ∼ D , and the current policy π , we can generate additional data using the following process: ˆ a it ∼ π ( ·| s it ) , ˆ s it +1 , ˆ r it +1 ∼ p ( · , ·| s it , ˆ a t ) .
These two options are orthogonal and can be applied jointly.
We implemented both techniques on top of the CRR exp variant (Wang et al., 2020) and show their combined effect in Figure 5.
The ﬁgure shows that autoregressive dynamics models also outperform feedforward ones in the policy optimization context.
Notably, in the case of cheetah run and ﬁsh swim, using autoregressive models for planning as well as data augmentation enables us to outperform the previous state-of-the-art on these ofﬂine datasets.
Additionally, when using autoregressive dynamics models, both techniques improve performance.
In the appendix, we show this result as well as more ablations.
C ONCLUSION
This paper shows the promise of autoregressive models in learning transition dynamics for continuous control, showing strong results for off-policy policy evaluation and ofﬂine policy optimization.
Our contributions to ofﬂine model-based policy optimization are orthogonal to prior work that uses ensembles to lower the values when ensemble components disagree (Kidambi et al., 2020).
Incorporating conservative value estimation into our method is an interesting avenue for future research.
We use relatively primitive autoregressive neural architectures in this paper to enable a fair comparison with existing feedforward dynamics models.
That said, it will be exciting to apply more sophisticated autoregressive neural network architectures with cross attention (Bahdanau et al., 2014) and self-attention
Vaswani et al., 2017) to Model-based RL for continuous control.
A O FFLINE P OLICY E VALUATION
We use the baseline results in Anonymous (2021).
For convenience, we replicate their description of the OPE baselines and metrics.
A. OPE M ETRICS
To evaluate the OPE algorithms, we compute three different metrics between the estimated returns and the ground truth returns:
Rank correlation
This metric assesses how well estimated values rank policies.
It is equal to the correlation between the ranking (sorted order) by the OPE estimates and the ranking by the ground truth values.
Absolute Error :
This metric measures the deviations of the estimates from the ground truth and does not directly access the usefulness for ranking.
Regret@k
This metric measures how much worse the best policies identiﬁed by the estimates are than the best policy in the entire set.
Regret@k is the difference between the actual expected return of the best policy in the entire set, and the actual value of the best policy in the top-k set.
A. OPE B ASELINES Fitted
Q-Evaluation (FQE)
As in Le et al.
(2019), we train a neural network to estimate the value of the evaluation policy π e by bootstrapping from Q ( s (cid:48) , π e ( s (cid:48) )) .
We tried two different implementations, one from Kostrikov and Nachum (2020) and another from Paine et al.
Importance Sampling (IS)
We perform importance sampling with a learned behavior policy.
We use the implementation from Kostrikov and Nachum (2020), which uses self-normalized (also known as weighted) step-wise importance sampling (Liu et al., 2018; Nachum et al., 2019).
Since the behavior policy is not known explicitly, we learn an estimate of it via a max-likelihood objective over the dataset D , as advocated by Hanna et al.
In order to be able to compute log-probabilities when the target policy is deterministic, we add artiﬁcial Gaussian noise with standard deviation 0 .
01 for all deterministic target policies.
Doubly-Robust (DR)
We perform weighted doubly-robust policy evaluation based on Thomas and
Brunskill (2016b) and using the implementation of Kostrikov and Nachum (2020).
Speciﬁcally, this method combines the IS technique above with a value estimator for variance reduction.
The value estimator is learned according to Kostrikov and Nachum (2020), using deep FQE with an L2 loss function.
DICE This method uses a saddle-point objective to estimate marginalized importance weights
d π ( s, a ) /d
π B ( s, a ) ; these weights are then used to compute a weighted average of reward over the ofﬂine dataset, and this serves as an estimate of the policy’s value in the MDP.
We use the implementation from Yang et al.
(2020) corresponding to the algorithm BestDICE .
Variational Power Method (VPM)
This method runs a variational power iteration algorithm to estimate the importance weights
d π ( s, a ) /d
π B ( s, a ) without the knowledge of the behavior policy.
It then estimates the target policy value using weighted average of rewards similar to the DICE method.
Our implementation is based on the same network and hyperparameters for OPE setting as in Wen et al.
We further tune the hyperparameters including the regularization parameter λ , learning rates α θ
and α v , and number of iterations on the Cartpole swingup task using ground-truth policy value, and then ﬁx them for all other tasks.
A. E NSEMBLING
As in Chua et al.
; Janner et al.
(2019), we can form an ensemble using our best-performing models.
We generate rollouts using the procedure detailed in Janner et al.
(2019), forming an ensemble with 4 models.
We see some improvement in policy evaluation results, as shown in Figure A.1.
Ensembling could likely be further improved by forcing unique hyperparameter settings and seeds.
[Table] [Table]
A
[Table]
s
[Table]
l u
[Table]
e
E rr o r b t w .
O P E a nd g r ound
[Table]
A b s o l u t e E rr o r b t w . O P E a nd g r ound t r u t h
[Table]
[Figure]
returns using the top model versus estimates of returns using an ensemble of the top-4 models.
Require: state s , policy π , dynamics model p , critic Q , temperature β , and noise variance σ 2 .
for m = 1 , ..., M do for n = 1 , ..., N do s
← s
R n ←for τ = 0 , ..., H − 1 do a τn ∼ π ( ·| s τn ) s τ +1 n , r τ +1 n ∼ π ( · , ·| s τn , a τn )
R n ← R n + γ τ r τ +1 n end for a H ∼ π ( ·| s H )
R n ← R n + γ H Q ( s Hn , a Hn ) end for
Re-deﬁne π such that π ( ·| ˆ s τ ) =
(cid:80) n [Equation] ) N ( ·
[Equation]
a τn , σ [Equation] I ) .
( π depends on τ and not ˆ s .)
end for sample ﬁnal action a ∼ (cid:80) n
[Equation]
) return a
B A DDITIONAL D ETAILS R EGARDING P OLICY O PTIMIZATION
To test dynamic models for policy optimization, we implement the two methods discussed in
Section 5.4 on top of CRR exp , one of the CRR variants (Wang et al., 2020).
We use the RL
Unplugged datasets (Gulcehre et al., 2020) for all environments studied in this section. When using data augmentation, we adopt a 1-to-1 ratio between the original dataset and the augmented dataset.
To take advantage of the dynamics models at test time, we use a variant of Model Predictive Path
Integral (MPPI) for planning.
To reduce the planning horizon, we truncate the model rollout using
CRR critics.
The details of the planning procedure is summarized in Algorithm 2.
All hyperparameter tuning for the planning process is conducted on the “cartpole swingup” task.
The hyperparameters used in the planning process are M = 3 , N = 16 , H = 10 , β = 0 .
1 , and σ 2 = 0 .
To match the temperature used in the planning component, we choose β = 0 .
1 for the CWP component of CRR.
This change, however, does not impact the baseline CRR agent performance much.
With the exception of β and the planning component, all hyperparameters are kept the same as CRR exp .
[Figure] [Figure]
We compare the agents’ performance with and without the planning procedure to test its effects.
As shown in Figure B.2, planning using an autoregressive model signiﬁcantly increases performance.
Data augmentation does not change the agents’ performance on cartpole swingup, ﬁsh swim, or ﬁnger turn hard.
It, however, boosts performance considerably on cheetah run.
In Figure B.3, we show the effects of data augmentation on cheetah run.
[Figure]