RAMBO-RL: Robust Adversarial Model-Based Ofﬂine Reinforcement Learning
Abstract
Ofﬂine reinforcement learning (RL) aims to ﬁnd performant policies from logged data without further environment interaction.
Model-based algorithms, which learn a model of the environment from the dataset and perform conservative policy optimisation within that model, have emerged as a promising approach to this problem.
In this work, we present Robust Adversarial Model-Based Ofﬂine RL (RAMBO), a novel approach to model-based ofﬂine RL.
We formulate the problem as a two-player zero sum game against an adversarial environment model.
The model is trained to minimise the value function while still accurately predicting the transitions in the dataset, forcing the policy to act conservatively in areas not covered by the dataset.
To approximately solve the two-player game, we alternate between optimising the policy and adversarially optimising the model.
The problem formulation that we address is theoretically grounded, resulting in a probably approximately correct (PAC) performance guarantee and a pessimistic value function which lower bounds the value function in the true environment.
We evaluate our approach on widely studied ofﬂine RL benchmarks, and demonstrate that it outperforms existing state-of-the-art baselines.
Introduction
Reinforcement learning (RL) [61] has achieved state-of-the-art performance on many sequential decision-making problems [40, 45, 59].
However, the need for extensive exploration prohibits the application of RL to many real world domains where such exploration is costly or dangerous.
Ofﬂine
RL [33, 35] overcomes this limitation by learning policies from static, pre-recorded datasets.
Online RL algorithms perform poorly in the ofﬂine setting due to the distributional shift between the state-action pairs in the dataset and those taken by the learnt policy.
Thus, an important aspect of ofﬂine RL is to introduce conservatism to prevent the learnt policy from executing state-action pairs which are out of distribution.
Model-free ofﬂine RL algorithms [15, 23, 27, 29, 31, 72] train a policy from only the data present in the ﬁxed dataset, and incorporate conservatism either into the value function or by directly constraining the policy.
On the other hand, model-based ofﬂine RL algorithms [76, 75, 25, 63, 39] use the dataset to learn a model of the environment, and train a policy using additional synthetic data generated from that model.
By training on additional synthetic data, model-based algorithms can potentially generalise better to states not present in the dataset, or to solving new tasks.
Previous approaches to model-based ofﬂine RL incorporate conservatism by estimating the uncertainty in the model and applying reward penalties for state-action pairs that have high uncertainty [76, 25].
However, uncertainty estimation can be unreliable for neural network models [75, 37].
Like recent work [75], we propose an approach for ofﬂine model-based RL which does not require uncertainty estimation .
In this work we present Robust Adversarial Model-Based Ofﬂine (RAMBO) RL, a new algorithm for model-based ofﬂine RL.
RAMBO incorporates conservatism by modifying the transition dynamics of the learnt Markov decision process (MDP) model in an adversarial manner.
We formulate the problem of ofﬂine RL as a zero-sum game against an adversarial environment.
To solve the resulting maximin optimisation problem, we alternate between optimising the agent and optimising the adversary in the style of Robust Adversarial RL (RARL) [50].
Unlike existing RARL approaches, our modelbased approach forgoes the need to deﬁne and train an adversary policy, and instead only learns an adversarial model of the MDP.
We train the agent policy with an actor-critic algorithm using synthetic data generated from the model in addition to data sampled from the dataset, similar to Dyna [60] and a number of recent methods [22, 76, 25, 75].
We update the environment model so that it reduces the value function for the agent policy, while still accurately predicting the transitions in the dataset.
As a result, our approach introduces conservatism by generating pessimistic synthetic transitions for state-action pairs which are out-of-distribution.
The theoretical formulation of ofﬂine RL that our algorithm addresses yields a PAC bound for the performance gap with respect to any policy covered by the dataset, and a pessimistic value function that lower bounds the value function in the true environment.
In summary, the main contributions of this work are:
• RAMBO, a novel and theoretically-grounded model-based ofﬂine RL algorithm which enforces conservatism by training an adversarial dynamics model.
• Adapting the Robust Adversarial RL approach to model-based ofﬂine RL by proposing a new formulation of RARL, where instead of deﬁning and training an adversary policy, we directly train the model adversarially.
In our experiments we demonstrate that RAMBO outperforms current state-of-the-art algorithms on the D4RL benchmarks [14]. Furthermore, we provide ablation results which show that training the model adversarially is crucial to the strong performance of RAMBO.
Related Work
Ofﬂine RL: Ofﬂine RL addresses the problem of learning policies from ﬁxed datasets, and has been applied to domains such as healthcare [42, 57], natural language processing [24, 23], and robotics [30, 38, 51].
Model-free ofﬂine RL algorithms do not require a learnt model.
Approaches for model-free ofﬂine RL include importance sampling algorithms [36, 41], constraining the learnt policy to be similar to the behaviour policy [16, 28, 72, 23, 58, 15], incorporating conservatism into the value function during training [9, 27, 31, 73], using uncertainty quantiﬁcation to generate more robust value estimates [1, 2, 29], or applying only a single iteration of policy iteration [7, 49].
In contrast, modelbased approaches learn a model of the environment and generate synthetic data from that model [60] to optimise a policy using either planning [4] or RL algorithms [25, 76, 75].
By training a policy on additional synthetic data, model-based approaches have the potential for broader generalisation and for solving new tasks [6, 76].
A simple approach to ensuring conservatism is to constrain the policy to be similar to the behaviour policy in the same fashion as some model-free approaches [8, 39, 63].
Another approach is to apply reward penalties for executing state-action pairs with high uncertainty in the environment model [25, 74, 76].
However, this requires explicit uncertainty estimates which may be unreliable for neural network models [17, 37, 46, 75].
COMBO [75] obviates the need for uncertainty estimation in model-based ofﬂine RL by adapting model-free techniques [31] to regularise the value function for out-of-distribution samples.
Like COMBO, our approach does not require uncertainty estimation.
Most approaches to model-based ofﬂine RL use maximum likelihood estimates (MLE) of the MDP trained using standard supervised learning [4, 39, 63, 76, 75].
However, other methods have been proposed to learn models which are more suitable for ofﬂine policy optimisation.
One approach is to reweight the loss function to ensure the model is accurate under the state-action distribution generated by the policy [34, 53, 20].
In contrast, our approach produces pessimistic synthetic transitions when out-of-distribution.
Most related to our work is a recent paper [66] which introduces the maximin formulation of ofﬂine
RL that we address.
This existing work motivates our approach theoretically by showing that the problem formulation obtains probably approximately correct (PAC) guarantees.
However, [66] only addresses the theoretical aspects of the problem formulation and does not propose a practical algorithm.
In this work, we propose a practical RL algorithm to solve the maximin formulation of model-based ofﬂine RL.
Robust RL: Algorithms for Robust MDPs [5, 12, 21, 43, 54, 64, 70]
ﬁnd the policy with the best worst-case performance over a set of possible MDPs.
Typically, it is assumed that the uncertainty set of MDPs is speciﬁed a priori.
To eliminate the need to specify the set of possible MDPs, model-free approaches to Robust MDPs [68, 55] instead assume that samples can be drawn from a misspeciﬁed
MDP which is similar to the true MDP.
As our work addresses ofﬂine RL, we assume that we have a ﬁxed dataset from the true MDP.
Our approach is conceptually similar to Robust Adversarial RL (RARL) [50], a method proposed to improve the robustness of RL policies in the online setting.
RARL is posed as a two-player zero-sum game where the agent plays against an adversary which perturbs the environment.
Formulations of model-free RARL differ in how they deﬁne the action space of the adversary.
Options include allowing the adversary to apply perturbation forces to the simulator [50], add noise to the agent’s actions [65], or periodically take over control [48].
A model-based approach to RARL is proposed in [13], which learns an optimistic and pessimistic model to encourage online exploration.
However, this existing approach requires uncertainty estimation as well as an adversarial policy to be learnt in addition to the model .
Our work follows the paradigm of RARL and alternates between agent and adversarial updates in a maximin formulation.
We adapt model-based RARL to the ofﬂine setting and propose an alternative formulation: we eliminate the need to learn an adversary policy and instead directly modify the MDP model adversarially .
Preliminaries
MDPs and Ofﬂine RL: An MDP is deﬁned by the tuple, M = ( S, A, T, R, µ 0 , γ ) .
S and A denote the state and action spaces respectively, R ( s, a ) is the reward function, T ( s (cid:48) | s, a ) is the transition function, µ 0 is the initial state distribution, and γ ∈ (0 , 1) is the discount factor.
In this work we consider Markovian policies, π ∈ Π , which map from each state to a distribution over actions.
We denote the (improper) discounted state visitation distribution of a policy by d πM ( s ) :
= (cid:80) ∞
t =0 γ t Pr( s t = s | π, M ) , where Pr( s t = s | π, M ) is the probability of reaching state s at time t by executing policy π in M .
The improper state-action visitation distribution is d πM ( s, a ) =
π ( a | s ) · d πM ( s ) .
We also denote the normalised state-action visitation distribution by ˜ d πM ( s, a ) = (1 − γ ) · d πM ( s, a ) .
The value function, V πM ( s ) , represents the expected discounted return from executing π from state s in M : V πM ( s )
=
E π,M (cid:2) (cid:80) ∞
t =
γ t R ( s t , a t ) (cid:3) .
We write V πM to indicate the value function under the initial state distribution, i.e. V πM = (cid:80) s ∈ S µ 0 ( s ) V πM ( s ) .
The standard objective for MDPs is to ﬁnd the policy which maximises V πM .
The state-action value function, Q πM ( s, a ) , is the expected discounted cumulative reward from taking action a at state s and then executing π thereafter.
In ofﬂine RL we only have access to a ﬁxed dataset of transitions from the MDP, D
= { ( s i , a i , r i , s (cid:48)
i ) } |D| i =1 .
The goal of ofﬂine RL is to ﬁnd the best possible policy using the ﬁxed dataset.
Model-Based Ofﬂine RL Algorithms: Model-based approaches to ofﬂine RL use a model of the
MDP to help train a policy.
The dataset is used to learn a dynamics model, (cid:98) T , which is typically trained via maximum likelihood estimation: min (cid:98) T E ( s,a,s (cid:48) )
∼D (cid:2) − log (cid:98) T ( s (cid:48) | s, a ) (cid:3) .
A model of the reward function, (cid:98) R ( s, a ) , can also be learnt if it is unknown.
The estimated MDP, (cid:99) M
=
( S, A, (cid:98) T, (cid:98) R, µ 0 , γ ) , has the same state and action space as the true MDP but uses the learnt transition and reward functions.
Thereafter, any planning or RL algorithm can be used to recover optimal policy in the learnt model,
[Equation]
Unfortunately, directly applying this approach to the ofﬂine RL setting does not perform well due to distributional shift.
In particular, if the dataset does not cover the entire state-action space, the model will inevitably be inaccurate for some state-action pairs.
Thus, naive policy optimisation on a learnt model in the ofﬂine setting can result in model exploitation [22, 32, 53].
To mitigate this issue, we propose the novel approach of enforcing conservatism by adversarially modifying the transition dynamics of (cid:99) M .
In line with existing works [76, 75, 8], we use model-based policy optimisation (MBPO)
[22] to learn the optimal policy for (cid:99) M .
MBPO utilises a standard actor-critic RL algorithm.
However, the value function is trained using an augmented dataset D ∪ D (cid:99) M , where D (cid:99) M is synthetic data generated by simulating rollouts in the learnt model.
To generate the synthetic data, MBPO performs k -step rollouts in (cid:99) M starting from states s ∈ D , and adds this data to D (cid:99) M .
To train the policy, minibatches of data are drawn from D ∪ D (cid:99) M , where each datapoint is sampled from the real data, D , with probability f , and from D (cid:99) M with probability 1 − f.
Robust Adversarial Reinforcement Learning: RARL addresses the problem of ﬁnding a robust agent policy, π , in the online RL setting by posing the problem as a two-player zero sum game against adversary policy, ¯ π :
[Equation]
where V π, ¯ π M is the expected value from executing π and ¯ π in environment M .
Different approaches deﬁne the action space for ¯ π in different ways, as discussed in Section 2.
For a scalable approximation to the optimisation problem in Equation 1, algorithms for RARL alternate between applying steps of stochastic gradient ascent to the agent’s policy to increase the expected value, and stochastic gradient descent to the adversary’s policy to decrease the expected value.
In our work, we follow the RARL paradigm of alternating between agent and adversarial updates and adapt it to the model-based ofﬂine setting.
Instead of deﬁning a separate adversary policy, we treat the model itself as the policy to be adversarially trained.
Problem Formulation
For the sake of generality, we assume that both the transition function and reward function are unknown.
Hereafter, we write (cid:98) T to denote both the learnt dynamics and reward function, where (cid:98) T ( s (cid:48) , r | s, a ) is the probability of receiving reward r and transitioning to s (cid:48)
after executing ( s, a ) .
We address the maximin formulation of ofﬂine RL recently proposed by [66]:
Problem 1.
For some dataset, D , and some ﬁxed constant ξ > 0 , ﬁnd the policy π deﬁned by
[Equation]
where
[Equation]
where TV ( P 1 , P 2 ) is the total variation distance between distributions P 1 and P 2 , and (cid:98) T MLE denotes the maximum likelihood estimate of the MDP given the ofﬂine dataset, D .
Thus, the set deﬁned in Equation 3 contains MDPs which are similar to the maximum likelihood estimate under state-action pairs in D .
However, because the expectation in Equation 3 is taken under D , there is no restriction on (cid:98) T for regions of the state-action space not covered by D .
We present a brief overview of the theoretical guarantees from [66] in the following subsection.
Remark 1.
Note that Problem 1 differs from the pessimistic MDP formulations introduced by
MOPO [76] and MOReL [25].
Problem 1 considers the worst-case transition dynamics, while the pessimistic MDPs constructed by MOPO and MOReL only modify the reward function by applying reward penalties for state-action pairs with high uncertainty.
4.1 Theoretical Motivation
The theoretical analysis from [66] shows that solving Problem 1 outputs a policy that with high probability is approximately as good as any policy with a state-action distribution that is covered by the dataset.
This is formally stated in the following theorem.
Theorem 1 (PAC guarantee from [66], Theorem 5) .
Denote the true MDP transition function by T , and let M denote a hypothesis class of MDP models such that T ∈ M .
Let π denote the solution to
Problem 1 for dataset D .
Then with probability 1 − δ for any policy, π ∗ ∈ Π , we have
[Equation]
[Equation]
where ρ is the state-action distribution from which D was sampled, and c and c 1 are universal constants.
We refer the reader to Appendix A of [66] for the deﬁnitions of G M 1 , G M 2 , and ξ n .
The quantity C π ∗ is upper bounded by the maximum density ratio between the comparator policy, π ∗ , and the ofﬂine distribution, i.e. C π ∗ ≤ max ( s,a ) ˜
d π ∗ T ( s, a ) /ρ
( s, a ) .
It represents the discrepancy between the distribution of data in the dataset compared to the visitation distribution of policy π ∗ .
Theorem 1 shows that if we ﬁnd a policy by solving Problem 1, the performance gap of that policy is bounded with respect to any other policy π ∗ that has a state-action distribution which is covered by the dataset.
Furthermore, the value function under the worst-case model in the set deﬁned by Problem 1 is a lower bound on the value function in the true environment, as stated by Proposition 1.
Proposition 1 (Pessimistic value function) .
Let T denote the true transition function for some MDP, and let M D be the set of MDP models deﬁned in Equation 3.
Then for any policy π , with probability 1 − δ we have that
[Equation]
Proposition 1 follows from the fact that T ∈ M D with high probability, which is proven in [66] (Appendix E.2).
Proposition 1 shows that we can expect the performance of any policy in the true
MDP to be at least as good as the value in the worst-case model deﬁned in Problem 1.
While [66] provides the theoretical motivation for solving Problem 1, it does not propose a practical algorithm.
In this work, we focus on developing a practical approach to solving Problem 1.
RAMBO-RL
In this section, we present Robust Adversarial Model-Based Ofﬂine RL (RAMBO), our algorithm for solving Problem 1.
The main difﬁculty with solving Problem 1 is that it is unclear how to ﬁnd the worst-case MDP in the set deﬁned in Equation 3.
To arrive at a scalable solution, we propose a novel approach which is in the spirit of RARL.
We alternate between optimising the agent policy to increase the expected value, and adversarially optimising the model to decrease the expected value.
In this section, we ﬁrst describe how we compute the gradient to adversarially train the model.
Then, we discuss how to ensure that the model remains approximately within the constraint set deﬁned in
Problem 1.
Finally, we present our overall algorithm.
5.1 Model Gradient
We propose a policy gradient-inspired approach to adversarially optimise the model.
Typically, policy gradient algorithms are used to modify the distribution over actions taken by a policy at each state [62, 71].
In contrast, the update that we propose modiﬁes the likelihood of the successor states and rewards in the MDP model.
We assume that the MDP model is deﬁned by parameters, φ , and we write (cid:98) T φ to indicate this.
We denote by V πφ the value function for policy π in model (cid:98) T φ .
To approximately ﬁnd min (cid:98) T φ ∈M D V πφ as required by Problem 1 via gradient descent, we wish to compute the gradient of the model parameters that reduces the value of the policy within the model, i.e. ∇ φ V πφ .
Proposition 2 (Model Gradient) .
Let φ denote the parameters of a parametric MDP model (cid:98) T φ , and let V πφ denote the value function for policy π in (cid:98) T φ .
Then:
[Equation]
The proof of Proposition 2 is given in Appendix A. We can subtract the baseline Q πφ ( s, a ) without biasing the gradient estimate (see Appendix A.1 for details):
[Equation]
The Model Gradient differs from the standard policy gradient in that a) it is used to update the likelihood of successor states in the model, rather than actions in a policy, and b) the “advantage” term r + γV πφ ( s (cid:48) )
− Q πφ ( s, a ) compares the utility of receiving reward r and transitioning to s (cid:48) to the expected value of state action pair ( s, a ) .
In contrast, the standard advantage term, Q ( s, a ) − V ( s ) [56], compares the value of executing state-action pair ( s, a ) to the value at s .
To estimate V πφ and Q πφ , we use the critic learnt by the actor-critic algorithm used for policy optimisation.
Thus, we use the critic both for training the policy and adversarially training the model.
Remark 2.
The Model Gradient can be thought of as a speciﬁc instantiation of the policy gradient if we view (cid:98) T φ as a adversarial “policy” on an augmented MDP, M + , i.e. (cid:98) T φ : S + → Dist( A + ) .
The augmented state space, S + = S
×
A includes the state in the original MDP augmented by the action taken by the agent.
The augmented action space consists of the reward applied and the successor state,
A + = S ×
[ R min , R max ] .
Thus, we can think of this approach as an instantiation of RARL in which the adversary policy ( ¯ π in Equation 1) that we train is the model itself .
5.2 Adversarial Model Training
If we were to update the model using Equation 5 alone, this would allow the model to be modiﬁed arbitrarily such that the value function in the model is reduced.
However, the set of plausible MDPs given by Equation 3 states that over the dataset, D , the model (cid:98) T φ should be close to the maximum likelihood estimate, (cid:98) T MLE .
Speciﬁcally, in the inner optimisation of Problem 1 we wish to ﬁnd a solution to the constrained optimisation problem
[Equation]
The Lagrangian relaxation leads to the unconstrained problem
[Equation]
where λ is the Lagrange multiplier.
Rather than optimising the Lagrange multiplier, we ﬁnd that in practice ﬁxing λ to apply a constant weighting between the two terms works well with minimal tuning.
To facilitate easier tuning of the learning rate, in our implementation we apply the weighting constant to the value function term rather than the model term, which is equivalent up to a scaling factor.
This leads to
[Equation]
We aim to make our algorithm efﬁcient and simple to implement. Therefore, rather than minimising the TV distance between the model and MLE model as prescribed by Equation 8, we directly optimise the standard MLE loss.
This leads to the ﬁnal loss function:
[Equation]
Thus, the loss function for the model in Equation 9 simply adds the adversarial term to the standard MLE loss. By minimising the loss function in Equation 9, the model is trained to a) predict the transitions within the dataset, and b) reduce the value function of the policy, with λ determining the tradeoff between these two objectives.
Choosing λ to be small ensures that the MLE term dominates for transitions within D , ensuring that the model ﬁts the dataset accurately.
Because the MLE term is only computed over D , the adversarial term dominates outside of the dataset meaning that the model is modiﬁed adversarially for transitions outside of the dataset.
To estimate the gradient of the loss function in Equation 9 for stochastic gradient descent, we sample a minibatch of transitions from D to estimate the MLE term.
The gradient for the value function term is computed using the Model Gradient.
The transitions used to compute the Model Gradient term must be sampled under the current policy and model (Equation 5).
Therefore, to estimate the Model
Gradient term, we generate a minibatch of transitions by simulating the current policy in (cid:98) T φ .
Like previous works [10, 76, 75, 8], we represent the dynamics model using an ensemble of neural networks.
Each neural network produces a Gaussian distribution over the next state and reward:
(cid:98) T φ ( s (cid:48) , r | s, a ) =
N ( µ φ ( s, a ) , Σ φ ( s, a )) .
A visualisation of the result of adversarially training the dynamics model can be found in Appendix C.3.
Normalisation The composite loss function in Equation 9 comprises two terms which may have different magnitudes across domains depending on the scale of the states and rewards.
To enable easier tuning of the adversarial loss weighting, λ , across different domains we perform the following normalisation procedure.
Prior to training, we normalise the states in D in the manner proposed in [15], by subtracting the mean and dividing by the standard deviation of each state dimension in the dataset.
Additionally, when computing the gradient in Equation 5 we normalise the advantage terms, r + γV πφ ( s (cid:48) )
− Q πφ ( s, a ) , according to the mean and standard deviation across each minibatch.
Advantage normalisation is common practice in policy gradient RL implementations [3, 52].
5.3 Algorithm
We are now ready to present our overall approach in Algorithm 1.
The ﬁrst step of RAMBO is to pretrain the environment dynamics model using standard MLE (Line 1).
Thereafter, the algorithm follows the format of RARL.
At each iteration, we apply gradient updates to the agent to increase the expected value, followed by gradient updates to the model to decrease the expected value.
Prior to each agent update, we generate synthetic k -step rollouts starting from states in D by simulating rollouts in the current MDP model (cid:98) T φ .
This data is added to the synthetic dataset D (cid:98) T φ (Line 3).
Following previous approaches [76, 75, 22] we only store data from recent iterations in D (cid:98) T φ .
The agent’s policy and value functions are trained with an off-policy actor-critic algorithm using samples from D ∪ D (cid:98) T φ (Line 4).
In our implementation, we use soft actor-critic (SAC) [19] for agent training.
To update the model to minimise the loss in Equation 9 (Line 5), we sample data from D to estimate the gradient for the MLE component.
To compute the adversarial component we generate samples by simulating the current policy and model, and utilise the value function learnt by the agent to compute the gradient according to Equation 5.
Require:
Normalised dataset, D ;
Experiments
In our experiments, we aim to: a) evaluate how well RAMBO performs compared to state-ofthe-art baselines, b) examine whether RAMBO can be tuned ofﬂine, c) determine the impact of adversarial training on the performance of the algorithm, and d) investigate the difference between RAMBO and COMBO, the most similar prior algorithm.
The code for our experiments is available at github.com/marc-rigter/rambo.
We evaluate our approach on the following domains.
MuJoCo There are three different environments representing different robots ( HalfCheetah, Hopper,
Walker2D ), each with 4 datasets ( Random, Medium, Medium-Replay, Medium-Expert ).
Random contains transitions collected by a random policy.
Medium contains transitions collected by an early-stopped SAC policy.
Medium-Replay consists of the replay buffer generated while training the
Medium policy.
The Medium-Expert dataset contains a mixture of suboptimal and expert data.
AntMaze The agent controls a robot and navigates to reach a goal, receiving a sparse reward only if the goal is reached.
There are three different layouts of maze ( Umaze, Medium, Large ), and different dataset types ( Fixed, Play, Diverse ) which differ in terms of the variety
Hyperparameter Details
The base hyperparameters that we use for RAMBO mostly follow those used in SAC [19] and COMBO [75].
We ﬁnd that the performance of RAMBO is sensitive to the choice of rollout length, k , consistent with ﬁndings in previous works [22, 37].
The other critical parameter for RAMBO is the choice of the adversarial weighting, λ .
For each dataset, we choose the rollout length and the adversarial weighting from one of three possible conﬁgurations: ( k, λ ) ∈ {(2, 3e-4), (5, 3e-4), (5, 0) } .
We included ( k, λ ) =
(5, 0) as we found that an adversarial weighting of 0 worked well for some datasets.
For the MuJoCo datasets we performed model rollouts using the current policy, and initialised the policy using behaviour cloning (BC) which is a common practice in ofﬂine RL [25, 74]. Ablation results in Appendix C.4 indicate that the BC initialisation results in a small improvement. For the AntMaze datasets we used a random rollout policy and a randomly initialised policy as we found that this performed better.
Further details about the hyperparameters are in Appendix B.
Evaluation We present two different evaluations of our approach: RAMBO and RAMBO OFF . For RAMBO, we ran each of the three hyperparameter conﬁgurations for ﬁve seeds each, and report the best performance across the three conﬁgurations.
Thus, our evaluation of RAMBO utilises limited online tuning which is the most common practice among existing model-based ofﬂine RL algorithms [25, 37, 39, 76].
The performance obtained for each of the hyperparameter conﬁgurations is included in Appendix C.2.
Ofﬂine hyperparameter selection is an important topic in ofﬂine RL [47, 77].
Therefore, we present additional results for RAMBO OFF where we select between the three choices of hyperparameters ofﬂine using a simple heuristic (details in Appendix B.5) based on the magnitude and stability of the
Q -values during ofﬂine training.
We ﬁrst select the hyperparameters using the heuristic, and then rerun each dataset for 5 seeds to generate the results for RAMBO OFF .
Baselines We compare RAMBO against state-of-the-art model-based (COMBO [75], RepBSDE [34], MOReL [25], and MOPO [76]) and model-free (CQL [31], IQL [28], and TD3+BC [15]) ofﬂine RL algorithms.
We provide results for all algorithms for the MuJoCo-v2 D4RL datasets and the AntMaze-v0 datasets (details in Appendix B.7).
6.1 Results
D4RL Performance The results in Table 1 show that RAMBO achieves the best total score for the MuJoCo locomotion domains, outperforming existing state-of-the-art methods. Furthermore,
RAMBO obtains the best overall score on both the Medium and Medium-Replay dataset types. For the random datasets, RAMBO is outperformed only by MOReL. This shows that RAMBO performs very well for datasets that are either noisy or consist of suboptimal data. For the Medium-Expert datasets, RAMBO is outperformed by most of the baseline algorithms, suggesting that RAMBO is less suitable for high-quality datasets.
However, for the Medium-Expert datasets, simpler approaches such as performing behaviour cloning on the best 10% of trajectories can be used to achieve stronger performance than ofﬂine RL methods [28].
Therefore, the suboptimal performance of RAMBO on the Medium-Expert datasets is less of a concern, as applying ofﬂine RL algorithms may not be the most suitable approach for these high-quality datasets.
For AntMaze, the model-based algorithms perform considerably less well than the model-free approaches. Unlike the other model-based approaches, RAMBO at least scores greater than zero for most of the datasets.
Our results echo previous ﬁndings that model-based approaches struggle to perform well in the AntMaze domains, potentially because model-based algorithms are too aggressive and collide with walls [67].
Recent work [67] showed that reverse rollouts can lead to stronger performance for model-based methods in these domains.
In future work, we wish to investigate whether combining RAMBO with reverse rollouts improves the performance for AntMaze.
Ofﬂine Tuning In Table 1, we also present results for RAMBO OFF where the ﬁnal hyperparameters are chosen using the heuristic described in Appendix B.5. We see that there is a slight degradation in the performance relative to RAMBO, which uses online tuning. However, RAMBO OFF still achieves comparable performance to the best existing approaches on the MuJoCo datasets. This suggests that suitable hyperparameters for RAMBO can reliably be chosen using our ofﬂine heuristic.
[Table] [Table] [Table] Ablation of Adversarial Training In Table 2 we present results for RAMBO with no adversarial updates.
These results demonstrate that overall performance degrades if the adversarial training is removed.
This parallels previous ﬁndings that mitigating the issue of model exploitation is crucial to obtaining strong performance in model-based ofﬂine RL.
Interestingly however, for some speciﬁc datasets we obtain the best performance with no adversarial training (Appendix C.2).
This suggests that a potential direction for future work could be trying to identify which types of problems do not require regularisation for a successful policy to be trained ofﬂine with model-based RL.
Comparison to COMBO We focus especially on comparing our approach to COMBO, as it is the most similar existing algorithm. We compare RAMBO and COMBO on the Single Transition toy example which is described in detail in Appendix C.1.
This domain has a one-dimensional state and action space, and several distinct regions of the action space are covered by the dataset.
We use this domain to investigate whether the policies optimised by RAMBO and COMBO tend to become stuck in local optima.
Table 3 compares the performance of RAMBO and COMBO on the Single Transition Example.
Further analysis in Appendix C.1 shows that for this problem, the pessimistic value function updates used by COMBO create local maxima in the Q -function which are present throughout training.
Policy optimisation can become stuck in these local maxima.
On the other hand, the value function produced by RAMBO is initially optimistic, and pessimism is introduced into the value function gradually as the transition function is modiﬁed adversarially. Adversarial modiﬁcation of the transition function is visualised in Appendix C.3.
As a result of this gradual introduction of pessimism, we observe that the policy produced by RAMBO is less likely to become stuck in poor local maxima, and better overall performance is obtained in Table 3.
This observation may help to explain why RAMBO is able to achieve consistently strong performance relative to existing algorithms in the MuJoCo domains.
Gradually increasing the level of pessimism could be a useful modiﬁcation for existing ofﬂine RL algorithms to be investigated in future work.
Conclusion and Future Directions
RAMBO is a promising new approach to ofﬂine RL which imposes conservatism by adversarially modifying the transition dynamics of a learnt model.
Our approach is theoretically justiﬁed, and achieves state-of-the-art performance on standard benchmarks.
There are a number of possible extensions to RAMBO, some of which we have already discussed.
In addition, we would like to apply RAMBO to image-space domains by using deep latent variable models to compress the state space [18, 51] and adversarially perturbing the transition dynamics in the latent space representation. Another direction that we would like to investigate is the use of adversarially trained models to aid interpretability in deep RL [44] by generating imagined worst-case trajectories.
Finally, we wish to investigate applying the ideas developed in this work to the online
RL setting to improve robustness. Acknowledgements
This work was supported by a Programme Grant from the Engineering and Physical Sciences Research
Council (EP/V000748/1), the Clarendon Fund at the University of Oxford, and a gift from Amazon
Web Services. Additionally, this project made use of time on Tier 2 HPC facility JADE2, funded by the Engineering and Physical Sciences Research Council (EP/T022205/1). The authors would like to thank Raunak Bhattacharyya, Paul Duckworth, and Matthew Budd for their feedback on an earlier draft of this work.
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We included results for the AntMaze datasets where our approach performs less well than model-free methods.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] We include a complete proof of Proposition 2.
For the theoretical results from [66], we provide references to the appropriate parts of that paper.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] A URL to the code is provided.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Detailed information about implementation details and hyperparameters is in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Standard deviation with respect to seeds is reported for all results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Details of compute used per run and total compute used for full evaluation is in Appendix B.8.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the D4RL benchmark datasets [14].
(b) Did you mention the license of the assets?
[Yes] In Appendix B.7.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Code for our work is in included via a URL.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] Appendices
A Proof of Proposition
Proposition 2 (Model Gradient) .
Let φ denote the parameters of a parametric MDP model (cid:98) T φ , and let V πφ denote the value function for policy π in (cid:98) T φ . Then:
[Equation] Proof: The proof is analogous to the proof of the policy gradient theorem [61, 62]. We start by expressing the value function using Bellman’s equation [Equation]
for some state s :
[Equation]
Applying the product rule:
[Equation]
Deﬁne
[Equation]
a
[Equation]
and additionally deﬁne ρ πφ ( s → x, n ) as the probability of transitioning from state s to x by executing π for n steps in the MDP model deﬁned by (cid:98) T φ .
Using these deﬁnitions, we can rewrite the above equation as:
[Equation]
(continuing to unroll)
[Equation]
[Equation]
We have that V πφ
= (cid:80) s µ 0 ( s )
· V πφ ( s ) by deﬁnition.
Therefore,
[Equation]
[Equation]
where d πφ ( s ) is the (improper) discounted state visitation distribution of the policy π in the MDP model (cid:98) T φ , under initial state distribution µ 0 .
Finally, we apply the log-derivative trick to arrive at the ﬁnal result:
[Equation]
A.1 Model Gradient with State-Action Value Baseline
We can subtract Q πφ ( s, a ) as a baseline without biasing the gradient estimate:
[Equation]
Proof : We can subtract any quantity which does not depend on s (cid:48) or r without changing the value of the expression because the subtracted quantity is zero.
Speciﬁcally, for the baseline of Q πφ ( s, a ) we have that:
[Equation]
B Implementation Details
The code for RAMBO is available at github.com/marc-rigter/rambo.
Our implementation builds upon the ofﬁcial MOPO codebase.
B.1 Model Training
We represent the model as an ensemble of neural networks that output a Gaussian distribution over the next state and reward given the current state and action:
[Equation]
Following previous works [75, 76], during the initial maximum likelihood model training (Line 1 of
Algorithm 1) we train an ensemble of 7 such dynamics models and pick the best 5 models based on the validation error on a held-out test set of 1000 transitions from the dataset D .
Each model in the ensemble is a 4-layer feedforward neural network with 200 hidden units per layer.
After the maximum likelihood training, RAMBO updates all of these 5 models adversarially according to Algorithm 1.
For each model rollout, we randomly pick one of the 5 dynamics models to simulate
[Table]
the rollout.
The learning rate for the model training is 3e-4 for both the MLE pretraining, and the adversarial training.
The model is trained using the Adam optimiser [26]. For the MLE pretraining, the batch size is 256. For each adversarial update in Equation 9 we sample a batch of 256 transitions from D to compute the maximum likelihood term, and 256 synthetic transitions for the model gradient term. The hyperparameters used for model training are summarised in Table 4.
B.2 Policy Training
We sample a batch of 256 transitions to train the policy and value function using soft actor-critic (SAC) [19].
We set the ratio of real data to f = 0 .
5 for all datasets, meaning that we sample 50% of the batch transitions from D and the remaining 50% from D (cid:98) T φ .
We chose this value because it was used for most datasets by COMBO [75] and we found that it worked well.
We represent the
Q -networks and the policy as three layer neural networks with 256 hidden units per layer.
For SAC [19] we use automatic entropy tuning, where the entropy target is set to the standard heuristic of − dim ( A ) .
The only hyperparameter that we modify from the standard implementation of SAC is the learning rate for the actor, which we set to 1e-4 as this was reported to work better in the CQL paper [31] which also utilised SAC.
For reference, the hyperparameters used for SAC are included in
Table 4.
B.3 RAMBO Implementation Details
For each iteration of RAMBO we perform 1000 gradient updates to the actor-critic algorithm, and 1000 adversarial gradient updates to the model.
For each run, we perform 2000 iterations.
The base hyperparameter conﬁguration for RAMBO that is shared across all runs is shown in Table 4.
The only parameters that we vary between datasets are the length of the synthetic rollouts, k , the weighting of the adversarial term in the model update, λ , and the choice of rollout policy.
Details are included in the Hyperparameter Tuning section below.
B.4 Hyperparameter Tuning
We found that the hyperparameters that have the largest inﬂuence on the performance of RAMBO are the length of the synthetic rollouts ( k ), and the weighting of the adversarial term in the model update ( λ ).
For the rollout length, we found that a value of either 2 or 5 worked well across most datasets.
This is a slight modiﬁcation to the values of k ∈ { 1 , 5 } that are typically used in previous model-based ofﬂine RL algorithms [8, 75, 76].
To arrive at a suitable value for the adversarial weighting, we used the intuition that we must have λ (cid:28) 1 .
This is necessary so that the MLE term dominates in the loss function in Equation 9 for indistribution samples.
This ensures that the model still ﬁts the dataset accurately despite the adversarial
[Table]
training.
We observed that if λ is set too high, the training can be unstable and the Q -function can become extremely negative.
If λ is too small, the adversarial training has no effect as the model is updated too slowly.
We found that a value of 3e-4 generally obtained good performance.
For all MuJoCo datasets we performed model rollouts using the current policy, but for some AntMaze datasets we used a random rollout policy as we found this performed better.
The rollout policies used are listed in Table 5.
For the MuJoCo datasets, we initialised the policy using behaviour cloning (BC) which is common practice in ofﬂine RL [25, 74].
Ablation results without the BC initialisation are in
Appendix C.4.
We did not use the BC initialisation for the AntMaze problems.
For the rollout length and the adversarial weighting, we tested the following three conﬁgurations on each dataset: ( k, λ ) ∈ { (2, 3e-4), (5, 3e-4), (5, 0) } .
We included ( k, λ ) =
(5 , 0) as we found that an adversarial weighting of 0 worked well for some datasets.
As described in Section 6, to evaluate
RAMBO we ran each of the three hyperparameter conﬁgurations for ﬁve seeds each, and reported the best performance across the three conﬁgurations.
The results for the best conﬁguration are in Table 1, and the corresponding ﬁnal hyperparameters chosen can be found in Table 5.
The results for each of the three conﬁgurations can be found in Table 6.
Ofﬂine selection of hyperparameters is an important topic in ofﬂine RL [47, 77].
This is because in some applications the selection of hyperparameters based on online performance may be infeasible.
Therefore, we present additional results in Table 1 where we select between the three choices of hyperparameters ofﬂine using a simple heuristic based on the magnitude and stability of the Q -values during training.
After selecting the hyperparameters using the heuristic, we ran a further ﬁve seeds using these parameters to produce the ﬁnal result.
We refer to this approach as RAMBO OFF .
The hyperparameters used by RAMBO OFF are also included in Table 5.
Details of the heuristic used to select the hyperparameter conﬁguration for RAMBO OFF can be found in the following subsection.
B.5 Ofﬂine Hyperparameter Selection Method
Deep RL algorithms are highly sensitive to the choice of hyperparameters [3].
There is no consensus on how parameters should be selected for ofﬂine RL [47].
Some possibilities include a) selection based on online evaluation [8, 9, 25, 39, 72, 76], b) selection based on an ofﬂine heuristic [47, 75], and c) methods based on off-policy evaluation [47, 77].
To address applications in which online evaluation is possible, we evaluated RAMBO using approach a) which is the most common practice among model-based ofﬂine RL algorithms [8, 25, 37, 39, 76].
To consider situations where online tuning is not possible, we also evaluate using approach b) to select the hyperparameters for each dataset using a simple heuristic in a similar vein to [75].
We refer to this second approach as RAMBO OFF .
To arrive at a heuristic that can be evaluated ofﬂine, we use the intuition that the value function should be regularised (i.e. low) as well as stable during training.
Therefore, our heuristic makes the ﬁnal hyperparameter selection from the set of candidate parameters according to: min (cid:0) Q avg + Q var ) , where Q avg is the average Q -value at the end of training, and Q var is the variance of the average
Q -value over the ﬁnal 100 iterations of training.
The values of this heuristic for each of the three conﬁgurations that we test are the values in the brackets provided in Table 6.
The bolded values in the table indicate the lowest value for the heuristic, which is the parameter conﬁguration chosen for RAMBO OFF .
After choosing the hyperparameters in this manner, we rerun the algorithm for a further ﬁve seeds per dataset.
The performance for these additional runs are the results reported in Table 1 for RAMBO OFF .
The results in Table 1 indicate that RAMBO OFF obtains strong performance compared to existing baselines on the MuJoCo domains.
This indicates that it is possible to obtain solid performance with RAMBO by choosing the ﬁnal hyperparameters according to the magnitude and stability of the Q -values. Unsurprisingly however, RAMBO performs slightly better than RAMBO OFF . This shows that better performance is obtained using online hyperparameter tuning.
B.6 Evaluation Procedure
We report the average undiscounted normalized return averaged over the last 10 iterations of training, with 10 evaluation episodes per iteration.
We evaluated the SAC policy by deterministically taking the mean action.
The normalization procedure is that proposed by [14], where 100 represents an expert policy and 0 represents a random policy.
B.7 Baselines and Domains
We compare RAMBO against state of the art model-based (COMBO [75], MOReL [25], RepBSDE [34] and MOPO [76]) and model-free (CQL [31], IQL [28], and TD3+BC [15]) ofﬂine RL algorithms on the D4RL benchmarks [14].
The D4RL benchmarks are released with the Apache
License 2.0.
To facilitate a fair comparison, we provide results for all algorithms for the MuJoCo-v
D4RL datasets which contain more performant data than v0 [37].
Because the original COMBO, MOPO, and CQL papers use the MuJoCo-v0 datasets we report the results for these algorithms on the MuJoCo-v2 datasets from [67].
The MOReL, IQL, RepB-SDE, and TD3+BC papers include evaluations on the MuJoCo-v2 datasets, so for these algorithms we include the results from the original papers.
For AntMaze-v0, we report the results from the original papers for CQL, IQL, and TD3+BC.
For
COMBO, MOPO, RepB-SDE, and MOReL, the results are taken from [67].
Following the IQL paper [28], we subtract 1 from the rewards of the AntMaze datasets for our experiments.
B.8 Computational Resources
During our experiments, each run of RAMBO has access to 2 CPUs of an Intel Xeon Platinum 8259CL processor at 3.1GHz, and half of an Nvidia T4 GPU.
With this hardware, each run of
RAMBO takes 24-30 hours.
For our main evaluation of RAMBO, we ran ﬁve seeds for each of three parameter conﬁgurations for 18 tasks resulting in a total of 270 runs.
This required approximately 150 GPU-days of compute.
C Additional Results C.1 Single Transition Example
Description In this domain, the state and action spaces are one-dimensional.
The agent executes a single action, a ∈ [ − 1 , 1] , from initial state s 0 .
After executing the action, the agent transitions to s (cid:48) and receives a reward equal to the successor state, i.e. r ( s (cid:48) )
=
s (cid:48) , and the episode terminates.
The actions in the dataset are sampled uniformly from a ∈ [ − 0 .
7] ∪ [ − 0 .
15 , − 0 .
1] ∪ [0 . 1 , 0 . 15] ∪ [0 . 7 , 0 . 75] .
In the MDP for this domain, the transition distribution for successor states is as follows:
• s (cid:48) ∼ N ( µ = 1 , σ = 0 . 2) , for a ∈ [ − 0 .
8 , − 0 .
• s (cid:48) ∼ N ( µ = 0 .
5 , σ = 0 . 2) , for a ∈ [ − 0 .
2 , − 0 .
• s (cid:48) ∼ N ( µ = 1 . 25 , σ = 0 . 2) , for a ∈ [0 . 05 , 0 . 2] .
• s (cid:48) ∼ N ( µ = 1 . 5 , σ = 0 . 2) , for a ∈ [0 . 65 , 0 . 8] .
• s (cid:48)
= 0 .
5 , for all other actions.
The transitions to successor states from the actions in the dataset are illustrated in Figure 1.
Comparison between RAMBO and COMBO To generate the comparison, we run both algorithms for 50 iterations.
To choose the regularisation parameter for COMBO, we sweep over β ∈ { 0 . 1 , 0 . 2 , 0 . 5 , 1 , 5 } and choose the parameter with the best performance.
Note that 0.5, 1, andare the values used in the original COMBO paper [75], so we consider lower values of regularisation than in the original paper.
For RAMBO we use λ = 3e-2.
We used the implementation of COMBO
Figure 2 shows the Q -values produced by RAMBO throughout training.
We can see that afteriterations, the Q -values for actions which are outside of the dataset are initially optimistic , and over-estimate the true values.
However, after 50 iterations the Q -values for out of distribution actions have been regularised by RAMBO.
The action selected by the policy after 50 iterations is the optimal action in the dataset, a ∈ [0 . 7 , 0 . 75] .
Thus, we see that RAMBO introduces pessimism gradually as the adversary is trained to modify the transitions to successor states.
For COMBO, the best performance averaged over 20 seeds was obtained for β = 0 .
2 , and therefore we report results for this value of the regularisation parameter.
Figure 3 illustrates the Q -values produced by COMBO throughout training (with β = 0 . 2 ).
We see that at both 5 iterations anditerations, the value estimates for actions outside of the dataset are highly pessimistic.
In the run illustrated for COMBO, the action selected by the policy after 50 iterations is a ∈ [0 . 1 , 0 . 15] , which is not the optimal action in the dataset.
Figure 3 shows that the failure to ﬁnd an optimal action is due to the gradient-based policy optimisation becoming stuck in a local maxima of the Q -function.
These results highlight a difference in the behaviour of RAMBO compared to COMBO.
For RAMBO, pessimism is introduced gradually as the adversary is trained to modify the transitions to successor states.
For COMBO, pessimism is introduced at the outset as it is part of the value function update throughout the entirety of training.
Additionally, the regularisation of the Q -values for out of distribution actions appears to be less aggressive for RAMBO than for COMBO.
The results averaged over 20 seeds in Table 3 show that RAMBO consistently performs better than
COMBO for this problem.
This suggests that the gradual introduction of pessimism produced by
RAMBO means that the policy optimisation procedure is less likely to get stuck in poor local maxima for this example.
The downside of this behaviour is that it may take more iterations for RAMBO to ﬁnd a performant policy.
Modifying other algorithms to gradually introduce pessimism may be an interesting direction for future research.
[Figure]
[Figure]
C.2 Results for each Hyperparameter Conﬁguration
In Table 6, we report the results for each of the three parameter conﬁgurations that we test.
The highlighted values indicate the best performance for each dataset, and are the results reported for
RAMBO in Table 1.
In Table 6 we also report the values of the ofﬂine hyperparameter selection heuristic in brackets.
The bold values indicate the lowest value of the heuristic, which is the hyperparameter conﬁguration selected for RAMBO OFF .
[Table]
R a ndo m
[Table]
M e d i u m
[Table]
M e d i u m
R e p l a y
[Table]
m
[Table]
C.3 Visualisation of Adversarially Trained Dynamics Model
To visualise the inﬂuence of training the transition dynamics model in an adversarial manner as proposed by RAMBO, we consider the following simple example MDP.
In the example, the state space ( S ) and action space ( A ) are 1-dimensional with, A
= [ − 1 , 1] .
For this example, we assume that the reward function is known and is equal to the current state, i.e. R ( s, a )
= s , meaning that greater values of s have greater expected value.
The true transition dynamics to the next state s (cid:48) depend on the action but are the same regardless of the initial state, s .
In Figure 4 we plot the data present in the ofﬂine dataset for this MDP.
Note that the actions in the dataset are sampled from a subset of the action space: a ∈ [ − 0 .
Because greater values of s (cid:48) correspond to greater expected value, the transition data indicates that the optimal action covered by the dataset is a = 0 .
In Figure 5 we plot the output of running naïve model-based policy optimisation (MBPO) on this illustrative ofﬂine RL example.
Figure 5a illustrates the MLE transition function used by MBPO.
The transition function ﬁts the dataset for a ∈ [ − 0 .
Outside of the dataset, it predicts that an action of a ≈ 1 transitions to the best successor state.
Figure 5b shows that applying a reinforcement learning algorithm (SAC) to this model results in the value function being over-estimated, and a ≈being predicted as the best action.
This illustrates that the policy learns to exploit the inaccuracy in the model and choose an out-of-distribution action.
In Figure 6 we show plots generated after running RAMBO on this example for 15 iterations, using an adversary weighting of λ = 3e-2.
Figure 6a shows that the transition function still accurately predicts the transitions within the dataset.
This is because choosing λ (cid:28) 1 means that the MLE loss dominates for state-action pairs within the dataset.
Due to the adversarial training, for actions a / ∈ [ − 0 . 3 , 0 . 3] which are out of the dataset distribution, the transition function generates pessimistic transitions to low values of s , which have low expected value.
As a result, low values are predicted for out-of-distribution actions, and the SAC agent illustrated in 6b learns to take action a ≈ 0 . 3 , which is the best action which is within the distribution of the dataset.
This demonstrates that by generating pessimistic synthetic transitions for out-of-distribution actions, RAMBO enforces conservatism and prevents the learnt policy from taking state-action pairs which have not been observed in the dataset.
For the MuJoCo locomotion experiments we initialised the policy using behaviour cloning, as noted in Section 6 and Appendix B.4. This is a common initialisation step used in previous works [25, 74]. In Table 7 we present ablation results where the behaviour cloning initialisation is removed, and the policy is initialised randomly.
Table 7 indicates that the behaviour cloning initialisation results in a small improvement in the performance of RAMBO.
[Table] ndo m [Table] e d i u m
R e p l a y [Table] M [Table]
d i u m
E xp e r t
[Table]