RAMBO-RL: Robust Adversarial Model-Based Ofﬂine Reinforcement Learning
Abstract
Ofﬂine reinforcement learning (RL) aims to ﬁnd performant policies from logged data without further environment interaction.
Model-based algorithms, which learn a model of the environment from the dataset and perform conservative policy optimisation within that model, have emerged as a promising approach to this problem.
In this work, we present Robust Adversarial Model-Based Ofﬂine RL (RAMBO), a novel approach to model-based ofﬂine RL.
We formulate the problem as a two-player zero sum game against an adversarial environment model.
The model is trained to minimise the value function while still accurately predicting the transitions in the dataset, forcing the policy to act conservatively in areas not covered by the dataset.
To approximately solve the two-player game, we alternate between optimising the policy and adversarially optimising the model.
The problem formulation that we address is theoretically grounded, resulting in a probably approximately correct (PAC) performance guarantee and a pessimistic value function which lower bounds the value function in the true environment.
We evaluate our approach on widely studied ofﬂine RL benchmarks, and demonstrate that it outperforms existing state-of-the-art baselines.
Introduction
Reinforcement learning (RL) [61] has achieved state-of-the-art performance on many sequential decision-making problems [40, 45, 59].
However, the need for extensive exploration prohibits the application of RL to many real world domains where such exploration is costly or dangerous.
Ofﬂine
RL [33, 35] overcomes this limitation by learning policies from static, pre-recorded datasets.
Online RL algorithms perform poorly in the ofﬂine setting due to the distributional shift between the state-action pairs in the dataset and those taken by the learnt policy.
Thus, an important aspect of ofﬂine RL is to introduce conservatism to prevent the learnt policy from executing state-action pairs which are out of distribution.
Model-free ofﬂine RL algorithms [15, 23, 27, 29, 31, 72] train a policy from only the data present in the ﬁxed dataset, and incorporate conservatism either into the value function or by directly constraining the policy.
On the other hand, model-based ofﬂine RL algorithms [76, 75, 25, 63, 39] use the dataset to learn a model of the environment, and train a policy using additional synthetic data generated from that model.
By training on additional synthetic data, model-based algorithms can potentially generalise better to states not present in the dataset, or to solving new tasks.
Previous approaches to model-based ofﬂine RL incorporate conservatism by estimating the uncertainty in the model and applying reward penalties for state-action pairs that have high uncertainty [76, 25].
However, uncertainty estimation can be unreliable for neural network models [75, 37].
Like recent work [75], we propose an approach for ofﬂine model-based RL which does not require uncertainty estimation .
In this work we present Robust Adversarial Model-Based Ofﬂine (RAMBO) RL, a new algorithm for model-based ofﬂine RL.
RAMBO incorporates conservatism by modifying the transition dynamics of the learnt Markov decision process (MDP) model in an adversarial manner.
We formulate the problem of ofﬂine RL as a zero-sum game against an adversarial environment.
To solve the resulting maximin optimisation problem, we alternate between optimising the agent and optimising the adversary in the style of Robust Adversarial RL (RARL) [50].
Unlike existing RARL approaches, our modelbased approach forgoes the need to deﬁne and train an adversary policy, and instead only learns an adversarial model of the MDP.
We train the agent policy with an actor-critic algorithm using synthetic data generated from the model in addition to data sampled from the dataset, similar to Dyna [60] and a number of recent methods [22, 76, 25, 75].
We update the environment model so that it reduces the value function for the agent policy, while still accurately predicting the transitions in the dataset.
As a result, our approach introduces conservatism by generating pessimistic synthetic transitions for state-action pairs which are out-of-distribution.
The theoretical formulation of ofﬂine RL that our algorithm addresses yields a PAC bound for the performance gap with respect to any policy covered by the dataset, and a pessimistic value function that lower bounds the value function in the true environment.
In summary, the main contributions of this work are:
• RAMBO, a novel and theoretically-grounded model-based ofﬂine RL algorithm which enforces conservatism by training an adversarial dynamics model.
• Adapting the Robust Adversarial RL approach to model-based ofﬂine RL by proposing a new formulation of RARL, where instead of deﬁning and training an adversary policy, we directly train the model adversarially.
In our experiments we demonstrate that RAMBO outperforms current state-of-the-art algorithms on the D4RL benchmarks [14].
Furthermore, we provide ablation results which show that training the model adversarially is crucial to the strong performance of RAMBO.
Related Work
Ofﬂine RL: Ofﬂine RL addresses the problem of learning policies from ﬁxed datasets, and has been applied to domains such as healthcare [42, 57], natural language processing [24, 23], and robotics [30, 38, 51].
Model-free ofﬂine RL algorithms do not require a learnt model.
Approaches for model-free ofﬂine RL include importance sampling algorithms [36, 41], constraining the learnt policy to be similar to the behaviour policy [16, 28, 72, 23, 58, 15], incorporating conservatism into the value function during training [9, 27, 31, 73], using uncertainty quantiﬁcation to generate more robust value estimates [1, 2, 29], or applying only a single iteration of policy iteration [7, 49].
In contrast, modelbased approaches learn a model of the environment and generate synthetic data from that model [60] to optimise a policy using either planning [4] or RL algorithms [25, 76, 75].
By training a policy on additional synthetic data, model-based approaches have the potential for broader generalisation and for solving new tasks [6, 76].
A simple approach to ensuring conservatism is to constrain the policy to be similar to the behaviour policy in the same fashion as some model-free approaches [8, 39, 63].
Another approach is to apply reward penalties for executing state-action pairs with high uncertainty in the environment model [25, 74, 76].
However, this requires explicit uncertainty estimates which may be unreliable for neural network models [17, 37, 46, 75].
COMBO [75] obviates the need for uncertainty estimation in model-based ofﬂine RL by adapting model-free techniques [31] to regularise the value function for out-of-distribution samples.
Like COMBO, our approach does not require uncertainty estimation.
Most approaches to model-based ofﬂine RL use maximum likelihood estimates (MLE) of the MDP trained using standard supervised learning [4, 39, 63, 76, 75].
However, other methods have been proposed to learn models which are more suitable for ofﬂine policy optimisation.
One approach is to reweight the loss function to ensure the model is accurate under the state-action distribution generated by the policy [34, 53, 20].
In contrast, our approach produces pessimistic synthetic transitions when out-of-distribution.
Most related to our work is a recent paper [66] which introduces the maximin formulation of ofﬂine
RL that we address.
This existing work motivates our approach theoretically by showing that the problem formulation obtains probably approximately correct (PAC) guarantees.
However, [66] only addresses the theoretical aspects of the problem formulation and does not propose a practical algorithm.
In this work, we propose a practical RL algorithm to solve the maximin formulation of model-based ofﬂine RL.
Robust RL: Algorithms for Robust MDPs [5, 12, 21, 43, 54, 64, 70]
ﬁnd the policy with the best worst-case performance over a set of possible MDPs.
Typically, it is assumed that the uncertainty set of MDPs is speciﬁed a priori.
To eliminate the need to specify the set of possible MDPs, model-free approaches to Robust MDPs [68, 55] instead assume that samples can be drawn from a misspeciﬁed
MDP which is similar to the true MDP.
As our work addresses ofﬂine RL, we assume that we have a ﬁxed dataset from the true MDP.
Our approach is conceptually similar to Robust Adversarial RL (RARL) [50], a method proposed to improve the robustness of RL policies in the online setting.
RARL is posed as a two-player zero-sum game where the agent plays against an adversary which perturbs the environment.
Formulations of model-free RARL differ in how they deﬁne the action space of the adversary.
Options include allowing the adversary to apply perturbation forces to the simulator [50], add noise to the agent’s actions [65], or periodically take over control [48].
A model-based approach to RARL is proposed in [13], which learns an optimistic and pessimistic model to encourage online exploration.
However, this existing approach requires uncertainty estimation as well as an adversarial policy to be learnt in addition to the model .
Our work follows the paradigm of RARL and alternates between agent and adversarial updates in a maximin formulation.
We adapt model-based RARL to the ofﬂine setting and propose an alternative formulation: we eliminate the need to learn an adversary policy and instead directly modify the MDP model adversarially .
Preliminaries
MDPs and Ofﬂine RL: An MDP is deﬁned by the tuple, M = ( S, A, T, R, µ 0 , γ ) .
S and A denote the state and action spaces respectively, R ( s, a ) is the reward function, T ( s (cid:48) | s, a ) is the transition function, µ 0 is the initial state distribution, and γ ∈ (0 , 1) is the discount factor.
In this work we consider Markovian policies, π ∈ Π , which map from each state to a distribution over actions.
We denote the (improper) discounted state visitation distribution of a policy by d πM ( s ) :
= (cid:80) ∞
t =0 γ t Pr( s t = s | π, M ) , where Pr( s t = s | π, M ) is the probability of reaching state s at time t by executing policy π in M .
The improper state-action visitation distribution is d πM ( s, a ) =
π ( a | s ) · d πM ( s ) .
We also denote the normalised state-action visitation distribution by ˜ d πM ( s, a ) =
(1 − γ ) · d πM ( s, a ) .
The value function, V πM ( s ) , represents the expected discounted return from executing π from state s in M : V πM ( s )
=
E π,M (cid:2) (cid:80) ∞
t =
γ t R ( s t , a t ) (cid:3) .
We write V πM to indicate the value function under the initial state distribution, i.e. V πM = (cid:80) s ∈ S µ 0 ( s ) V πM ( s ) .
The standard objective for MDPs is to ﬁnd the policy which maximises V πM .
The state-action value function, Q πM ( s, a ) , is the expected discounted cumulative reward from taking action a at state s and then executing π thereafter.
In ofﬂine RL we only have access to a ﬁxed dataset of transitions from the MDP, D
= { ( s i , a i , r i , s (cid:48)
i ) } |D| i =1 .
The goal of ofﬂine RL is to ﬁnd the best possible policy using the ﬁxed dataset.
Model-Based Ofﬂine RL Algorithms: Model-based approaches to ofﬂine RL use a model of the
MDP to help train a policy.
The dataset is used to learn a dynamics model, (cid:98) T , which is typically trained via maximum likelihood estimation: min (cid:98) T E ( s,a,s (cid:48) )
∼D (cid:2) − log (cid:98) T ( s (cid:48) | s, a ) (cid:3) .
A model of the reward function, (cid:98) R ( s, a ) , can also be learnt if it is unknown.
The estimated MDP, (cid:99) M
=
( S, A, (cid:98) T, (cid:98) R, µ 0 , γ ) , has the same state and action space as the true MDP but uses the learnt transition and reward functions.
Thereafter, any planning or RL algorithm can be used to recover optimal policy in the learnt model,
[Equation]
Unfortunately, directly applying this approach to the ofﬂine RL setting does not perform well due to distributional shift.
In particular, if the dataset does not cover the entire state-action space, the model will inevitably be inaccurate for some state-action pairs.
Thus, naive policy optimisation on a learnt model in the ofﬂine setting can result in model exploitation [22, 32, 53].
To mitigate this issue, we propose the novel approach of enforcing conservatism by adversarially modifying the transition dynamics of (cid:99) M .
In line with existing works [76, 75, 8], we use model-based policy optimisation (MBPO)
[22] to learn the optimal policy for (cid:99) M .
MBPO utilises a standard actor-critic RL algorithm.
However, the value function is trained using an augmented dataset D ∪ D (cid:99) M , where D (cid:99) M is synthetic data generated by simulating rollouts in the learnt model.
To generate the synthetic data, MBPO performs k -step rollouts in (cid:99) M starting from states s ∈ D , and adds this data to D (cid:99) M .
To train the policy, minibatches of data are drawn from D ∪ D (cid:99) M , where each datapoint is sampled from the real data, D , with probability f , and from D (cid:99) M with probability 1 − f.
Robust Adversarial Reinforcement Learning: RARL addresses the problem of ﬁnding a robust agent policy, π , in the online RL setting by posing the problem as a two-player zero sum game against adversary policy, ¯ π :
[Equation]
where V π, ¯ π M is the expected value from executing π and ¯ π in environment M .
Different approaches deﬁne the action space for ¯ π in different ways, as discussed in Section 2.
For a scalable approximation to the optimisation problem in Equation 1, algorithms for RARL alternate between applying steps of stochastic gradient ascent to the agent’s policy to increase the expected value, and stochastic gradient descent to the adversary’s policy to decrease the expected value.
In our work, we follow the RARL paradigm of alternating between agent and adversarial updates and adapt it to the model-based ofﬂine setting.
Instead of deﬁning a separate adversary policy, we treat the model itself as the policy to be adversarially trained.
Problem Formulation
For the sake of generality, we assume that both the transition function and reward function are unknown.
Hereafter, we write (cid:98) T to denote both the learnt dynamics and reward function, where (cid:98) T ( s (cid:48) , r | s, a ) is the probability of receiving reward r and transitioning to s (cid:48)
after executing ( s, a ) .
We address the maximin formulation of ofﬂine RL recently proposed by [66]:
Problem 1.
For some dataset, D , and some ﬁxed constant ξ > 0 , ﬁnd the policy π deﬁned by
[Equation]
where
[Equation]
where TV ( P 1 , P 2 ) is the total variation distance between distributions P 1 and P 2 , and (cid:98) T MLE denotes the maximum likelihood estimate of the MDP given the ofﬂine dataset, D .
Thus, the set deﬁned in Equation 3 contains MDPs which are similar to the maximum likelihood estimate under state-action pairs in D .
However, because the expectation in Equation 3 is taken under D , there is no restriction on (cid:98) T for regions of the state-action space not covered by D .
We present a brief overview of the theoretical guarantees from [66] in the following subsection.
Remark 1.
Note that Problem 1 differs from the pessimistic MDP formulations introduced by
MOPO [76] and MOReL [25].
Problem 1 considers the worst-case transition dynamics, while the pessimistic MDPs constructed by MOPO and MOReL only modify the reward function by applying reward penalties for state-action pairs with high uncertainty.
4.1 Theoretical Motivation
The theoretical analysis from [66] shows that solving Problem 1 outputs a policy that with high probability is approximately as good as any policy with a state-action distribution that is covered by the dataset.
This is formally stated in the following theorem.
Theorem 1 (PAC guarantee from [66], Theorem 5) .
Denote the true MDP transition function by T , and let M denote a hypothesis class of MDP models such that T ∈ M .
Let π denote the solution to
Problem 1 for dataset D .
Then with probability 1 − δ for any policy, π ∗ ∈ Π , we have
[Equation]
[Equation]
where ρ is the state-action distribution from which D was sampled, and c and c 1 are universal constants.
We refer the reader to Appendix A of [66] for the deﬁnitions of G M 1 , G M 2 , and ξ n .
The quantity C π ∗ is upper bounded by the maximum density ratio between the comparator policy, π ∗ , and the ofﬂine distribution, i.e. C π ∗ ≤ max ( s,a ) ˜
d π ∗ T ( s, a ) /ρ
( s, a ) .
It represents the discrepancy between the distribution of data in the dataset compared to the visitation distribution of policy π ∗ .
Theorem 1 shows that if we ﬁnd a policy by solving Problem 1, the performance gap of that policy is bounded with respect to any other policy π ∗ that has a state-action distribution which is covered by the dataset.
Furthermore, the value function under the worst-case model in the set deﬁned by Problem 1 is a lower bound on the value function in the true environment, as stated by Proposition 1.
Proposition 1 (Pessimistic value function) .
Let T denote the true transition function for some MDP, and let M D be the set of MDP models deﬁned in Equation 3.
Then for any policy π , with probability 1 − δ we have that
[Equation]
Proposition 1 follows from the fact that T ∈ M D with high probability, which is proven in [66] (Appendix E.2).
Proposition 1 shows that we can expect the performance of any policy in the true
MDP to be at least as good as the value in the worst-case model deﬁned in Problem 1.
While [66] provides the theoretical motivation for solving Problem 1, it does not propose a practical algorithm.
In this work, we focus on developing a practical approach to solving Problem 1.
RAMBO-RL
In this section, we present Robust Adversarial Model-Based Ofﬂine RL (RAMBO), our algorithm for solving Problem 1.
The main difﬁculty with solving Problem 1 is that it is unclear how to ﬁnd the worst-case MDP in the set deﬁned in Equation 3.
To arrive at a scalable solution, we propose a novel approach which is in the spirit of RARL.
We alternate between optimising the agent policy to increase the expected value, and adversarially optimising the model to decrease the expected value.
In this section, we ﬁrst describe how we compute the gradient to adversarially train the model.
Then, we discuss how to ensure that the model remains approximately within the constraint set deﬁned in
Problem 1.
Finally, we present our overall algorithm.
5.1 Model Gradient
We propose a policy gradient-inspired approach to adversarially optimise the model.
Typically, policy gradient algorithms are used to modify the distribution over actions taken by a policy at each state [62, 71].
In contrast, the update that we propose modiﬁes the likelihood of the successor states and rewards in the MDP model.
We assume that the MDP model is deﬁned by parameters, φ , and we write (cid:98) T φ to indicate this.
We denote by V πφ the value function for policy π in model (cid:98) T φ .
To approximately ﬁnd min (cid:98) T φ ∈M D V πφ as required by Problem 1 via gradient descent, we wish to compute the gradient of the model parameters that reduces the value of the policy within the model, i.e. ∇ φ V πφ .
Proposition 2 (Model Gradient) .
Let φ denote the parameters of a parametric MDP model (cid:98) T φ , and let V πφ denote the value function for policy π in (cid:98) T φ .
Then:
[Equation]
The proof of Proposition 2 is given in Appendix A. We can subtract the baseline Q πφ ( s, a ) without biasing the gradient estimate (see Appendix A.1 for details):
[Equation]
The Model Gradient differs from the standard policy gradient in that a) it is used to update the likelihood of successor states in the model, rather than actions in a policy, and b) the “advantage” term r + γV πφ ( s (cid:48) )
− Q πφ ( s, a ) compares the utility of receiving reward r and transitioning to s (cid:48) to the expected value of state action pair ( s, a ) .
In contrast, the standard advantage term, Q ( s, a ) − V ( s ) [56], compares the value of executing state-action pair ( s, a ) to the value at s .
To estimate V πφ and Q πφ , we use the critic learnt by the actor-critic algorithm used for policy optimisation.
Thus, we use the critic both for training the policy and adversarially training the model.
Remark 2.
The Model Gradient can be thought of as a speciﬁc instantiation of the policy gradient if we view (cid:98) T φ as a adversarial “policy” on an augmented MDP, M + , i.e. (cid:98) T φ : S + → Dist( A + ) .
The augmented state space, S + = S
×
A includes the state in the original MDP augmented by the action taken by the agent.
The augmented action space consists of the reward applied and the successor state,
A + = S ×
[ R min , R max ] .
Thus, we can think of this approach as an instantiation of RARL in which the adversary policy ( ¯ π in Equation 1) that we train is the model itself .
5.2 Adversarial Model Training
If we were to update the model using Equation 5 alone, this would allow the model to be modiﬁed arbitrarily such that the value function in the model is reduced.
However, the set of plausible MDPs given by Equation 3 states that over the dataset, D , the model (cid:98) T φ should be close to the maximum likelihood estimate, (cid:98) T MLE .
Speciﬁcally, in the inner optimisation of Problem 1 we wish to ﬁnd a solution to the constrained optimisation problem
[Equation]
The Lagrangian relaxation leads to the unconstrained problem
[Equation]
where λ is the Lagrange multiplier.
Rather than optimising the Lagrange multiplier, we ﬁnd that in practice ﬁxing λ to apply a constant weighting between the two terms works well with minimal tuning.
To facilitate easier tuning of the learning rate, in our implementation we apply the weighting constant to the value function term rather than the model term, which is equivalent up to a scaling factor.
This leads to
[Equation]
We aim to make our algorithm efﬁcient and simple to implement.
Therefore, rather than minimising the TV distance between the model and MLE model as prescribed by Equation 8, we directly optimise the standard MLE loss.
This leads to the ﬁnal loss function:
[Equation]
Thus, the loss function for the model in Equation 9 simply adds the adversarial term to the standard
MLE loss.
By minimising the loss function in Equation 9, the model is trained to a) predict the transitions within the dataset, and b) reduce the value function of the policy, with λ determining the tradeoff between these two objectives.
Choosing λ to be small ensures that the MLE term dominates for transitions within D , ensuring that the model ﬁts the dataset accurately.
Because the MLE term is only computed over D , the adversarial term dominates outside of the dataset meaning that the model is modiﬁed adversarially for transitions outside of the dataset.
To estimate the gradient of the loss function in Equation 9 for stochastic gradient descent, we sample a minibatch of transitions from D to estimate the MLE term.
The gradient for the value function term is computed using the Model Gradient.
The transitions used to compute the Model Gradient term must be sampled under the current policy and model (Equation 5).
Therefore, to estimate the Model
Gradient term, we generate a minibatch of transitions by simulating the current policy in (cid:98) T φ .
Like previous works [10, 76, 75, 8], we represent the dynamics model using an ensemble of neural networks.
Each neural network produces a Gaussian distribution over the next state and reward:
(cid:98) T φ ( s (cid:48) , r | s, a ) =
N ( µ φ ( s, a ) , Σ φ ( s, a )) .
A visualisation of the result of adversarially training the dynamics model can be found in Appendix C.3.
Normalisation The composite loss function in Equation 9 comprises two terms which may have different magnitudes across domains depending on the scale of the states and rewards.
To enable easier tuning of the adversarial loss weighting, λ , across different domains we perform the following normalisation procedure.
Prior to training, we normalise the states in D in the manner proposed in [15], by subtracting the mean and dividing by the standard deviation of each state dimension in the dataset.
Additionally, when computing the gradient in Equation 5 we normalise the advantage terms, r + γV πφ ( s (cid:48) )
− Q πφ ( s, a ) , according to the mean and standard deviation across each minibatch.
Advantage normalisation is common practice in policy gradient RL implementations [3, 52].
5.3 Algorithm
We are now ready to present our overall approach in Algorithm 1.
The ﬁrst step of RAMBO is to pretrain the environment dynamics model using standard MLE (Line 1).
Thereafter, the algorithm follows the format of RARL.
At each iteration, we apply gradient updates to the agent to increase the expected value, followed by gradient updates to the model to decrease the expected value.
Prior to each agent update, we generate synthetic k -step rollouts starting from states in D by simulating rollouts in the current MDP model (cid:98) T φ .
This data is added to the synthetic dataset D (cid:98) T φ (Line 3).
Following previous approaches [76, 75, 22] we only store data from recent iterations in D (cid:98) T φ .
The agent’s policy and value functions are trained with an off-policy actor-critic algorithm using samples from D ∪ D (cid:98) T φ (Line 4).
In our implementation, we use soft actor-critic (SAC) [19] for agent training.
To update the model to minimise the loss in Equation 9 (Line 5), we sample data from D to estimate the gradient for the MLE component.
To compute the adversarial component we generate samples by simulating the current policy and model, and utilise the value function learnt by the agent to compute the gradient according to Equation 5.
Require:
Normalised dataset, D ;
Experiments
In our experiments, we aim to: a) evaluate how well RAMBO performs compared to state-ofthe-art baselines, b) examine whether RAMBO can be tuned ofﬂine, c) determine the impact of adversarial training on the performance of the algorithm, and d) investigate the difference between
RAMBO and COMBO, the most similar prior algorithm.
The code for our experiments is available at github.com/marc-rigter/rambo.
We evaluate our approach on the following domains.
MuJoCo There are three different environments representing different robots ( HalfCheetah, Hopper,
Walker2D ), each with 4 datasets ( Random, Medium, Medium-Replay, Medium-Expert ).
Random contains transitions collected by a random policy.
Medium contains transitions collected by an early-stopped SAC policy.
Medium-Replay consists of the replay buffer generated while training the
Medium policy.
The Medium-Expert dataset contains a mixture of suboptimal and expert data.
AntMaze The agent controls a robot and navigates to reach a goal, receiving a sparse reward only if the goal is reached.
There are three different layouts of maze ( Umaze, Medium, Large ), and different dataset types ( Fixed, Play, Diverse ) which differ in terms of the variety
Hyperparameter Details
The base hyperparameters that we use for RAMBO mostly follow those used in SAC [19] and COMBO [75].
We ﬁnd that the performance of RAMBO is sensitive to the choice of rollout length, k , consistent with ﬁndings in previous works [22, 37].
The other critical parameter for RAMBO is the choice of the adversarial weighting, λ .
For each dataset, we choose the rollout length and the adversarial weighting from one of three possible conﬁgurations: ( k, λ ) ∈ {(2, 3e-4), (5, 3e-4), (5, 0) } .
We included ( k, λ ) =
(5, 0) as we found that an adversarial weighting of 0 worked well for some datasets.
For the MuJoCo datasets we performed model rollouts using the current policy, and initialised the policy using behaviour cloning (BC) which is a common practice in ofﬂine RL [25, 74].
Ablation results in Appendix C.4 indicate that the BC initialisation results in a small improvement.
For the AntMaze datasets we used a random rollout policy and a randomly initialised policy as we found that this performed better.
Further details about the hyperparameters are in Appendix B.
Evaluation We present two different evaluations of our approach: RAMBO and RAMBO OFF .
For
RAMBO, we ran each of the three hyperparameter conﬁgurations for ﬁve seeds each, and report the best performance across the three conﬁgurations.
Thus, our evaluation of RAMBO utilises limited online tuning which is the most common practice among existing model-based ofﬂine RL algorithms [25, 37, 39, 76].
The performance obtained for each of the hyperparameter conﬁgurations is included in Appendix C.2.
Ofﬂine hyperparameter selection is an important topic in ofﬂine RL [47, 77].
Therefore, we present additional results for RAMBO OFF where we select between the three choices of hyperparameters ofﬂine using a simple heuristic (details in Appendix B.5) based on the magnitude and stability of the
Q -values during ofﬂine training.
We ﬁrst select the hyperparameters using the heuristic, and then rerun each dataset for 5 seeds to generate the results for RAMBO OFF .
Baselines We compare RAMBO against state-of-the-art model-based (COMBO [75], RepBSDE [34], MOReL [25], and MOPO [76]) and model-free (CQL [31], IQL [28], and TD3+BC [15]) ofﬂine RL algorithms.
We provide results for all algorithms for the MuJoCo-v2 D4RL datasets and the AntMaze-v0 datasets (details in Appendix B.7).
6.1 Results
D4RL Performance The results in Table 1 show that RAMBO achieves the best total score for the MuJoCo locomotion domains, outperforming existing state-of-the-art methods.
Furthermore,
RAMBO obtains the best overall score on both the Medium and Medium-Replay dataset types.
For the random datasets, RAMBO is outperformed only by MOReL.
This shows that RAMBO performs very well for datasets that are either noisy or consist of suboptimal data.
For the Medium-Expert datasets, RAMBO is outperformed by most of the baseline algorithms, suggesting that RAMBO is less suitable for high-quality datasets.
However, for the Medium-Expert datasets, simpler approaches such as performing behaviour cloning on the best 10% of trajectories can be used to achieve stronger performance than ofﬂine RL methods [28].
Therefore, the suboptimal performance of RAMBO on the Medium-Expert datasets is less of a concern, as applying ofﬂine RL algorithms may not be the most suitable approach for these high-quality datasets.
For AntMaze, the model-based algorithms perform considerably less well than the model-free approaches.
Unlike the other model-based approaches, RAMBO at least scores greater than zero for most of the datasets.
Our results echo previous ﬁndings that model-based approaches struggle to perform well in the AntMaze domains, potentially because model-based algorithms are too aggressive and collide with walls [67].
Recent work [67] showed that reverse rollouts can lead to stronger performance for model-based methods in these domains.
In future work, we wish to investigate whether combining RAMBO with reverse rollouts improves the performance for AntMaze.
Ofﬂine Tuning In Table 1, we also present results for RAMBO OFF where the ﬁnal hyperparameters are chosen using the heuristic described in Appendix B.5.
We see that there is a slight degradation in the performance relative to RAMBO, which uses online tuning.
However, RAMBO OFF still achieves comparable performance to the best existing approaches on the MuJoCo datasets.
This suggests that suitable hyperparameters for RAMBO can reliably be chosen using our ofﬂine heuristic.
[Table] [Table]
[Table]
Ablation of Adversarial Training In Table 2 we present results for RAMBO with no adversarial updates.
These results demonstrate that overall performance degrades if the adversarial training is removed.
This parallels previous ﬁndings that mitigating the issue of model exploitation is crucial to obtaining strong performance in model-based ofﬂine RL.
Interestingly however, for some speciﬁc datasets we obtain the best performance with no adversarial training (Appendix C.2).
This suggests that a potential direction for future work could be trying to identify which types of problems do not require regularisation for a successful policy to be trained ofﬂine with model-based RL.
Comparison to COMBO We focus especially on comparing our approach to COMBO, as it is the most similar existing algorithm.
We compare RAMBO and COMBO on the Single Transition toy example which is described in detail in Appendix C.1.
This domain has a one-dimensional state and action space, and several distinct regions of the action space are covered by the dataset.
We use this domain to investigate whether the policies optimised by RAMBO and COMBO tend to become stuck in local optima.
Table 3 compares the performance of RAMBO and COMBO on the Single Transition Example.
Further analysis in Appendix C.1 shows that for this problem, the pessimistic value function updates used by COMBO create local maxima in the Q -function which are present throughout training.
Policy optimisation can become stuck in these local maxima.
On the other hand, the value function produced by RAMBO is initially optimistic, and pessimism is introduced into the value function gradually as the transition function is modiﬁed adversarially.
Adversarial modiﬁcation of the transition function is visualised in Appendix C.3.
As a result of this gradual introduction of pessimism, we observe that the policy produced by RAMBO is less likely to become stuck in poor local maxima, and better overall performance is obtained in Table 3.
This observation may help to explain why RAMBO is able to achieve consistently strong performance relative to existing algorithms in the MuJoCo domains.
Gradually increasing the level of pessimism could be a useful modiﬁcation for existing ofﬂine RL algorithms to be investigated in future work.
Conclusion and Future Directions
RAMBO is a promising new approach to ofﬂine RL which imposes conservatism by adversarially modifying the transition dynamics of a learnt model.
Our approach is theoretically justiﬁed, and achieves state-of-the-art performance on standard benchmarks.
There are a number of possible extensions to RAMBO, some of which we have already discussed.
In addition, we would like to apply RAMBO to image-space domains by using deep latent variable models to compress the state space [18, 51] and adversarially perturbing the transition dynamics in the latent space representation.
Another direction that we would like to investigate is the use of adversarially trained models to aid interpretability in deep RL [44] by generating imagined worst-case trajectories.
Finally, we wish to investigate applying the ideas developed in this work to the online
RL setting to improve robustness.
Acknowledgements
This work was supported by a Programme Grant from the Engineering and Physical Sciences Research
Council (EP/V000748/1), the Clarendon Fund at the University of Oxford, and a gift from Amazon
Web Services.
Additionally, this project made use of time on Tier 2 HPC facility JADE2, funded by the Engineering and Physical Sciences Research Council (EP/T022205/1).
The authors would like to thank Raunak Bhattacharyya, Paul Duckworth, and Matthew Budd for their feedback on an earlier draft of this work.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We included results for the AntMaze datasets where our approach performs less well than model-free methods.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] We include a complete proof of Proposition 2.
For the theoretical results from [66], we provide references to the appropriate parts of that paper.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] A URL to the code is provided.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Detailed information about implementation details and hyperparameters is in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Standard deviation with respect to seeds is reported for all results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Details of compute used per run and total compute used for full evaluation is in Appendix B.8.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the D4RL benchmark datasets [14].
(b) Did you mention the license of the assets?
[Yes] In Appendix B.7.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Code for our work is in included via a URL.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]