RAMBO-RL: Robust Adversarial Model-Based Ofﬂine Reinforcement Learning s
Abstract
Ofﬂine reinforcement learning (RL) aims to ﬁnd near-optimal policies from loggeddata without further environment interaction.
Model-based algorithms, which learna model of the environment from the dataset and perform conservative policy optimisation within that model, have emerged as a promising approach to this problem.
In this work, we present Robust Adversarial Model-Based Ofﬂine RL (RAMBO), anovel approach to model-based ofﬂine RL.
To achieve conservatism, we formulatethe problem as a two-player zero sum game against an adversarial environmentmodel.
The model is trained to minimise the value function while still accuratelypredicting the transitions in the dataset, forcing the policy to act conservatively inareas not covered by the dataset.
To approximately solve the two-player game, wealternate between optimising the policy and adversarially optimising the model.
The problem formulation that we address is theoretically grounded, resulting in a probably approximately correct (PAC) performance guarantee and a pessimisticvalue function which lower bounds the value function in the true environment.
Weevaluate our approach on widely studied ofﬂine RL benchmarks, and demonstratethat it outperforms existing state-of-the-art baselines.
Introduction
Reinforcement learning (RL) [64] has achieved state-of-the-art performance on many sequentialdecision-making problems [41, 61, 46].
However, the need for extensive exploration prohibits theapplication of RL to many real world domains where such exploration is costly or dangerous.
Ofﬂine
RL [33, 35] overcomes this limitation by learning policies from static, pre-recorded datasets.
Online RL algorithms perform poorly in the ofﬂine setting due to the distribution shift between thestate-action pairs in the dataset and those taken by the learnt policy.
Thus, an important aspect ofofﬂine RL is to introduce conservatism to prevent the learnt policy from executing state-action pairswhich are out of distribution.
Model-free ofﬂine RL algorithms [29, 31, 75, 23, 27, 14] train a policyfrom only the data present in the ﬁxed dataset, and incorporate conservatism either into the valuefunction or by directly constraining the policy.
On the other hand, model-based ofﬂine RL algorithms [78, 77, 25, 66, 40] use the dataset to learna model of the environment, and train a policy using additional synthetic data generated from thatmodel.
By training on additional synthetic data, model-based algorithms can potentially generalisebetter to states not present in the dataset, or to solving new tasks.
Previous approaches to model-basedofﬂine RL incorporate conservatism by estimating the uncertainty in the model and applying rewardpenalties for state-action pairs that have high uncertainty [78, 25].
However, uncertainty estimationcan be unreliable for neural network models [77, 37].
Like recent work [77], we propose an approachfor ofﬂine model-based RL which does not require uncertainty estimation .
In this work we present Robust Adversarial Model-Based Ofﬂine (RAMBO) RL, a new algorithm formodel-based ofﬂine RL.
RAMBO incorporates conservatism by modifying the transition dynamics ofthe learnt Markov decision process (MDP) model in an adversarial manner.
We formulate the problemof ofﬂine RL as a zero-sum game against an adversarial environment.
To solve the resulting maximinoptimisation problem, we alternate between optimising the agent and optimising the adversary inthe style of Robust Adversarial RL (RARL) [52].
Unlike existing RARL approaches, our modelbased approach forgoes the need to deﬁne and train an adversary policy, and instead only learns anadversarial model of the MDP.
We train the agent policy with an actor-critic algorithm using syntheticdata generated from the model in addition to data sampled from the dataset, similar to Dyna [63] anda number of recent methods [22, 78, 25, 77].
We update the environment model so that it reducesthe value function for the agent policy, whilst still accurately predicting the transitions in the dataset.
As a result, our approach introduces conservatism by generating pessimistic synthetic transitions forstate-action pairs which are out-of-distribution.
The theoretical formulation of ofﬂine RL that ouralgorithm addresses yields a PAC guarantee, and a pessimistic value function that lower bounds thevalue function in the true environment.
In summary, the main contributions of this work are:
• RAMBO, a novel and theoretically-grounded model-based ofﬂine RL algorithm which enforcesconservatism by training an adversarial dynamics model.•
Adapting the Robust Adversarial RL approach to model-based ofﬂine RL by proposing a newformulation of RARL, where instead of deﬁning and training an adversary policy, we directly trainthe model adversarially.
In our experiments we demonstrate that RAMBO outperforms current state-of-the-art algorithms onthe D4RL benchmarks [13]. We provide ablation results which show that if the model is not trainedadversarially, the performance of RAMBO degrades signiﬁcantly.
This echoes previous ﬁndings thatconservatism is crucial to obtaining strong performance in ofﬂine RL.
Related Work
Ofﬂine RL: Ofﬂine RL addresses the problem of learning policies from ﬁxed datasets, and hasbeen applied to domains such as healthcare [43, 59], natural language processing [24, 23], androbotics [30, 38, 53].
Model-free ofﬂine RL algorithms do not require a learnt model.
Approachesfor model-free ofﬂine RL include importance sampling algorithms [36, 42], constraining the learntpolicy to be similar to the behaviour policy [15, 28, 75, 23, 60, 14], incorporating conservatisminto the value function during training [8, 27, 31, 76], using uncertainty quantiﬁcation to generatemore robust value estimates [1, 29], or applying only a single iteration of policy iteration [6, 51].
In contrast, model-based approaches learn a model of the environment and generate synthetic datafrom that model [63] to optimise a policy using either planning [3] or RL algorithms [25, 78, 77].
By training a policy on additional synthetic data, model-based approaches have the potential forbroader generalisation and for solving new tasks [5, 78].
A simple approach to ensuring conservatismis to constrain the policy to be similar to the behaviour policy in the same fashion as some modelfree approaches [7, 40, 66].
The approach proposed by MOPO [78] and MOReL [25] is to applyreward penalties for executing state-action pairs with high uncertainty in the environment model.
However, this requires explicit uncertainty estimates which may be unreliable for neural networkmodels [16, 37, 47, 77].
COMBO [77] obviates the need for uncertainty estimation in modelbased ofﬂine RL by adapting model-free techniques [31] to regularise the value function for out-ofdistribution samples.
Like COMBO, our approach does not require uncertainty estimation.
Most approaches to model-based ofﬂine RL use maximum likelihood estimates (MLE) of the MDPtrained using standard supervised learning [3, 40, 66, 78, 77].
However, other methods have beenproposed to learn models which are more suitable for ofﬂine policy optimisation.
One approachis to reweight the loss function to ensure the model is accurate under the state-action distributiongenerated by the policy [34, 55, 19].
A reverse model is used by [70] to generate rollouts to goalstates which are in-distribution.
In contrast, our approach produces pessimistic synthetic transitionswhen out-of-distribution.
Most related to our work is a recent paper [69] which introduces the maximin formulation of ofﬂine
RL that we address.
This existing work motivates our approach theoretically by showing that theproblem formulation obtains probably approximately correct (PAC) guarantees.
However, [69] only addresses the theoretical aspects of the problem formulation and does not propose a practicalalgorithm.
In this work, we propose a practical RL algorithm to solve the maximin formulation ofmodel-based ofﬂine RL.
Robust RL: Algorithms for Robust MDPs [4, 21, 44, 56, 67, 73]
ﬁnd the policy with the best worstcase performance over a set of possible MDPs.
Typically, it is assumed that the uncertainty set of
MDPs is speciﬁed a priori.
To eliminate the need to specify the set of possible MDPs, model-freeapproaches to Robust MDPs [71, 57] instead assume that samples can be drawn from a misspeciﬁed
MDP which is similar to the true MDP.
As our work addresses ofﬂine RL, we assume that we have aﬁxed dataset from the true MDP.
Our approach is conceptually similar to Robust Adversarial RL (RARL) [52], a method proposed toimprove the robustness of RL policies in the online setting.
RARL is posed as a two-player zero-sumgame where the agent plays against an adversary which perturbs the environment.
Formulationsof model-free RARL differ in how they deﬁne the action space of the adversary.
Options includeallowing the adversary to apply perturbation forces to the simulator [52], add noise to the agent’sactions [68], or periodically take over control [49].
A model-based approach to RARL is proposedin [11], which learns an optimistic and pessimistic model to encourage online exploration.
However,this existing approach requires uncertainty estimation as well as an adversarial policy to be learntin addition to the model .
Our work follows the paradigm of RARL and alternates between agent andadversarial updates in a maximin formulation.
We adapt model-based RARL to the ofﬂine settingand propose an alternative formulation: we eliminate the need to learn an adversary policy andinstead directly modify the MDP model adversarially .
Related to our work is [39] which adversariallymodiﬁes trajectories while training a policy.
However, [39] assumes that the dynamics are known inthe form of a differential equation, whereas we assume that we only have access to sampled data.
Works on adversarial attacks in both online [12, 20, 50, 45] and ofﬂine [80] RL consider adversarialperturbations to the state observations.
We assume that the agent has access to the correct observations,but it is the transition function of the environment model which is adversarially modiﬁed.
Preliminaries
MDPs and Ofﬂine RL: An MDP is deﬁned by the tuple, M = ( S, A, T, R, µ 0 , γ ) .
S and Adenote the state and action spaces respectively, R ( s, a ) is the reward function, T ( s (cid:48) | s, a ) is thetransition function, µ 0 is the initial state distribution, and γ ∈ (0 , 1) is the discount factor.
Inthis work we consider Markovian policies, π ∈ Π , which map from each state to a distributionover actions.
We denote the (improper) discounted state visitation distribution of a policy byd πM ( s ) :
= (cid:80) ∞
t =0 γ t Pr( s t = s | π, M ) , where Pr( s t = s | π, M ) is the probability of reaching state sat time t by executing policy π in M .
The improper state-action visitation distribution is d πM ( s, a ) =
π ( a | s ) · d πM ( s ) .
We also denote the normalised state-action visitation distribution by ˜ d πM ( s, a ) =(1 − γ ) · d πM ( s, a ) .
The value function, V π M ( s ) , represents the expected discounted return from executing π from state sin M : V πM ( s )
=
E π,M (cid:2) (cid:80) ∞
t =
γ t R ( s t , a t ) (cid:3) .
The standard objective for MDPs is to ﬁnd the policywhich maximises the value function.
The state-action value function, Q πM ( s, a ) , is the expecteddiscounted cumulative reward from taking action a at state s and then executing π thereafter.
In ofﬂine RL we only have access to a ﬁxed dataset of transitions from the MDP, D
={ ( s i , a i , r i , s (cid:48)
i ) } |D| i =1 .
The goal of ofﬂine RL is to ﬁnd the best possible policy using the ﬁxed dataset.
Model-Based Ofﬂine RL Algorithms: Model-based approaches to ofﬂine RL use a model of the
MDP to help train a policy.
The dataset is used to learn a dynamics model, (cid:98) T , which is typically trainedvia maximum likelihood estimation: min (cid:98) T E ( s,a,s (cid:48) )
∼D (cid:2) − log (cid:98) T ( s (cid:48) | s, a ) (cid:3) .
A model of the rewardfunction, (cid:98) R ( s, a ) , can also be learnt if it is unknown.
The estimated MDP, (cid:99) M
=
( S, A, (cid:98) T, (cid:98) R, µ 0 , γ ) ,has the same state and action space as the true MDP but uses the learnt transition and reward functions.
Thereafter, any planning or RL algorithm can be used to recover optimal policy in the learnt model,(cid:98)
π = arg max π ∈ Π V π (cid:99) M .
Unfortunately, directly applying this approach to the ofﬂine RL setting does not perform well dueto distribution shift.
In particular, if the dataset does not cover the entire state-action space, the model will inevitably be inaccurate for some state-action pairs.
Thus, naive policy optimisation on alearnt model in the ofﬂine setting can result in model exploitation [22, 32, 55].
To mitigate this issue,we propose the novel approach of enforcing conservatism by adversarially modifying the transitiondynamics of (cid:99) M .
In line with existing works [78, 77, 7], we use model-based policy optimisation (MBPO)
[22] tolearn the optimal policy for (cid:99) M .
MBPO utilises a standard actor-critic RL algorithm.
However, thevalue function is trained using an augmented dataset D ∪ D (cid:99) M , where D (cid:99) M is synthetic data generatedby simulating rollouts in the learnt model.
To generate the synthetic data, MBPO performs k -steprollouts in (cid:99) M starting from states s ∈ D , and adds this data to D (cid:99) M .
To train the policy, minibatchesof data are drawn from D ∪ D (cid:99) M , where each datapoint is sampled from the real data, D , withprobability f , and from D (cid:99) M with probability 1 − f.
Robust Adversarial Reinforcement Learning: RARL addresses the problem of ﬁnding a robustagent policy, π , in the online RL setting by posing the problem as a two-player zero sum game againstadversary policy, ¯ π :
[Equation]
where V π, ¯ π M is the expected value from executing π and ¯ π in environment M .
Different approachesdeﬁne the action space for ¯ π in different ways, as discussed in Section 2.
For a scalable approximationto the optimisation problem in Equation 1, algorithms for RARL alternate between applying steps ofstochastic gradient ascent to the agent’s policy to increase the expected value, and stochastic gradientdescent to the adversary’s policy to decrease the expected value.
In our work, we follow the RARLparadigm of alternating between agent and adversarial updates and adapt it to the model-based ofﬂinesetting.
Instead of deﬁning a separate adversary policy, we treat the model itself as the policy to beadversarially trained.
Problem Formulation
For the sake of generality, we assume that both the transition function and reward function areunknown.
Hereafter, we write (cid:98) T to denote both the learnt dynamics and reward function, where(cid:98) T ( s (cid:48) , r | s, a ) is the probability of receiving reward r and transitioning to s (cid:48)
after executing ( s, a ) .
Weaddress the maximin formulation of ofﬂine RL recently proposed by [69]:
Problem 1.
For some dataset, D , and some ﬁxed constant ξ > 0 , ﬁnd the policy π deﬁned by
[Equation]
where
[Equation] [Equation]
where TV ( P 1 , P 2 ) is the total variation distance between distributions P 1 and P 2 , and (cid:98) T MLE denotesthe maximum likelihood estimate of the MDP given the ofﬂine dataset, D .
Thus, the set deﬁned in Equation 3 contains MDPs which are similar to the maximum likelihoodestimate under state-action pairs in D .
However, because the expectation in Equation 3 is takenunder D , there is no restriction on (cid:98) T for regions of the state-action space not covered by D .
Wepresent a brief overview of the theoretical guarantees from [69] in the following subsection.
Remark 1.
Note that Problem 1 differs from the pessimistic MDP formulations introduced by
MOPO [78] and MOReL [25].
Problem 1 considers the worst-case transition dynamics, while thepessimistic MDPs constructed by MOPO and MOReL only modify the reward function by applyingreward penalties for state-action pairs with high uncertainty.
4.1 Theoretical Motivation
The theoretical analysis from [69] shows that solving Problem 1 outputs a policy that with highprobability is approximately as good as any policy with a state-action distribution that is covered bythe dataset.
This is formally stated in the following theorem.
Theorem 1 (PAC guarantee from [69], Theorem 5) .
Denote the true MDP transition function by T , and let M denote a hypothesis class of MDP models such that T ∈ M .
Let π denote the solution to
Problem 1 for dataset D .
Then with probability 1 − δ for any policy, π ∗ ∈ Π , we have
[Equation]
, where
[Equation]
where ρ is the state-action distribution from which D was sampled, and c and c 1 are universalconstants.
We refer the reader to Appendix A of [69] for the deﬁnitions of G M 1 , G M 2 , and ξ n .
The quantity C π ∗ is upper bounded by the maximum density ratio between the comparator policy,π ∗ , and the ofﬂine distribution, i.e. C π ∗ ≤ max ( s,a ) ˜
d π ∗ T ( s, a ) /ρ
( s, a ) .
It represents the discrepancybetween the distribution of data in the dataset compared to the visitation distribution of policy π ∗ .
Theorem 1 shows that if we ﬁnd a policy by solving Problem 1, that policy will be almost as good asany other policy π ∗ which has a state-action distribution which is covered by the dataset.
Furthermore, the value function under the worst-case model in the set deﬁned by Problem 1 is a lowerbound on the value function in the true environment, as stated by Proposition 1.
Proposition 1 (Pessimistic value function) .
Let T denote the true transition function for some MDP, and let M D be the set of MDP models deﬁned in Equation 3.
Then for any policy π , with probability 1 − δ we have that
[Equation]
Proposition 1 follows from the fact that T ∈ M D with high probability, which is proven in [69](Appendix E.2).
Proposition 1 shows that we can expect the performance of any policy in the true
MDP to be at least as good as the value in the worst-case model deﬁned in Problem 1.
While [69] provides the theoretical motivation for solving Problem 1, it does not propose a practicalalgorithm.
In this work, we focus on developing a practical approach to solving Problem 1.
RAMBO-RL
In this section, we present Robust Adversarial Model-Based Ofﬂine RL (RAMBO), our algorithmfor solving Problem 1.
The main difﬁculty with solving Problem 1 is that it is unclear how to ﬁndthe worst-case MDP in the set deﬁned in Equation 3.
To arrive at a scalable solution, we propose anovel approach which is in the spirit of RARL.
We alternate between optimising the agent policy toincrease the expected value, and adversarially optimising the model to decrease the expected value.
In this section, we ﬁrst describe how we compute the gradient to adversarially train the model.
Then,we discuss how to ensure that the model remains approximately within the constraint set deﬁned in
Problem 1.
Finally, we present our overall algorithm.
5.1 Model Gradient
We propose a policy gradient-inspired approach to adversarially optimise the model.
Typically,policy gradient algorithms are used to modify the distribution over actions taken by a policy at eachstate [64, 65, 74].
In contrast, the update that we propose modiﬁes the likelihood of the successorstates and rewards in the MDP model.
We assume that the MDP model is deﬁned by parameters, φ , and we write (cid:98) T φ to indicate this.
Wedenote by V πφ the value function for policy π in model (cid:98) T φ .
To approximately ﬁnd min (cid:98) T φ ∈M D V πφ asrequired by Problem 1 via gradient descent, we wish to compute the gradient of the model parameterswhich reduces the value of the policy within the model, i.e. ∇ φ V πφ .
Proposition 2 (Model Gradient) .
Let φ denote the parameters of a parametric MDP model (cid:98) T φ , andlet V πφ denote the value function for policy π in (cid:98) T φ .
Then:
[Equation]
The proof of Proposition 2 is given in Appendix A. We can subtract the baseline Q πφ ( s, a ) withoutbiasing the gradient estimate (see Appendix A.1 for details):
[Equation]
The Model Gradient differs from the standard policy gradient in that a) it is used to update thelikelihood of successor states in the model, rather than actions in a policy, and b) the “advantage”term r + γV πφ ( s (cid:48) )
− Q πφ ( s, a ) compares the utility of receiving reward r and transitioning to s (cid:48) to theexpected value of state action pair ( s, a ) .
In contrast, the standard advantage term, Q ( s, a ) − V ( s )
[58],compares the utility of executing state-action pair ( s, a ) to the expected value at s .
To estimate V πφand Q πφ , we use the critic learnt by the actor-critic algorithm used for policy optimisation.
Thus, weuse the critic both for training the policy and adversarially training the model.
Remark 2.
The Model Gradient can be thought of as a speciﬁc instantiation of the policy gradient ifwe view (cid:98) T φ as a adversarial “policy” on an augmented MDP, M + , i.e. (cid:98) T φ : S + → Dist( A + ) .
Theaugmented state space, S + = S
×
A includes the state in the original MDP augmented by the actiontaken by the agent.
The augmented action space consists of the reward applied and the successorstate,
A + = S ×
[ R min , R max ] .
Thus, we can think of this approach as an instantiation of RARL inwhich the adversary policy that we train is the model itself .
Remark 3.
In principle, we could have instead adapted backpropagation through time (BPTT) [10,72] to adversarially train the model.
However, BPTT is known to have issues with numericalinstability [10].
5.2 Adversarial Model Training
If we were to update the model using Equation 5 alone, this would allow the model to be modiﬁedarbitrarily such that the value function in the model is reduced.
However, the set of plausible MDPsgiven by Equation 3 states that over the dataset, D , the model (cid:98) T φ should be close to the maximumlikelihood estimate, (cid:98) T MLE .
Speciﬁcally, in the inner optimisation of Problem 1 we wish to ﬁnd asolution to the constrained optimisation problem
[Equation]
The Lagrangian relaxation leads to the unconstrained problem
[Equation]
(7) where λ is the Lagrange multiplier.
Rather than optimising the Lagrange multiplier, we ﬁnd thatin practice ﬁxing λ to apply a constant weighting between the two terms works well with minimaltuning.
To facilitate easier tuning of the learning rate, in our implementation we apply the weightingconstant to the value function term rather than the model term, which is equivalent up to a scalingfactor.
This leads to
[Equation]
Rather than explicitly minimising the TV distance between the model and MLE model, we apply thestandard MLE loss for the model term.
This leads to the ﬁnal loss function:
[Equation]
By minimising the loss function in Equation 9, the model is trained to a) predict the transitions withinthe dataset, and b) reduce the value function of the policy, with λ determining the tradeoff betweenthese two objectives.
Choosing λ to be small ensures that the MLE term dominates for transitionswithin D , ensuring that the model ﬁts the dataset accurately.
Because the MLE term is only computedover D , the value function term dominates outside of the dataset meaning that the model is modiﬁedadversarially for transitions outside of the dataset.
To estimate the gradient of the loss function in Equation 9 for stochastic gradient descent, we samplea minibatch of transitions from D to estimate the MLE term.
The gradient for the value function term is computed using the Model Gradient.
The transitions used to compute the Model Gradient termmust be sampled under the current policy and model (Equation 5).
Therefore, to estimate the Model
Gradient term, we generate a minibatch of transitions by simulating the current policy in (cid:98) T φ .
Like previous works [9, 78, 77, 7], we represent the dynamics model using an ensemble of neuralnetworks.
Each neural network produces a Gaussian distribution over the next state and reward:(cid:98)
T φ ( s (cid:48) , r | s, a ) =
N ( µ φ ( s, a ) , Σ φ ( s, a )) .
A visualisation of the result of adversarially training thedynamics model can be found in Appendix C.3.
Normalisation The composite loss function in Equation 9 comprises two terms which may havedifferent magnitudes across domains depending on the scale of the states and rewards.
To enableeasier tuning of the adversarial loss weighting, λ , across different domains we perform the followingnormalisation procedure.
Prior to training, we normalise the states in D in the manner proposedby [14], by subtracting the mean and dividing by the standard deviation of each state dimension inthe dataset.
Additionally, when computing the gradient in Equation 5 we normalise the advantageterms, r + γV πφ ( s (cid:48) )
− Q πφ ( s, a ) , according to the mean and standard deviation across each minibatch.
Advantage normalisation is already common practice in policy gradient RL implementations [2, 54].
Algorithm
We are now ready to present our overall approach in Algorithm 1.
The ﬁrst step of RAMBO is topretrain the environment dynamics model using standard MLE (Line 1).
Thereafter, the algorithmfollows the format of RARL.
At each iteration, we apply gradient updates to the agent to increase theexpected value, followed by gradient updates to the model to decrease the expected value.
Prior to each agent update, we generate synthetic k -step rollouts starting from states in D by simulatingrollouts in the current MDP model (cid:98) T φ .
This data is added to the synthetic dataset D (cid:98) T φ (Line 3).
Following previous approaches [78, 77, 22] we store data in D (cid:98) T φ in a ﬁrst in, ﬁrst out manner so thatonly data generated from recent iterations is stored in D (cid:98) T φ .
The agent’s policy and value functionsare trained using an off-policy actor critic algorithm using samples from D ∪ D (cid:98) T φ (Line 4).
In ourimplementation, we use soft actor-critic (SAC) [18] for agent training.
To update the model to minimise the loss in Equation 9 in Line 5, we sample data from D to estimatethe gradient for the MLE component.
To compute the adversarial component, we generate samples bysimulating the current policy and model and utilise the value function learnt by the agent to computethe gradient according to Equation 5.
Experiments
In our experiments, we examine how well RAMBO performs compared to state-of-the-art baselines,and explore the importance of adversarial training to the performance of the algorithm.
The code forour experiments is included in the supplementary material and will be made publicly available.
Weevaluate our approach on the following domains from the D4RL benchmarks [13].
MuJoCo There are three different environments representing different robots ( HalfCheetah, Hopper,
Walker2D ), each with 4 datasets ( Random, Medium, Medium-Replay, Medium-Expert ).
Randomcontains transitions collected by a random policy.
Medium contains transitions collected by an early-stopped SAC policy.
Medium-Replay consists of the replay buffer generated while training the
Medium policy.
The Medium-Expert dataset contains a mixture of suboptimal and expert data.
AntMaze The agent controls a robot and navigates to reach a goal, receiving a sparse reward only ifthe goal is reached.
There are three different layouts of maze ( Umaze, Medium, Large ), and differentdataset types ( Fixed, Play, Diverse ) which differ in terms of the variety of start and goal locationsused to collect the dataset.
Hyperparameter Selection and Evaluation
The base hyperparameters that we use for RAMBOmostly follow those used in SAC [18] and COMBO [77].
We ﬁnd that the performance of RAMBOis sensitive to the choice of rollout length, k , consistent with ﬁndings in previous works [22, 37].
Theother critical parameter for RAMBO is the choice of the adversarial weighting, λ .
For each dataset, we choose the rollout length and the adversarial weighting from one of three possibleconﬁgurations: ( k, λ ) ∈ {(2, 3e-4), (5, 3e-4), (5, 0) } .
We included ( k, λ ) =
(5, 0) as we found thatan adversarial weighting of 0 worked well for some datasets.
For all MuJoCo datasets we performedmodel rollouts using the current policy, but for some AntMaze datasets we used a random rolloutpolicy as we found this performed better.
Further details about the hyperparameters can be found in
Appendix B.
To evaluate RAMBO, we ran each of the three hyperparameter conﬁgurations for ﬁve seeds each, andreport the best performance across the three conﬁgurations.
Thus, our evaluation of RAMBO utiliseslimited online hyperparameter tuning to choose hyperparameters for each dataset.
Note that this is themost common practice among existing model-based ofﬂine RL algorithms [7, 25, 37, 40, 78].
Theperformance obtained for each of the hyperparameter conﬁgurations is included in Appendix C.1.
Ofﬂine hyperparameter selection is an important topic in ofﬂine RL [48, 79].
This is becausein some applications selecting hyperparameters based on online performance may be infeasible.
Therefore, we present additional results in Appendix C.2 where we select between the three choicesof hyperparameters ofﬂine using a simple heuristic based on the magnitude and stability of the
Q -values during training.
Computational Resources During our experiments, each run of RAMBO has access to 2 CPUs ofan Intel(R) Xeon(R) Platinum 8259CL processor at 3.1GHz, and half of an Nvidia T4 GPU.
Withthis hardware, each run of RAMBO takes 24-30 hours.
Baselines We compare RAMBO against state-of-the-art model-based (COMBO [77], MOReL [25],and MOPO [78]) and model-free (CQL [31], IQL [28], and TD3+BC [14]) ofﬂine RL algorithms.
Tofacilitate a fair comparison, we provide results for all algorithms for the MuJoCo-v2 D4RL datasetswhich contain more performant data than v0 [5] (details in Appendix B.6).
We report results for the
AntMaze-v0 datasets.
Results
The results in Table 1 show that RAMBO outperforms existing state-of-the-art methodson the MuJoCo locomotion domains.
Relative to existing algorithms, RAMBO obtains the bestperformance on the Medium-Replay datasets, and performs less well on the Medium-Expert datasets.
For the high-quality data in the Medium-Expert datasets, simpler approaches such as performingbehaviour cloning on the best 10% of trajectories can be used to achieve stronger performance thanofﬂine RL methods [28].
Therefore, the suboptimal performance of RAMBO on the Medium-Expertdatasets is less of a concern, as applying ofﬂine RL algorithms may not be the most suitable approachfor these datasets.
For AntMaze, RAMBO performs considerably less well than the model-free baselines.
This echoesthe ﬁndings of [70] that model-based approaches that use forward rollouts struggle to perform well inthe AntMaze domains, potentially because model-based algorithms are too aggressive and collidewith walls.
The authors of [70] show that reverse rollouts can lead to stronger performance formodel-based methods in these domains.
In future work, we wish to investigate whether combining
RAMBO with reverse rollouts can lead to stronger performance on these domains.
In Table 2 we present ablations comparing the performance of RAMBO to the same approach withoutany adversarial updates to the model.
These results demonstrate that overall performance degradesif the adversarial training is removed.
This parallels previous ﬁndings that conservatism is crucialto obtaining strong performance in ofﬂine RL.
Interestingly however, for some speciﬁc datasets
[Table]
i.e. λ = 0 ).
The scores are averaged over 5 seeds.
To produce these results, we gave runs where the value function diverged a score of 0.
[Table]
we obtain the best performance with no adversarial training (Appendix C.1).
This suggests that apotential direction for future work could be trying to identify which types of problems do not requireconservatism for a successful policy to be trained ofﬂine with model-based RL.
Conclusion and Future Directions
RAMBO is a promising new approach to ofﬂine RL which imposes conservatism by adversariallymodifying the transition dynamics of a learnt model.
Our approach is theoretically justiﬁed, buildingon a formulation for which PAC bounds are guaranteed, and achieves state-of-the-art performance onstandard benchmarks.
There are a number of possible extensions to RAMBO, some of which we have already discussed.
In addition, we would like to apply RAMBO to image-space domains by using deep latent variablemodels to compress the state space [17, 53] and adversarially perturbing the transition dynamics inthe latent space representation.
Furthermore, we wish to improve the computational efﬁciency of
RAMBO.
RAMBO is more computationally demanding than many prior ofﬂine RL methods due tothe need to simultaneously train the model as well as the policy.
Potential directions to improve theefﬁciency include warmstarting the policy with behaviour cloning [25, 62] or reducing the frequencyof model updates.
Finally, another potential research direction is to apply the idea of adversariallytrained dynamics models to improve robustness in the online setting.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We described two main limi-
tations: a) our approach performs less well on the AntMaze datasets (discussed inexperiments), and b) requires more computation than many other approaches (discussedin conclusion).
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] We include a complete
proof of Proposition 2.
For the theoretical results from [69], we provide references tothe appropriate parts of that paper.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See thesupplementary material.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] Detailed information about implementation details and hyperparametersis in the appendix.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Standard deviation with respect to seeds is reported forall results.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Details of compute used per run isin the experiments section.
Details of total compute used for full evaluation is in theappendix.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the D4RL
benchmark datasets [13].
(b) Did you mention the license of the assets?
[Yes] In Appendix B.6.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Code for our work is in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]