RAMBO-RL: Robust Adversarial Model-Based Ofﬂine Reinforcement Learning
Abstract
Ofﬂine reinforcement learning (RL) aims to ﬁnd performant policies from loggeddata without further environment interaction.
Model-based algorithms, which learna model of the environment from the dataset and perform conservative policy optimisation within that model, have emerged as a promising approach to this problem.
In this work, we present Robust Adversarial Model-Based Ofﬂine RL (RAMBO), anovel approach to model-based ofﬂine RL.
To enforce conservatism, we formulatethe problem as a two-player zero sum game against an adversarial environmentmodel.
The model is trained to minimise the value function while still accuratelypredicting the transitions in the dataset, forcing the policy to act conservatively inareas not covered by the dataset.
To approximately solve the two-player game, wealternate between optimising the policy and adversarially optimising the model.
The problem formulation that we address is theoretically grounded, resulting in aprobably approximately correct (PAC) performance guarantee and a pessimisticvalue function which lower bounds the value function in the true environment.
Weevaluate our approach on widely studied ofﬂine RL benchmarks, and demonstratethat it outperforms existing state-of-the-art baselines.
Introduction
Reinforcement learning (RL) [61] has achieved state-of-the-art performance on many sequentialdecision-making problems [40, 58, 44].
However, the need for extensive exploration prohibits theapplication of RL to many real world domains where such exploration is costly or dangerous.
Ofﬂine
RL [32, 34] overcomes this limitation by learning policies from static, pre-recorded datasets.
Online RL algorithms perform poorly in the ofﬂine setting due to the distribution shift between thestate-action pairs in the dataset and those taken by the learnt policy.
Thus, an important aspect ofofﬂine RL is to introduce conservatism to prevent the learnt policy from executing state-action pairswhich are out of distribution.
Model-free ofﬂine RL algorithms [28, 30, 72, 22, 26, 14] train a policyfrom only the data present in the ﬁxed dataset, and incorporate conservatism either into the valuefunction or by directly constraining the policy.
On the other hand, model-based ofﬂine RL algorithms [76, 75, 24, 63, 39] use the dataset to learna model of the environment, and train a policy using additional synthetic data generated from thatmodel.
By training on additional synthetic data, model-based algorithms can potentially generalisebetter to states not present in the dataset, or to solving new tasks.
Previous approaches to model-basedofﬂine RL incorporate conservatism by estimating the uncertainty in the model and applying rewardpenalties for state-action pairs that have high uncertainty [76, 24].
However, uncertainty estimationcan be unreliable for neural network models [75, 36].
Like recent work [75], we propose an approachfor ofﬂine model-based RL which does not require uncertainty estimation .
In this work we present Robust Adversarial Model-Based Ofﬂine (RAMBO) RL, a new algorithm formodel-based ofﬂine RL.
RAMBO incorporates conservatism by modifying the transition dynamics ofthe learnt Markov decision process (MDP) model in an adversarial manner.
We formulate the problemof ofﬂine RL as a zero-sum game against an adversarial environment.
To solve the resulting maximinoptimisation problem, we alternate between optimising the agent and optimising the adversary inthe style of Robust Adversarial RL (RARL) [49].
Unlike existing RARL approaches, our modelbased approach forgoes the need to deﬁne and train an adversary policy, and instead only learns anadversarial model of the MDP.
We train the agent policy with an actor-critic algorithm using syntheticdata generated from the model in addition to data sampled from the dataset, similar to Dyna [60] anda number of recent methods [21, 76, 24, 75].
We update the environment model so that it reducesthe value function for the agent policy, whilst still accurately predicting the transitions in the dataset.
As a result, our approach introduces conservatism by generating pessimistic synthetic transitionsfor state-action pairs which are out-of-distribution.
The theoretical formulation of ofﬂine RL thatour algorithm addresses yields a PAC bound for the performance gap with respect to any policycovered by the dataset, and a pessimistic value function that lower bounds the value function in thetrue environment.
In summary, the main contributions of this work are:
• RAMBO, a novel and theoretically-grounded model-based ofﬂine RL algorithm which enforcesconservatism by training an adversarial dynamics model.•
Adapting the Robust Adversarial RL approach to model-based ofﬂine RL by proposing a newformulation of RARL, where instead of deﬁning and training an adversary policy, we directly trainthe model adversarially.
In our experiments we demonstrate that RAMBO outperforms current state-of-the-art algorithmson the D4RL benchmarks [13], and we provide ablation results which show that if the model is nottrained adversarially, the performance of RAMBO degrades signiﬁcantly.
Related Work
Ofﬂine RL: Ofﬂine RL addresses the problem of learning policies from ﬁxed datasets, and hasbeen applied to domains such as healthcare [42, 56], natural language processing [23, 22], androbotics [29, 37, 50].
Model-free ofﬂine RL algorithms do not require a learnt model.
Approaches formodel-free ofﬂine RL include importance sampling algorithms [35, 41], constraining the learnt policyto be similar to the behaviour policy [15, 27, 72, 22, 57, 14], incorporating conservatism into the valuefunction during training [9, 26, 30, 73], using uncertainty quantiﬁcation to generate more robust valueestimates [1, 2, 28], or applying only a single iteration of policy iteration [7, 48].
In contrast, modelbased approaches learn a model of the environment and generate synthetic data from that model [60]to optimise a policy using either planning [4] or RL algorithms [24, 76, 75].
By training a policy onadditional synthetic data, model-based approaches have the potential for broader generalisation andfor solving new tasks [6, 76].
A simple approach to ensuring conservatism is to constrain the policyto be similar to the behaviour policy in the same fashion as some model-free approaches [8, 39, 63].
Another approach is to apply reward penalties for executing state-action pairs with high uncertaintyin the environment model [24, 74, 76].
However, this requires explicit uncertainty estimates whichmay be unreliable for neural network models [16, 36, 45, 75].
COMBO [75] obviates the need foruncertainty estimation in model-based ofﬂine RL by adapting model-free techniques [30] to regularisethe value function for out-of-distribution samples.
Like COMBO, our approach does not requireuncertainty estimation.
Most approaches to model-based ofﬂine RL use maximum likelihood estimates (MLE) of the MDPtrained using standard supervised learning [4, 39, 63, 76, 75].
However, other methods have beenproposed to learn models which are more suitable for ofﬂine policy optimisation.
One approachis to reweight the loss function to ensure the model is accurate under the state-action distributiongenerated by the policy [33, 52, 19].
A reverse model is used by [67] to generate rollouts to goalstates which are in-distribution.
In contrast, our approach produces pessimistic synthetic transitionswhen out-of-distribution.
Most related to our work is a recent paper [66] which introduces the maximin formulation of ofﬂine
RL that we address.
This existing work motivates our approach theoretically by showing that theproblem formulation obtains probably approximately correct (PAC) guarantees.
However, [66]only addresses the theoretical aspects of the problem formulation and does not propose a practical algorithm.
In this work, we propose a practical RL algorithm to solve the maximin formulation ofmodel-based ofﬂine RL.
Robust RL: Algorithms for Robust MDPs [5, 20, 43, 53, 64, 70]
ﬁnd the policy with the best worstcase performance over a set of possible MDPs.
Typically, it is assumed that the uncertainty set of
MDPs is speciﬁed a priori.
To eliminate the need to specify the set of possible MDPs, model-freeapproaches to Robust MDPs [68, 54] instead assume that samples can be drawn from a misspeciﬁed
MDP which is similar to the true MDP.
As our work addresses ofﬂine RL, we assume that we have aﬁxed dataset from the true MDP.
Our approach is conceptually similar to Robust Adversarial RL (RARL) [49], a method proposed toimprove the robustness of RL policies in the online setting.
RARL is posed as a two-player zero-sumgame where the agent plays against an adversary which perturbs the environment.
Formulationsof model-free RARL differ in how they deﬁne the action space of the adversary.
Options includeallowing the adversary to apply perturbation forces to the simulator [49], add noise to the agent’sactions [65], or periodically take over control [47].
A model-based approach to RARL is proposedin [12], which learns an optimistic and pessimistic model to encourage online exploration.
However,this existing approach requires uncertainty estimation as well as an adversarial policy to be learntin addition to the model .
Our work follows the paradigm of RARL and alternates between agent andadversarial updates in a maximin formulation.
We adapt model-based RARL to the ofﬂine settingand propose an alternative formulation: we eliminate the need to learn an adversary policy andinstead directly modify the MDP model adversarially .
Related to our work is [38] which adversariallymodiﬁes trajectories while training a policy.
However, [38] assumes that the dynamics are known inthe form of a differential equation, whereas we assume that we only have access to sampled data.
Preliminaries
MDPs and Ofﬂine RL: An MDP is deﬁned by the tuple, M = ( S, A, T, R, µ 0 , γ ) .
S and Adenote the state and action spaces respectively, R ( s, a ) is the reward function, T ( s (cid:48) | s, a ) is thetransition function, µ 0 is the initial state distribution, and γ ∈ (0 , 1) is the discount factor.
Inthis work we consider Markovian policies, π ∈ Π , which map from each state to a distributionover actions.
We denote the (improper) discounted state visitation distribution of a policy byd πM ( s ) :
= (cid:80) ∞
t =0 γ t Pr( s t = s | π, M ) , where Pr( s t = s | π, M ) is the probability of reaching state sat time t by executing policy π in M .
The improper state-action visitation distribution is d πM ( s, a ) =
π ( a | s ) · d πM ( s ) .
We also denote the normalised state-action visitation distribution by ˜ d πM ( s, a ) =(1 − γ ) · d πM ( s, a ) .
The value function, V πM ( s ) , represents the expected discounted return from executing π from states in M : V πM ( s )
=
E π,M (cid:2) (cid:80) ∞
t =
γ t R ( s t , a t ) (cid:3) .
We write V πM to indicate the value function underthe initial state distribution, i.e. V πM = (cid:80) s ∈ S µ 0 ( s ) V πM ( s ) .
The standard objective for MDPs is toﬁnd the policy which maximises V πM .
The state-action value function, Q πM ( s, a ) , is the expecteddiscounted cumulative reward from taking action a at state s and then executing π thereafter.
In ofﬂine RL we only have access to a ﬁxed dataset of transitions from the MDP, D
={ ( s i , a i , r i , s (cid:48)
i ) } |D| i =1 .
The goal of ofﬂine RL is to ﬁnd the best possible policy using the ﬁxed dataset.
Model-Based Ofﬂine RL Algorithms: Model-based approaches to ofﬂine RL use a model of the
MDP to help train a policy.
The dataset is used to learn a dynamics model, (cid:98) T , which is typically trainedvia maximum likelihood estimation: min (cid:98) T E ( s,a,s (cid:48) )
∼D (cid:2) − log (cid:98) T ( s (cid:48) | s, a ) (cid:3) .
A model of the rewardfunction, (cid:98) R ( s, a ) , can also be learnt if it is unknown.
The estimated MDP, (cid:99) M
=
( S, A, (cid:98) T, (cid:98) R, µ 0 , γ ) ,has the same state and action space as the true MDP but uses the learnt transition and reward functions.
Thereafter, any planning or RL algorithm can be used to recover optimal policy in the learnt model,
[Equation]
Unfortunately, directly applying this approach to the ofﬂine RL setting does not perform well dueto distribution shift.
In particular, if the dataset does not cover the entire state-action space, themodel will inevitably be inaccurate for some state-action pairs.
Thus, naive policy optimisation on alearnt model in the ofﬂine setting can result in model exploitation [21, 31, 52].
To mitigate this issue,we propose the novel approach of enforcing conservatism by adversarially modifying the transitiondynamics of (cid:99) M .
In line with existing works [76, 75, 8], we use model-based policy optimisation (MBPO)
[21] tolearn the optimal policy for (cid:99) M .
MBPO utilises a standard actor-critic RL algorithm.
However, thevalue function is trained using an augmented dataset D ∪ D (cid:99) M , where D (cid:99) M is synthetic data generatedby simulating rollouts in the learnt model.
To generate the synthetic data, MBPO performs k -steprollouts in (cid:99) M starting from states s ∈ D , and adds this data to D (cid:99) M .
To train the policy, minibatchesof data are drawn from D ∪ D (cid:99) M , where each datapoint is sampled from the real data, D , withprobability f , and from D (cid:99) M with probability 1 − f.
Robust Adversarial Reinforcement Learning: RARL addresses the problem of ﬁnding a robustagent policy, π , in the online RL setting by posing the problem as a two-player zero sum game againstadversary policy, ¯ π :
[Equation]
where V π, ¯ π M is the expected value from executing π and ¯ π in environment M .
Different approachesdeﬁne the action space for ¯ π in different ways, as discussed in Section 2.
For a scalable approximationto the optimisation problem in Equation 1, algorithms for RARL alternate between applying steps ofstochastic gradient ascent to the agent’s policy to increase the expected value, and stochastic gradientdescent to the adversary’s policy to decrease the expected value.
In our work, we follow the RARLparadigm of alternating between agent and adversarial updates and adapt it to the model-based ofﬂinesetting.
Instead of deﬁning a separate adversary policy, we treat the model itself as the policy to beadversarially trained.
Problem Formulation
For the sake of generality, we assume that both the transition function and reward function areunknown.
Hereafter, we write (cid:98) T to denote both the learnt dynamics and reward function, where(cid:98) T ( s (cid:48) , r | s, a ) is the probability of receiving reward r and transitioning to s (cid:48)
after executing ( s, a ) .
Weaddress the maximin formulation of ofﬂine RL recently proposed by [66]:
Problem 1.
For some dataset, D , and some ﬁxed constant ξ > 0 , ﬁnd the policy π deﬁned by
[Equation]
T , where
[Equation] [Equation]
where TV ( P 1 , P 2 ) is the total variation distance between distributions P 1 and P 2 , and (cid:98) T MLE denotesthe maximum likelihood estimate of the MDP given the ofﬂine dataset, D .
Thus, the set deﬁned in Equation 3 contains MDPs which are similar to the maximum likelihoodestimate under state-action pairs in D .
However, because the expectation in Equation 3 is takenunder D , there is no restriction on (cid:98) T for regions of the state-action space not covered by D .
Wepresent a brief overview of the theoretical guarantees from [66] in the following subsection.
Remark 1.
Note that Problem 1 differs from the pessimistic MDP formulations introduced by
MOPO [76] and MOReL [24].
Problem 1 considers the worst-case transition dynamics, while thepessimistic MDPs constructed by MOPO and MOReL only modify the reward function by applyingreward penalties for state-action pairs with high uncertainty.
4.1 Theoretical Motivation
The theoretical analysis from [66] shows that solving Problem 1 outputs a policy that with highprobability is approximately as good as any policy with a state-action distribution that is covered bythe dataset.
This is formally stated in the following theorem.
Theorem 1 (PAC guarantee from [66], Theorem 5) .
Denote the true MDP transition function by T , and let M denote a hypothesis class of MDP models such that T ∈ M .
Let π denote the solution to
Problem 1 for dataset D .
Then with probability 1 − δ for any policy, π ∗ ∈ Π , we have
[Equation]
where
[Equation]
where ρ is the state-action distribution from which D was sampled, and c and c 1 are universalconstants.
We refer the reader to Appendix A of [66] for the deﬁnitions of G M 1 , G M 2 , and ξ n .
The quantity C π ∗ is upper bounded by the maximum density ratio between the comparator policy,π ∗ , and the ofﬂine distribution, i.e. C π ∗ ≤ max ( s,a ) ˜
d π
[Equation]
T ( s, a ) /ρ
( s, a ) .
It represents the discrepancybetween the distribution of data in the dataset compared to the visitation distribution of policy π ∗ .
Theorem 1 shows that if we ﬁnd a policy by solving Problem 1, the performance gap of that policy isbounded with respect to any other policy π ∗ which has a state-action distribution which is covered bythe dataset.
Furthermore, the value function under the worst-case model in the set deﬁned by Problem 1 is a lowerbound on the value function in the true environment, as stated by Proposition 1.
Proposition 1 (Pessimistic value function) .
Let T denote the true transition function for some MDP, and let M D be the set of MDP models deﬁned in Equation 3.
Then for any policy π , with probability 1 − δ we have that
[Equation]
Proposition 1 follows from the fact that T ∈ M D with high probability, which is proven in [66](Appendix E.2).
Proposition 1 shows that we can expect the performance of any policy in the true
MDP to be at least as good as the value in the worst-case model deﬁned in Problem 1.
While [66] provides the theoretical motivation for solving Problem 1, it does not propose a practicalalgorithm.
In this work, we focus on developing a practical approach to solving Problem 1.
RAMBO-RL
In this section, we present Robust Adversarial Model-Based Ofﬂine RL (RAMBO), our algorithmfor solving Problem 1.
The main difﬁculty with solving Problem 1 is that it is unclear how to ﬁndthe worst-case MDP in the set deﬁned in Equation 3.
To arrive at a scalable solution, we propose anovel approach which is in the spirit of RARL.
We alternate between optimising the agent policy toincrease the expected value, and adversarially optimising the model to decrease the expected value.
In this section, we ﬁrst describe how we compute the gradient to adversarially train the model.
Then,we discuss how to ensure that the model remains approximately within the constraint set deﬁned in
Problem 1.
Finally, we present our overall algorithm.
5.1 Model Gradient
We propose a policy gradient-inspired approach to adversarially optimise the model.
Typically,policy gradient algorithms are used to modify the distribution over actions taken by a policy at eachstate [61, 62, 71].
In contrast, the update that we propose modiﬁes the likelihood of the successorstates and rewards in the MDP model.
We assume that the MDP model is deﬁned by parameters, φ , and we write (cid:98) T φ to indicate this.
Wedenote by V πφ the value function for policy π in model (cid:98) T φ .
To approximately ﬁnd min (cid:98) T φ ∈M D V πφ asrequired by Problem 1 via gradient descent, we wish to compute the gradient of the model parameterswhich reduces the value of the policy within the model, i.e. ∇ φ V πφ .
Proposition 2 (Model Gradient) .
Let φ denote the parameters of a parametric MDP model (cid:98) T φ , andlet V πφ denote the value function for policy π in (cid:98) T φ .
Then:
[Equation]
The proof of Proposition 2 is given in Appendix A. We can subtract the baseline Q πφ ( s, a ) withoutbiasing the gradient estimate (see Appendix A.1 for details):
[Equation]
The Model Gradient differs from the standard policy gradient in that a) it is used to update thelikelihood of successor states in the model, rather than actions in a policy, and b) the “advantage”term r + γV πφ ( s (cid:48) )
− Q πφ ( s, a ) compares the utility of receiving reward r and transitioning to s (cid:48) to theexpected value of state action pair ( s, a ) .
In contrast, the standard advantage term, Q ( s, a ) − V ( s )
[55],compares the utility of executing state-action pair ( s, a ) to the expected value at s .
To estimate V πφand Q πφ , we use the critic learnt by the actor-critic algorithm used for policy optimisation.
Thus, weuse the critic both for training the policy and adversarially training the model.
Remark 2.
The Model Gradient can be thought of as a speciﬁc instantiation of the policy gradient ifwe view (cid:98) T φ as a adversarial “policy” on an augmented MDP, M + , i.e. (cid:98) T φ : S + → Dist( A + ) .
Theaugmented state space, S + = S
×
A includes the state in the original MDP augmented by the actiontaken by the agent.
The augmented action space consists of the reward applied and the successorstate,
A + = S ×
[ R min , R max ] .
Thus, we can think of this approach as an instantiation of RARL inwhich the adversary policy ( ¯ π in Equation 1) that we train is the model itself .
Remark 3.
In principle, we could have instead adapted backpropagation through time (BPTT) [11,69] to adversarially train the model.
However, BPTT is known to have issues with numericalinstability [11].
5.2 Adversarial Model Training
If we were to update the model using Equation 5 alone, this would allow the model to be modiﬁedarbitrarily such that the value function in the model is reduced.
However, the set of plausible MDPsgiven by Equation 3 states that over the dataset, D , the model (cid:98) T φ should be close to the maximumlikelihood estimate, (cid:98) T MLE .
Speciﬁcally, in the inner optimisation of Problem 1 we wish to ﬁnd asolution to the constrained optimisation problem
[Equation]
The Lagrangian relaxation leads to the unconstrained problem
[Equation]
where λ is the Lagrange multiplier.
Rather than optimising the Lagrange multiplier, we ﬁnd thatin practice ﬁxing λ to apply a constant weighting between the two terms works well with minimaltuning.
To facilitate easier tuning of the learning rate, in our implementation we apply the weightingconstant to the value function term rather than the model term, which is equivalent up to a scalingfactor.
This leads tomin
[Equation] [Equation]
Rather than explicitly minimising the TV distance between the model and MLE model, we apply thestandard MLE loss for the model term.
This leads to the ﬁnal loss function:
[Equation]
By minimising the loss function in Equation 9, the model is trained to a) predict the transitions withinthe dataset, and b) reduce the value function of the policy, with λ determining the tradeoff betweenthese two objectives.
Choosing λ to be small ensures that the MLE term dominates for transitionswithin D , ensuring that the model ﬁts the dataset accurately.
Because the MLE term is only computedover D , the value function term dominates outside of the dataset meaning that the model is modiﬁedadversarially for transitions outside of the dataset.
To estimate the gradient of the loss function in Equation 9 for stochastic gradient descent, we samplea minibatch of transitions from D to estimate the MLE term.
The gradient for the value function termis computed using the Model Gradient.
The transitions used to compute the Model Gradient termmust be sampled under the current policy and model (Equation 5).
Therefore, to estimate the Model
Gradient term, we generate a minibatch of transitions by simulating the current policy in (cid:98) T φ .
Like previous works [10, 76, 75, 8], we represent the dynamics model using an ensemble of neuralnetworks.
Each neural network produces a Gaussian distribution over the next state and reward:
(cid:98) T φ ( s (cid:48) , r | s, a ) =
N ( µ φ ( s, a ) , Σ φ ( s, a )) .
A visualisation of the result of adversarially training thedynamics model can be found in Appendix C.4.
Normalisation The composite loss function in Equation 9 comprises two terms which may havedifferent magnitudes across domains depending on the scale of the states and rewards.
To enableeasier tuning of the adversarial loss weighting, λ , across different domains we perform the followingnormalisation procedure.
Prior to training, we normalise the states in D in the manner proposedby [14], by subtracting the mean and dividing by the standard deviation of each state dimension inthe dataset.
Additionally, when computing the gradient in Equation 5 we normalise the advantageterms, r + γV πφ ( s (cid:48) )
− Q πφ ( s, a ) , according to the mean and standard deviation across each minibatch.
Advantage normalisation is already common practice in policy gradient RL implementations [3, 51].
Algorithm
We are now ready to present our overall approach in Algorithm 1.
The ﬁrst step of RAMBO is topretrain the environment dynamics model using standard MLE (Line 1).
Thereafter, the algorithmfollows the format of RARL.
At each iteration, we apply gradient updates to the agent to increase theexpected value, followed by gradient updates to the model to decrease the expected value.
Prior to each agent update, we generate synthetic k -step rollouts starting from states in D by simulatingrollouts in the current MDP model (cid:98) T φ .
This data is added to the synthetic dataset D (cid:98) T φ (Line 3).
Following previous approaches [76, 75, 21] we store data in D (cid:98) T φ in a ﬁrst in, ﬁrst out manner so thatonly data generated from recent iterations is stored in D (cid:98) T φ .
The agent’s policy and value functionsare trained using an off-policy actor critic algorithm using samples from D ∪ D (cid:98) T φ (Line 4).
In ourimplementation, we use soft actor-critic (SAC) [18] for agent training.
To update the model to minimise the loss in Equation 9 in Line 5, we sample data from D to estimatethe gradient for the MLE component.
To compute the adversarial component, we generate samples bysimulating the current policy and model and utilise the value function learnt by the agent to computethe gradient according to Equation 5.
Algorithm 1 RAMBO-RL
Require:
Normalised dataset, D ; n iter do
Adversarial to Eq.
9, using samples from D for the
MLE component and the current critic Q πφ and synthetic data sampled from π and
T for the adversarial component.
Experiments
In our experiments, we examine how well RAMBO performs compared to state-of-the-art baselines,and explore the importance of adversarial training to the performance of the algorithm.
The codefor our experiments is included in the supplementary material.
We evaluate our approach on thefollowing domains from the D4RL benchmarks [13].
MuJoCo There are three different environments representing different robots ( HalfCheetah, Hopper,
Walker2D ), each with 4 datasets ( Random, Medium, Medium-Replay, Medium-Expert ).
Randomcontains transitions collected by a random policy.
Medium contains transitions collected by anearly-stopped SAC policy.
Medium-Replay consists of the replay buffer generated while training the
Medium policy.
The Medium-Expert dataset contains a mixture of suboptimal and expert data.
AntMaze The agent controls a robot and navigates to reach a goal, receiving a sparse reward only ifthe goal is reached.
There are three different layouts of maze ( Umaze, Medium, Large ), and differentdataset types ( Fixed, Play, Diverse ) which differ in terms of the variety of start and goal locationsused to collect the dataset.
Single Transition Example
This domain has a one-dimensional state and action space, and severaldistinct regions of the action space are covered by the dataset.
We use this domain to illustrate thedifference between RAMBO and the most similar prior algorithm, COMBO.
See Appendix C.1 for amore detailed description.
Hyperparameter Selection and Evaluation
The base hyperparameters that we use for RAMBOmostly follow those used in SAC [18] and COMBO [75].
We ﬁnd that the performance of RAMBOis sensitive to the choice of rollout length, k , consistent with ﬁndings in previous works [21, 36].
Theother critical parameter for RAMBO is the choice of the adversarial weighting, λ .
For each dataset, we choose the rollout length and the adversarial weighting from one of three possibleconﬁgurations: ( k, λ ) ∈ {(2, 3e-4), (5, 3e-4), (5, 0) } .
We included ( k, λ ) =
(5, 0) as we found thatan adversarial weighting of 0 worked well for some datasets.
For all MuJoCo datasets we performedmodel rollouts using the current policy, but for some AntMaze datasets we used a random rolloutpolicy as we found that this performed better.
Further details about the hyperparameters are in
Appendix B.
To evaluate RAMBO, we ran each of the three hyperparameter conﬁgurations for ﬁve seeds each, andreport the best performance across the three conﬁgurations.
Thus, our evaluation of RAMBO utiliseslimited online hyperparameter tuning to choose hyperparameters for each dataset.
Note that this is themost common practice among existing model-based ofﬂine RL algorithms [8, 24, 36, 39, 76].
Theperformance obtained for each of the hyperparameter conﬁgurations is included in Appendix C.2.
Ofﬂine hyperparameter selection is an important topic in ofﬂine RL [46, 77].
This is becausein some applications selecting hyperparameters based on online performance may be infeasible.
Therefore, we present additional results in Appendix C.3 where we select between the three choicesof hyperparameters ofﬂine using a simple heuristic based on the magnitude and stability of the
Q -values during training.
Computational Resources During our experiments, each run of RAMBO has access to 2 CPUsof an Intel Xeon Platinum 8259CL processor at 3.1GHz, and half of an Nvidia T4 GPU.
With thishardware, each run of RAMBO takes 24-30 hours.
Baselines We compare RAMBO against state-of-the-art model-based (COMBO [75], MOReL [24],and MOPO [76]) and model-free (CQL [30], IQL [27], and TD3+BC [14]) ofﬂine RL algorithms.
Tofacilitate a fair comparison, we provide results for all algorithms for the MuJoCo-v2 D4RL datasetswhich contain more performant data than v0 [6] (details in Appendix B.6).
We report results for the
AntMaze-v0 datasets.
Results
The results in Table 1 show that RAMBO outperforms existing state-of-the-art methodson the MuJoCo locomotion domains.
Relative to existing algorithms, RAMBO obtains the bestperformance on the Medium-Replay datasets, and performs less well on the Medium-Expert datasets.
For the high-quality data in the Medium-Expert datasets, simpler approaches such as performingbehaviour cloning on the best 10% of trajectories can be used to achieve stronger performance thanofﬂine RL methods [27].
Therefore, the suboptimal performance of RAMBO on the Medium-Expertdatasets is less of a concern, as applying ofﬂine RL algorithms may not be the most suitable approachfor these datasets.
For AntMaze, RAMBO performs considerably less well than the model-free baselines.
This echoesthe ﬁndings of [67] that model-based approaches that use forward rollouts struggle to perform well inthe AntMaze domains, potentially because model-based algorithms are too aggressive and collidewith walls.
The authors of [67] show that reverse rollouts can lead to stronger performance formodel-based methods in these domains.
In future work, we wish to investigate whether combining
RAMBO with reverse rollouts can lead to stronger performance on these domains.
In Table 2 we present ablations comparing the performance of RAMBO to the same approach withoutany adversarial updates to the model.
These results demonstrate that overall performance degradesif the adversarial training is removed.
This parallels previous ﬁndings that conservatism is crucialto obtaining strong performance in ofﬂine RL.
Interestingly however, for some speciﬁc datasetswe obtain the best performance with no adversarial training (Appendix C.2).
This suggests that apotential direction for future work could be trying to identify which types of problems do not requireconservatism for a successful policy to be trained ofﬂine with model-based RL.
[Table]
These results use the same rollout length for each dataset as
RAMBO but with no adversarial updates (i.e. λ = 0 ).
The scores are averaged over 5 seeds.
To produce these results, we gave runs where the value function diverged a score of 0.
[Table] [Table]
Table 3 compares the performance of RAMBO and COMBO on the Single Transition Example.
Further analysis in Appendix C.1 shows that for this problem, the pessimistic value function updatesused by COMBO create local maxima in the Q -function throughout training.
Policy optimisation canbecome stuck in these local maxima.
On the other hand, the value function produced by RAMBO isinitially optimistic, and pessimism is introduced into the value function gradually as the transitionfunction is modiﬁed adversarially.
As a result, the likelihood of the policy becoming stuck in poorlocal maxima is reduced.
This may explain why RAMBO is able to achieve strong performancerelative to existing algorithms in the MuJoCo domains.
Gradually increasing the level of pessimismcould be a useful modiﬁcation for existing ofﬂine RL algorithms.
Conclusion and Future Directions
RAMBO is a promising new approach to ofﬂine RL which imposes conservatism by adversariallymodifying the transition dynamics of a learnt model.
Our approach is theoretically justiﬁed, andachieves state-of-the-art performance on standard benchmarks.
There are a number of possible extensions to RAMBO, some of which we have already discussed.
In addition, we would like to apply RAMBO to image-space domains by using deep latent variablemodels to compress the state space [17, 50] and adversarially perturbing the transition dynamics inthe latent space representation.
Furthermore, we wish to improve the computational efﬁciency of
RAMBO.
RAMBO is more computationally demanding than many prior ofﬂine RL methods due tothe need to simultaneously train the model as well as the policy.
Potential directions to improve theefﬁciency include warmstarting the policy with behaviour cloning [24, 59] or reducing the frequencyof model updates.
Finally, another potential research direction is to apply the idea of adversariallytrained dynamics models to improve robustness in the online setting.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We described two main limi-
tations: a) our approach performs less well on the AntMaze datasets (discussed inexperiments)
and b) requires more computation than many other approaches (discussed
in conclusion).
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] We include a complete
proof of Proposition 2.
For the theoretical results from [66], we provide references to
the appropriate parts of that paper.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] Detailed information about implementation details and hyperparameters
is in the appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] Standard deviation with respect to seeds is reported for
all results.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Details of compute used per run isin the experiments section.
Details of total compute used for full evaluation is in theappendix.4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the D4RL
benchmark datasets [13].
(b) Did you mention the license of the assets?
[Yes] In Appendix B.6.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]Code for our work is in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects..
[Table]
) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable
[Table]
you describe any potential participant risks, with links to Institutional Review
[Table]
Board
approvals,
applicable
N
A
[Table]
you include the estimated hourly
paid
participants and the total amountspent on participant compensation
N
A]
[Table]