Challenges and Opportunities in Ofﬂine Reinforcement Learning from Visual Observations
Abstract
Ofﬂine reinforcement learning has shown great promise in leveraging large precollected datasets for policy learning, allowing agents to forgo often-expensive online data collection.
However, to date, ofﬂine reinforcement learning from visual observations in the continuous control domain has been relatively under-explored, and there is a lack of understanding of where the remaining challenges lie.
In this paper, we seek to establish simple baselines for continuous control in the visual domain.
We show that simple modiﬁcations to two state-of-the-art visionbased online reinforcement learning algorithms, DreamerV2 and DrQ-v2, sufﬁce to outperform prior work and establish a competitive baseline.
We rigorously evaluate these algorithms on both existing ofﬂine datasets and a new testbed for ofﬂine reinforcement learning from visual observations that better represents the data distributions present in real-world ofﬂine reinforcement learning problems, and open-source our code and data to facilitate progress in this important domain.
Finally, we present and analyze several key desiderata unique to ofﬂine RL from visual observations, including visual distractions and visually identiﬁable changes in dynamics.
Introduction
Many prominent recent successes in deep learning have come from training models on large datasets [6, 10, 30].
By contrast, the reinforcement learning (RL, Sutton and Barto [54]) paradigm is typically characterized by agents learning from online interaction , which is often expensive or potentially even impractical to collect in abundant quantities.
Ofﬂine (or batch) reinforcement learning [11, 38] aims to address these issues by leveraging pre-collected datasets to train and deploy autonomous agents without requiring any online interaction with an environment.
While ofﬂine reinforcement learning algorithms, both model-based [2, 26, 62] and model-free [14, 16, 28, 31, 32, 56], have mastered challenging continuous control tasks, most prior works have relied on access to proprioceptive states [13].
By contrast, there exists only a single study of ofﬂine reinforcement learning from visual observations [47] for continuous control tasks, a problem setting for which there exist neither well-designed benchmarking tasks, nor any carefully evaluated baselines.
Training agents ofﬂine from visual observations provides an opportunity to make reinforcement learning more widely applicable to real-world settings, where we have access to vast quantities of visual observations of desirable behaviors.
For example, in autonomous driving [25], large quantities [Figure] of visual ofﬂine data already exist but have not been fully utilized [41, 61].
Similarly, in robotics, data collection is expensive due to costs associated with set-up and human supervision.
Effective, transferable ofﬂine reinforcement learning could allow us to reuse datasets gathered previously for different tasks or settings for unseen new problems [8].
Unlocking this potential would represent signiﬁcant progress towards learning general-purpose agents for realistic problems.
To enable the development of effective, robust, and adaptive algorithms for ofﬂine RL from visual observations, we present a suite of carefully designed datasets and benchmarking tasks for this burgeoning domain.
We use these tasks to establish simple performance baselines, to study how the composition of vision-based ofﬂine datasets affects performance of different types of RL algorithms, and to evaluate the extent to which algorithms for ofﬂine RL from visual observations satisfy a set of desiderata , including robustness to visual distractions, generalization across environment dynamics, and improved performance at scale.
Our evaluation identiﬁes clear failure modes of the baseline methods and highlights opportunities and open problems for future work which can be tackled with our benchmark.
Recent progress in ofﬂine reinforcement learning from proprioceptive observations has been driven by well-designed and easy-to-use evaluation testbeds and baselines.
We hope that V - D 4 RL and the analysis in this paper will help facilitate the development of robust RL agents that leverage large, diverse, and often imperfect ofﬂine datasets of visual observations across tasks and deployment settings.
We open-source our code and data at https://github.com/conglu1997/v-d4rl .
The core contributions of this paper are as follows:
We present a benchmark for ofﬂine RL from visual observations of DMC ONTROL S UITE (DMC) tasks [55].
This benchmark, Vision Datasets for Deep Data-Driven RL ( V - D 4 RL ), follows the design principles of the popular D 4 RL benchmark [13].
We identify three key desiderata for realistic ofﬂine RL from visual observations: robustness to distractions [53], generalization across dynamics [64], and improved performance for ofﬂine reinforcement learning at scale.
We present a suite of evaluation protocols designed to test whether ofﬂine RL algorithms satisfy these desiderata.
We establish model-based and model-free baselines for ofﬂine RL from visual observations.
We do so by modifying state-of-the-art online RL algorithms, DreamerV2 [21] and DrQ-v2 [58], which showcase the relative strengths of model-based and model-free algorithms for ofﬂine RL from visual observations.
We use these algorithms to provide simple baselines for the aforementioned desiderata to serve as a measure of progress for future advances in this domain.
Preliminaries
We model the environment as a Markov Decision Process (MDP), deﬁned as a tuple M = ( S , A , P, R, ρ 0 , γ ) , where S and A denote the state and action spaces respectively, P ( s (cid:48) | s, a ) the transition dynamics, R ( s, a ) the reward function, ρ 0 the initial state distribution, and γ ∈ (0 , 1) the discount factor.
The standard goal in online reinforcement learning is to optimize a policy π ( a | s ) that maximizes the expected discounted return E π,P,ρ
[ (cid:80) ∞ t =
γ t R ( s t , a t )] through interactions with the environment.
In ofﬂine reinforcement learning , the policy is not deployed in the environment until test time.
Instead, the algorithm has access to a ﬁxed dataset D env =
{ ( s i , a i , r i , s i +1 ) }
Ni =1 , collected by one or more behavioral policies π b .
Following Yu et al.
[62], we refer to the distribution from which D env was sampled as the behavioral distribution .
We ﬁrst describe recent advancements in ofﬂine RL and RL from visual observations through the lens of model-based and model-free methods.
2.1 Ofﬂine Reinforcement Learning Paradigms
Model-based.
A central problem in ofﬂine reinforcement learning is over-estimation of the value function [54] due to incomplete data [31].
Model-based methods in ofﬂine RL provide a natural solution to this problem by penalizing the reward from model-rollouts by a suitable measure of uncertainty.
Yu et al.
[62] provide theoretical justiﬁcation for this approach by constructing a pessimistic MDP (P-MDP) by lower-bounding the expected true return, η M ( π ) , using the error between the estimated and true model dynamics.
However, this quantity is usually not available without access to the true environment dynamics, so algorithms such as MOPO and MOReL in practice [26, 62] penalize reward with a surrogate measure of uncertainty.
Those algorithms train an ensemble of K probabilistic dynamics models [44], and deﬁne a heuristic based on the ensemble predictions.
Recent work [40] has shown that a more optimal choice to approximate true dynamics error is the standard deviation of the ensemble’s mixture distribution, as proposed by Lakshminarayanan et al.
Model-free.
In the model-free paradigm, we lose the natural measure of uncertainty provided by the model.
In lieu of this, algorithms such as CQL [32] attempt to avoid catastrophic over-estimation by penalizing actions outside the support of the ofﬂine dataset with a wide sampling distribution over the action bounds.
Recently, Fujimoto and Gu [14] have shown that a minimal approach to ofﬂine reinforcement learning works in proprioceptive settings, where ofﬂine policy learning with TD3 [15] can be stabilized by augmenting the loss with a behavioral cloning term.
2.2 Reinforcement Learning from Visual Observations
Recent advances in reinforcement learning from visual observations have been driven by use of data augmentation, contrastive learning and learning powerful recurrent world models of the environment.
We describe the current state-of-the-art in model-based and model-free methods.
Model-based (DreamerV2, Hafner et al.
DreamerV2 learns a powerful model of the environment using a Recurrent State Space Model (RSSM, Hafner et al.
[19, 20], and predicts ahead using compact model latent states.
The particular instantiation used in DreamerV2 uses model states s t containing a deterministic component h t , implemented as the recurrent state of a Gated Recurrent
Unit (GRU, [9]), and a stochastic component z t with categorical distribution.
The actor and critic are trained from imagined trajectories of latent states, starting at encoded states of previously encountered sequences.
Model-free (DrQ-v2, Yarats et al.
DrQ-v2 is an off-policy algorithm for vision-based continuous control, which uses data-augmentation [36, 60] of the state and next state observations.
The base policy optimizer is DDPG [39], and the algorithm uses a convolutional neural network (CNN) encoder to learn a low-dimensional feature representation.
Baselines for Ofﬂine Reinforcement Learning from Visual Observations
In this section, we begin by motivating our creation of a new benchmark ( V - D 4 RL ), introduce our new simple baselines combining recent advances in ofﬂine RL and vision-based online RL, and present a comparative evaluation of current methods on V - D 4 RL .
Comprehensive and rigorous benchmarks are crucial to progress in nascent ﬁelds.
To our knowledge, the only prior work that trains vision-based ofﬂine RL agents on continuous control tasks is LOMPO [47].
We analyze their datasets in Section {+3.3.
+} and ﬁnd they do not conform to standard D 4 RL convention.
3.1 Adopting D 4 RL Design Principles
In this section, we outline how to generate D 4 RL -like vision-based datasets for V - D 4 RL .
To generate ofﬂine datasets of visual observations, we consider the following three DMC ONTROL
S UITE (DMC) environments:
• walker-walk : a planar walker is rewarded for being upright and staying close to a target velocity.
[Figure]
• cheetah-run : a planar biped agent is rewarded linearly proportional to its forward velocity.
• humanoid-walk : a 21-jointed humanoid is rewarded for staying close to a target velocity.
Due to the huge range of motion styles possible, this environment is extremely challenging with many local minima and is included as a stretch goal. From these environments
[Figure]
we follow a D 4 RL -style procedure in considering ﬁve different behavioral policies for gathering the data.
As in D 4 RL [Figure] the base policy used to gather the data is Soft Actor–Critic (SAC, Haarnoja et al.
[18]) on the proprioceptive states.
• random : Uniform samples from the action space.
• medium-replay (mixed) :
The initial segment of the replay buffer until the SAC agent reaches medium-level performance.
• medium : Rollouts of a ﬁxed medium-performance policy.
• expert : Rollouts of a ﬁxed expert-level policy.
• medium-expert (medexp) : Concatenation
of medium and expert datasets above.
By default, each dataset consists of 100,000 total transitions (often 10 × less than in D 4 RL ) in order to respect the memory demands of vision-based tasks.
The cheetah and humanoid medium-replay datasets consist of 200,000 and 600,000 transitions respectively due to the increased number of samples required to train policies on these environments.
Full statistics of each dataset are given in Appendix A.
Further details on data generation are given in Appendix B.
3.2 Baselines
We show that for the two state-of-the-art online vision-based RL algorithms described in Section 2.2, simple adjustments from the proprioceptive literature sufﬁce to transfer them to the ofﬂine setting.
In Section 3.3, we demonstrate that these baselines provide a new frontier on our benchmark and on prior datasets.
Additional details and hyperparameters for our algorithms are given in Appendix C.
Model-based.
For DreamerV2, the latent stochastic states are sampled from an ensemble, which allows us to naturally deﬁne a penalty for the reward based on the dynamics uncertainty as in [62].
We adopt a similar approach to that studied in [40] and use the mean-disagreement of the ensemble.
Thus, the reward at each step becomes:
[Equation]
where λ is a normalization constant and µ ∗ ( s, a ) = 1 K (cid:80)
Kk =1 µ kφ ( s, a ) is the mean over the dynamics ensemble.
Instead of interleaving model-training steps and policy optimization steps, we simply perform one phase of each.
We refer to this algorithm as Ofﬂine DV2 .
Model-free.
For DrQ-v2, we note that the base policy optimizer shares similarities with TD3 [15], which has recently been applied effectively in ofﬂine settings from proprioceptive states by simply adding a regularizing behavioral-cloning term to the policy loss, resulting in the algorithm [Table]
TD3+BC [14].
Concretely, the policy objective becomes:
[Equation]
where λ is a normalization term, Q is the learned value function and π is the learned policy.
We apply the same regularization to DrQ-v2, and call this algorithm: DrQ+BC .
Prior work.
Since DrQ-v2 is an actor-critic algorithm, we may also use it to readily implement the CQL [32] algorithm by adding the CQL regularizers to the Q-function update.
We additionally compare against LOMPO [47], and behavioral cloning ( BC , [3, 5]), where we apply supervised learning to mimic the behavioral policy.
Ofﬂine DV2 is closely related to LOMPO as both use an
RSSM [19, 20] as the fundamental model, however Ofﬂine DV2 is based on the newer discrete
RSSM with an uncertainty penalty more closely resembling the ensemble penalties in supervised learning [35] and uses KL balancing during training [21].
3.3 Comparative Evaluation
We now evaluate the ﬁve algorithms described in Section 3.2 on a total of ﬁfteen datasets.
To provide a fair evaluation, we provide full training curves for each algorithm in Figure 2 and summarize ﬁnal performance with error in Table 1.
Since no online data collection is required, we measure progress through training via an “ofﬂine epochs” metric which we deﬁne in Appendix D.
Table 1 shows a clear trend: Ofﬂine DV2 is the strongest on the random and mixed datasets, consisting of lower-quality but diverse data, DrQ+BC is the best on datasets with higher-quality but still widelydistributed data and pure BC outperforms on the high-quality narrowly-distributed expert data.
We see from Table 1 and Figure 2 that DrQ+BC is extremely stable across seeds and training steps and has the highest overall performance.
CQL is also a strong baseline, especially on expert data, but requires signiﬁcant hyperparameter tuning per dataset, often has high variance across seeds, and is also far slower than DrQ+BC to train.
Finally, no algorithm achieves strong performance on the challenging humanoid datasets, mirroring the online RL challenges [21, 58], with only the supervised BC and, by extension, DrQ+BC showing marginal positive returns (for more graphs, see Appendix E.1).
Perhaps surprisingly, Ofﬂine DV2 learns mid-level policies from random data on DMC environments.
Furthermore, the random data are more challenging than their D 4 RL equivalents because there is no early termination, and thus mostly consists of uninformative failed states; this shows the strength of model-based methods in extracting signal from large quantities of suboptimal data.
On the other hand,
Ofﬂine DV2 is considerably weaker on the expert datasets that have narrow data distributions.
For these environments, we ﬁnd the uncertainty penalty is uninformative, as discussed in Appendix F.2.
Taking all these ﬁndings into consideration leads us to our ﬁrst open problem, which we believe continued research using our benchmark can help to answer:
[Table]
Open Problem 1 : Can a single algorithm outperform both the model-free and model-based baselines, and produce strong performance across all ofﬂine datasets?
3.3.1 Comparison to the LOMPO Benchmark
For a fair comparison to LOMPO we also benchmark on the data used in Rafailov et al.
[47] on the DMC Walker-Walk task.
In the LOMPO benchmark, the datasets are limited to three types: {medium-replay, medium-expert and expert}.
We provide ﬁnal scores in Table 2.
While LOMPO struggles on V - D 4 RL , it performs reasonably on its own benchmark.
However, LOMPO is still outperformed by Ofﬂine DV2 on all datasets, whereas DrQ+BC is the best on two datasets.
We may explain the relative strength of LOMPO on this benchmark by noting that the medium-expert dataset used by Rafailov et al.
[47] is described as consisting of the second half of the replay buffer after the agent reaches medium-level performance, thus containing far more diverse data than a bimodal D 4 RL -style concatenation of two datasets.
Furthermore, the expert data is far more widely distributed than that of a standard SAC expert, as we conﬁrm in the statistics in Table 6 of Appendix A.
Desiderata for Ofﬂine Reinforcement Learning from Visual Observations
A key ingredient in recent advances in deep learning is the use of large and diverse datasets, which are particularly prevalent in vision, to train models.
Such datasets enable the learning of general features that can be successfully transferred to downstream tasks, even when the original task bears little immediate similarity with the transferred task [46, 57].
This is a clear rationale for adopting visual observations in ofﬂine RL; by leveraging large quantities of diverse high-dimensional inputs, we should be able to learn generalizable features and agents for control.
However, combining rich visual datasets with RL presents its own unique challenges.
In this section
[Figure]
we present important desiderata that highlight this
[Figure]
and conclude each with an open problem that requires further research
[Figure] [Figure]
As CQL is quite sensitive to hyperparameters
4.1 Robustness to Visual Distractions
One desideratum for ofﬂine RL from visual observations is the ability to learn a policy from data collected under multiple different settings.
For example, quadrupedal robots deployed at different times of day (e.g., morning and night) will likely gather data having signiﬁcantly different visual properties, such as brightness and range of visibility.
Although the robot may produce similar behaviors in both deployments, superﬁcial differences in visual outputs present a host of opportunities for the agent to learn spurious correlations that prevent generalization [48, 52].
A key opportunity that arises is the potential to disentangle sources of distractions through training on multiple settings, facilitating the learning of general features.
By contrast, proprioceptive observations do not generally have distractions, as they are typically low-dimensional and engineered for the task of interest [37].
This also limits their ability to transfer, as it is unclear how to incorporate features learned under one set of agent geometries to another.
To test a simpliﬁed version of this challenge, we train our baseline agents using newly created datasets featuring varying visual augmentations from the Distracting Control Suite [53].
This suite provides three levels of distractions (i.e., low, moderate, high), and each distraction represents a shift in the data distribution.
We subsequently refer to the level of distraction as the “shift severity” [50].
The ofﬂine datasets are then constructed as mixtures of samples from the original environment without distractions and samples from an environment with a ﬁxed distraction level.
Further details of how we generate this data are given in Appendix B.
The learned policies are then evaluated on test environments featuring unseen distractions of the same shift severity.
We compare the baseline algorithms, Ofﬂine DV2 and DrQ+BC on datasets that they excel on in
Section 3 and Section 4.3: walker-walk random with 100,000 datapoints and cheetah-run mediumexpert with 1 million respectively.
The returns normalized by unshifted performance are shown in Figure 3.
Ofﬂine DV2 is able to accommodate datasets with mixed distractions and generalizes reasonably well to unseen test distractions, especially when trained with ‘low’ and ‘moderate’ levels of shift severity.
Similarly, DrQ+BC is robust to multiple training distractions, with little to no degradation in performance.
However, the policy learned is brittle with respect to unseen distractions, and performs signiﬁcantly worse on the test environments.
Overall, both Ofﬂine DV2 and DrQ+BC represent strong baselines for mixed datasets.
Interestingly,
Ofﬂine DV2 demonstrates strong generalization to unseen distractions.
This can be explained by generalization that occurs in the trained world-model, which uses a self-supervised loss; we discuss reasons behind this in Appendix F.3.
This could be improved even further with recent reconstructionfree variants of DreamerV2 [43, 45] which have shown robustness to distractions.
On the other hand, we observe DrQ+BC generalizes poorly to unseen distractions, presenting a direction for future work using our datasets to learn robust model-free agents.
This directly leads us to our next open problem:
Open Problem 2 : How can we improve the generalization performance of ofﬂine model-free methods to unseen visual distractions?
4.2 Generalization Across Environment Dynamics
Another desideratum of ofﬂine RL from visual observations is learning policies that can generalize across multiple dynamics.
This challenge manifests in three clear ways.
Firstly, we will likely collect data from multiple agents that each have different dynamics, and must therefore learn a policy that can perform well when deployed on any robot that gathered the data (i.e., train time robustness).
Secondly, we may be provided with asymmetric data, featuring scarce coverage of particular dynamics, and therefore require the ability to leverage data from more abundant sources (i.e., transferability).
Thirdly, we may be presented with unseen dynamics at deployment time, and must therefore learn a policy that is robust to these changes (i.e., test time robustness).
A key opportunity that arises in visual observations is the improved richness of the underlying dataset compared to proprioceptive data.
For instance, some dynamics changes may be visually obvious (e.g., changed limb sizes, broken actuators), whereas in the proprioceptive setting such information may not be available.
Without this information, we must turn to meta-RL [49, 65] or HiPMDP [27, 64] approaches that try to infer the missing information from gathered trajectories, adding complexity to the RL process.
In contrast, this information can be contained explicitly in visual observations, and should allow adaption to a range of downstream tasks without complex inference methods.
To test this hypothesis, we consider two settings from the MTEnv benchmark [51] which adapts
DMC: cheetah-run with modiﬁed torso length and walker-walk with modiﬁed leg length.
We follow an analogous approach to [64] where we consider eight different settings {A - H} ordered in terms of limb length, and construct new ofﬂine datasets using {B, C, F, G} in equal proportions as our training data.
The settings {A, H} are considered the extrapolation generalization environments and {D, E} interpolation generalization.
Data generation details are provided in Appendix B.
[Table]
We evaluate Ofﬂine DV2 and DrQ+BC on medium-expert datasets of size 100,000 and 1 million respectively and show the results in Table 3.
We see that DrQ+BC learns policies that are suitable for transfer across multiple tasks in both walker and cheetah, and maintains that performance on the interpolation test environments.
For the extrapolation environments, we see an average drop of around 30%.
While this may represent adequate performance, especially compared to a medium policy, it is a striking drop when compared to performance on in-distribution dynamics.
This suggests there is a dynamics generalization gap that remains for model-free methods when extrapolating, and represents clear opportunities for further research.
Ofﬂine DV2 displays similar trends (results on random datasets are in Appendix E.3).
On the random data, Ofﬂine DV2 learns a similar quality policy to that on the base environment, but experiences no deterioration in performance on the test environments in walker or cheetah.
Thus, we demonstrate the sufﬁciency of both Ofﬂine DV2 and DrQ+BC as baselines in multitask ofﬂine RL without any modiﬁcation ; model-based approaches further admit opportunities for zero-shot generalization [4].
We now contrast our work to that of [64], where a multitask policy was trained using a total of 3. million timesteps of online data collection.
Whilst it is hard to compare ofﬂine and online results, our
DrQ+BC algorithm uses less data, with 1 million total timesteps, and obtains similar extrapolation return on the walker environments.
This supports a similar conclusion reached by [34] who show that our approach, simply minimizing the sum of the task losses, is drastically underestimated in the literature.
As noted before, we suffer a comparatively larger drop in performance, lending further evidence that closing this generalization gap should be prioritized.
In conclusion, we believe there are many further avenues for future research using these benchmarks; an immediate open problem we have identiﬁed is as follows:
Open Problem 3 : How can we improve generalization to new dynamics that are not contained in the ofﬂine dataset?
4.3 Improved Performance with Scale
[Figure]
Learning from large datasets presents huge opportunities for learning general agents for control.
To make use of them, we need to understand how our baselines scale with dataset size.
We analyze our base choice of 100,000 observations for VD 4 RL in Figure 4, where we vary the size of the walker-walk random dataset for Ofﬂine DV2 and the cheetah-run medexp dataset for DrQ+BC in the range of { 0 . 25 × , . . .
, 4 ×} the size of the original dataset.
We observe a monotonic increase in the performance of both Ofﬂine DV2 and DrQ+BC with increasing dataset size but hit diminishing returns past 2 × the original size.
We note, perhaps surprisingly, that Ofﬂine DV2 can reach ≈ 500 total return from random data that average 10 × less.
For the walker and cheetah datasets with ﬁxed distributions—random, medium, medium-expert and expert—Table 4 shows an average increase of 42.1% for Ofﬂine DV2 and 32.5% for DrQ+BC compared to 10.8% for BC when we scale the dataset size to 500,000, showing that the reinforcement learning algorithms make far better use of increased data than supervised behavioral cloning.
However, a crucial difference between Ofﬂine DV2 and DrQ+BC is that DrQ+BC handles larger ofﬂine datasets far more readily.
DrQ+BC policy training for the same number of epochs on 500,000 and 100, [Table] observations takes 8 and 1.6 hours respectively on a V100 GPU.
This is signiﬁcantly quicker than Ofﬂine DV2, which takes 10 hours to train on 100,000 observations; we discuss this further in Appendix F.2.
This signiﬁcant computational discrepancy leads to a clear open problem:
Open Problem 4 : How can we scale model-based methods to larger datasets?
Related Work
There has been signiﬁcant recent progress in ofﬂine RL, giving rise to many benchmarks.
We list several here; to our knowledge, no contemporary works feature tasks related to distractions or changed dynamics.
Benchmarks for continuous control on states.
D 4 RL [13] is the most prominent benchmark for continuous control with proprioceptive states.
The large variety of data-distributions have allowed for comprehensive benchmarking [26, 29, 32, 62, 63] and understanding the strengths and weaknesses of ofﬂine algorithms.
Our work aims to establish a similar benchmark for vision-based tasks.
RL
Unplugged [17] also provides data on the DMControl Suite, but focuses on proprioceptive states.
Analysis on characteristics of ofﬂine datasets.
Recent work [12] has sought to understand when ofﬂine RL algorithms outperform behavioral cloning in the proprioceptive setting.
Kumar et al.
[33] recommend BC for many settings but showed theoretically that ofﬂine RL was preferable in settings combining expert and suboptimal data, which we conﬁrm in Tables 1 and 4.
Vision-based discrete control datasets.
Whilst there has been a lack of suitable benchmarks for vision-based ofﬂine continuous control, vision-based datasets for discrete control have been created for Atari [1].
However, at 50M samples per environment, this poses signiﬁcant challenges; we believe that V - D 4 RL ’s 100K benchmark represents a far more approachable challenge for practitioners.
Ofﬂine vision-based robotics.
Ofﬂine vision-based learning is an active area of robotics research.
In [24], policies are ﬁne-tuned on real interactions, after ofﬂine pre-training on visual data of robot interactions.
[8] follows a similar setup, and explores the representation learning for controllers.
Our work complements this, is not focused on goal-conditioned learning, and is open source.
Conclusion
In this paper, we took the ﬁrst steps towards establishing benchmarking tasks and competitive baselines for ofﬂine reinforcement learning from visual observations.
Until now, work in this space has been nascent, with ad-hoc analyses leading to unclear comparisons.
To address the lack of meaningful evaluations and comparative analyses in this space, we provided a set of straightforward and standardized benchmarking tasks that follow popular low-dimensional equivalent experiment setups and presented competitive model-based and model-free baselines.
We analyzed key factors that help explain the performance of these approaches, while also demonstrating their ability to generalize in more challenging settings that are unique to visual observations.
We hope this work will be useful to practitioners and researchers alike and that it will provide a springboard for developing ofﬂine reinforcement learning methods for real-world continuous-control problems and spark further progress in this space. Acknowledgments and Disclosure of Funding Cong Lu and Tim G. J. Rudner are funded by the Engineering and Physical Sciences Research Council (EPSRC).
Philip J. Ball is funded through the Willowgrove Studentship.
Tim G. J. Rudner is also funded by the Rhodes Trust and by a Qualcomm Innovation Fellowship. We are grateful to Rafael Rafailov for sharing the data used with LOMPO and Edoardo Cetin for his NumPy Array replay buffer implementation. We thank Samuel Daulton, Clare Lyle, Muhammad Faaiz Tauﬁq for detailed feedback on an early draft.
The authors would also like to thank the anonymous ICMLreviewers for constructive feedback which helped improve the paper A Ofﬂine Dataset Characteristics B Ofﬂine Data Generation Details B.1 Standard Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Visually Distracted Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Multitask Datasets . . .
B.4 Choice of Behavioral Policy for Ofﬂine Data . . . . . . . . . . . . . . . . . . . . .
B.5 Broader Issues with Visual Data . . . . . . . . . . . . . . . . . . . . . . . . . . .
C Algorithmic Details
C.1 Ofﬂine DV2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 DrQ+BC . . .
C.3 CQL . . .
D Hyperparameter and Experiment Setup
D.1 Ofﬂine DV2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 DrQ+BC . .
D.3 Behavioral Cloning . .
D.4 CQL
D.5 LOMPO
D.6 Computational Cost
E Further Tabular Results and Training Curves
E.1 Training Curves for the Humanoid Environment
E.2 Full Tabular Results for Experiments with Distractions
E.3 Further Ofﬂine DV2 Results on Multitask Datasets
F Further Ablation Studies
F
DrQ
+
BC Random
-Expert
Ablation Studies . . . . . . . . . . . . . . . . . . . . .
F
Ofﬂine DV2 Training Time
and
Uncertainty
Penalty . . . . . . . . . . . . . . . . .
F. Understanding Model-Based Extrapolation
A Ofﬂine Dataset Characteristics
We provide explicit statistics on the returns of each episode for the datasets used in our main evaluation.
This provides a reasonable proxy to how diverse each dataset is.
[Table]
We compare this to the LOMPO datasets and ﬁnd that they are more widely distributed, due to the differing data-collection method
[Table]
[Table] As we see in Table 6 and Figure [Table] the LOMPO walker-walk expert dataset has a standard deviation roughly 8x higher than our expert dataset and has an extremely wide [min, max] range.
Furthermore, whilst our medexp dataset is bimodal [Table] the LOMPO medexp dataset’s returns are a continuous progression.
This reﬂects that the LOMPO data is sampled from the second half of a replay buffer after medium-level performance is attained, akin to the medium-replay (mixed) datasets
[Figure]
B Ofﬂine Data Generation Details B.1 Standard Datasets
For the standard ofﬂine medium and expert datasets, we ﬁrst train SAC [18] policies on the proprioceptive states until convergence, taking checkpoints every 10,000 frames of interaction.
We use a frame skip of 2, the default in other state-of-the-art vision RL algorithms [21, 58].
We deﬁne expert policies as those that have converged in the limit.
For DMControl tasks, this typically means near 1,000 reward on the environment they were trained on.
We deﬁne medium policies as the ﬁrst saved agent during training that is able to consistently achieve above 500 reward in the environment.
We conﬁrm these thresholds are reasonable, as we observe a noticeable gap between the behavior of the medium and expert level policies.
In order to generate the ofﬂine visual observations, we deploy the proprioceptive agents in the environment, and save the visual observation rendered from the simulator instead of the proprioceptive state.
This provides us with the ﬂexibility to generate observations of any size without having to retrain for that resolution (e.g., 84 × 84 or 64 × 64 ).
As was done in D4RL, we generate data using a stochastic actor, which involves sampling actions from the Gaussian posterior distribution of the SAC policy, featuring a parameterized variance head that determines the amount of action stochasticity at each state.
For the mixed datasets, we simply store the replay buffer of the medium agent when it ﬁnishes training, and convert the proprioceptive observations into visual observations.
The cheetah-run mixed dataset is larger because the SAC agent takes longer to reach medium-level performance.
B.2 Visually Distracted Datasets
The visual distractions from the Distracting Control Suite do not manifest themselves in the proprioceptive states, so we can naturally generate the shifted visual observations by simply rendering the distractions on top of the existing visual observations.
Thus, we can simply use the same “medium” and “expert” level policies as before, trained on proprioceptive states.
B.3 Multitask Datasets
For the multitask dataset, we follow the same procedure described in Appendix B.1 but instead train a SAC on the proprioceptive states from the modiﬁed tasks (e.g., {A-H}).
The medium and expert policies are deﬁned in the same way.
B.4 Choice of Behavioral Policy for Ofﬂine Data
Interestingly, we found that using online DrQ-v2 as the ofﬂine behavioral policy made the tasks signiﬁcantly easier to learn for all agents.
This suggests that the proprioceptive agent may learn behavior modes that are less biased towards being easy under vision-based methods; for instance,
DrQ-v2 may be biased towards behavior modes that induce fewer visual occlusions compared to proprioceptive SAC.
B.5 Broader Issues with Visual Data
Large datasets consisting of images often contain systematic biases, which can damage generalization.
The datasets constructed in this paper are all synthetic from simulated reinforcement learning environments.
However, as we move towards applying ofﬂine RL from visual observations to realworld tasks, it is important to take these potential dangers into account and extend existing work in algorithmic fairness from computer vision to our setting.
C Algorithmic Details
We provide additional details for both algorithms here and indicate where our modiﬁcations have been made.
C.1 Ofﬂine DV
In the ofﬂine setting, it sufﬁces to simply perform one phase of model training and one phase of policy training for the DreamerV2 [21] algorithm.
Each episode in the ofﬂine dataset is ordered sequentially to facilitate sequence learning.
To this instantiation of DreamerV2, we simply add a reward penalty corresponding to the mean disagreement of the dynamics ensemble.
During standard DreamerV policy training, imagined latent trajectories { ( s τ , a τ ) } t
+ H τ = t are assigned reward r τ = E
[ q θ ( · | s τ )] according to the mean output of the reward predictor.
The imagined latent states s t consist of a deterministic component h t , implemented as the recurrent state of a GRU, and a stochastic component z t with categorical distribution.
The logits of the categorical distribution are computed from an ensemble (with input h t ) over which we compute the mean disagreement.
C.2 DrQ+BC
Here, we simply modify the policy loss term in DrQ-v2 [58] to match the loss given in Fujimoto and Gu [14].
Following the notation from Yarats et al.
[58], the DrQ-v2 actor π φ is trained with the following loss:
[Equation]
where h t = f ξ (aug ( s t )) is the encoded augmented visual observation, a t = π φ ( h t )
+ (cid:15) is the action with clipped noise to smooth the targets (cid:15) ∼ clip (cid:0) N (cid:0) 0 , σ 2 (cid:1) , − c, c (cid:1) .
Note that we also do not update encoder weights with the policy gradient.
We also train an ensemble of two fully connected
Q-networks, which both use features from a single encoder, and take their minimum when calculating target values and actor losses.
The resultant algorithm can be viewed as TD3 [15], but with decaying smoothing noise parameters c .
In DrQ+BC, this loss becomes:
[Equation]
where λ = α 1 N (cid:80) ( hi,ai ) | Q ( h i ,a i ) | is an adaptive normalization term computed over minibatches.
α is a behavioral cloning weight, always set to 2.5 in Fujimoto and Gu [14], which we also adopt.
We experimented performing an extensive grid search over α , but did not observe any noticeable beneﬁt in our ofﬂine datasets by deviating away from the default value.
C.3 CQL
Our CQL implementation was also built on top of the DrQv2 codebase for comparability.
We use the
CQL( H ) objective with ﬁxed weight, as we found this to be the most performant.
This corresponds to choosing the KL-divergence to a uniform prior as the regularizer R ( µ ) in Kumar et al.
Concretely, the Q-function objective becomes
[Equation]
where α CQL is a trade-off factor, ˆ π β refers to the empirical behavioral policy and ˆ B π k to the empirical
Bellman operator that backs up a single sample.
This is approximated by taking gradient steps and sampling actions from the given bounds.
D Hyperparameter and Experiment Setup D.1 Ofﬂine DV
Our Ofﬂine DV2 implementation was built on top of the ofﬁcial DreamerV2 repository at: https: //github.com/danijar/dreamerv2 with minor modiﬁcations.
The code was released under the
MIT License.
Table 7 lists the hyperparameters used for Ofﬂine DV2.
For other hyperparameter values, we used the default values in the DreamerV2 repository.
We found a default value of λ = 10 works for most settings.
The only settings where this changes are λ = 3 for both random datasets and λ = 8 for the walker-walk mixed dataset.
[Table]
[Table]
For the penalty choice
[Table]
we chose mean disagreement of the ensemble because it comprises one half of the ensemble variance, which was shown to be an optimal choice for ofﬂine model-based reinforcement learning in Lu et al.
We found that the other component of the ensemble variance, the average variance over the ensemble, was uninformative and so discarded it
[Table]
D.2 DrQ+BC
Our DrQ+BC implementation was built on top of the ofﬁcial DrQ-v2 repository at: https:// github.com/facebookresearch/drqv2 .
The code was released under the MIT License.
Tablelists the hyperparameters used for DrQ+BC.
For other hyperparameter values, we used the default values in the DrQ-v2 repository.
Due to the size of some of our ofﬂine datasets, we found the default replay buffer would not scale to the ofﬂine datasets.
Thus, we used a NumPy [22] array implementation instead.
[Table]
We tuned α within {1.5, 2.5, 3.5} but as Fujimoto and Gu [14] found, we did not observe any noticeable beneﬁt from deviating away from the default value for α .
D.3 Behavioral Cloning
Our BC implementation shares the exact same policy network and hyperparameters in DrQ+BC but just minimizes MSE on the ofﬂine data.
Consequently, we must also optimize the learned encoder using the supervised learning loss (unlike in DrQ+BC, where the TD-loss only contributes to the encoder representation learning, and not the policy loss).
D.4 CQL
Similarly to BC, our CQL implementation is also based on the same networks and hyperparameters in DrQ+BC.
CQL introduces one extra hyperparameter, the trade-off factor α CQL .
We perform a hyperparameter sweep for this over the range: { 0 . 5 , 1 , 2 , 5 , 10 , 20 } .
We chose the following values per environment:
[Table]
D.5 LOMPO
For our LOMPO evaluation, we used the ofﬁcial repository at: https://github.com/ rmrafailov/LOMPO .
The code was open-sourced without license.
We perform a hyperparameter search over the uncertainty weight λ in the range { 1 , 5 } .
The default value used in Rafailov et al. [47] is λ = 5 , but we found that λ
= 1 worked better for random datasets.
The accompanying data for the DMC Walker-Walk data from Rafailov et al.
[47] was very kindly provided by the authors.
D.6 Computational Cost
The experiments in this paper were run on NVIDIA V100 GPUs.
On the standard V - D 4 RL 100K datasets, DrQ+BC took 1.6 hours, Ofﬂine DV2 took 10 hours, and CQL took 12 hours.
Since we wish to compare several ofﬂine algorithms using the same dataset, we deﬁne a notion of “ofﬂine epoch” for all algorithms to show training performance over time.
We simply normalize the total number of gradient steps, so training progress falls within [0 , 1000] .
E Further Tabular Results and Training Curves E.1 Training Curves for the Humanoid Environment
We include additional training curves for the Humanoid environment, as in Figure 2 for all our evaluated algorithms in Figure 6.
As we noted previously, only the supervised BC and by extension
DrQ+BC achieve meaningful return on this environment
[Figure]
We give the full un-normalized results for Section 4.1 in Table 10 and Table 11. The highlighted base results are the same as in Table 1 for Ofﬂine DV2 and Table 4 for DrQ+BC. The Ofﬂine DV2 results use 100K datapoints, and the DrQ+BC results use 1 million, in light of the discussion in Section 4.3.
[Table]
[Table]
E.3 Further Ofﬂine DV2 Results on Multitask Datasets On the random multitask data, Ofﬂine DV2 learns a similar quality policy to that on the base environment, but experiences no deterioration in performance on the test environments in walker or cheetah. [Table] F Further Ablation Studies F.1 DrQ+BC Random-Expert Ablation Studies
[Figure]
Analogously to Fujimoto and Gu [14], we continue to see that DrQ+BC does well on mixed datasets with high reward with experiments on the concatenated random-expert datasets.
Surprisingly, we see behavioral cloning also does reasonably well on cheetah-run with random-expert data.
This is likely to be because there is signiﬁcant distribution shift between the states of random and expert trajectories for that environment.
This would lead to minimal destructive interference between similar states which contain completely different actions, as these state distributions largely do not overlap.
This is in contrast to the medium-expert dataset, which experiences higher state overlap between its two modes (i.e., states generated by a medium and an expert policy respectively), resulting in a marginal “average" action being learned which is likely suboptimal, as can be seen by the poor performance of the BC agent in Tables 1 and 4.
F.2 Ofﬂine DV2 Training Time and Uncertainty Penalty
One of the major factors preventing algorithms which use a RSSM like Ofﬂine DV2 and LOMPO from scaling to larger datasets is the time required for model training.
The standard number of epochs of model training for our 100K datasets in Section 3 is 800 epochs, which takes around 6 hours on a V100 GPU.
This scales linearly with number of training points if we maintain the same batch size.
As we show in Figure 8, this is mandatory for performance and is a fundamental limitation of model-based methods.
We can see that evaluated return increases and becomes more tightly distributed as the number of training epochs increases until 800.
In Table 1, we see that Ofﬂine DV2 is considerably weaker on the expert datasets that have narrow data distributions.
On these expert datasets, we found that the model uncertainty often had lower variance across a trajectory compared to models trained on other datasets, and was thus uninformative.
We give summary statistics for the uncertainty penalty on states generated by a random policy in the walkerwalk environment in Table 13.
Since we followed the author’s DreamerV2 implementation [21],
[Figure]
only the stochastic portion of the latent state is predicted by the ensemble; this may mean crucial calibration is lost when ignoring the impact of the deterministic latent. Future work could involve investigating SVSG [23], an extension to DreamerV2 with a purely stochastic latent state
[Table]
F.3 Understanding Model-Based Extrapolation
We further investigate the reasons why our model-based baseline, Ofﬂine DV2, appears to generalize to unseen distractions.
The RSSM includes a latent decoder for the visual observations, which is primarily used during training to provide a self-supervised reconstruction loss.
During deployment, the policy solely relies on the latent states from the encoder, and the decoder is effectively discarded.
However, we can still use the decoder at test-time in order to understand the information contained in the latent states.
To do this, we take the latent states generated during rollouts in the extrapolation experiments, and ‘translate’ them into natural images by passing them through the decoder.
We ﬁrst investigate the setting where each RSSM is trained only on data from a single distraction and then transferred.
This setting is the most successful, as can be seen in Figure 3.
Thus, in Figure 9 we ﬁrst show the ground truth observation provided to the agent, then below this we show the decoder output reconstructed from the latent.
In many cases, we can recover the original pose despite ending up in different visual surroundings.
This is an indication that despite the decoder overﬁtting to the exogenous factors in the visual input (e.g., background, colors), the latent captures the salient state information of the agent (e.g., joint positions and angles), explaining the strong test-time transfer performance.
In Figure 3, we note that settings with a mixture of distractors often had worse test-time transfer performance than those with a single distractor.
To explain this, we consider an RSSM trained on images with a combination of two ﬁxed distractors, and examine the reconstruction of an episode under a third distraction.
We see in Figure 10, that the RSSM latents are split between the two modes of the data and the reconstruction switches robot color and background midway.
This signiﬁcantly confuses the recovered pose of the walker and likely causes the degradation in performance.
Disentangling the factors of variation [7, 42] represents important future work for ofﬂine RL from visual observations.