Paper: D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The proposed benchmark does not include much more diverse domains compared to D4RL. I think with some past works such as RoboMimic [1] and some recent works such as RL-ViGen [2], similar and even more diverse benchmarks could be easily made, by running RL agents to collect offline datasets.

Author response:
**The proposed benchmark does not include much more diverse domains compared to D4RL.
I think with some past works such as RoboMimic [1] and some recent works such as RL-ViGen [2], similar and even more diverse benchmarks could be easily made, by running RL agents to collect offline datasets.**

We believe our tasks are significantly more diverse than RoboMimic. In particular:
1. Our environments use a number of different objects for the manipulation setting
2. We use two different types of robot arms - a Franka Emika and a WidowX, as well as an A1 quadruped robot. 
3. The tasks we consider are more complex and involve achieving multiple objectives (i.e. arranging multiple objects in a scene).
4. Furthermore, our tasks have a number of continuous distribution shifts, such as scene, texture and camera randomization. 
5. Moreover we have collected over 20 hours of realistic human manipulation data beyond pure expert demonstrations, such as play, failures, and sub-optimal execution, which was not analyzed by RoboMimic. 

Thank you for bringing RL-ViGen to our attention. We view this work as complementary to ours, since our manipulation environments do not overlap. 

1. We aimed to create a lightweight and accessible benchmark for data-centric RL, which does not involve heavy simulators, such as CARLA and Habitat. 
2. Our proposed environments have naturally occurring distribution shifts, such as diverse objects, visuals, lighting and camera angles, while the ones in RL-ViGen are rather artificial.
3. We also aimed to utilize realistic human-collected datasets, rather than replay-buffer based ones.

Reviewer comment (Experiments):
Possibly unfeasible settings. As shown by evaluation results, most SOTA algorithms are not working well on this benchmark. This could possibly be attributed to the difficulties of the task, but from my perspective, there could also be some key factors that are not right, such as the problem formulation and the framework of offline RL. (This point might be not significant, but it is my personal concern about the benchmark.)

Author response:
**Possibly unfeasible settings. As shown by evaluation results, most SOTA algorithms are not working well on this benchmark. This could possibly be attributed to the difficulties of the task, but from my perspective, there could also be some key factors that are not right, such as the problem formulation and the framework of offline RL. (This point might be not significant, but it is my personal concern about the benchmark.)**

We have included improved results on some of the tasks, achieved by a more extensive hyper-parameter search (see table below). The fact that SoTA RL methods achieve good performance on other benchmarks (such as D4RL) but do not achieve good performance on our benchmark is echoed by results on many real world robot manipulation tasks, in which behavior-cloning based methods tend to out-perform current RL methods. We hope that our benchmark can drive the development of RL methods which are more successful in real robot applications. For example, a recent model-based RL method [1] is able to fully solve the Standard Franka Kitchen from images with online fine-tuning. 

| Task Name         | BC    | DDPM + BC | IQL   | IDQL  | CQL | TD3 + BC |
|-------------------|-------|-----------|-------|-------|-----|----------|
| Sorting           | 32.54 | 28.47     | 33.56 | 35.59 | 0   | 14.24    |
| Sorting Bin Noise | 44.75 | 34.58     | 45.76 | 49.49 | 0   | 21.36    |


[1] MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning 
Rafael Rafailov, et. al, Conference on Robot Learning, 2023

Reviewer comment (Novelty):
Overclaim of the paper title. This work names the newly proposed benchmark as D5RL, which is obviously following the name of D4RL. However, this could be misleading for the entire community to keep pursuing the limited improvements on a new but essentially similar benchmark.

Author response:
**Overclaim of the paper title. This work names the newly proposed benchmark as D5RL, which is obviously following the name of D4RL. However, this could be misleading for the entire community to keep pursuing the limited improvements on a new but essentially similar benchmark.**

We believe there are substantial differences between d4rl and our proposal. The main ones being:
1. Focusing on realistic robotic tasks versus simple simulated embodiments.
2. Learning from realistic observation spaces rather than ground-truth states.
3. We introduced a significant amount of diversity in terms of objects to manipulate (which require different grasps) and visual environments.
4. We introduced a number of new realistic human-collected datasets, such as undirected manipulation and sub-optimal execution data, rather than using RL replay buffers.

Reviewer comment (Experiments):
Limited number of tasks for the A1 legged locomotion. Tasks such as jumping, obstacle avoidance, etc. can also be of interest in this.

Author response:
Here we present an initial version of our benchmark. If there is sufficient interest in the community, it is possible to add additional locomotion tasks, including more complex embodiments in the future.

Reviewer comment (Evaluation):
My impression from Tables 1 and 2 is the proposed benchmark is too difficult for existing offline RL methods to achieve meaningful performances, since most methods except BC and IQL have near-zero returns. While having a challenging benchmark is good, a too-challenging benchmark will not provide much signal to improve the performance of current methods.

Author response:
Thank you for your suggestion. We have added some updated results to improve the performance on the WidowX task (see Table below), one of these challenging tasks that are referred to. 
To increase task performance, we refactored the environment by modifying the layout of the environment, the tractability of grasping objects in the scene and collecting longer demos:

- **Augmenting the layout of the environment** so that there is more room to grasp the objects found in the center of the scene

- **Reducing the range of the rotation of objects and the size of objects in the scene** to increase grasp success (as the simulated robot doesn’t have a rotation in gripper space)

- **Collecting for a longer horizon** to ensure that the full sorting procedure occurs for the scripted sorting policy for each episode collected.

| Task Name         | BC    | DDPM + BC | IQL   | IDQL  | CQL | TD3 + BC |
|-------------------|-------|-----------|-------|-------|-----|----------|
| Sorting           | 32.54 | 28.47     | 33.56 | 35.59 | 0   | 14.24    |
| Sorting Bin Noise | 44.75 | 34.58     | 45.76 | 49.49 | 0   | 21.36    |

We have refactored the tasks to be (1) sorting, where the agent has to sort two objects into their respective bins and (2) sorting bin noise, where the agent still has to sort the two objects but there is a probability that each object is placed into the wrong bin.
Along with this modified task, we have included the performance of a random policy on our tasks, which achieves a 0% success rate across all tasks.

Reviewer comment (Experiments):
I would like to see a random policy as a simple baseline. I wonder if the benchmarked methods even do better than a random policy.

Author response:
Random policies achieve zero success on our manipulation environments. We have added the result to the experiment table.

Reviewer comment (Evaluation):
Why is there such a big gap between IQL and other similar methods such as CQL and TD3+BC? These methods often perform very similarly in D4RL, at least in the locomotion tasks.

Author response:
The fact that value-based RL methods perform worse than behavior cloning-based methods on our benchmark is an important insight/contribution, not a disadvantage. In many real robot manipulation tasks, behavior cloning-based methods tend to outperform offline value-based-RL methods, too. While value-based RL methods have achieved SoTA performance on simpler benchmarks, such as D4RL, we hope that our benchmark can help drive the development of offline RL methods that perform well on more realistic robot manipulation tasks

Reviewer comment (Experiments):
Section 5.2 states that the goal of the Frank Kitchen environment is to study offline RL and online fine-tuning, but the following text does not mention how the proposed task facilitates online finetuning.

Author response:
As far as we are aware no current method can solve the environment fully with only the provided offline data. Because of this, additional online fine-tuning is a necessary step to be able to learn to fully solve it. We hope that our benchmark can provide a good way to evaluate methods for offline-to-online fine-tuning, where the available offline data is insufficient to fully solve a task, but can provide a good starting off point for initializing a policy for online learning. We carry online fine-tuning experiments on a number of our environments, results are shown in Table 2.

Reviewer comment (Novelty):
The benchmark does not specifically address current challenges in offline RL. While the paper broadly covers various challenges of offline RL, some of these challenges could already be observed in previous benchmarks. Evaluating temporal compositionality seems possible using previous benchmarks (D4RL Maze2D, AntMaze, FrankaKitchen-Mixed, Calvin). The proposed benchmark has less diversity than D4RL, and different levels of data distribution are also discussed in D4RL. However, the text does not explicitly claim why using this new benchmark is better than using previous benchmarks for each challenge.

Author response:
**Some of these challenges could be already observed in previous benchmarks. Evaluating temporal compositionality seems possible using previous benchmarks (D4RL Maze2D, AntMaze, FrankaKitchen-Mixed, Calvin).**

We agree that these environments test temporal compositionality to some degree. However, performance on the D4RL maze tasks is becoming saturated. While the existing Franka and Calvin environments still pose a challenge, they do not test other scenarios, such as the multimodality of the data distribution (Franka kitchen is composed entirely of demonstration data, and CALVIN is composed entirely of play data). In our “play” datasets tasks can be executed partially and in arbitrary order and the agent needs to learn how to compose these together, which proves to be quite challenging. 

**The proposed benchmark has less diversity than D4RL, and different levels of data distribution are also discussed in D4RL.**

In this proposed benchmark we focus on realistic robotics tasks. While it is true that D4RL considered different levels of data quality, it did so by using different parts of an online RL replay buffer. Instead in this work we focus on more realistic data distributions, such as sub-optimal demonstrations and play data.

Reviewer comment (Experiments):
The paper does not clarify under what circumstances this benchmark should be chosen. Questions remain such as: when to choose this benchmark? Are previous works successful on prior benchmarks likely to fail on this new benchmark specifically due to one of the specific challenges?

Author response:
**The text does not explicitly claim why using this new benchmark is better than using previous benchmarks for each challenge. To encourage using this benchmark, the paper should address questions like: when to choose this benchmark? Are previous works successful on prior benchmarks likely to fail on this new benchmark specifically due to one of specific challenges?**

We cover the difference between d5rl and other benchmarks in our Related Works section. In particular we believe it addresses the following challenges:

1. Uses models of real robot systems.
2. Uses realistic datasets, such as demonstrations with varying quality provided by haman tele-operators, play data and scripted planners.
3. Addresses realistic challenges, such as learning from high-dimensional observations and robustness to distribution shifts.
4. Compared to other works we believe it strikes a better balance addressing the above challenges and time/complexity/computational requirements.

Reviewer comment (Novelty):
Addressing offline-to-online learning adaptation appears to be incremental. As almost all previous offline RL benchmarks are based on simulation environments and rule-based reward functions, implementing online fine-tuning on top of existing benchmarks is feasible.

Author response:
**Addressing offline-to-online learning adaption appears to be incremental. As almost all previous offline RL benchmarks are based on simulation environment and rule-based reward function, implementing online fine-tuning on the top of exisiting benchmarks is feasible.**

We do not claim that our work proposes or formalizes the online fine-tuning problem. Instead we aim to demonstrate that our environments and datasets are suitable for that application.

Reviewer comment (Experiments):
More information or resources should be provided to challenge the community to solve the image observation version of Franka Kitchen. It is unclear whether the failure of existing works is attributed to the incompleteness of offline RL algorithms or the lack of image representation learning. Can this task be intuitively completed solely with the provided offline data? If not, is there an alternative data source that the community can leverage to address this challenge?

Author response:
**More information or resource should be provided to challenge the community to solve the image observation version of Franka Kitchen. Is the failure of existing works attributed to the incompleteness of offline RL algorithms or the lack of image representation learning? Can this task be intuitively completed solely with the provided offline data? If not, is there an alternative data source that the community can leverage to address this challenge?**

The kitchen environments are quite challenging due to a variety of factors. As far as we are aware at the time of submission no offline RL or online fine-tuning approach had fully solved even the fully observable low-dimensional state versions. However, the standard (non-randomized) environments are feasible to solve, since a recent model-based work provided a complete solution [1].