Paper: Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators (ICLR-2024)

Reviewer comment (Novelty):
The main problem to me is that the work proposed is too incremental for a main track paper at ICLR. While there is value in the identification of the main challenges faced by the RL agent in the offline data+simulator scenario, it is a very incremental step from already-existing datasets/works, and procedurally, generating those benchmarks are relatively easy given Mujoco is a simple to execute and easily-available environment. For instance, someone interested in generating a similar benchmark might not even have to run the environment too many times by leveraging already-existing datasets. Starting from (Fu et al, 2020) datasets, one could modify the datasets by simply (i) changing an state variable; (ii) omitting a recorded state variable (iii) adding noise to the recorded action; (iv) adding noise to observation and action, to replicate a similar dataset generation as the one done by the authors, maybe 2 or 3 of the generated configurations only would require running all the experiment again (and even those, represent a small change in the Mujoco environment).

Author response:
While some of our benchmarks might seem easy to implement, we believe that our contribution does not stop at the technical implementations of these benchmarks, but in the formulation of the benchmark as a whole and in identifying the distinct challenges associated with hybrid modeling. To this day, there is no standardized way to benchmark hybrid-RL algorithms, and indeed we believe we are the first to even outline the issues at play. We hope that our benchmark could provide researchers a starting point when crafting new hybrid algorithms, and inspire novel research in that area.

Reviewer comment (Experiments):
A lot of work can be done to improve clarity of the work as well. It's not very clear how the datasets replicate the "structure" in Figure 1. The offline data and simulator construction is explained, but how is the agent performance validated? Is there a ground truth environment that is not modified from which performance measures are extracted?

Author response:
We are sorry for the confusion. In all experiments, the ground truth environment is the unbiased simulator. For example, in the modeling error experiment, the training agents have access to the biased simulator (e.g., with erroneous gravitational parameter) and some form of offline data. The offline data was collected from the ground truth environment but suffers from one of the biases of offline data. To test the algorithm, we evaluate it’s capabilities on the unbiased simulator (e.g., with the correct gravitational parameter). We will clarify this point in the text.

Reviewer comment (Novelty):
I was surprised the authors didn't mention the very related multi-fidelity MDP formulation. The formulation models exactly one of the problems the authors are coping with, imprecise simulators. Silva, F. L., Yang, J., Landajuela, M., Goncalves, A., Ladd, A., Faissol, D., & Petersen, B. (2023). Toward Multi-Fidelity Reinforcement Learning for Symbolic Optimization. Adaptive and Learning Agents (ALA) Workshop.

Author response:
Thank you for the helpful feedback. We will add this relevant paper to our revised version.

Reviewer comment (Evaluation):
How is the agent performance validated? Is there a ground truth environment that is not modified from which performance measures are extracted?

Author response:
We are sorry for the confusion. In all experiments, the ground truth environment is the unbiased simulator. For example, in the modeling error experiment, the training agents have access to the biased simulator (e.g., with erroneous gravitational parameter) and some form of offline data. The offline data was collected from the ground truth environment but suffers from one of the biases of offline data. To test the algorithm, we evaluate it’s capabilities on the unbiased simulator (e.g., with the correct gravitational parameter). We will clarify this point in the text.

Reviewer comment (Experiments):
The experiments are also very limited on HalfCheetah and there is no convincing justification for why this paper would be a good benchmarking paper that the RL community can benefit from.

Author response:
Regarding limitations of experiments. We agree that having more experiments could be beneficial to our benchmarks, and would add more with time.

Reviewer comment (Experiments):
For some environment modifications, the authors mention that "the choice of values was done based on the results of the SAC algorithm". Could this introduce a certain bias if solely based on one algorithm?

Author response:
Regarding SAC algorithm:
The calibration of our algorithms was indeed based on the SAC algorithm, which as you point out can introduce some bias to the noise levels chosen. However, as seen in the experiments section, the chosen noise levels affect other baselines similarly to SAC.

Reviewer comment (Theory):
**Lacks the theoretical analysis of the four challenges.** The paper describes the intuition of the four challenges in using a simulator and offline data, however, its influence on the performance is unclear. For example, providing the upper bound of the performance decrease caused by the modeling error of the simulator and that caused by the bias of logged data can be informative for the readers. The manuscript is ambiguous about how the simulator and offline logged data compensate for errors with each other.

Author response:
We agree that a theoretical analysis could make a valuable addition to understanding this problem. However, we believe that as a benchmark paper, developing novel theoretical bounds is beyond the scope of this specific work. We hope that our outlining of the challenges of the hybrid setting will spur future theoretical analyses.

Reviewer comment (Novelty):
**Lacks the discussion of related work.** From the manuscript, I could not figure out if the proposed method is the only approach trying to combine the use of a simulator and offline data due to the lack of discussion of existing approaches. In addition, the advantage of the proposed method over model-based offline RL approaches was not discussed. For example, is the proposed method more effective than a representative model-based offline RL method proposed (e.g., COMBO (Yu et al., 21)) and why the proposed method performs better?

Author response:
As we mention in section 4.1, to the best of our knowledge there is currently only a single hybrid algorithm (except the one we proposed), which is named H2O. Regarding HyCORL vs. MOPO, as we write in section 4.1 and in appendix B (which can be found in the supplementary material), the only difference between HyCORL and MOPO is that HyCORL can utilize a given simulator. It does so by training the transition function to fix the simulator's outputs instead of learning the transition entirely from data. We will clarify this point in the main text.

Reviewer comment (Evaluation):
What kind of errors do the simulator and offline data compensate for each other?

Author response:
There are two potential sources of error compensation: The simulator is not affected by confounding errors and can be used extensively for exploration. The offline data was gathered in the real world so it does not suffer from errors in the model (transition errors, action errors etc.).
Many current research efforts are made to either build an online algorithm that can overcome simulator errors, or to build an offline algorithm that is less affected by data biases. A hybrid algorithm should be able to balance between the disadvantages of both sources of information. For example, HyCORL was able to use D4RL offline data to fix an erroneous simulator (which has wrong parameters for example), and learn a very reliable transition function, which was then used to achieve better rewards.

Reviewer comment (Novelty):
What are the advantages of the proposed method over existing model-based offline RL methods?

Author response:
As mentioned above, the only advantage HyCORL has over MOPO is that it can utilize an online simulator, even if it is biased. In case the simulator simulates the environment perfectly, HyCORL could learn to use the simulator's predictions and not use data at all. When the simulator is highly noisy or biased, HyCORL should collapse to produce results such as MOPO (as mentioned in section 4.1 – HyCORL).