Paper: RAVL: Reach-Aware Value Learning for the Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
The authors first reveal the key overestimation issue of values on edge-of-reach states in offline MBRL. However, as a direct combination of EDAC and MOPO, the proposed method RAVL is largely independent of the revealed issue. In particular, the only difference between RAVL and MOPO is that RAVL exhibits the Q-update rule in EDAC. However, such a Q-update rule is performed on all states in the buffer and is not tailored for edge-of-reach states. RAVL does not identify edge-of-reach states and correct the corresponding values to address the claimed issue.

Author response:
**(Weakness 1) Does RAVL actually target edge-of-reach states?**

This is a good question and gets to the crux of our method. Please see part A of our general response where we have included a detailed step-by-step walk-through of the intuition of why we can expect our RAVL approach to detect and penalize edge-of-reach states. We would be more than happy to provide further explanation, so please let us know if it is still unclear.

---

Dear Reviewer 34p6,

The OOD (high uncertainty) states that RAVL detects are **with respect to $D_{\text{rollouts}}$ and not $D_{\text{offline}}$**. Therefore, the scenario the reviewer describes could not happen and RAVL specifically targets edge of reach states which are by definition OOD with respect to $s\in D_{\text{rollouts}}$. This allows us to use existing Q-uncertainty methods but **with respect to the model rollout distribution and not the original offline data distribution**, and is the key novelty of our method.

Please let us know if we can help clarify further.

Best,
The authors.

Reviewer comment (Evaluation):
The empirical performance of RAVL is not strong. In D4RL, the existing method MOBILE beats RAVL regarding the number of best-performance tasks, implying that the improvement of RAVL is limited.

Author response:
**(Weakness 3) Empirical performance of RAVL**

Thank you for raising this comment, however, we respectfully disagree with the assessment that the performance of RAVL is not strong, as it is state-of-the-art in 5/9 of the environments we evaluated on. In addition, we are pleased to be able to report that RAVL gives significant performance boosts on the more complex V-D4RL benchmark *(please see part B of our general response)*. We also note that SOTA scores on the D4RL benchmark are likely saturated, already being significantly over 100 (normalized expert level) on many of the environments. As such, future work is unlikely to be able to demonstrate substantial improvements on this benchmark.

More importantly, however, we have demonstrated that MOBILE (the only algorithm that beats RAVL), along with many other prominent offline model-based algorithms, is derived based on significantly incomplete theoretical foundations (and we explain why they work despite this). It is the new insight, and the resulting much more complete understanding of offline RL, that we view as the main exciting new contribution of our work. We anticipate this should be hugely valuable in the future development of robust, strong-performing offline RL methods, potentially also resulting in new improved ways of tackling the critical, previously overlooked edge-of-reach problem.

Reviewer comment (Theory):
In Eq.(3), $+\gamma$ should be $-\gamma$.

Author response:
**(Question 1) Typo in Equation 3**

Thank you for spotting the typo in Equation 3. We have now fixed this in the updated paper.

Reviewer comment (Novelty):
The problem addressed in this paper may not be influential.

Author response:
**(Weakness 1) Influence of the paper**

Thank you for this question. We strongly believe *(like reviewer y8bf)* that the analysis in our paper has the potential to have a substantial impact on future work in this area.

The main contribution of our paper is to expose and explain a significant gap in the current understanding of the issues underlying offline model-based RL. This current incomplete *“dynamics model error”*-based view forms the basis for many current offline model-based algorithms, such as MOPO [1], MOReL [2], and MOBILE [3]. Combined, these works alone have a total of over 1000 citations and have influenced a whole sub-field of offline model-based RL. Our paper demonstrates a significant misunderstanding in these works, provides and verifies an explanation (the edge-of-reach problem), as well as providing a solution (RAVL). We believe the resulting much more complete picture of offline model-based RL should be hugely valuable in the development of more accurately motivated and stronger-performing offline algorithms.

Furthermore, our detailed experiments show the edge-of-reach problem is not just an issue in theory, but that it is actually very prevalent in practice (on the simple environment, on the standard offline D4RL benchmark (see Table 1 and Figure 5), and even in latent space models *(see our new results in part B of our general response*).

Finally, as discussed in Section 6 and Appendix A, our work provides a unified view of model-based and model-free approaches, which we hope will enable more ideas from model-free literature to be directly applied to model-based algorithms (as we have done with RAVL), and we anticipate that our work may inspire further future connections to be drawn between the two subfields of offline RL.

Reviewer comment (Experiments):
The benchmark environment used in this research is relatively simplistic. It would be interesting to investigate whether or not the proposed method performs satisfactorily in more complex environments.

Author response:
**(Weakness 2) Evaluation on harder environments**

Thank you for raising this concern. We are pleased to present additional state-of-the-art results on the more challenging pixel-based V-D4RL benchmark *(please see part B of our general response)*.

Reviewer comment (Experiments):
The modifications made to the experiments in this paper do not appear to be particularly substantial.

Author response:
**(Weakness 3) Simplicity of our algorithm**

Thank you for noting that the approach in RAVL is quite simple. We agree, and we see this as a strength. Our aim was to make as few modifications to the base algorithm as possible, and our unified view of model-based and model-free approaches (introduced in Section 6 and Appendix A) enables us to directly take a robust out-of-the-box solution from model-free literature and apply it to solve the edge-of-reach problem. We found this worked extremely well and found no need to add anything to complicate it.

Reviewer comment (Presentation):
I do not fully understand the Figure 3. How is the figure generated? Why is the conclusion drawn: "As desired, RAVL is effective at capturing the value uncertainty for state-actions which transition to edge-of-reach nextstates."?

Author response:
**(Question 1) How was Figure 3 generated?**

Thank you for raising this question. It has brought to our attention that the caption and labels in Figure 3 were unnecessarily unclear and we have edited them in light of your comment. In short: our construction of the simple environment allows us to exactly define whether a state is edge-of-reach or not (checking whether it is within fixed bounds). We are therefore able to compare the Q-ensemble variance for edge-of-reach vs. not edge-of-reach states. Figure 3 shows a histogram of the Q-ensemble variance (RAVL’s effective penalty) for all the state-action pairs evaluated over training, separated into whether the state is edge-of-reach (orange) or not edge-of-reach (ie. within-reach, blue). It shows that, as we hoped, the Q-ensemble variance (RAVL’s effective penalty) is significantly higher on states which are edge-of-reach, thus verifying that our RAVL method is effective at detecting and penalizing edge-of-reach states. (In part A of our general response we reiterate the intuition on how RAVL is able to do this.)

Reviewer comment (Experiments):
The D4RL evaluation includes only the basic MuJoCo tasks. The authors could try a more complex environment such as AntMaze or Adroit to further demonstrate the strength of the algorithm over previous methods.

Author response:
**(Weakness 1) Evaluation on harder environments**

Thank you for raising this concern. We have run RAVL on the more challenging pixel-based V-D4RL benchmark and are pleased to report new state-of-the art results *(please see Part B of our general response)*.

Reviewer comment (Experiments):
While RAVL seems to be effective at addressing the edge-of-reach problem, it may require a large ensemble (e.g. 50 models) to achieve sufficient pessimism, which could be computationally expensive.

Author response:
**(Question 1, part 1) Can the number of critics be reduced?**

In our original submission, we stated that RAVL required N=50 critics to achieve sufficient edge-of-reach awareness on the hopper environments (while N=10 was enough for all the other environments). We are pleased to report that we have run additional experiments and have found that N=10 is mostly sufficient for hopper as well, with only hopper-medexp needing N=30, meaning that all but one environment uses N=10.

| **Environment** |        | **RAVL (N=50, in the paper)** |  **RAVL (Updated)**  |
|:---------------:|:------:|:-----------------------------:|:--------------------:|
| hopper-v2       | mixed  | 103.1±1.0                     | 102.6±1.1 **(N=10)** |
| hopper-v2       | med    | 88.1±15.4                     | 90.6±11.9 **(N=10)** |
| hopper-v2       | medexp | 110.1±1.8                     | 107.0±5.4 **(N=30)** |

Reviewer comment (Experiments):
I was curious if you considered/experimented with adding a small explicit penalty to the values of states at the final step of the rollouts?

Author response:
**(Question 1, part 2) Did you try adding an explicit penalty to states at the end of rollouts as an alternative approach to the edge-of-reach problem?**

Yes, this was an idea we had initially as well! We considered this for quite some time, but from our experiments it did not seem to work well. Our intuition on why this may not work is: Final step rollout states may not actually be edge-of-reach and hence may also appear earlier in a different rollout. This means the target for the state will sometimes have a penalty and sometimes have no penalty. From our investigations, it seems that the resulting contradictory target function or reward signal leads to significant optimization issues. As such there is an extremely delicate trade-off between balancing the penalty to be (1) large enough to address the edge-of-reach issue, but (2) small enough to not cause optimization issues. This highlights the challenge of tackling the edge-of-reach problem! We found our RAVL approach to work much better and more robustly and avoid this optimization difficulty and delicate balancing act.

Reviewer comment (Writing):
This paper use two similar expressions, 'out-of-sample' and 'out-of-distribution', are these two expressions the same or different?

Author response:
We refer the reviewer to Footnote 1 on page 3 of the main paper where we clarify that we use “out-of-sample” and “out-of-distribution” to mean the same thing. We chose to use both terms to highlight connections to past work. IQL [1], for example, contains a very clear explanation of the model-free version of these issues in which they use the term “out-of-sample”, while many other papers (and wider literature in general) use the term “out-of-distribution”.

Reviewer comment (Experiments):
It's confusing that, Table.1 aims to illustrate that it is the edge-of-reach problem that causes the failure for model-based offline RL methods, while the simple experiment in Figure.2 don't include any model-based methods (SAC is a typical model-free methods).

Author response:
We believe there may be a misunderstanding of the meaning of “SAC” in Figure 2. This figure is designed to illustrate that the MBPO procedure [4] (which is the base algorithm in most prominent offline model-based methods) causes the edge-of-reach problem. The MPBO procedure can be described as:

1. generate a dataset of truncated rollouts according to the agent’s policy, and then

2. train the agent (most papers use SAC) on this dataset of rollouts ($\mathcal{D}_{\text{rollouts}}$), and then repeat.

We therefore simply use “SAC” to refer to the base MBPO+SAC procedure (used in MOPO, MOReL, MOBILE…) without our RAVL modification. Please see Table 8 for a summary of the setups. Your question has also prompted us to add Algorithm 1, so thank you for this valuable feedback.