Paper: Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
Although there seems no issue with the technical part of the paper, I do want to bring the attention to a recent paper: Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning (https://arxiv.org/abs/2305.10282). I believe this paper, from the theory perspective, proposes the same intuition as the current paper: after running model-based offline RL on the offline dataset, one could use the model to estimate the occupancy measure of the offline policy, and thus have the knowledge of the uncovered directions from the offline data, and during online fine-tuning, one could use exploration to collect the data in the remaining directions. To me the current paper shares a lot of intuition with this earlier paper, minus many empirical considerations, which are also good contributions.

Author response:
We thank Reviewer Q4z6 for pointing us to this interesting reference. We were not aware of this work, but we have now included it in our Related Work section.

Reviewer comment (Experiments):
Other than CEB, there might be many other methods for measuring the out-of-distributioness. It would improve the paper if there are more ablations.

Author response:
In principle, representation learning methods other than the CEB can be used within PTGOOD. However, as we highlight in Section 4.2, CEB-learned representations have been shown to work well in out-of-distribution detection tasks, which is a property that is useful for PTGOOD’s purposes.

Reviewer comment (Experiments):
The experiment section seems to miss of of the benchmarks that are tested in cal-QL.

Author response:
We specifically avoid using datasets that contain demonstrations of expert or hand-crafted policies (i.e., policies that successfully complete the task in the case of binary success tasks like Kitchen or optimal policies like in HalfCheetah Expert.) We motivate this setting in our introduction and reiterate it in our Related Work section. Below, we will go through each of the datasets used in the Cal-QL paper’s main results (Figure 6), and highlight that each of these datasets contains trajectories that violate this dataset selection criterion.  

All four of the AntMaze datasets used in the Cal-QL paper are collected by expert policies. The D4RL documentation (https://github.com/Farama-Foundation/d4rl/wiki/Tasks#antmaze) describes the trajectories as being generated by “commanding random goal locations in the maze and navigating the ant to them.” Also, the D4RL paper (Fu et al., 2021) states that the AntMaze datasets are generated by first “training a goal reaching policy.”  

All three of the Kitchen datasets are filled with trajectories that successfully complete the task. The documentation (https://github.com/Farama-Foundation/d4rl/wiki/Tasks#frankakitchen) states that the datasets “includes demonstrations of all 4 target subtasks being completed” and the D4RL paper describes the Kitchen datasets as coming from “human demonstrations” that are either “performing all of the desired tasks” or “performing subtasks.”  

The Pen, Door, and Relocate datasets are also filled with trajectories that successfully complete the task. The Cal-QL paper describes these datasets with: “Each of these tasks only provides a narrow offline dataset 	consisting of 25 demonstrations collected via human teleoperation and additional trajectories collected by a BC policy.” Additionally, the D4RL paper describes these datasets with: “This domain was selected to measure the effect of a narrow expert data distributions and human demonstrations.”  

The Cal-QL authors do not provide the dataset used in the Visual Manipulation environment, but they do refer to its datasets as containing “narrow” behavior, which is verbiage often used to describe datasets collected by expert policies. These datasets appear to come from Ebert et al., (2021), who describe the datasets as being collected via teleoperation specifically for imitation learning purposes, which suggests that the trajectories within the dataset successfully complete the tasks.

Reviewer comment (Experiments):
Although the authors tried to approach the offline-to-online problem from an exploratory perspective, they did not prove their claims through extensive experiments. For example, the experiments in Table 2 are too limited and only include 6 tasks. I suggest the authors provide additional experimental results in complete D4RL tasks to verify their claims.

Author response:
**Request for more environments from D4RL**


We specifically do not use all of the environment-dataset combinations in the D4RL suite because many of them violate our selection criteria (explained in section 6.2.) Many of the environment-dataset combinations in D4RL can be solved to (near-) optimality during the offline pretraining phase. Such environment-dataset combinations are uninteresting in the OtO setting and violate our criterion 1, which is defined in Section 6.2. On the other hand, some of the environment-dataset combinations provide such poor coverage of the MDP (e.g., the Random dataset in D4RL’s Walker2d) that offline pretraining provides virtually no benefit (violating criterion 2 which is defined in Section 6.2.) As a result, we collect several of our own datasets in environments that are not represented in D4RL, such as Ant, Humanoid, and DMC Walker, for a total of 7 environment-dataset combinations. These datasets comprise a portion of our work’s contribution to the research community, and we will release them to the public in our work’s code repository.

Reviewer comment (Novelty):
In recent years, there has been rapid development in the field of offline-to-online RL, with numerous relevant works published. It is crucial for the authors to include and discuss these more recent works in the related works section, like AWAC[1], E2O[2], PROTO[3], SUNG[4] and PEX[5].

Author response:
**Request for more baselines**


We have added two of the requested baselines (PROTO and PEX) to the main results in Table 2, Figure 11, and Figure 12. Two of the other requested baselines (E2O and SUNG) do not have code provided by the authors. The final baseline (AWAC) is outperformed by the two baselines we have already added, and therefore it would likely not provide additional information. We highlight that PTGOOD, our contribution, significantly outperforms both PROTO and PEX in all of our benchmarks.

Reviewer comment (Experiments):
The author claims that traditional exploration methods do not work, such as internal rewards and UCB. What will happen if naive exploration methods are used, such as epsilon exploration?

Author response:
**Traditional exploration methods**


We display in our experiment section (e.g., Table 2) that traditional exploration methods, such as intrinsic rewards and UCB exploration, work fine, but not as well as PTGOOD. Also, we perform hyperparameter tuning for each one of these baselines, as seen in Figures 3-7. We also show that following the policy without any exploration guidance (Naive) works fine but also significantly underperforms PTGOOD.

Reviewer comment (Experiments):
In the online stage, does the author use standard online RL algorithms, such as SAC and TD3, or does he use the online version of the offline learning algorithm?

Author response:
**Algorithm used during online fine-tuning phase**


All of our baselines (except for Cal-QL, PROTO, and PEX) and PTGOOD use model-based policy optimization (MBPO) and Soft-Actor critic (SAC), as we mention in the first paragraph of Section 6.1. Combining SAC with MBPO is standard practice and works well, as shown in the original MBPO paper (Janner et al., 2019).

Reviewer comment (Writing):
The core of the proposed method is to "target online exploration in relatively high-reward regions of the state-action space unlikely to be visited by the behavior policy". However, it is unclear what is "relatively high-reward regions", e.g., what is the criterial for choosing those regions.

Author response:
We highlight in Section 6.4 that the relatively high-reward regions are found by keeping exploration “close” to the improving policy. Due to the agent’s learning process (via RL), it should visit increasingly higher-reward regions of the MDP’s state-action space naturally. As we show in the experiment in Section 6.4 and Figure 2, keeping the exploration process “close” to this improving policy is critical. This is shown by the large decrease in returns throughout training when the planning noise is too high.

Reviewer comment (Evaluation):
What is the explanation of the low variance in Figure 1 when choose small $\lambda$.

Author response:
Could Reviewer Ftkq please clarify their question? In Figure 1, sometimes the variance across seeds is small when lambda is small, but sometimes the variance across seeds is large when lambda is small. The same is also true for relatively large values of lambda. So we are unsure what the reviewer is referring to specifically.