Paper: Guided Decoupled Exploration for Offline Reinforcement Learning Fine-tuning (ICLR-2024)

Reviewer comment (Novelty):
Introductory analysis and experiments are helpful for understanding and motivating the method but they are mostly not new as authors also already mentioned in the paper. They are mostly already covered in works like [Fujimoto'18; Lee'22; Luo'23]

Author response:
Thank you for the suggestion. Indeed, we acknowledge that certain aspects discussed in Section 3 have been covered in prior works. In Section 3, we intended to summarize and revisit some practical challenges in offline RL fine-tuning. In our view, this introductory analysis contributes to presenting a more comprehensive background and enhances the overall coherence of the paper.

Contrasting our work with prior research, notable distinctions include:

- [Fujimoto '18] delved into the bootstrapping error issue within offline RL training.
- [Lee '22] focused on addressing the bootstrapping error during the online fine-tuning stage.
- [Luo '23] explored the policy crash issue during the online fine-tuning stage.

In this work, we synthesized these distinct elements and introduced a straightforward algorithm to tackle these challenges. Unlike [Fujimoto '18, Lee '22, Luo '23], which proposed specific algorithms in their respective papers, we presented a general framework. The experiments illustrate its effectiveness in collaboration with various backbone offline RL agents.

Reviewer comment (Evaluation):
The proposed method needs evaluation rollouts for updating the teacher policy. Then the number of environment interactions required for this evaluation should also be incorporated into the sample count for the proposed method. It's not clear if this is reflected in the current results, and this should be properly computed if they are not because it's not a fair evaluation. Correctly doing this would also make Figure 6(b) analysis more meaningful because it's an important trade-off.

Author response:
The evaluation trials were not considered in the budget for online fine-tuning. Consequently, we made an error by utilizing additional online samples for policy selection.

As discussed in the common question section, we have rectified this issue. The overall results align closely with the previous findings. The results in Table 2, Table 3, Table 4, and Figure 6 are updated accordingly.

Reviewer comment (Presentation):
The paper is a bit difficult to parse in some parts. There is a room for improving the readability. For instance, Table 4 is difficult to read because it's missing the results of GDE with all the components. Including this could help improving the readability by making not scroll the paper up and down. Figure 5 is not very helpful for understanding the main method and could be improved to intuitively help the readers to understand the main idea. Augmenting the Algorithm 1 to be more self-contained could be helpful for better readability.

Author response:
Appreciate the suggestions. We have revised Table 4, Figure 5, and Algorithm 1 to enhance readability.

Reviewer comment (Novelty):
Could you clarify what's the unique observation that could be further emphasized in the paper?

Author response:
We acknowledge that some observations in our paper overlap with prior works. However, we believe certain unique observations can be highlighted:

- The optimal way to leverage offline samples during fine-tuning remains an open question. In some cases, dropping the offline buffer proves to be a surprisingly strong baseline.
- It might be helpful to use varying degrees of conservatism in exploration and training, as demonstrated by the decoupling method.

Reviewer comment (Experiments):
Is the number of samples required for evaluating the policy is incorporated for counting the samples? It's important for a fair comparison.

Author response:
The evaluation trials were not considered in the budget for online fine-tuning. Consequently, we made an error by utilizing additional online samples for policy selection.

As discussed in the common question section, we have rectified this issue. The overall results align closely with the previous findings. The results in Table 2, Table 3, Table 4, and Figure 6 are updated accordingly.

Reviewer comment (Experiments):
Is n-step used for all the methods including both the proposed method and the baselines? It's not a new component proposed in the method so it needs to be included for a fair evaluation.

Author response:
Thank you for the reminder. In our previous experiments, we didn't utilize the n-step return for the baselines, and we acknowledge that this could lead to an unfair comparison. To address this concern, we have compared GDE to the two best-performing baselines using n-step returns. We can observe that GDE continues to outperform QDagger and PEX.

|             |  QDagger   |    PEX     |    GDE     |
| :---------: | :--------: | :--------: | :--------: |
|  antmaze-u  | 96.6 (1.0) | 97.4 (1.0) | 97.6 (1.0) |
| antmaze-u-d | 85.4 (2.9) | 90.2 (1.6) | 90.2 (2.5) |
| antmaze-m-p | 90.6 (2.4) | 92.0 (2.3) | 94.8 (2.1) |
| antmaze-m-d | 91.4 (2.1) | 94.8 (1.7) | 94.4 (2.1) |
| antmaze-l-p | 73.0 (3.6) | 71.6 (2.9) | 73.2 (6.7) |
| antmaze-l-d | 67.8 (4.9) | 69.6 (4.0) | 80.4 (2.3) |
|    Total    |   504.8    |   515.6    |   530.6    |

Reviewer comment (Presentation):
Improving some parts of Table 4, Figure 4 as in Weaknesses could be useful for improving the readability of the paper.

Author response:
Thank you for the suggestions. We have revised Table 4 to highlight the worst-case performance.

Reviewer comment (Experiments):
In Section 5.5, it's not clear how would the method that ablates a component work. For instance, how would the method work without the exploitation policy? Then what is the main policy you are updating with? Such things are not clear so it's difficult to understand what's going on in the analysis.

Author response:
Thank you for the suggestions. We have revised Section 5.5 for clarity.

In the ablation study on not using an exploitation policy, we experimented with two policies, utilizing the checkpoint of the exploration policy as the teacher policy. The exploration policy is updated according to Eqn (5).

Reviewer comment (Evaluation):
Please consider only making the numbers be bold when they are statistically significant, i.e., when errors do not overlap

Author response:
Thank you for the suggestion. We have revised the tables in the paper accordingly.

Reviewer comment (Novelty):
There is uncertainty about the generality of the framework, while the paper claims to be general. For example, if the actor loss for the base algorithm already uses a TD3 update, it is not obvious that the decoupled exploration does anything different.

Author response:
In stating, 'It is notable that the proposed GDE is a general method that can be combined with different offline RL algorithms,' our intention was to convey that GDE can be employed to fine-tune various offline RL agents.

Unlike certain other approaches, such as CAL-QL, which is a specific offline RL fine-tuning algorithm limited to a single backbone offline RL algorithm, GDE is a framework indifferent to the choice of backbone algorithms.

Indeed, our method aligns with TD3+BC in using the same TD3 actor loss. The distinction lies in GDE introducing adaptive conservatism for the exploration and exploitation policy. Here, we encourage the exploration policy to take dissimilar actions while maintaining that the exploitation policy remains more conservative to avoid the policy crash issue.

Reviewer comment (Experiments):
The main benefit of GDE likely comes from the ability to allow the behavior policy to be updated using TD3, which cannot be done with IQL naively, as Q-values aren’t trained on OOD actions in IQL. The policy crash at low levels of $\rho$ is likely because of using TD3 loss with an IQL trained Q-value function. This makes the comparison with [2] quite critical. They report fairly sample efficient results, but beyond empirical gains, it is suspected that most of the benefit in GDE is derived from the fact that using a TD3 update for exploration policy improves the policy much faster than AWR update and thus generates better exploration data. CalQL shows that calibrated Q-value functions can allow direct usage of TD3 style updates for policy improvement, without the whole decoupled framework.

Author response:
Excellent question. We concur with the reviewer that the primary advantages of GDE stem from improved exploration data. As highlighted in the concluding section of the introduction, 'We then focus on addressing the exploration issue to achieve sample-efficient fine-tuning,' our emphasis is on mitigating the inefficient exploration problem arising from an overly conservative policy in offline RL fine-tuning.

In this paper, utilizing a TD3-like update for the exploration policy serves as a straightforward method to validate our assumptions. We acknowledge that the TD3-like update is a common factor contributing to the effectiveness of both GDE and CAL-QL.

We introduced the decoupled teacher policy to prevent policy crash issues, whereas CAL-QL addresses this matter through a modified CQL critic loss. GDE and CAL-QL aim to resolve this issue using different methods.

The key distinction lies in our decoupled framework's ease of integration with other offline RL algorithms.

Reviewer comment (Experiments):
The phenomenon in 3.2 should be established in more environments, with algorithms beyond IQL.

Author response:
Thank you for the feedback. We have incorporated the results of fine-tuning a TD3+BC agent with and without the offline buffer into Figure 3(a), and the outcomes align with those of IQL and SAC.

As mentioned in our response to Q1 for R1, we have conducted similar experiments where we trained a SAC agent from scratch using a preload buffer. Our observations indicate that omitting the offline replay buffer proves beneficial when the quality of offline data is low, such as in the case of a random dataset.

To further investigate this phenomenon across multiple environments, we fine-tuned a TD3+BC agent and a SAC agent using the pretrained checkpoint of a TD3+BC and a CQL offline RL agent, respectively. The average evaluation scores at the end of 2.5e5 steps are reported below:

|                  | TD3+BC without buffer | TD3+BC with buffer | SAC without buffer | SAC with buffer |
| :--------------: | :-------------------: | :----------------: | ------------------ | --------------- |
|  halfcheetah-r   |      49.1 (2.3)       |     35.5 (1.6)     | 80.7 (2.4)         | 68.7 (1.6)      |
|     hopper-r     |      17.1 (3.4)       |     9.9 (0.3)      | 102.4 (10.8)       | 43.8 (21.8)     |
|    walker2d-r    |       7.0 (2.6)       |     5.1 (5.7)      | 39.1 (14.6)        | 15.9 (6.8)      |
|  halfcheetah-m   |      71.0 (1.5)       |     56.6 (0.5)     | 85.6 (1.8)         | 66.6 (0.9)      |
|     hopper-m     |      102.3 (0.3)      |     60.1 (3.2)     | 100.4 (12.9)       | 105.7 (0.2)     |
|    walker2d-m    |      70.7 (40.5)      |     86.0 (0.7)     | 72.1 (24.3)        | 97.6 (0.3)      |
| halfcheetah-m-re |      64.4 (3.0)       |     51.7 (0.4)     | 78.9 (7.1)         | 71.3 (0.3)      |
|   hopper-m-re    |      103.9 (1.2)      |    79.2 (29.7)     | 108.2 (0.9)        | 102.8 (3.8)     |
|  walker2d-m-re   |      100.4 (3.5)      |     91.1 (0.3)     | 109.0 (7.3)        | 99.0 (1.3)      |
|      Total       |         585.9         |       475.2        | 776                | 671.4           |

We note that fine-tuning a TD3+BC or SAC agent without the replay buffer tends to yield strong performance on locomotion tasks. However, these methods often struggle in antmaze tasks, resulting in 0 rewards, aligning with the observations discussed in the response to Q1 for R1.

Reviewer comment (Experiments):
Clarifying the writing in Section 4 is needed, and it should be confirmed that the evaluation rollouts for exploitation policy are duly counted in the fine-tuning budget.

Author response:
We mainly followed the experiment setups from some prior work, evaluating the agent in the locomotion and antmaze tasks for 10 and 100 trajectories, respectively.

The evaluation trials were not counted in the budget for online fine-tuning. Therefore, we made a mistake by using extra online samples to select the policy. We have corrected this issue, as discussed in the common question section. The overall results are similar to the previous ones.

We evaluate every 2500 steps for the locomotion tasks and every 25000 steps for the antmaze tasks. This is mainly because we followed some prior work to plot training curves with 100 points and 10 points for the locomotion and antmaze tasks, respectively.

Reviewer comment (Experiments):
The analysis of challenges can be made more in-depth. The current experiments are more like proof-of-concept and the results can be expected.

Author response:
Thank you for the valuable suggestion. We acknowledge that our focus has been primarily on illustrating the challenges in current offline RL fine-tuning. We provided analysis on these challenges in Appendix B.5.

Given the constraints of space, delving deeply into the three challenges might be challenging. Our approach in this work has been to analyze key observations and demonstrate the efficacy of the proposed method. We are open to the prospect of conducting a more in-depth theoretical analysis of these challenges in future work.

Reviewer comment (Experiments):
GDE maintains 3 policies, rendering extra computation and memory costs. Although the authors emphasize the minimalist algorithm design, I feel that the current design is not necessarily the most efficient. For example, can the exploration policy directly be a function of the exploitation policy (one can just adjust the output action distribution by exploration objectives, without training an extra policy.)

Author response:
Yes, GDE indeed maintains three policies: a teacher policy, an exploration policy, and an exploitation policy. It's important to note that the teacher policy is essentially a checkpoint of the previous exploitation policy. Consequently, during training, we only need to train two policies. Moreover, the exploration policy is implemented as a simple 2-layer MLP, which means the additional computation overhead is relatively light.

Reviewer comment (Experiments):
There are a lot of exploration approaches, is there a reason of selecting Eq (5) as the loss for the exploration policy? Will other methods work here, such as curiosity-driven ones?

Author response:
One of the main reasons for using Equation 5 lies in its simplicity. Beginning with a minimalist perspective in designing our algorithm, we opted to add a single regularization term to the canonical actor loss in the TD3 and SAC.

Our primary goal was to demonstrate that by introducing this new actor loss, we could address the issue of inefficient exploration while staying relatively close to the behavior policy. We believe that other actor losses with a similar concept could yield positive results as well, and we chose one of the simplest forms to validate our assumptions. For instance, incorporating a Random Network Distillation (RND) module to augment the task reward and encourage exploration could be beneficial in certain tasks, as illustrated in Table 1.

However, such methods typically entail learning an additional model and introduce additional computational complexity. Given the effective performance of our current simple design in experiments, we opted to retain it.

Reviewer comment (Experiments):
Why is the performance of train from scratch with SAC so low? Given enough samples, shouldn't SAC be able to learn a good policy in many of these tasks?

Author response:
The seemingly low performance of 'Fromscratch SAC' can be attributed to two primary reasons:

- **Limited Interaction Steps:** Our focus in this study was on the challenge of sample-efficient offline RL fine-tuning. Consequently, we constrained the agent to interact with the environment for only 2.5e5 environmental steps in the experiments. The suboptimal performance of Fromscratch SAC suggests that the randomly initialized policy is less efficient in collecting useful samples during the early stages of learning.
- **Normalized Evaluation Scores:** The reported scores are normalized evaluation scores, and as a result, they may appear low. If we were to run Fromscratch SAC for 1e6 environmental steps, it would likely achieve scores near 100 in the locomotion tasks.