Paper: Offline RL for Online RL: Decoupled Policy Learning for Mitigating Exploration Bias (ICLR-2024)

Reviewer comment (Novelty):
The Offline-to-Online-to-Offline (OOO) paradigm can be divided into two parts: offline-to-online and online-to-offline. The former is a well-established concept with numerous dedicated algorithms, lacking in novelty.

Author response:
Thanks for raising this point and providing the two ways to view our work. Our work is better viewed as a framework for both online and the well established offline-to-online RL problem. Our primary contribution is addressing the exploration bias introduced by the use of exploration bonuses. As our experiments show, directly using state of the art offline-to-online algorithms (Tables 1-2) and online-only algorithms that use prior data (Table 3) are not able to explore efficiently without exploration bonuses. Naive, yet standard, use of exploration bonuses is not able to achieve optimal performance as the policy is incentivized to continue exploring (Figures 10 and 12). This motivated the construction of OOO as a framework, where a policy extraction step at the end of online fine-tuning using offline RL can counteract the exploration bias, and thus meaningfully improve the final performance.

We have revised the paper to emphasize this novel aspect of OOO, i.e. counteracting the exploration bias using offline retraining, and thus allowing more effective use of exploration bonuses in online fine-tuning and online RL.

Reviewer comment (Novelty):
The online-to-offline segment essentially resembles standard offline reinforcement learning, as D4RL datasets are drawn from replay buffers of policies trained by online algorithms.

Author response:
Thanks for raising this important point. Indeed offline datasets in popular benchmarks are created using online RL, and one may see the online-to-offline part of our framework as a generalization of this. However, note that none of the prior offline-to-online RL methods utilize the offline retraining subroutine. Moreover, the primary benefit of OOO is to mitigate the exploration bias. To investigate this further, we fine-tune using IQL on the D4RL locomotion environments, and after the fine-tuning budget for online steps is completed, we train an offline RL policy on all the data collected offline and online (i.e. OOO but no exploration bonuses). We observe the following results:

| Domain     | Task                         | IQL       | OOO (IQL) |
|------------|------------------------------|-----------|-----------|
| locomotion | halfcheetah-random-v2        | 56 (1.2)  | 59 (1.8)  |
|            | halfcheetah-medium-v2        | 57 (0.3)  | 59 (0.5)  |
|            | halfcheetah-medium-expert-v2 | 95 (0.1)  | 95 (0.1)  |
|            | hopper-random-v2             | 37 (17.5) | 37 (17)   |
|            | hopper-medium-v2             | 83 (7.1)  | 89 (10.1) |
|            | hopper-medium-expert-v2      | 90 (9.3)  | 85 (11.8) |
|            | walker2d-random-v2           | 13 (3.7)  | 17 (4)    |
|            | walker2d-medium-v2           | 86 (4.6)  | 80 (4.1)  |
|            | walker2d-medium-expert-v2    | 111 (3)   | 116 (0.9) |
|	 | maze2d-missing-data-large-v1 | 127 (50.7) | 130 (60.4) |
| Average    | -                            | 75.5      | 76.7      |

This evaluation suggests that offline retraining provides limited improvements over the final fine-tuned policy, further suggesting that the improved performance in our main experiments arises from counteracting the exploration bias. We have revised the paper to include these results in Table 5 in Appendix C, which reinforces our point that our proposed decoupled scheme specifically mitigates the exploration bias when using exploration bonuses in fine-tuning.

Reviewer comment (Novelty):
The practical implementation of OOO combines existing off-the-shelf algorithms, specifically IQL/Cal-QL for the offline phase and RLPD and RND for the online phase. These algorithms themselves are not new and may not significantly contribute to the field. The design of the algorithms for the interplay is still not very new, more like some tricks without theoretical support or in-depth investigation.

Author response:
We chose to present OOO as a framework, because it can be combined with a variety of algorithms (IQL, CalQL, RLPD) and exploration bonuses (RND, count-bonuses). We are happy to revise the presentation if you have suggestions for alternatives to framework.

Please note that the methods we have compared to are state-of-the-art methods for the respective problems (IQL, PEX, CalQL for offline-to-online fine-tuning, and RLPD for online RL). Moreover, very limited prior work has used exploration bonuses for online fine-tuning. Can you clarify what alternative approach you are referring to?

We agree that offline-to-online is a well studied problem, but none of the prior works (a) use an offline retraining step as a part of the algorithm and (b) demonstrate improved performance from the offline retraining of the exploitation policy.

We want to clarify, the first parts of the OOO method (offline pre-training and online fine-tuning with exploration bonus) **do not** surpass the SOTA algorithms. In fact, without the third and final step of OOO, it simply reduces to the base algorithm + exploration bonus, which sometimes can perform worse than the base algorithm alone (e.g. Figure 12 top left, IQL + RND performs worse than IQL for relocate-binary-v0). It is the third step of offline extraction which enables us to surpass the performance of prior SOTA algorithms. All the reported results for OOO are for the policy produced by the final exploitation policy (i.e. after the third stage).

We are unaware of prior papers that show that combine exploration bonuses with online fine-tuning, and show that performance of both offline-to-online and online RL can be improved by re-training with offline RL on the resulting buffer.

Reviewer comment (Experiments):
In Figure 2d, the authors solely compare to the IQL+Bonus case. To ensure a fair comparison, it is advisable to include a popular online algorithm as a baseline.

Author response:
Figure 2 presents a didactic example for the offline-to-online setting where the OOO framework is exemplified on top of the IQL algorithm. For our main results on standard and harder benchmarks (Tables 1-3), multiple popular online and offline-to-online algorithms are considered as baselines.

Reviewer comment (Experiments):
The comparison involving TD3+RND in an Offline-to-Online setting may not be suitable. A more reliable setting would entail loading both offline data and the value function from offline pre-training and then train TD3+RND.

Author response:
Thank you for the valuable suggestion. We present below results for TD3+RND initialized from a pre-trained IQL agent. We compare it with TD3+RND trained from scratch. Results are largely similar, with reduced performance for pen and hammer environments.

| Domain             | Task                          | TD3+RND (from scratch) | TD3+RND (pre-trained with IQL) |
|--------------------|-------------------------------|------------------------|--------------------------------|
| Binary             | relocate-binary-v0            | 1 (0.8)                | 0 (0)                          |
|                    | door-binary-v0                | 2 (1.5)                | 0 (0)                          |
|                    | pen-binary-v0                 | 59 (18)                | 33 (9.2)                       |
| FrankaKitchen      | kitchen-partial-v0            | 0 (0)                  | 3 (3)                          |
|                    | kitchen-mixed-v0              | 0 (0)                  | 0 (0)                          |
|                    | kitchen-complete-v0           | 0 (0.3)                | 3 (1.8)                        |
| Harder exploration | antmaze-goal-missing-large-v2 | 0 (0)                  | 0 (0)                          |
|                    | maze2d-missing-data-large-v1  | 233 (5.8)              | 231 (6.6)                      |
|                    | hammer-truncated-expert-v1    | 13 (5.3)               | 3 (0.2)                        |
| -                  | average                       | 34.2                   | 30.3                           |

Reviewer comment (Novelty):
The underlying paradigm, while interesting, may not be considered highly novel.

Author response:
Thanks for raising this point and providing the two ways to view our work. Our work is better viewed as a framework for both online and the well established offline-to-online RL problem. Our primary contribution is addressing the exploration bias introduced by the use of exploration bonuses. As our experiments show, directly using state of the art offline-to-online algorithms (Tables 1-2) and online-only algorithms that use prior data (Table 3) are not able to explore efficiently without exploration bonuses. Naive, yet standard, use of exploration bonuses is not able to achieve optimal performance as the policy is incentivized to continue exploring (Figures 10 and 12). This motivated the construction of OOO as a framework, where a policy extraction step at the end of online fine-tuning using offline RL can counteract the exploration bias, and thus meaningfully improve the final performance.

We have revised the paper to emphasize this novel aspect of OOO, i.e. counteracting the exploration bias using offline retraining, and thus allowing more effective use of exploration bonuses in online fine-tuning and online RL.

Reviewer comment (Theory):
A deeper exploration of the intricate dynamics between the three stages is ideally underpinned by robust theoretical analysis.

Author response:
Thanks for clarifying. Our submission includes some relevant analysis in the paper already:

First, the exploitation performance over time in our experiments can be seen as an ablation over replay buffers (as exploitation policies are trained with varying amounts of data over time). Figuring out a sufficient online budget apriori to training is a non-trivial and general problem, beyond the scope of this work.

Figure 5 shows a relevant ablation in which we ask “given the replay buffer generated by the > second stage, does the exploitation training from the third stage need to be pessimistic?” The experiment compares training the final exploitation policy with a standard RL algorithm, compared to using an offline RL algorithm, and finds that training a standard, non-pessimistic RL algorithm results in exploding Q values and poor performance.

Figure 14 shows another relevant experiment about the interplay between stages 2 and 3, where we try continuing the online exploration with the exploitation policy each time we train one. We find that this can drastically improve online exploration performance, but doesn’t substantially change the final exploitation performance.

Moreover, we evaluate the role of data rebalancing in Figure 11 for the final stage of OOO, another important aspect that can be critical to recover a performant policy. The rebalancing becomes important specifically because the second stage (online fine-tuning) can result in an imbalanced replay buffer.

Reviewer comment (Experiments):
It is unclear whether it is possible to combine other exploration algorithms instead of relying solely on full space coverage exploration algorithms, such as the probing policies used in [1,2].

Author response:
We believe this should be possible to do, and should form an interesting extension of our work.

Reviewer comment (Experiments):
How to make a trade-off between the size of offline datasets and the online fine-tuning steps, so that the framework fits more into an offline setting?

Author response:
In general, the hope is that we can use as much offline data as is available (though data quality control and filtering are important topics). As we collect more and more data, potentially from different fine-tuning runs, the offline datasets will grow larger and the amount of fine-tuning required would hopefully go down – making the problem increasingly offline. However, determining the exact number of fine-tuning steps given the size of offline datasets is an interesting problem that we do not have definitive answers to.

Reviewer comment (Experiments):
some important baselines seem missed: baseline of offlineRL + exploration (without decoupling) should be added; some offlineRL + decoupled exploration methods should be added (see below);

Author response:
> baseline of offlineRL + exploration (without decoupling) should be added

Figure 4 and 12 (in Appendix) show results for IQL + RND, which corresponds to your suggestion of offline RL + exploration bonuses without decoupling. We find consistent improvements in performance when decoupling the performance across environments, where we have highlighted some of the interesting cases in Figure 4.

> Some offlineRL + decoupled exploitation methods should be added; some related works should be discussed

Thanks for the suggestions for related works on decoupling exploration and exploitation policies, we have added a discussion in Section 2 (Related Work) on how our work differs from prior works. We will update the rebuttal with DEEP (Decoupled Exploration and Exploitation Policies) as a baseline, and we have revised the related works to discuss the suggested papers. We note that the decoupling strategy in DEEP is different from OOO: DEEP trains two decoupled policy, one for the task reward and one *exclusively* for the exploration bonus, and mixes them to collect data. DEEP samples the product of these two policies (restricting to actions from the support of the task policy), while OOO trains a single policy on the mixture of task and exploration reward, and trains a decoupled exploitation policy on task reward using offline RL.

Reviewer comment (Evaluation):
the evidence of exploration bonuses bias in the learned policy should be discussed more explicitly. Maybe giving some visualizations for OOO and a standard offlineRL + RND algorithm is good.

Author response:
> The evidence of exploration bonuses bias in the learned policy should be discussed more explicitly. Maybe giving some visualizations for OOO and a standard offlineRL + RND algorithm is good.

Figure 2 provides a visualization of the exploration bias (as it is easy to visualize for 2D environments). Figure 2c shows the replay buffer obtained by IQL + exploration bonus, which contains enough exploration data for OOO to be able to recover optimal behavior, but the policy continues to explore due to incomplete coverage of the state space. Moreover, Figure 4a shows a specific case where IQL + RND underperforms IQL even though OOO (IQL) outperforms both of them, suggesting that the exploration bias is at play. We welcome recommendations for how we can emphasize this better. What other visualizations would be informative here?

Reviewer comment (Experiments):
In Figure 3, why use "frozen" RND and a baseline to demonstrate the effects of exploration bias?

Author response:
> In Figure 3, why use "frozen" RND and a baseline to demonstrate the effects of exploration bias?

Primacy bias [1] has been hypothesized as a phenomenon for poor sample efficiency of deep RL algorithms. Reinitializing the Q-value and policy networks has been shown to improve the performance. In OOO, since we train the exploitation from scratch at the end, the removal of primacy bias (if any) is a competing hypothesis to removal of exploration bias. To investigate these competing hypotheses, the RND weights are frozen to get the true exploration bonuses being used at each online timestep, and we do an offline RL training step with and without the RND rewards (the latter being equivalent to OOO). If we do not freeze the RND model weights, exploration bonuses may decay to 0 during the offline retraining step in OOO, which would not allow us to rule out primacy bias as an explanation for improved performance of OOO. We have revised the text to make this clearer.

Reviewer comment (Evaluation):
I am a bit confused about the paragraph "What explains the improved performance in OOO RL?". How can we derive the conclusion that "These ablations suggest that mitigating the exploration bias by removing the intrinsic reward when training the exploitation policy leads to improved performance under OOO" just by excluding the two hypotheses?

Author response:
> How can we derive the conclusion that "These ablations suggest that mitigating the exploration bias by removing the intrinsic reward when training the exploitation policy leads to improved performance under OOO" just by excluding the two hypotheses?

We agree that excluding two hypotheses does not necessarily imply our conclusion, and we have revised it to be more clear. Are there any alternate hypotheses that would be valuable to investigate? Figures 3 and 4 argue that increased state coverage and primacy bias mitigation cannot fully explain the performance improvement, and that removing the exploration bias is also part of what improves performance. Additionally, the hypothesis that improved performance under OOO results from mitigating exploration bias is tested in the ablation in Figure 3, where the only difference between two exploitation policies is whether the intrinsic rewards are included.

Reviewer comment (Theory):
An open question for discussion: What do you think are the essential differences, the extra challenges, and the benefits of the standard exploration problem in online RL and the exploration problem in offline-to-online RL?

Author response:
> An open question for discussion: What do you think are the essential differences, the extra challenges, and the benefits of the standard exploration problem in online RL and the exploration problem in offline-to-online RL?

While they are definitely closely related and similar techniques can be leveraged for either, the prior offline data can inform the exploration strategy in offline-to-online RL, exploring new parts of the state space. Further, exploration-exploitation trade-off may be different between online and offline-to-online RL, as one may want to explore conservatively in the latter, staying closer to the offline distribution.