Paper: A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
This work utilizes VAE to characterize the uncertainty of state-actions, whereas the paper does not provide the specific definition of uncertainty, and it’s not convincing why VAE is superior to other methods, e.g., Q-ensemble learning.

Author response:
> **Q1: Definition of uncertainty.**

In this work, we define the uncertainty as the approximate negative log likelihood of the state-action density as shown in Page 4:
$\mathcal{U}(s,a)\overset{\rm{def}}{=}-\log p(s,a)\approx\mathcal{L}_{\mathrm{ELBO}}(s,a;\psi,\varphi).$

> **Q2: Advantage of VAE over ensemble-based technique for uncertainty quantification.**

We detail the reasons at the beginning of Section 4.1 as below.

> Many prior works typically utilize Q ensembles to qualify uncertainty. However, the ensemble technique may significantly increase computational costs. Thus, in this work, we utilize a simple yet effective approach by adopting VAE  as a state-action visitation density estimator for uncertainty quantification.

Furthermore, we also derive some initial empirical results on the ensemble setting in Table 9 in the appendix. From Table 9, we can observe that uncertainty quantification with VAE can attain comparable performance to using the variance of 10 Q functions, where the latter would significantly increase computational costs.

Reviewer comment (Experiments):
There is a lack of clear explanation towards the unique challenges in the exploration-exploitation of the offline-to-online problem. For instance, a major issue could be the “forgetting” - during the model updates the fine-tuned policy can quickly forget what it has learned from offline, which is neglected by this work.

Author response:
> **Q4: Explanations on exploration-exploitation issue in offline-to-online RL.**

We observe that aggressive exploration may exacerbate the distribution shift, while conservative exploitation can hinder agents from efficient online finetuning. Specifically, as stated at the end of Section 4.2, to solve the constrained exploratory behavior issue, we introduce optimistic exploration strategy. However, this may also bring negative effects by further increasing state-action distribution shift, as a result of the principle of optimism in the face of uncertainty. Thus, simultaneously and efficiently addressing both challenges brings additional improvement.

> **Q5: "Forgetting" issue.**

We want to clarify that the forgetting issue is one outcome of failing to handle state-action distribution shift. As stated in Introduction, 

> Accordingly, the state-action distribution shift between offline and online data occurs, leading to the well-known extrapolation error during exploitation, which may wipe out the good initialization obtained from the pretraining stage.

Thus, we solve this issue by proposing the adaptive exploitation method with OOD sample identification.

Reviewer comment (Experiments):
The motivation and evaluation are isolated. In the Introduction, the paper claims “the proposed method provides a generic solution for offline-to-online RL to enable finetuning agents pretrained with different offline RL objectives.” However, in the experiment, the tuning policy is pretrained from the same task.

Author response:
> **Q6: Evaluation of "generic" solution for offline-to-online RL.**

We want to clarify that the genericness of SUNG means that SUNG can be seamlessly combined with multiple existing offline RL methods. We verify this by combing SUNG with four backbone offline RL methods, including TD3+BC, CQL, CQL-10, and SPOT.

Reviewer comment (Theory):
As shown in Eq. (9), is the fine-tuning method dependent on the pretraining algorithm?

Author response:
> **Q8: Relationship between finetuning method and pretraining algorithm.**

Yes, you are right. In this work, we utilize the corresponding online RL objectives for online finetuning to keep consistent.

Reviewer comment (Theory):
I find the formulation of the SUNG framework a bit counterintuitive: The authors mention that they want to have high-uncertainty actions, yet at the same time they only sample "near-on-policy actions for exploration", which appears contradicting. Further, during the optimization / policy improvement part (green arrows in fig 1), the same percentage p of the batch is always labeled as OOD, which is not consistent, since the absolute uncertainty value at which a sample could be labeled OOD can vary (it could even be the case that the same sample is sometimes OOD and sometimes not, depending on the other samples in the batch).

Author response:
> **Q1: Contradiction between high-uncertainty and near-on-policy actions.**

As shown in Eq. (7), SUNG involves selecting actions with both high Q values and high uncertainty to guide exploration. Specifically, we restrict our exploration to near-on-policy actions to prevent significant deviations. This restriction offers two key advantages:

(1) Ensuring finetuning stability: By exploring near-on-policy actions during finetuning, we maintain stability in the learning process. The policy improvement relies on experiences collected during exploration. Deviating too far from the current policy could lead to abrupt and undesirable changes, rendering the learning process unstable and less sample-efficient.

(2) Avoiding inaccurate Q value and uncertainty estimation: Sampling actions far from the on-policy actions may result in less reliable estimates of Q values and uncertainty. This unreliability has the potential to throw the exploration direction out of control, hindering the overall effectiveness and efficiency of exploration.

> **Q2: The absolute uncertainty value at which a sample could be labeled as OOD can vary.**

We fully agree with your perspectives. SUNG detects OOD samples via percentage-based filtering strategy out of simplicity. This approach is intentionally designed to minimize the complexity associated with hyper-parameter tuning. However, we want to highlight that empirical results demonstrate the effectiveness of SUNG, as evidenced by the SOTA finetuning performance achieved by SUNG through the proposed adaptive exploitation. We acknowledge that there is room for future exploration in refining adaptive and precise identification of OOD samples.

Reviewer comment (Theory):
Generally, I have a bit of an issue with the additional OOD detection in the adaptive exploitation - the regularizer in offline RL algorithms is mostly already doing OOD detection (often also based on some measure of uncertainty), so it should already automatically detect low and high uncertainty samples and thus assign low or high regularization accordingly. While this is normally not done in such a binary nature, I don't see how the adaptive exploitation scheme is not doing pretty much the same thing again. It seems to just amplify the regularization tendency that is already there. The empirical ablation in fig 2d shows it to be important, but to me it is unclear why that is the case - is the normal regularization simply not yet enough? or is the VAE based OOD detection in some qualitative way better than the base algorithm's own regularization?

Author response:
> **Q3: Why introducing additional OOD detection in the adaptive exploitation.**

Although many offline RL methods integrate OOD detection in the adaptive exploitation, they still necessitate heavy conservatism in the objective to avoid the extrapolation error. In this paper, SUNG introduces OOD sample identification to adaptively relax such conservatism for higher sample efficiency during online finetuning. It is noteworthy that certain offline RL algorithms, including TD3+BC, CQL, and SPOT, lack explicit OOD detection. Hence, the proposed adaptive exploitation holds significant importance for these methods to attain desirable fine-tuning performance.

Reviewer comment (Experiments):
In the exploration scheme, where actions are ranked & selected based on value and then sampled based on uncertainty (or the other way around), you mention the possibility to adapt the trade-off preference between value and uncertainty - what do you mean by that? Would you adaptively change the trade-off during the training process or is it something you set in advance (and based on what information)? I'm also not sure whether you mention which way around you ended up doing filtering + sampling (first value or first uncertainty or maybe some combination) and why.

Author response:
> **Q4: Explanation on trade-off between value and uncertainty.**

In the suggested optimistic exploration strategy, choosing the ranking criteria as "value first" implies that a low value of k indicates a preference for high value, whereas a high value of k indicates a preference for high uncertainty. Conversely, if the ranking criteria are set as "uncertainty first", the scenario is precisely the opposite. As such, by altering ranking criteria and k, the algorithm allows for the customization of preferences between uncertainty and Q value to varying extents. This flexibility empowers users to finely adjust the trade-off according to the specific requirements of their realistic scenarios.

> **Q5: How to select the ranking criteria?**

We detail this problem in Appendix A.2 and A.3. Specifically, we choose uncertainty first for value regularization based offline RL methods, whereas Q first for other offline RL methods. This choice is underpinned by the following considerations. Firstly, we want to highlight the empirical finding that value regularization based offline RL methods (e.g., CQL) are only compatible with preference for high uncertainty, whereas other offline RL methods are compatible with preference for both high uncertainty and high value. We give a detailed analysis for the reason of this finding in Appendix B. As articulated in Q4, irrespective of the chosen ranking criteria, we possess the flexibility to adeptly adjust k to attain the desired preference. Consequently, we select the aforementioned criteria to fix the tuning range within a small value of k, only aiming for consistency.

Reviewer comment (Evaluation):
In the empirical evaluation I'm assuming you report final performances (i.e. after the 100k environment steps). Prior works that you mention (like Cal-QL, AWAC) have shown that during O2O, algorithms commonly suffer from immediate performance drops right after online learning starts due to distribution drift - it would be extremely interesting to see how SUNG performs in this context, which is why another performance metric reporting during the beginning of online learning could be useful or directly plotting the test returns over training time.

Author response:
> **Q6: Other evaluation metrics.**

Since this paper focuses on the improving the sample efficiency of offline-to-online RL, we choose the final average performance of 100k environment steps as the evaluation metric in the main body of this paper for clear comparison. Note that we also provide the full training curve for 30 settings in Appendix D.4 to further demonstrate the effectiveness of SUNG.

Reviewer comment (Novelty):
I believe some other prior works that are currently not contained should also be considered in the related work section:

[1] Swazinna, P., Udluft, S., & Runkler, T. (2021). Overcoming model bias for robust offline deep reinforcement learning. Engineering Applications of Artificial Intelligence (EAAI), 104.

[2] Ghosh, D., Ajay, A., Agrawal, P., & Levine, S. (2022). Offline rl policies should be trained to be adaptive. ICML 2022

[3] Hong, J., Kumar, A., & Levine, S. (2022). Confidence-Conditioned Value Functions for Offline Reinforcement Learning. ICLR 2023

[4] Swazinna, P., Udluft, S., & Runkler, T. (2022). User-Interactive Offline Reinforcement Learning. ICLR 2023

[1] introduces MOOSE, which is a model-based offline RL method that uses a VAE as a regularizer. While your adaptive exploitation scheme is using it in a different way, I still believe it should be mentioned. The works [2-4] are also concerned with offline to online learning, just that their online phase is a little shorter and their adaptations thus look a little different than the one you consider. Still, when thinking about O2O they are closely related and should be considered.

Author response:
> **Q7: Some related works.**

Thanks for providing these related works! They are all included for discussion in the revised version.

Reviewer comment (Presentation):
More a sidenote: Fig. 2b is labeled "Optimistic Exploration without Uncertainty". As far as I understand, this method is greedily selecting best actions based on value - it is unclear to me how that is optimistic (i.e. optimism in the face of what) so maybe the different ablations should be named more clearly.

Author response:
> **Q8: Label of "Optimistic Exploration without Uncertainty"**

Here, the optimism refers to exploration with optimism in the face of Q value. We give detailed experiment settings of the ablation study in Appendix D.5.