Paper: Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor Critic (ICLR-2024)

Reviewer comment (Theory):
The terminology can be misleading. The overestimation issue in the Q-value approximation generally is due to the changing order of expectation and $\max$. It is incorrect to say that the $Q$-function will have "underestimation when encountering successes" in Fig 1 (a). The authors need to clarify the context and difference of the statement in order to avoid confusion.

Author response:
Thanks for your suggestions. We have refined it in our revised paper.

* We removed the terminology in Fig 1(a).
* We refined the caption of Fig 1(a) as, "When current policy generated action is inferior to the best action in the replay buffer, which usually occurs in the later stage of training (referred to as the under-exploitation stage), SAC is more prone to underestimation pitfalls than BAC."

Reviewer comment (Theory):
In order to investigate on the under-exploitation, the metric $\Delta(\cdot,\cdot)$ is defined on the current Q-function approximation. Intuitively, $\Delta(\cdot,\cdot)$ shows that the current Q-function approximation can be either overestimate or underestimate given different policy, i.e., $\mu_k$ and $\pi_k$. It is unclear what is the meaning of this metric. Considering most of the algorithm will update the policy and Q-function approximation at the same time, e.g., Actor-Critic, the Q-function should be evaluated under the current policy instead of the policy obtained earlier. The authors need to clarify why the definition here makes sense for the under-exploitation investigation.

Author response:
Sorry for any confusion in our initial presentation. We've revised the notation for clarity:


$\Delta(\mu_k, \pi_k) = \mathbb{E}_s[\max\_{a\sim \mu_k}Q^{\mu_k}(s,a) - \mathbb{E}\_{a\sim \pi_k} [Q^{\pi_k}(s,a)-\omega(s,a\vert \pi_k)]]
$.

This formula shows the expected difference between two different types of calculated Q-values. Specifically:

* $\max_{a\sim \mu_k}Q^{\mu_k}(s,a)$:  This is the maximum Q-value according to the policy mixture $\mu_k$ learned using offline RL techniques from the replay buffer. It approximates the value of the best policy that could be derived from the replay buffer.
* $\mathbb{E}_{a\sim \pi_k} Q^{\pi_k}(s,a)$: This corresponds to the expected Q-value under the current policy $\pi_k$. And the exploration bonus $-\omega(s,a\vert \pi_k)$ could also be considered when computing.




Thus, $\Delta(\mu_k, \pi_k)$ quantifies the gap between the optimal actions in the replay buffer and those generated by the current policy $\pi$. A positive $\Delta(\mu_k, \pi_k)$ indicates the existence of better actions in the replay buffer that have not been fully exploited.



Finally, we hope we resolve all of your concerns. We have refined our explanations in the rebuttal revision according to your suggestions. Thanks again for your suggestions!

Reviewer comment (Novelty):
The novelty of the proposed approach is limited.

Author response:
Our novelty could be summarized below:

* While existing off-policy RL algorithms primarily focused on alleviating value over-estimation, our work reveals the long-neglected underestimation phenomenon in the latter stage of the online RL training process. We have provided extensive investigations into this issue (more in Appendix E), offering fresh insights into this overlooked area.
* Our findings shed light on a new way to cure underestimation: incorporating sufficient exploitation by leveraging techniques from offline RL. Our proposed BEE operator is simple and versatile and achieved great performance in our empirical experiments.
* Furthermore, our study **demystifies the misperception** that offline RL is over-conservative and incompatible with the online RL setting. It suggests a new paradigm that incorporates exploitation ingredients from offline RL to enhance pure online RL.

Reviewer comment (Experiments):
The choice of $\lambda$ is largely empirical and requires extra manipulation in new tasks.

Author response:
There might be a misunderstanding here. Simply using $\lambda=0.5$ is sufficient for strong performance across all the 50 tasks in our experiments.  We provide clarification on the hyperparameter robustness; please refer to our **General Response**.

Reviewer comment (Theory):
The paper only provides basic theoretical analysis, such as the accurate policy evaluation and the guarantee of policy improvement. The benefit of linearly combining two Q-value functions is not discussed theoretically.

Author response:
The focus of our study is to provide a strong practical algorithm rather than presenting a theoretical study to provide lots of performance bounds. Actually, in order to provide a versatile framework that is capable of leveraging wide choices of exploration terms $\omega(\cdot|\pi)$ and Bellman exploitation operator evaluation methods (see Section 4.4 for details), it becomes pretty challenging for anyone to provide further theoretical analysis. Since different instantiation choices will introduce different theoretical properties. However, as mentioned by the reviewer, we have empirically verified the effectiveness via extensive experiments in both the simulated environments and a real robot. We hope this can justify the strengths of our algorithm.

Reviewer comment (Experiments):
The experiments are conducted in continuous control tasks with dense rewards. The exploration ability can be better evaluated in environments with sparse rewards.

Author response:
Thanks for your constructive comment! 
* We have conducted experiments on six sparse reward tasks across various control types and robotic platforms, including manipulation and locomotion tasks. And BAC outperforms the baselines a lot. Please refer to Figure 31 in our revised paper.
- Besides, the illustrative example in Appendix F.3 is also a sparse reward setting, showcasing the BEE's effectiveness.

Reviewer comment (Novelty):
There is a lack of discussions with related papers (See Question 3).

Author response:
Thank you for pointing out these interesting works. 
* **Prioritized replay methods**: 
    * We have included a discussion on prioritized replay methods in our revision and cited them accordingly. We'd like to emphasize that although prioritized replay methods follow a similar intuition of leveraging better samples from the replay buffer for enhanced value learning, BAC provides a much simpler and more lightweight manner. There is also no need to change the data pipeline as in prioritized replay methods.
    * Second, we'd like to emphasize that BAC is compatible with prioritized replay methods. Recall the Bellman error used in policy evaluation, $\mathbb{E}_d[w(s,a)(Q(s,a)-\mathcal{B}Q(s,a))^2]$.  Here, the BEE operator designs a hybrid operator $\mathcal{B}^{\mu,\pi}$ focusing on blending exploitation and exploration, and prioritized replay methods concentrate on designing effective weights $w(s, a)$.  

* **Advantage-based methods**:  This class of methods has a different setting from ours. As we discussed in Related Works, AWAC is under a two-stage paradigm, performing offline RL on offline datasets and then followed by online fine-tuning. BAC is a **pure online RL** algorithm.

Reviewer comment (Evaluation):
Empirically, the BAC algorithm will only be more efficient in exploiting the replay buffer. The exploration still relies on the maximum-entropy formulation in SAC. Then why can BAC perform significantly better than SAC in failure-prone scenarios such as HumanoidStandup, as if BAC can better explore the unknown regions?

Author response:
First, we'd like to clarify that our exploration term $\omega(\cdot|\pi)$ is not restricted to the entropy term as in SAC; other exploration designs are also applicable, as discussed in Section 4.4. We use the entropy term as in SAC mainly due to its simplicity.
- Second, as we have discussed in our paper, SAC is not very effective to exploit the high-quality samples in the replay buffer. This can have a great impact on failure-prone tasks, as informative, successful samples can be scarce in policy-generated samples. Blindly sampling batches of data from the replay buffer can be very ineffective in extracting useful information for value updates. For example, in the HumanoidStandup task,  SAC can only maintain short-standing postures before failing, as seen from the trajectory visualization (Figure 33 or Video in https://beeauthors.github.io/ ). This indicates that while SAC has explored high-value transitions, it struggles to exploit them effectively. In this case, fully exploiting the high-quality samples in the replay buffer for value updates is essential to achieve efficient policy learning.
- We present more illustrative results to showcase the effectiveness of our method in failure-prone scenarios in our revision; please refer to Appendix F.3.

Reviewer comment (Experiments):
Can you discuss or exhibit the performance of BAC in some tasks with sparse rewards? This can demonstrate the generalizability of the proposed approach.

Author response:
Thanks for your constructive comment! 
* We have conducted experiments on six sparse reward tasks across various control types and robotic platforms, including manipulation and locomotion tasks. And BAC outperforms the baselines a lot. Please refer to Figure 31 in our revised paper.
- Besides, the illustrative example in Appendix F.3 is also a sparse reward setting, showcasing the BEE's effectiveness.

Reviewer comment (Experiments):
The BAC method tunes its $\lambda$ and $\tau$ differently for tasks in MuJoCo and DMC (Table 1 & 5). It's questionable to claim superiority over other state-of-the-art (SOTA) methods like SAC and TD3, which use consistent hyperparameters (HP) across tasks. Adjusting HP for each task can inflate results as seen in Figure 5, which can be misleading. Why not showcase the automatic $\lambda$ tuning methods from Appendix B.3.3 in the main text if they're effective?

Author response:
> **Q1: The BAC method tunes its $\lambda$ and $\tau$ differently for tasks in MuJoCo and DMC (Table 1 & 5). It's questionable to claim superiority over other state-of-the-art (SOTA) methods like SAC and TD3, which use consistent hyperparameters (HP) across tasks. Adjusting HP for each task can inflate results as seen in Figure 5, which can be misleading.**

We believe there is a misunderstanding about the HP tuning of our work. 
* As discussed in our **General Response**, simply using a single set of HP in our method can achieve SOTA performance throughout all 50 environments.
* We did tune the HPs of SAC and TD3 baselines across benchmarks/tasks for improved performance, following suggestions from the original paper [3,4] (for MuJoCo), TD-MPC [5] paper (for DMC), Meta-World [6] paper (for MetaWorld), as detailed in Appendix D.  For clarity, we attach the HPs table for SAC and TD3 into our revision, please refer to Table 5 and Table 6.

> **Q2: Why not showcase the automatic $\lambda$ tuning methods from Appendix B.3.3 in the main text if they're effective?**

- We find that fixed $\lambda=0.5$ is already sufficient for strong performance, validated by extensive experiments across 50 tasks.
- A simple yet efficient algorithm has been long advocated. Hence, we present BAC with minimal designs in the main text.  The automatic $\lambda$ tuning mechanism in the appendix is intended for interested readers to explore potential future refinements.

Reviewer comment (Novelty):
Presuming the Q value underestimation problem is widely recognized (which I invite the authors to contest), the paper seems to lack innovation. The BEE operator, at its core, appears to be a fusion of existing Bellman operators.

Author response:
> **Q4: Presuming the Q value underestimation problem is widely recognized (which I invite the authors to contest), the paper seems to lack innovation. The BEE operator, at its core, appears to be a fusion of existing Bellman operators.**

We would like to emphasize that our method is not simply fusing existing Bellman operators. We revealed a long-neglected underestimation issue in an existing method, which could potentially hurt performance and sample efficiency (elaborated in our response to the **major concern**). To address this, we provide a natural and very simple solution to enable fully exploitation of high-quality samples in the replay buffer. More importantly, our work demystifies the misperception that offline RL is over-conservative and incompatible with the online RL setting, and also provides a new paradigm to incorporate exploitation ingredients from offline RL to enhance the performance of online RL.

Reviewer comment (Theory):
The statement "BEE exhibits no extra overestimation" seems conditional on specific $\lambda$ and $\tau$ values. For instance, using $\lambda = 1$ and $\tau = 1$ could induce overestimation.

Author response:
> **Q5: The statement "BEE exhibits no extra overestimation" seems conditional on specific $\lambda$ and $\tau$ values. For instance, using $\lambda$=1 and $\tau$=1 could induce overestimation.**

* For $\lambda=1$: this case actually reduces to an in-sample learning offline RL algorithm (e.g., IQL in our instantiation), which is inherently conservative. We show in Appendix E.6 that this case does not exhibit overestimation. Besides, note that our BEE operator is defined on $\lambda\in (0,1)$, as stated in Definition 4.1. 
* For $\tau=1$:  BEE operator is not bounded to hyperparameter $\tau$. As detailed in Appendix B.3.1, implementing BEE with other in-sample learning offline techniques, such as SQL and EQL instead of IQL would not have $\tau$ at all.  We chose IQL primarily for its simplicity, and the original IQL paper also restricts $\tau < 1$.