Paper: Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR-2024)

Reviewer comment (Evaluation):
Unclear empirical motivation in Figure 1. What do these variants (Conservatism, Constraint, Off-policy) mean in (a)? How does Q value compare with V value in (b)? Moreover, from (b), it seems that CQL->SAC shows faster improvement than On-policy (V). How does this conclude that Q values of SAC exhibit slow improvement? Furthermore, CQL->CQL and CQL->SAC are naïve solutions for offline-to-online RL. What about advanced offline-to-online RL algorithms, such as off2on?

Author response:
**Q1.1: What do these variants (Conservatism, Constraint, Off-policy) mean in (a)?**

**A1.1:** Here we use offline -> online method to specify the offline-to-online mehtod. In the previous version of Fig. 1(a), the conservatism, constraint, and off-policy represent the offline-to-online methods CQL->CQL, IQL->IQL, and CQL->SAC. We have revised the legend to make it more clear in the updated manuscript.

**Q1.2: How does the Q value compare with the V value in (b)? Moreover, from (b), it seems that CQL->SAC shows faster improvement than On-policy (V). How does this conclude that Q values of SAC exhibit slow improvement?**

**A1.2:** In the motivating example, we aim to compare the naïve off-policy (SAC), conservative method (CQL), and naïve on-policy (ours). During the offline-to-online transition, naïve off-policy methods often encounter distribution shifts when the policy explores out-of-distribution regions. This can result in evaluation errors and a subsequent drop in performance. A simple method to address this challenge is to inherit the conservatism employed during the offline phase. However, relying solely on this conservative strategy can introduce suboptimality during the online phase, resulting in data inefficiency. Therefore, we aim to investigate whether a naïve on-policy RL algorithm can effectively tackle this challenge. Uni-O4 initialized the policy and $V$-function from the offline phase for a standard online PPO. 

*Both the $V$ and $Q$ functions can evaluate the performance of the policy. Therefore, we consider it acceptable to compare the $V$-value and $Q$-value across these methods to investigate how the value function guides performance improvement.* In the case of the naïve off-policy method, the $Q$-value exhibits a significant drop and unstable improvement during online training, leading to a decline in policy performance. The conservative method, on the other hand, shows slow improvement in the $ Q$ value, resulting in suboptimality. In contrast, the on-policy method demonstrates stable improvement in the $ V$ value, leading to consistent and efficient fine-tuning.

**Q1.3: How does this conclude that Q values of SAC exhibit slow improvement?**

**A1.3:** The $Q$-value of the off-policy method improves from 27 to 41, while the $V$-value of the on-policy method improves from approximately 36.5 to 47. It is true that the improvement in the $Q$-value is slightly faster than its counterpart. However, two points should be considered: *1) The on-policy method (ours) has a significantly higher initial performance from the offline phase compared to the off-policy method, resulting in higher initial scores for the value function. Therefore, a slightly slower convergence speed is reasonable. 2) The off-policy method experiences a noticeable drop and unstable improvement in the value, whereas this phenomenon is absent in the on-policy method.*

**Q1.4: Furthermore, CQL->CQL and CQL->SAC are naïve solutions for offline-to-online RL. What about advanced offline-to-online RL algorithms, such as off2on?**

**A1.4:** Thank you for your insightful suggestion. It is interesting to investigate how the ensemble value function guides policy improvement. Thus, we have added the off2on as a comparison. Our findings reveal that the $Q$ value of the off2on method demonstrates faster improvement and can eventually converge to the same level as the $V$ value of Uni-O4. However, it is worth noting that the off2on approach experiences performance drops and unstable training during the initial stages, primarily due to the unstable improvement of the $Q$ value.

Reviewer comment (Novelty):
The technique seems incremental by just combining BPPO with OPE.

Author response:
**Q2: The technique seems incremental by just combining BPPO with OPE.**

**A2**: Uni-O4 offers a significant insight by demonstrating how an on-policy RL algorithm can seamlessly unify online and offline RL, enabling flexible combinations of pretraining, fine-tuning, offline, and online learning. In the offline phase, a crucial step in offline RL, we replace online evaluation with Off-Policy Evaluation (OPE) based on BPPO. This substitution is vital as it eliminates the need for costly online interactions and enhances safety.

Furthermore, in Uni-O4, we replace the value function trained by SARSA with one trained by IQL. This simple replacement, albeit derived from existing methods, plays a crucial role in providing more accurate OPE and advantage function estimations. Additionally, we introduce the ensemble behavior cloning (BC) method with disagreement penalty in order to effectively capture the multi-modality present within the dataset. These seemingly simple components are of paramount importance when it comes to exploring optimal policies.

Together, these components within Uni-O4 contribute significantly to the exploration of optimal policies, providing a comprehensive and precise framework for offline RL. This is precisely why Uni-O4 demonstrates significantly superior performance compared to BPPO, even though we replace online evaluation with OPE.

Reviewer comment (Experiments):
Insufficient comparison of offline RL methods, including but not limited to:  
[1] RORL: Robust Offline Reinforcement Learning via Conservative Smoothing.  
[2] Extreme Q-Learning: MaxEnt RL Without Entropy.  
[3] Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization.

Author response:
**Q3.1: I think there is a slight overclaiming of the experimental results in the Introduction without sufficient comparison of SOTA algorithms. Insufficient comparison of offline RL, including but not limited to [1, 2, 3]:** 

**A3.1**: Thank you for the suggestion. We have incorporated Diffusion-QL [1], RORL [2], XQL [3], and SQL [4] as additional comparisons in the offline phase. Below are the main findings, with most of the results extracted from the original papers. Results marked with an asterisk (*) indicate that they were reproduced using open-source code and tuned with hyperparameters.

Upon inspecting the table below, it becomes evident that Uni-O4 outperforms all other methods in terms of the total score across all tasks. Furthermore, Uni-O4 outperforms all other methods in 16 out of 26 individual tasks. While RORL surpasses Uni-O4 in the total score for MoJoCo Locomotion tasks, it performs worse than Uni-O4 in the other three domains and exhibits limited effectiveness in the Kitchen domain.

|Environment|Diffusion-QL[1]|RORL[2]|XQL[3]|SQL[4]|Uni-O4 (Ours)| |
|:----|:----|:----|:----|:----|:----|:----|
|halfcheetah-medium-v2|51.1|**66.8**|48.3|48.3|52.6 $\pm$ 0.4|
|hopper-medium-v2|90.5|**104.8**|74.2|75.5|**104.4 $\pm$ 0.6**|
|walker2d-medium-v2|87.0|**102.4**|84.2|84.2|90.2 $\pm$ 1.4|
|halfcheetah-medium-replay|47.8|**61.9**|45.2|44.8|44.3 $\pm$ 0.7|
|hopper-medium-replay|101.3|**102.8**|100.7|99.7|**103.2 $\pm$ 0.8**|
|walker2d-medium-replay|95.5|90.4|82.2|81.2|**98.4 $\pm$ 1.6**|
|halfcheetah-medium-expert|96.8|**107.8**|94.2|94.0|93.8 $\pm$ 1.3|
|hopper-medium-expert|**111.1**|**112.7**|**111.2**|**111.8**|**111.4 $\pm$ 1.5**|
|walker2d-medium-expert|110.1|121.2|112.7|110.0|**118.1 $\pm$ 2.2**|
|***locomotion total***|*791.2*|***870.8***|*752.9*|*749.5*|*816.4 $\pm$ 10.5*|
|Umaze-v2|93.4|**96.7**|93.8|92.2|93.7 $\pm$ 3.2|
|Umaze-diverse-v2|66.2|**90.7**|82.0|74.0|83.5 $\pm$ 11.1|
|Medium-play-v2|76.6|76.3|76.0|**80.2**|75.2 $\pm$ 4.4|
|Medium-diverse-v2|78.6|69.3|73.6|**79.1**|72.2 $\pm$ 3.8|
|Large-play-v2|46.4|16.3|46.5|53.2|**64.9 $\pm$ 2.5**|
|Large-diverse-v2|56.6|41.0|49.0|52.3|**58.7 $\pm$ 3.0**|
|***Antmaze total***|*417.8*|*390.3*|*420.9*|*431.0*|***448.2 $\pm$ 28.0***|
|pen-human|72.8|33.7|85.5|89.2*|**108.2 $\pm$ 10.7**|
|hammer-human|4.3* |2.3|8.2|3.8*|**24.7 $\pm$ 4.4**|
|door-human|6.9* |3.8|11.5|7.2*|**27.1 $\pm$ 1.3**|
|relocate-human|0.0* |0.0|0.2|0.2*|**1.7 $\pm$ 0.6**|
|pen-cloned|57.3*|35.7|53.9|69.8*|**101.3 $\pm$ 19.3**|
|hammer-cloned|2.1* |1.7|4.3|2.1*|**7.0 $\pm$ 0.9**|
|door-cloned|4.1* |-0.1|5.9|4.8*|**10.2 $\pm$ 2.6**|
|relocate-cloned|0.0* |0.0|-0.2|-0.1*|**1.4 $\pm$ 0.2**|
|***Adroit total***|*147.5* |*77.1*|*169.3*|*177.0*|***281.6 $\pm$ 40.0***|
|kitchen-complete|84.0|0.3*|82.4|76.4|**93.6 $\pm$ 2.5**|
|kitchen-partial|60.5|0.0*|**73.7**|72.5|58.3 $\pm$ 3.6|
|kitchen-mixed|62.6|0.0*|62.5|**67.4**|65.0 $\pm$ 4.6|
|***kitchen total***|*207.0*|*0.3*|***218.6***|***216.3***|***216.9 $\pm$ 10.7***|
|***Total***|*1563.5* |*1338.5*|*1140.8*|*1573.8*|***1763.1 $\pm$ 89.2***|

Reviewer comment (Reproducibility):
The behavior cloning method proposed in section 3.1 requires training multiple policy networks, which incurs significant computational overhead. At the same time, it does not mention how to get $\hat{pi}_{\beta}$ from a policy set.

Author response:
**A1:** Thank you for your comment. We have thoroughly examined the computational overhead and performance implications associated with different ensemble sizes of policies in Section 5.3 and **Appendix A.11**. Our findings indicate that an ensemble size of 4 strikes a balance between performance and computational overhead. Specifically, the total training time for the offline phase is approximately 265 minutes (with an ensemble size of 4), compared to 200 minutes (with a single policy) using our PyTorch implementation. Despite the slight increase in training time, this trade-off is acceptable considering the significant performance improvement achieved through the ensemble approach.

Reviewer comment (Theory):
Definition error, the definition of f used in formulas 6 and 7 is incorrect. Taking the maximum value of multiple distributions cannot guarantee a single distribution (the sum cannot be guaranteed to be 1), and analysis based on this definition is also meaningless. If the code is truly implemented based on this definition, I am skeptical about the final performance of the algorithm.

Author response:
**A2:** Thank you for pointing this out. We have taken measures to address this concern. Firstly, we have normalized the distribution of the combined policies, as outlined in Proposition 1. Additionally, we have derived a lower bound over the defined distance, as described in Theorem 1. By optimizing this lower bound, we are able to enhance the diversity among behavior policies. Notably, the derived lower bound corresponds to the penalty term mentioned in Equation 7 of the previous version. Therefore, the implementation can be guaranteed based on these derived results.

Reviewer comment (Evaluation):
The proposed offline strategy evaluation method relies on the accuracy of the probability transfer model T, and using the transfer model for evaluation will introduce more errors.

Author response:
**A3:** Thank you for this suggestion. We have reevaluated the approximation employed throughout the entire pipeline and included a thorough analysis to address this concern. Firstly, we have revised the notion of value functions trained by IQL manner form $Q_{\pi_{\beta}}$ and $V_{\pi_{\beta}}$ to $\widehat{Q_{\tau}}$ and $\widehat{V_{\tau}}$. It is helpful to distinct the value functions trained by SARSA and IQL which offers an optimal solution $Q^*$ under the dataset constraints [1], i.e., $\lim_{\tau \rightarrow 1} Q_{\tau} (s, a) = Q^{\ast} (s, a)$. $\widehat{Q_{\tau}}$ and $\widehat{V_{\tau}}$ are used to express the value functions obtained through gradient-based optimization. This revision helps to make more clear error analysis. 

During the multi-step policy optimization stage, the decision for behavior policy replacement is based on the defined metric AM-Q, which incorporates the true transition model $T$ and optimal value function $Q^*$. In the offline setting, however, the agent does not have access to the true model and $Q^*$. But $Q_{\tau}$ approaches $Q^*$ based on the dataset support constraints. We fit $\hat{T}$ and $\widehat{Q_{\tau}}$ by gradient-based optimization. In this way, the practical AM-Q can be expressed as $\widehat{J_{\tau}} (\pi)$. Thus, the OPE bias is mainly coming from the transition model approximation. We have analyzed and derived the bound of the offline policy evaluation, described by Theorem 2 in Section 3.2, please refer to check out the details.

BPPO has derived the offline monotonical improvement bound (their Theorem 3) between the target policy and behavior policy, i.e., $\pi_{k+1}$ and $\pi_k$. However, since the behavior policy is updated iteratively, this bound can result in cumulative errors that disrupt the desired monotonicity. Consequently, this bound cannot be directly utilized in the offline setting. This limitation is the reason why BPPO requires online evaluation to ensure performance improvement when replacing the behavior policy with the target policy. Given the OPE bound, we can replace the online evaluation with AM-Q to guarantee monotonicity.

Additionally, we compute the advantage function by $(\widehat{Q_{\tau}}-\widehat{V_{\tau}})$, which provides an approximation of the optimal $Q^*$ and $V^*$ based on the dataset constraint assumption. In **Appendix A.8**, we conduct experiments to demonstrate this replacement is a superior choice compared to iteratively fitting the value function of the target policy in the offline setting. The results from these experiments prove that iteratively updating the value function will lead to overestimation which causes unstable training performance and potential crashes.

Reviewer comment (Presentation):
The legend in Figure 3 is missing to know the correspondence between curves and algorithms.

Author response:
**A4:** Thank you for pointing this out. We have addressed this issue in the revised paper.

Reviewer comment (Reproducibility):
Can you provide a detailed reconstruction method for policy $\hat{\pi}_{\beta}$, whether to select any one from the policy set $\Pi_n$ or integrate it using the f function to obtain a policy?

Author response:
**A5:** In this work, we leverage the learned diverse behavior policies to capture the multi-modality in the dataset. During the offline multi-step policy optimization stage, each behavior policy is optimized by the PPO objective and guided by the AM-Q OPE metric. Thus, the behavior policy is not selected from the ensemble or using a function to obtain it. On the other hand, sampling the action from mixed policies or other policies conflicts with the PPO objective, as it aims to restrict the distribution distance between the target policy and behavior policy. In other words, the target policy needs to be close to the behavior policy because PPO is an on-policy algorithm that incorporates important sampling techniques.

Reviewer comment (Evaluation):
Is there a way to evaluate the quality of behavior cloning? Can you compare your proposed method of behavior cloning with previous methods?

Author response:
**A6:** Thank you for sharing your insightful suggestion. In this work, our motivation for utilizing a diverse ensemble of policies is to effectively capture the multi-modality inherent in the dataset, thereby enabling exploration of the high-return action region. Evaluating the quality of the ensemble behavior cloning directly is challenging, as our primary objective is to improve the final performance of the offline multi-step policy optimization process. Therefore, in Section 5.3, we evaluate the offline performance of the proposed ensemble behavior cloning approach with a disagreement penalty, comparing it to the standard behavior cloning (BC) method and the ensemble BC approach without the regularization term. The hyper-parameter analysis demonstrates that the proposed ensemble BC with disagreement penalty, when properly tuned, outperforms the alternative methods. Additionally, we have conducted an ablation study to answer the "Do diverse policies help in exploring the high-return action region to discover optimal policies" in **Appendix A.5**. The results show that it is helpful for exploring the optimal policies.

Reviewer comment (Theory):
Can we analyze the errors in the approximate part? You can cite the results of previous work to prove it. For this article, you do not need to prove the size of the approximation error. You only need to quantify the approximation error to a certain extent, analyze the potential impact, and find ways to avoid negative effects.

Author response:
**A3:** Thank you for this suggestion. We have reevaluated the approximation employed throughout the entire pipeline and included a thorough analysis to address this concern. Firstly, we have revised the notion of value functions trained by IQL manner form $Q_{\pi_{\beta}}$ and $V_{\pi_{\beta}}$ to $\widehat{Q_{\tau}}$ and $\widehat{V_{\tau}}$. It is helpful to distinct the value functions trained by SARSA and IQL which offers an optimal solution $Q^*$ under the dataset constraints [1], i.e., $\lim_{\tau \rightarrow 1} Q_{\tau} (s, a) = Q^{\ast} (s, a)$. $\widehat{Q_{\tau}}$ and $\widehat{V_{\tau}}$ are used to express the value functions obtained through gradient-based optimization. This revision helps to make more clear error analysis. 

During the multi-step policy optimization stage, the decision for behavior policy replacement is based on the defined metric AM-Q, which incorporates the true transition model $T$ and optimal value function $Q^*$. In the offline setting, however, the agent does not have access to the true model and $Q^*$. But $Q_{\tau}$ approaches $Q^*$ based on the dataset support constraints. We fit $\hat{T}$ and $\widehat{Q_{\tau}}$ by gradient-based optimization. In this way, the practical AM-Q can be expressed as $\widehat{J_{\tau}} (\pi)$. Thus, the OPE bias is mainly coming from the transition model approximation. We have analyzed and derived the bound of the offline policy evaluation, described by Theorem 2 in Section 3.2, please refer to check out the details.

BPPO has derived the offline monotonical improvement bound (their Theorem 3) between the target policy and behavior policy, i.e., $\pi_{k+1}$ and $\pi_k$. However, since the behavior policy is updated iteratively, this bound can result in cumulative errors that disrupt the desired monotonicity. Consequently, this bound cannot be directly utilized in the offline setting. This limitation is the reason why BPPO requires online evaluation to ensure performance improvement when replacing the behavior policy with the target policy. Given the OPE bound, we can replace the online evaluation with AM-Q to guarantee monotonicity.

Additionally, we compute the advantage function by $(\widehat{Q_{\tau}}-\widehat{V_{\tau}})$, which provides an approximation of the optimal $Q^*$ and $V^*$ based on the dataset constraint assumption. In **Appendix A.8**, we conduct experiments to demonstrate this replacement is a superior choice compared to iteratively fitting the value function of the target policy in the offline setting. The results from these experiments prove that iteratively updating the value function will lead to overestimation which causes unstable training performance and potential crashes.

Reviewer comment (Reproducibility):
The complexity of the method, especially regarding the ensemble behavior cloning and disagreement-based regularization, may present a steep learning curve for practitioners.

Author response:
**A1:** We have added a more intuitive explanation and algorithm description, aiming to provide a clearer understanding for practitioners. Uni-O4 adopts a straightforward approach by combining online and offline RL learning without introducing additional regularization during their transfer. 

In detail, the online RL algorithm employed is a standard PPO. While the offline stage involves multiple components such as value functions, transition models, and ensemble BC, it is worth noting that these components are designed to be implementation-friendly, straightforward, and efficient. For the ensemble BC, we train each policy using a standard loss, incorporating an approximate KL-divergence penalty with a combined policy. This penalty term is implemented by sampling data from the offline dataset. These components remain fixed during the multi-step policy optimization stage. 
During this stage, AM-Q consists of the well-trained transition model, and the $Q$-function is used to evaluate the performance of the target policy when replacing the behavior policy. By permitting the replacement of the current behavior policy with the target policy, it achieves multi-step optimization. It differs from iteratively updating the behavior policy which leads to accumulated errors that disrupt the desired monotonicity. In contrast, the multi-step method can guarantee monotonicity due to the AM-Q error is bounded.

We'll also fully open-source our code with detailed documentation to help implement our method, including in a new real-world robot scenario.

Reviewer comment (Evaluation):
What are the computational overheads associated with the ensemble policies, and how do they impact the method's scalability?

Author response:
**A2:** Thank you for this suggestion. We've conducted the computational overheads and the performance associated with the ensemble size of policies in Section 5.3 and **Appendix A.11**. We found that the ensemble size $4$ is a trade-off between performance and the computational overheads. Specifically, the whole training time of the offline phase is around 265 minutes (ensemble size $4$) vs. 200 minutes (single policy) based on Pytorch implementation. It's acceptable due to the significant performance improvement. For offline training, the primary runtime of the offline phase is attributed to the supervised learning stage. To evaluate the runtime, we conducted 2 million steps for $Q, V$ training, 0.5 million steps for ensemble policy training, and 1 million steps for transition model training. However, in practical scenarios, these training steps can be halved, resulting in reduced runtime.

Reviewer comment (Experiments):
Why don't use the ensemble approach to mitigate mismatches instead of other methods for handling the diverse behaviors in the datasets? For example, Diffusion-QL [1] demonstrates that Diffusion model can be used to learn multimodal policy.

Author response:
**A3:** Thank you for the insightful comment. Diffusion-QL [1] is a representative work that uses diffusion policies to handle the multi-modal dataset. It uses a diffusion model as the policy to predict actions and trains the policy with a behavior cloning loss and an extra state-action value term. It is a value-based training paradigm. Indeed, the diffusion policy has the capability to replace ensemble BC during the supervised stage. However, it cannot directly be optimized by the PPO objective during the multi-step policy optimization stage. This kind of policy-based algorithm needs to model the policy's distribution explicitly and rely on differentiating through the policy distribution to compute gradients and perform updates. It is not suitable for optimizing this kind of generative model directly. 

Additionally, we have added diffusion-QL as a baseline for comparison. The performance of Uni-O4 significantly outperforms the Diffusion-QL over several domain tasks. Moreover, Uni-O4 is more computationally efficient than Diffusion-QL, consuming 12.6 hours (Diffusion-QL) vs. 4.4 hours (Uni-O4).
| Environment/method | Diffusion-QL [1]    | Uni-O4    |
| ------------------ | ------- | ------- |
| MuJoCo locomotion average           | 88.0    | 90.7   |
| AntMaze average   | 69.6    | 74.7    |
| Adroit-Pen average    | 65.1    | 108.1    |
| kitchen average  | 69.0    | 72.3    |
| All average | *72.9* | *86.4* |

[1] Wang Z, Hunt J J, Zhou M. Diffusion policies as an expressive policy class for offline reinforcement learning[J]. arXiv preprint arXiv:2208.06193, 2022.

Reviewer comment (Writing):
Notions are confusing in this paper, especially after the overloading in Equ. (8).

Author response:
**A1:** Thank you for pointing this out. We have rewritten Section 3.2 "Multi-step policy ensemble optimization" to enhance clarity in our presentation. In Equ. 8 (7 in the new version), specifically, we overload the notion of behavior policy for introducing the iteration number. It is necessary for showcasing how Uni-O4 performs multi-step optimization by adding the iteration number as subscript $k$ in behavior policies $\pi^i_k$ where $i$ specifies the policy index in the ensemble. Furthermore, we have revised the notion of the $Q$ and $V$ -function to be consistent with IQL as $\widehat{Q_{\tau}}$ and $\widehat{V_{\tau}}$ from $Q_{\pi_{\beta}}$ and $V_{\pi_{\beta}}$. Because we found that the previous version ($Q_{\pi_{\beta}}$ and $V_{\pi_{\beta}}$) will cause ambiguity. On the other hand, IQL has the capability to reconstruct the optimal value function, i.e., $\lim_{\tau \rightarrow 1} Q_{\tau} (s, a) = Q^{\ast} (s, a)$  based on dataset support constraints [1]. In this work, we exploit the desirable property to facilitate multi-step policy optimization and recover the optimal policy. $Q_{\tau}$ and $V_{\tau}$ are the optimal solution. We use $\widehat{Q_{\tau}}$ and $\widehat{V_{\tau}}$ to denote the value functions obtained through gradient-based optimization.

Reviewer comment (Presentation):
In Fig.2, it is hard to capture the Offline Multi-Step Optimization process, i.e. the sequence relationship of each step.

Author response:
**A2:** Based on this comment, we have updated Fig. 2 by incorporating sequence numbers and a detailed description for each step. During the offline multi-step optimization stage, the policy optimization begins with the learned behavior policies. Specifically, each policy undergoes optimization using the PPO loss for a designated number of gradient steps outlined in step 1 of Fig. 2. Subsequently, both the target policy and behavior policy are assessed using AM-Q to determine whether the behavior policy should be replaced by the target policy, as specified in step 2. If the evaluation results meet the OPE conditions, the behavior policy is substituted with its target policy, denoted as step 3. These three steps are iterated until the predetermined number of gradient steps is reached. Also, we sincerely invite the reviewer to check the GIF of Fig. 2 on our website for a clearer presentation.

Reviewer comment (Theory):
In Sec 3.1: "BPPO leads to a mismatch ... due to the presence of diverse behavior policies in the dataset D", could authors explain further why the diversity is blamed for the mismatch?

Author response:
**A3:** One example that can illustrate this motivation is the presence of multi-modality within a diverse dataset. In other words, the dataset exhibits multiple modes, with the main mode primarily composed of low-return actions. Conversely, the subdominant mode consists of low-density but high-return actions. In such a scenario, standard behavior cloning (BC) is susceptible to imitating the high-density but low-return actions, resulting in a bias towards fitting the main mode. However, during the offline multi-step optimization stage, the policy optimization is constrained by the clip function, making it difficult for the policy to escape this mode. Consequently, this can lead to a sub-optimal policy as it becomes unable to explore the high-return action region.

In contrast, our ensemble BC approach learns diverse behavior policies that are more likely to cover all modes present in the dataset. This facilitates exploration of the high-return region, enabling the discovery of the optimal policy. To support this claim, we have included a motivating example and conducted additional ablation studies in **Appendix A.5**. These findings demonstrate that the utilization of multiple diverse behavior policies allows for effective exploration of the high-return action region, particularly in the case of the "medium-replay" dataset, which contains more sub-optimal data than others.

Reviewer comment (Theory):
Lack of theoretical analysis (to support the motivation of technique details), but it has sufficient experiments thus this point is acceptable I think.

Author response:
**A4:** Thank you for this suggestion. We have reevaluated the approximation employed throughout the entire pipeline and included a thorough analysis to address this concern. During the multi-step policy optimization stage, the decision for behavior policy replacement is based on the proposed AM-Q metric, which incorporates the true transition model $T$ and optimal value function $Q^*$. In the offline setting, however, the agent does not have access to the true model and $Q^*$. But $Q_{\tau}$ approaches $Q^*$ based on the dataset support constraints. We fit $\hat{T}$ and $\widehat{Q_{\tau}}$ by gradient-based optimization. In this way, the practical AM-Q can be expressed as $\widehat{J_{\tau}}(\pi)$. Thus, the OPE bias is mainly coming from the transition model approximation. We have analyzed and derived the bound of the offline policy evaluation, described by Theorem 2 in Section 3.2, please refer to check out the detail.

BPPO [2] have derived the offline monotonical improvement bound (their Theorem 3) between the target policy and behavior policy, i.e., $\pi_{k+1}$ and $\pi_k$. However, since the behavior policy is updated iteratively, this bound can result in cumulative errors that disrupt the desired monotonicity. Consequently, this bound cannot be directly utilized in the offline setting. This limitation is the reason why BPPO requires online evaluation to ensure performance improvement when replacing the behavior policy with the target policy. Given the OPE bound, we can replace the online evaluation with AM-Q to guarantee monotonicity.

Additionally, we compute the advantage function by $(\widehat{Q_{\tau}}-\widehat{V_{\tau}})$, which provides an approximation of the optimal $Q^*$ and $V^*$ based on the dataset constraint assumption. In **Appendix A.8**, we conduct experiments to demonstrate this replacement is a superior choice compared to iteratively fitting the value function of the target policy in the offline setting. The results from these experiments prove that iteratively updating the value function will lead to overestimation which causes unstable training performance and potential crashes.

Reviewer comment (Presentation):
Suggest to add legends for Fig. 3 or bringing the legend in Fig. 4 forward.

Author response:
**A5:** Thank you for pointing this out. We have addressed this issue in the updated manuscript.