Paper: SERA: Sample Efficient Reward Augmentation in offline-to-online Reinforcement Learning (ICLR-2024)

Reviewer comment (Presentation):
From Fig. 1, it seems that CQL-SERA > Cal-QL-SERA > Cal-QL baseline > CQL baseline, which contradicts to the empirical findings in Fig. 4.

Author response:
$Q1:$ From Fig. 1, it seems that CQL-SERA > Cal-QL-SERA > Cal-QL baseline > CQL baseline, which contradicts to the empirical findings in Fig. 4.

$A1:$ We thank the reviewer's question. Firstly, this is a schematic diagram, and its primary purpose is to illustrate the training phase of SERA. Additionally, we will adjust the positions of the curves based on the final average experimental results.

Reviewer comment (Novelty):
Please conduct sufficient research investigation on offline-to-online RL. A lot of related works are not appropriately referenced:

[1] Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble, CoRL’22.  
[2] Adaptive policy learning for offline-to-online reinforcement learning, AAAI’23.  
[3] Policy Expansion for Bridging Offline-to-Online Reinforcement Learning, ICLR’23.  
[4] Sample Efficient Offline-to-Online Reinforcement Learning, TKDE’23.  
[5] Actor-Critic Alignment for Offline-to-Online Reinforcement Learning, ICML’23.  
[6] Fine-tuning offline policies with optimistic action selection, NeurIPS workshop.  
[7] A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning, arXiv preprint.  
[8] PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning, arXiv preprint.  
[9] Ensemble-based Offline-to-Online Reinforcement Learning: From Pessimistic Learning to Optimistic Exploration, arXiv preprint.  
[10] Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness, arXiv preprint.

Author response:
$Q3:$ Please conduct sufficient research investigation on offline-to-online RL. A lot of related works are not appropriately referenced:

$A3:$ We thank reviewer for pointing out the shortcomings. The referenced papers are indeed crucial recent studies. However, due to limited space in the main text, we prefer to introduce studies most directly related to our research.  If necessary, we may choose to add a section on advances in offline-to-online learning in the supplementary material.

---

Further discussion of Q3: We supplement related work in this section, and we have added such content in the Appendix of new manuscript. 

In the past, efforts have primarily focused on enhancing offline-to-online performance from two perspectives.

The first perspective involves adopting a conservative policy optimization during online fine-tuning, typically achieved through the incorporation of policy constraints. Specifically, there are three main approaches within this category. The first approach constrains the predictions of the fine-tuning policy within the scope of offline support during online fine-tuning [1]. While this method contributes to achieving stable online fine-tuning performance, it tends to lead to overly conservative policy learning, and the accuracy of the estimation of offline support also influences the effectiveness of online fine-tuning. The second approach utilizes an offline dataset to constrain policy learning [2,3,4,5]. However, the effectiveness of fine-tuning cannot be guaranteed if the dataset quality is poor. This method is sensitive to the quality of the dataset. The third approach employs pre-trained policies to constrain online fine-tuning, but this paradigm is influenced by the quality of the pre-trained policy [6,7].

The second perspective involves adopting a conservative approach during offline training, specifically using pessimistic constraints to learn Q to avoid OOD (Out-of-Distribution) issues. Research in this category primarily includes: Learning a conservative Q during offline pretraining and employing an appropriate experience replay method during online learning or using Q ensemble during offline pretraining to avoid OOD problems [8,10,11]. However, as this approach introduces conservative constraints during critic updates, the value estimates between offline and online are not aligned, leading to a decrease in performance during early online fine-tuning. Therefore, Cal-QL introduces a calibrated conservative term to ensure standard online fine-tuning [9].

Addtionally, there are also some other methods, such that ODT[12] combined sequence modeling with Goal conditioned RL to conduct offline-to-online RL.

reference:

(Policy constrain)

[1] Supported Policy Optimization for Offline Reinforcement Learning. (SPOT)

[2] AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. (AWAC)

[3] Offline Reinforcement Learning with Implicit Q-Learning. (IQL)

[4] The In-Sample Softmax for Offline Reinforcement Learning. (InAC)

[5] ine-Tuning Offline Policies With Optimistic Action Selection. (O3F)

[6] Policy Expansion for Bridging Offline-to-Online Reinforcement Learning. (PEX)

[7] Actor-Critic Alignment for Offline-to-Online Reinforcement Learning. (ACA)

(Pessimitic critic)

[8] Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble .

[9] Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning.

[10] Mildly conservative q-learning for offline reinforcement learning.

[11] Confidence-conditioned value functions for offline reinforcement learning.

(Additional method)

[12] Online Decision Transformer.

Reviewer comment (Novelty):
Exploration has been discussed a lot by previous works on offline-to-online RL [3,4,6,7]. Please discuss advantages of SERA over them.

Author response:
$Q4:$ Exploration has been discussed a lot by previous works on offline-to-online RL [3,4,6,7]. Please discuss advantages of SERA over them.

$A4:$ From the experimental results (average results on gym-mujoco (medium, medium-replay) and antmaze), our method outperforms PEX[3], BR[.1], APL[.2], SUNG[7]

| CQL+APL | CQL+PEX | CQL+BR |  CQL+SUNG|CQL+SERA
|:---------:| :---------:|:---------:|:---------:|:---------:|
|56.2 |27.6  |50.4|82.4 |83.8|

Next, we will compare SERA with [3, 4, 6, 7] in terms of methods and theories:

- Compared to PEX[3], SERA does not require training an additional policy, and experimental results show that SERA outperforms PEX.

- Compared to [4], SERA does not incorporate meta-adaptation into the framework. Instead, it focuses solely on enhancing offline-to-online performance from the perspective of exploration. Therefore, in theoretical terms, this represents an essential and unique approach.

- SERA and [6] represent two completely opposite approaches. SERA adopts an aggressive exploration strategy during online training, while [6] aims to explore only slightly beyond the replay buffer during online fine-tuning. Therefore, intuitively, SERA can collect a more diverse dataset more quickly.

- SERA and SUNG[7] share a commonality in that SUNG tends to favor optimistic exploration strategies, while SERA also encourages exploration by the agent. The difference lies in the fact that SERA does not require additional training, and SUNG's VAE may be influenced by data quality during pretraining. Additionally, experimental results show that SERA slightly outperforms SUNG

Reviewer comment (Writing):
In Section 3.1, in $\mathcal{B}_{\mathcal{M}}^{\pi}Q(s,a)$, the condition of the expectation is $s \sim \mathcal{D}$?

Author response:
$Q3:$In $\mathcal{B}_{M}^{\pi}Q(s,a)$, the condition of the expectation is $s\sim \mathcal{D}$?

$A3:$$\mathcal{B}$ is bellman operator, and $\mathcal{B}_{M}^{\pi}Q(s,a)=r(s,a)+\gamma\times Q(s',\pi(s'))$ where $(s,a,s')\sim \mathcal{D}$

Reviewer comment (Writing):
In Section 3.2, overall, the details are hard to follow. Please provide step-by-step instructions in Appendix to make it more clear.

Author response:
$Q4:$ Overall, I cannot follow details in Section 3.2. Please provide step-by-step instructions in Appendix to make it more clear.

$A4:$ The latest version has reorganized Section 3.2 in the main text and provided relevant proofs

Reviewer comment (Novelty):
In Section 4.1, isn’t SERA a generic offline-to-online RL algorithm? Why is the training objective constrained to the framework of CQL and Cal-QL?

Author response:
$Q5:$In Section 4.1, Isn’t SERA a generic offline-to-online RL algorithm? Why the training objective is constrained to the framework of CQL and Cal-QL?
Moreover, this paper claims to have an appendix pdf, but I cannot find the appendix in openreview.
Why experiments are only conducted on 8 selected tasks. In general, MuJoCo has random/medium/medium-replay/medium-expert/etc. datasets. Consider these settings.
It seems that there are only one random seed throughout the paper. Please repeat all the experiments with at least three different random seeds to control the randomness. Also, please report the mean and std value.

$A5:$ Firstly, our theorem 4.1 proves that SERA can ensure the policy optimization of soft Q, so we tend to test it on algorithms based on soft Q, thus we choose Cal-QL and CQL to conduct the test. Secondly, in Figure 4, we tested the performance of SERA combined with various model-free algorithms (TD3+BC, AWAC, IQL), and the improvement brought by SEAR is particularly evident for AWAC. Additionally, we used AWAC as the base algorithm, and in Figure 5, we compared the performance differences between various exploration algorithms (RND, SE) and SEAR. SERA performs the best overall. Additionally, the latest version we submitted has added an appendix and expanded the D4RL experiments to $\texttt{medium-replay}$.

Reviewer comment (Reproducibility):
This paper claims to have an appendix pdf, but the appendix is not found in OpenReview.

Author response:
$Q9:$ Moreover, this paper claims to have an appendix pdf, but I cannot find the appendix in openreview.

The supplementary material has been added after the main text. Specifically, 

- In Appendix B, we provide some theoretical proofs relevant to this paper. Specifically,  Appendix B.1 explains the connection between State entropy maximization and exploration. Appendix B.2 contains Theorems 4.1 and 4.2. Theorem 4.1 ensures policy improvement in Soft-Q algorithms when combined with SERA.

- In Appendix C$\sim$D, we have systematically organized the implementation details of SERA, including implementation, computing resources and hyperparameters.

- In Appendix E, We have added the performance of the algorithm on medium-replay, and further comparisons were made between SERA and various efficient offline-to-online algorithms.

Reviewer comment (Experiments):
Why experiments are only conducted on 8 selected tasks? In general, MuJoCo has random/medium/medium-replay/medium-expert/etc. datasets. Consider these settings.

Author response:
$Q10:$ Why experiments are only conducted on 8 selected tasks. In general, MuJoCo has random/medium/medium-replay/medium-expert/etc. datasets. Consider these settings.

$A10:$ We have supplemented the experimental results for medium-replay in Figure 1 (training curve) and Table 9 (fine-tuned results).

Reviewer comment (Experiments):
It seems that there is only one random seed used throughout the paper. Please repeat all experiments with at least three different random seeds to control randomness and report the mean and std values.

Author response:
$Q1:$ It seems that there are only one random seed throughout the paper. Please repeat all the experiments with at least three different random seeds to control the randomness. Also, please report the mean and std value.

$A1:$ In Table 1, we report results from at least three repeated experiments, with multiple runs tested for each repetition. The new experimental results include both the mean and variance. To facilitate simultaneous presentation, we have included the results below.

|Task Name| CQL | CQL+SERA | CalQL | CalQL+SERA
|:---------:|:---------:| :---------:|:---------:|:---------:|
| antmaze-large-diverse |89.2  |89.8$\pm$3.2|86.3$\pm$0.2|94.5$\pm$1.7|
| antmaze-large-play    |91.7  |&92.6$\pm$ 1.3|83.3$\pm$9.0|95.0$\pm$1.1|
| antmaze-medium-diverse|89.6 |98.9$\pm$0.2|96.8$\pm$1.0 |99.6$\pm$0.1|
| antmaze-medium-play   |97.7 |99.4$\pm$0.4|95.8$\pm$0.9|	98.9$\pm$0.6|
| halfcheetah-meidum    | 69.9  |87.9$\pm$2.3|45.6$\pm$0.0|46.9$\pm$0.0|
| walker2d-medium       |123.1 |130.0$\pm$0.0 |80.3$\pm$0.4 |90.0$\pm$3.6|
| hopper-medium         |56.4|62.4$\pm$ 1.3|55.8$\pm$0.7|61.7$\pm$2.6|
| ant-medium            |123.8|136.9$\pm$1.6|96.4$\pm$0.3|104.2$\pm$3.0|

Additionally, in Figure 3, we refer to [1] to conduct a statistical analysis of Table 1, and the improvement brought by SERA is statistically significant.

Reviewer comment (Experiments):
Please consider more sufficient comparison in Fig. 5. Besides, in ant-medium, where is TD3+BC? In ant, halfcheetah, and walker2d, IQL seems to perform better than IQL-SERA. Could you provide more explanations?

Author response:
$Q6:$ Please consider more sufficient comparison in Fig. 5. Besides, in ant-medium, where is TD3+BC? In ant, halfcheetah, and walker2d, IQL seems performs better than IQL-SERA. Could you provide more explanations?

$A6:$ Firstly, IQL combined with SERA generally achieves slightly better performance in most cases. We provide the reason for the poor performance of IQL on $\texttt{walker2d-medium}$. Beta in IQL is a crucial hyperparameter, where a small beta makes the training target close to behavioral cloning, and a large beta makes the training target close to Q learning. However, when testing SERA, we did not intentionally adjust the hyperparameters of various algorithms. Therefore, a too-small beta may limit the exploratory nature of SERA, thereby restricting the potential improvement in sample efficiency brought by SERA.

Reviewer comment (Experiments):
Why are only two tasks selected in Fig. 6(a)?

Author response:
$Q13:$ Why only two tasks are selected in Fig. 6 (a)?

$A13:$ We thank the reviewer's questions. Figure 6(a) serves as further verification of whether SERA can be applied to soft-Q-based algorithms (theorem 4.1 has the guarantee of soft-Q optimization). We have conducted extensive experiments based on CQL-SAC and Cal-QL, which demonstrate that SERA can be used in conjunction with Soft-Q.

Reviewer comment (Experiments):
Why is only IQL selected in Fig. 6(b) and only on two tasks?

Author response:
$Q14:$ Why only IQL is selected in Fig. 6 (b) on only two tasks?

$A14:$ In the new version, we have added AWAC and compared the performance of IQL and AWAC when combined with RND, SE, and SERA on these two tasks. SERA performs the best.

Reviewer comment (Writing):
Some typos:  
(1) Reference format is not well-handled throughout the paper. In ICLR template: xxx (Author, et al., Year)  
(2) On page 2: by maximizing no-conditioned -> non-conditioned; Anther reason -> Another.  
(3) On page 3: some researches penalty the -> penalize; both offline and online RL., we -> delete the comma; improving Model-free offline-to-online RL -> model-free.  
(4) On page 4: given N i.i.d samples -> $N$; consists of samples -> revise this sentence; add , in Eq.(4); Equation. 4 -> Equation 4.  
(5) On page 5: params -> parameters.  
(6) On page 6: Differing -> Different.

Author response:
$Q9:$ Some typos $\cdots$

$A9:$ Thanks for the reviewer's suggestions. We have submitted a new version of the manuscript and will systematically address the details based on the writing advice from the reviewers.

---

$Q16:$ Some typos: $\cdots$

$A16:$ We thank reviewers for pointing out these shortcomings. Your feedback is very helpful in improving our writing.

Reviewer comment (Writing):
The writing is not good enough, with many sentences that are not rigorous or are confusing. For example: In the first paragraph, "such paradigm can only learn similar or slightly better performance than behavioural policy" is not true, because model-based offline RL methods can sometimes significantly improve the performance with respect to the behavioural policy.

Author response:
$Q1:$ Firstly, the writing is not good enough. Many sentences are not rigorous or confusing. ;$Q2:$ There are too many typos and grammar errors

$Q1,Q2:$ We thank Reviewer for the series of writing suggestions. In the latest version we submitted, we have systematically organized the entire writing and we will further improve the expression of this paper.

Reviewer comment (Experiments):
There are some missing SOTA baselines for offline-to-online fine-tuning in the experiments: Reincarnating RL [1], PEX [2], InAC [3].  
[1] (Agarwal et al., NeurIPS' 22) Reincarnating reinforcement learning: Reusing prior computation to accelerate progress  
[2] (Zhang et al., ICLR' 23) Policy Expansion for Bridging Offline-to-Online Reinforcement Learning  
[3] (Xiao et al., ICLR' 23) The In-Sample Softmax for Offline Reinforcement Learning

Author response:
$Q3:$ There are some missing SOTA baselines for offline-to-online fine-tuning in the experiments.

$A3:$ We thank Reviewer's suggestion to add new comparison experiments (Compared with previous efficient algorithms). As shown in Figure 7 (new manuscript) and Table 10 (new manuscript) in the Appendix, CQL-SERA outperforms CQL-APL, CQL-PEX, and CQL-BR on (antmaze, and gym-mujoco (medium, medium-replay) domains). At the same time, for convenience, we have provided the average performance of the compared algorithms on all tasks below:

| CQL+APL | CQL+PEX | CQL+BR | CQL+SERA
|:---------:| :---------:|:---------:|:---------:|
|56.2 |27.6  |50.4|83.8 |

Reviewer comment (Theory):
The phrase "which is unbiased in the early online process" needs clarification — why is it unbiased?

Author response:
$Q1:$ "which is unbiased in the early online process" => why it's unbiased?

$A1:$  We thank the essiential question of Reviewer, and we would like to further clarify our use of 'un-biased' in reviwer's comments.  Such 'un-biased' are not related to the concept of distribution shift between offline and online. Instead, such 'un-based' are specifically referring to whether the Q used in calculating the state entropy of the Q condition is pre-trained. In VCSE [.1], the authors considered a purely online setting and used a value network with random initialization to compute the condition of value-conditioned entropy. However, a value network with random initialization may not accurately estimate the values of all states in the early stages of online training. Consequently, when calculating the value-conditioned state entropy, there may be situations where states with different true values correspond to the same output from the value network. 

Therefore, we use a value network with random initialization to calculate the value-conditioned entropy as a 'biased' prediction in the early stages of online training. Such biased prediction doesn't align with the original intention of VCSE because low-value states and high-value states may be assigned the same value condition in the early stages of online training, it could result in states from low-value regions being included in high-value regions, or states from high-value regions being included in low-value regions.

However, we can't immediately conclude that an offline pre-trained Q network is necessarily better than a randomly initialized Q network. There is a distribution shift issue between offline and online. Therefore, we chose AWAC as the base algorithm and combined with SERA (pretrained Q condition), SERA (from-scratch Q condition) to test on various offline-to-online tasks such as $\texttt{ant-medium}$, $\texttt{hopper-medium}$, $\texttt{walker-medium}$, and $\texttt{halfcheetah-medium}$. As shown in Figure 10 (Appendix F.2), using an offline-pretrained Q as a condition performs better compared to using a randomly initialized Q as a condition and consistently outperforms the baseline. In particular, using a randomly initialized Q as a condition even achieves lower performance than the baseline in $\texttt{walker-medium}$. These experimental results all demonstrate the advantage of using a pre-trained Q network when computing conditions

Reviewer comment (Experiments):
Since the main argument of this work is a new exploration method for fine-tuning offline RL agents, it should compare to other intrinsic reward baselines, i.e., state entropy, RND, ICM.

Author response:
$Q2:$ Since the main argument of this work is a new exploration method for fine-tuning offline RL agents. I think it should compare to other intrinsic reward baselines, i.e, state entropy, RND, ICM.

$A2:$ Thanks for the valuable suggestions from the reviewers. We chose AWAC and IQL as base algorithms, combined with SERA, RND, and state entropy (SE), and conducted offline-to-online testing on $\texttt{walker-medium}$ and $\texttt{hopper-medium}$. As shown in Figure 5 (b), we found that the algorithm combined with SERA performs the best.

[.1] Dongyoung Kim, et al. Accelerating reinforcement learning with value-conditional state entropy exploration, 2023

Reviewer comment (Theory):
I do not understand how to calculate Equation (3) in practice when the state is high dimensional continuous variables. Can the authors provide the analysis?

Author response:
Thanks for reviewer's question. Firstly, although this formula is in discrete form, it is independent of whether the variables are continuous or not. Additionally, this formula calculates entropy for a batch and does not cover all samples. However, with a sufficient number of training iterations, it is equivalent to maximizing the state entropy over the entire space.

Reviewer comment (Theory):
What is the relation between Equation (2) and Equation (4) when calculating the critic-conditioned intrinsic reward?

Author response:
Thanks for the reviewer's suggestions. Firstly, we have removed Equation 2 in the main text and systematically defined the concept and mathematical form of state entropy under the joint probability distribution (new version manuscript, Definition 2). We have also provided the implementation form of Q-conditioned state entropy (new version manuscript, Equation 2). In addition, based on the initial version, Formula 2 in the initial version cannot directly calculate Q conditioned state entropy. The initial version manuscript adopts the same method as the current version to calculate Q-conditioned state entropy, which is same as Equation 2 in new version manuscript.

Reviewer comment (Experiments):
Every intrinsic reward calculation must be calculated by KNN, so the efficiency of physic time consumption may be a little poor.

Author response:
We thank the reviewer's question. Firstly, the main advantage of mathematically approximating entropy is that it does not require training. Secondly, in the new version, we have supplemented the calculation method based on VAE (Equation 21). We plan to compare the speed and cost between the VAE-based and mathematical methods in the future.

Reviewer comment (Presentation):
I can not find the appendix mentioned in this paper.

Author response:
The current version includes the supplementary material directly appended to the end of the main text. (introduction see Response to Reviewer NgYG (Point by point manner, Part 2/3), Q9)

Reviewer comment (Presentation):
The format of some citations is wrong.

Author response:
We thank the reviewer for pointing out the shortcomings of writing. The current version of the writing has seen significant improvement compared to the initial version. The suggestions from the reviewers have been instrumental in enhancing the quality of the manuscript.

Reviewer comment (Reproducibility):
What about the random seed in the experiments?

Author response:
Thanks, we have presented the mean and variance of the final performance for each task in Table 1 (new version), we have included the results below (see: Response to Reviewer NgYG (Part 2: New extended and ablation studies )).

Reviewer comment (Writing):
The authors should explain all symbols that appear in this paper, e.g., in Section 3.1, the authors do not introduce $d_D(.|s)$ and $\mathcal{G}_{\mathcal{M}}$.

Author response:
We thank suggestions provided by the reviewer, and we believe these questions and recommendations will further refine our definitions and writing. In the new manuscript, we have organized and redefined several concepts essential for understanding this paper. Some symbols and previously defined content have been removed, but we still provide explanations for certain symbols used previously. $\mathcal{G}$ means the greedy operator,ie, $\pi \leftarrow argmax_{\pi} E_{s\sim D}[Q(\pi(s),s)]$, and $d(\cdot|s)$ denotes state-marginal distribution in dataset $D$.

Reviewer comment (Theory):
In Definition 1, why the critic-conditioned entropy does not contain “-”. Besides, if there are N states that are used for calculating the critic conditioned entropy,

Author response:
We thank suggestions provided by the reviewer, and we believe these questions and recommendations will further refine our definitions and writing. In the new manuscript, we have organized and redefined several concepts essential for understanding this paper. Some symbols and previously defined content have been removed, but we still provide explanations for certain symbols used previously. $\mathcal{G}$ means the greedy operator,ie, $\pi \leftarrow argmax_{\pi} E_{s\sim D}[Q(\pi(s),s)]$, and $d(\cdot|s)$ denotes state-marginal distribution in dataset $D$.

Reviewer comment (Theory):
The authors should provide the derivation about “Another reason is that maximizing Es∼ρ(s)[Hπ[s]] is equivalent to minimize DKL(ρπ(s)||p∗(s)) thus has the mathematical guarantee.”

Author response:
Good questions! In the new version paper, we have systematically defined and demonstrated that maximizing entropy serves two main purposes (Appendix.B.1): (1) Maximizing state entropy encourages the agent to explore the entire state space, enabling the acquisition of states not initially included in the dataset through increased exploration. (2) Maximizing state entropy establishes a trade-off, where one aspect aids in approximating any target density. Furthermore, our Theorem 4.1 ensures that, when combined with SERA in soft Q algorithms, policy improvement is guaranteed. Certainly, we acknowledge that in the past, our expression regarding this matter may have been too affirmative. Merely maximizing entropy cannot guarantee the absolute achievement of SMM (State Marginal Matching). Achieving SMM in an absolute sense requires combining term 3, ie, $\max E_{s\sim \rho_{\pi}}[\log p^*(s)]$. This term can offset term1 in the Appendix.B.1 (page 16, Equation.7), thereby achieving SMM.

Reviewer comment (Writing):
Section 3.1 “Model-free Offline RL”: “In particular, Model-free” -> “In particular, model-free”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 3.1 “Model-free Offline RL”: “Specifically, Model-free” -> “Specifically, model-free”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 3.1 “Model-free Offline RL”: “one step bellman equation i.e. …. which” -> “one step bellman equation, i.e. xxxx, which”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 3.1 “Model-free Offline RL”: “Previous studies have extensively studied such a problem, such that CQL was proposed to penalty the OOD state actions by conservative term (Equation 1), and IQL implicitly learns Q function with expected regression without explicit access to the value estimation of OOD state-actions.”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 3.1: “state entropy(Seo et al., 2021)” -> “state entropy (Seo et al., 2021)”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 3.1: “i.i.d”-> ““i.i.d.””

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 3.2: grammatical mistake: “Specifically, we first use the offline methods to …..”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 3.2: “\pi_{beta}” -> “\pi_{\beta}”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 3.2: “Equation. 4” -> “Equation (4)”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 3.2: “Only maximize” -> “Only maximizing”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 4.1: “… are the params of double Q Networks” -> “… are the parameters of double Q Networks”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Writing):
Section 4.1: “in addition to testing SERA” -> “in addition to test SERA”

Author response:
We thank the reviewers for pointing out the shortcomings in our initial manuscript. The suggestions from the reviewers are very helpful in improving the quality of our manuscript. The current version has shown significant improvement in writing compared to the initial version.

Reviewer comment (Experiments):
In the experimental section, the author conducted experiments solely on the medium dataset in MuJoCo. However, according to the consensus in the field of offline-to-online research, it is generally recommended to perform experiments on at least three types of datasets: medium, medium-replay, and medium-expert, in order to validate the effectiveness of the method.

Author response:
We thank the reviewer for the suggestion to supplement the experiments (D4RL). In the new version of the manuscript, we have added tests on "medium-replay" in D4RL. As shown in Table.9 (SERA improve CQL 21%, Cal-QL 12.2%), and CQL-SERA has the best overall performance (Compared with various baselines IQL, AWAC, TD3+BC, CQL, Cal-QL).

Reviewer comment (Experiments):
The method proposed in this paper is primarily an extension of CQL and Cal-QL. However, in the context of the offline-to-online field, the actual compared baselines are limited to AWAC and Cal-QL. It is advisable for the authors to consider comparing their method with other more efficient algorithms such as Balanced Replay[1], PEX[2], and ENOTO[3].

Author response:
Good suggestions! We have supplemented relevant tests. As shown in Figure 7 (new manuscript) and Table 10 in the appendix (new manuscript), CQL-SERA outperforms CQL-APL, CQL-PEX, and CQL-BR on (antmaze, and gym-mujoco {medium, medium-replay} domains). At the same time, for convenience, we have provided the average performance of the compared algorithms on all tasks below:

| CQL+APL | CQL+PEX | CQL+BR | CQL+SERA
|:---------:| :---------:|:---------:|:---------:|
|   56.2   |    27.6   |   50.4  |  83.8     |

Reviewer comment (Experiments):
The SERA algorithm, proposed in this paper, primarily enhances online performance by designing intrinsic rewards to encourage exploration. This concept has been mentioned in previous works such as O3F[4] and ENOTO, although SERA employs different exploration methods. While introducing exploration during the online phase can enhance performance, it may introduce another challenge: instability due to distribution shift, which can lead to performance degradation in the early stages of online learning. This issue has been discussed in many offline-to-online works and is a critical metric in this field. However, it might not be very evident on the medium dataset. Therefore, the authors should consider conducting additional experiments on the medium-replay and medium-expert datasets to verify whether performance degradation occurs.

Author response:
Nice suggestions! In question 1, we supplemented experiments related to D4RL. Additionally, in the supplementary pages, we provided mathematical proofs demonstrating that SERA ensures policy improvement during soft Q optimization (Theorem B.3 in Appendix.B.2) and encourages the agent to uniformly explore the entire observation space (Appendix.B.1 "Why does ASMM encourage covering the target density?"). Additionally, [.2] explains that diverse data can address the shortcomings of conservative policies, and SERA can encourage the agent to explore the environment as much as possible, facilitating the collection of more diverse dataset, therefore, from a theoretical perspective, SERA also helps improve various soft Q-based algorithms.

Reviewer comment (Presentation):
In Figure 4, the experimental results for the Antmaze environment are challenging to discern, as the curves for various algorithms are intertwined and unclear. The author should consider optimizing the representation of these experimental results for better clarity.

Author response:
Good suggestions! According to the reviewer's suggestion, we further optimized this figure using uniform sampling. The distinguishability between curves has significantly improved compared to the previous version. The trends of the curves remain consistent with the statements in the paper, showing that CQL-SERA and Cal-QL-SERA converge faster and achieve better fine-tuned results.

Reviewer comment (Evaluation):
In Table 1, only the mean values of the algorithm results are presented, with a lack of information regarding the errors or variances associated with these results.

Author response:
Thanks to the reviewer's suggestion, we have reported the results of experiments conducted in at least three repetitions, with multiple runs for each repetition. We have presented the mean and variance of the final performance for each task in Table 1. Additionally, we employed the statistical methods described in [.1] and further analyzed the experimental results from Table 1 in Figure 3. The results demonstrate that SERA significantly enhances the performance of CQL and Cal-QL (higher median, IQM, and mean scores are better).

[.1] Rishabh Agarwal, et.al. Deep reinforcement learning at the edge of the statistical precipice, 2022

[.2] Yicheng Luo, et.al. Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions, 2023.