Paper: PROTO: Iterative Policy Regularizied Offline-to-Online Reinforcement Learning (ICLR-2024)

Reviewer comment (Presentation):
The experiments in Figure 4 are hard to visualize, perhaps the author can change another color to demonstrate the curves (especially the `BC+PROTO+SAC`, `EQL+SAC`, `PEX+BC`).

Author response:
We thank the reviewer for this constructive comment! We have re-drawn almost all figures in our paper according to the suggestions of the reviewer.

Reviewer comment (Experiments):
Would it be ok if the author also add the Cal-QL experiments for comparison? As far as the reviewer is aware of, the Cal-QL paper also has open-sourced the [code](https://github.com/nakamotoo/Cal-QL).

Author response:
We thank the reviewer for this helpful comment. We have added the results in Figure 26 in Appendix M and the hyper-parameter details for Cal-QL in our revised paper. The comparisons show that PROTO can also outperform or obtain on-par performances as compared to Cal-QL.

Reviewer comment (Novelty):
I think this is the naive adaptation of TD3-BC [1] in offline-to-online settings, because KL regularization term results in BC objective in online settings.

Author response:
>**W1. I think this is the naive adaptation of TD3-BC in offline2online settings, because KL regularization term results in BC objective in online settings.**

- **No. PROTO is fundamentally different from TD3+BC**.
  - TD3+BC is over-conservative in offline2online settings. In TD3+BC, the BC regularization only imposes constraints on the slowly evolving data in replay buffer $\mathcal{B}$. However, relying only on $\mathcal{B}$ is too conservative to obtain near-optimal results. To address this, APL[1] and SUNG[2] proposed some adaptive pessimistic methods to reduce the conservatism during online finetuning, but they still cast constraints on the suboptimal $\mathcal{B}$.
  - In contrast, PROTO considers a gradually evolving trust region induced by $\pi_k$, which is notably superior in offline2online settings. As extensively discussed in our paper, the constraints w.r.t $\pi_k$ enable both stable and near-optimal results. See Figure 1, 2, 3, 6, 7, 8 for comprehensive details.
  - As thoroughly discussed in Appendix A, the introduction of $\pi_k$ as policy constraints represents a novel and versatile solution that tackles the challenges faced by existing offline2online methods.

>**Q2. How does PROTO conduct offline pre-training? PROTO also employs iterative policy constraint? or other algorithms (CQL, IQL, etc.)? I guess this algorithm results in TD3-BC [1] in offline settings.**

- We must clarify that PROTO is not a specific pretrain/finetune method but a **versatile component** that bridges a wide range of pretrain/finetune methods. It allows us to use almost any policy pretraining methods **without modifications**. Then, we augment actor and critic training in existing off-policy RL methods with the regularization term $\log(\pi/\pi_k)$ to create effective offline-to-online RL methods. 
For example,
  - EQL+PROTO+SAC involves using EQL for value and policy pretraining and incorporating the iterative policy regularization term $\log(\pi/\pi_k)$ from PROTO into SAC finetuning, bridging the offline2online gap; 
  - SQL+PROTO+SAC uses SQL to pretrain the policy and value networks, then we plug PROTO into SAC for online finetuning; 
  - EQL+PROTO+TD3 refers to using EQL for offline pretraining and then plugging PROTO into TD3 that uses a deterministic policy for online finetuning. Please see Appendix H for EQL+PROTO+TD3 details.
  - To the best of the authors' knowledge, achieving or evaluating such remarkable versatility has not been reported in prior studies.
- Also, PROTO and TD3+BC are fundamentally different as discussed above.

Reviewer comment (Experiments):
Similar to above, I'm not sure the difference between PROTO and PROTO-TD3 or PROTO-SAC presented in Figure 4/5. I think PROTO and TD3(-BC) is quite similar to each other. Because PROTO modifies the objective of policy and value function, it is not intuitive to incorporate with SAC.

Author response:
>**W2. Similar to above, I'm not sure the difference between PROTO and PROTO-TD3 or PROTO-SAC presented in Figure 4/5.**

- Our proposed PROTO is a versatile framework that can bridge a wide range of offline pretraining and online finetuning methods. The default PROTO refers to EQL pretrain+PROTO+SAC finetune as stated in the first paragraph in Section 4 (page 6), and it can be easily adapted to other versions by replacing different online finetuning methods, for example, PROTO+TD3 refers to EQL pretrain+PROTO+TD3 finetune. See Appendix H for PROTO+TD3 details.

Reviewer comment (Novelty):
Theorem 1 seems the same as what Vieillard et al. (2020) [2] have proven, and equation 7 is the same as what Scherrer et al. (2015) [3] have proven. I don't think there is novel and offline-to-online-specific discussion around the theorem. Please let me know if my understanding is not correct.

Author response:
>**W3. Theorem 1 seems the same as what Vieillard et al. (2020) [3] have proven, and equation 7 is the same as what Scherrer et al. (2015) [4] have proven. I don't think there is novel and offline-to-online-specific discussion around the theorem. Please let me know if my understanding is not correct.**

- No, there are some differences. Please see Appendix B for the differences between ours and [3][4].
- Specifically, [3] and [4] only study the conventional online RL settings without considering the offline pretrained policies and value functions, and thus **are not directly applicable in offline2online settings**. However, in our paper, we extend the analysis from [3] and [4] from the conventional online RL to our specific offline2online settings by introducing a mild assumption on $Q_0$ initialization and making several modifications.
- Note that we aim to provide insightful interpretations for PROTO, as stated in Appendix B, rather than aiming to develop more intricate or stringent bounds.

Reviewer comment (Novelty):
In the similar setting, [4] proposes iterative trust-region update from the offline pre-trained policy.

Author response:
>**W4. BREMEN[5] proposes iterative trust-region update from the offline pre-trained policy.**

We thank the reviewer for providing this related work and we are happy to include this paper in our revision. However, note that BREMEN and PROTO are very different and belong to different problem settings.
- BREMEN[5] focuses on the deployment-efficient RL setting which involves multiple stages of offline training and online interactions. But PROTO focuses on the offline2online RL setting. These two settings share some similarities but are not the same.
- BREMEN[5] constrains on a replay buffer $\mathcal{D}$ rather than $\pi_k$ since it periodically resets the policy using the BC policy w.r.t $\mathcal{D}$, see Eq.(3) and Algorithm 1 in [5] for details. The reason why BREMEN also uses trust-region updates is that BREMEN utilizes on-policy RL methods to train the policy using the model rollouts. As the authors stated, BREMEN actually implicitly constrains w.r.t $\mathcal{D}$, as discussed following Eq.(3) in [5]. In contrast, PROTO enforces constraints on $\pi_k$, which has been shown to be superior to the over-conservative replay buffer based constraint.
- BREMEN adopts a model-based approach and trains an ensemble of dynamics models, which can be computationally expensive. On the other hand, PROTO is model-free and extremely lightweight.

Reviewer comment (Novelty):
Because the theorem from Vieillard et al., (2020) and Scherrer et al., (2015) originally holds under any initialization (initialization of matrix) and thus offline-to-online RL has already been included in online RL, this paper does not have any novel theorem or statement. Insightful interpretations for PROTO do not exist. The same statements from Vieillard et al., (2020) and Scherrer et al., (2015) are repeated in the main text.

Author response:
>**4. Theoretical Analysis**

>**the theorem from Vieillard et al., (2020) and Scherrer et al., (2015) originally holds under any initialization and thus offline-to-online RL has already been included in online RL, this paper does not have any novel theorem or statement...I express strong concerns that the paper has the same theorem as the previous papers without deriving a novel theorem. Moreover, if the important discussion is only discussed in Appendix, that is very weird.**

The reviewer's primary concern appears to revolve around our theoretical contributions.
- First, we must clarify that **we never argued to provide some entirely new or even tighter bounds than previous works**, as clearly stated in our paper. We have revised our paper to make it more clear.
- Instead, we only argued that **we can extend some existing theories from online to our specific offline-to-online setting** to compare different regularizations and bring some theoretical insights. This is clearly stated in the paragraph preceding Eq. (6) in the main text and Appendix B. 
  - We have discussed these previous theories with proper context, citations, and did not claim to introduce entirely novel theoretical contributions in our paper, as we focus more on providing a strong, simple, versatile and efficient offline-to-online framework rather than presenting a theoretical study to provide lots of performance bounds. In this sense, we used existing theories to analyze the proposed methods. To clarify this, we revised our paper to call them Lemma rather than Theorem.
- Furthermore, we respectfully disagree that our extension to the offline-to-online setting is trivial.
  - While previous theories assume the $Q_0$ under any initialization, which is versatile and implicitly includes offline-to-online RL settings, we specifically consider such theory in offline-to-online RL settings by introducing one additional Q assumption that the offline pretrained policy should be obtained by one step policy improvement step upon the randomly-initialized $Q_0$. This aspect has not been explored in previous works.
- Moreover, note that PROTO offers a versatile framework that is capable of bridging diverse offline/online RL methods. In this sense, it becomes pretty **challenging for anyone to provide further detailed theoretical analysis**. Since different instantiation choices will introduce different theoretical properties. To address this, we have empirically verified the versatility of PROTO by extensive experiments in Figure 4, 5.
- Finally, the discussions in Appendix B primarily aim to explain how we can apply previous theories to our offline-to-online settings. We left this content in the Appendix to avoid overwhelming readers with these detailed explanations and due to the space limits. We provided many links to Appendix B in the main text to guide the readers to read these explanations.

**In summary, we respectfully disagree that the theoretical aspects of our paper are over-claimed.**

Reviewer comment (Novelty):
I express strong concerns that the paper has the same theorem as the previous papers without deriving a novel theorem. Moreover, if the important discussion is only discussed in Appendix, that is very weird.

Author response:
>**4. Theoretical Analysis**

>**the theorem from Vieillard et al., (2020) and Scherrer et al., (2015) originally holds under any initialization and thus offline-to-online RL has already been included in online RL, this paper does not have any novel theorem or statement...I express strong concerns that the paper has the same theorem as the previous papers without deriving a novel theorem. Moreover, if the important discussion is only discussed in Appendix, that is very weird.**

The reviewer's primary concern appears to revolve around our theoretical contributions.
- First, we must clarify that **we never argued to provide some entirely new or even tighter bounds than previous works**, as clearly stated in our paper. We have revised our paper to make it more clear.
- Instead, we only argued that **we can extend some existing theories from online to our specific offline-to-online setting** to compare different regularizations and bring some theoretical insights. This is clearly stated in the paragraph preceding Eq. (6) in the main text and Appendix B. 
  - We have discussed these previous theories with proper context, citations, and did not claim to introduce entirely novel theoretical contributions in our paper, as we focus more on providing a strong, simple, versatile and efficient offline-to-online framework rather than presenting a theoretical study to provide lots of performance bounds. In this sense, we used existing theories to analyze the proposed methods. To clarify this, we revised our paper to call them Lemma rather than Theorem.
- Furthermore, we respectfully disagree that our extension to the offline-to-online setting is trivial.
  - While previous theories assume the $Q_0$ under any initialization, which is versatile and implicitly includes offline-to-online RL settings, we specifically consider such theory in offline-to-online RL settings by introducing one additional Q assumption that the offline pretrained policy should be obtained by one step policy improvement step upon the randomly-initialized $Q_0$. This aspect has not been explored in previous works.
- Moreover, note that PROTO offers a versatile framework that is capable of bridging diverse offline/online RL methods. In this sense, it becomes pretty **challenging for anyone to provide further detailed theoretical analysis**. Since different instantiation choices will introduce different theoretical properties. To address this, we have empirically verified the versatility of PROTO by extensive experiments in Figure 4, 5.
- Finally, the discussions in Appendix B primarily aim to explain how we can apply previous theories to our offline-to-online settings. We left this content in the Appendix to avoid overwhelming readers with these detailed explanations and due to the space limits. We provided many links to Appendix B in the main text to guide the readers to read these explanations.

**In summary, we respectfully disagree that the theoretical aspects of our paper are over-claimed.**

Reviewer comment (Experiments):
I have some concerns regarding Polyak averaging. Is the initial policy $\bar{\pi}_0$ obtained by the offline pretraining? If so, does Polyak averaging potentially lead to PROTO deviating very slowly from the offline pretrained policy since the parameter is a very small value ($\tau=5e-5,5e-3$)? Additionally, it seems that the role of Polyak averaging might **overlap** with the addition of the KL term in Equation 3. Therefore, I believe that ablation experiments for this parameter are not sufficient. An experiment without this parameter should be conducted.

Author response:
>**W1.1 Is the initial policy $\pi_0$ obtained by the offline pretraining?**

Yes.

>**W1.2 If so, does a small Polyak averaging (tau=5e−5,5e−3) potentially lead to PROTO deviating very slowly from the offline pretrained policy?**
- No. Figure 8 shows that PROTO deviates a lot from $\pi_0$ and even obtains a similar deviation degree compared to Off2On, which poses no regularization. 
- Also, we added quantitative investigations on the distributional shift degrees in Figure 23, Appendix K in our revised paper. The results show that PROTO with a very small $\tau$ can still deviate fast from $\pi_0$. 

>**W1.3 The role of Polyak averaging might overlap with the addition of the KL term in Equation 3**
- No. The polyak averaging trick is only utilized to provide a stabilized trust region around $\bar\pi_k$, but is not used to directly slow down the policy updates. In that sense, the polyak averaging assists the KL term by smoothing the potential rapid evolution of the trust region.
- However, we believe the reviewer makes an excellent idea and insightful observation! Using the polyak averaging trick to directly slow down the policy updates may serve as some kind of implicit policy constraint, while the KL term in Eq.(3) performs as an explicit constraint. Therefore, it would be interesting if we could combine them together to develop more advanced and versatile offline2online RL methods.

>**W2.1 An experiment without polyak averaging should be conducted.**

We thank the reviewer for this helpful comment. 

- We added new results for $\tau=0.1$ and no polyak averaging (No $\tau$) in Figure 24, Appendix L in our revised paper.
  - Figure 24 shows that PROTO (No $\tau$) can obtain better results on offline datasets that have good coverage such as medium-replay and random datasets since a good data coverage can alleviate the instability and No $\tau$ can enhance the optimality. 
  - However, for datasets that have narrow data coverage such as medium datasets, PROTO (No $\tau$) may undergo some initial performance drop without polyak averaging. 
  - Nevertheless, with a stabilized $\bar\pi_k$ induced by polyak average, PROTO ($\tau=5e-3$) can consistently achieve stable and near-optimal finetuning results, achieving a robust trade-off between stability and optimality across all evaluated tasks.

Reviewer comment (Reproducibility):
Does the update process of EQL+SAC make use of this parameter $\tau$? If not, could this parameter potentially help prevent training collapse in EQL+SAC?

Author response:
>**W2.2 Does the update process of EQL+SAC make use of $\tau$?**
- No, the polyak averaging trick is only used to obtain a stable trust region $\bar\pi_k$ to regularize the online finetuning, but not to directly slow down the policy updates. EQL+SAC has no trust region update mechanisms as it directly finetunes without any regularization.

>**W2.3 Could this parameter potentially help prevent training collapse in EQL+SAC?**

- No, we added results for EQL+SAC using polyak averaging with the same $\tau$ as PROTO in Figure 25, Appendix L in our revised paper. The results show that simply using polyak averaging without explicit KL-regularization still suffers training instability in EQL+SAC. Although this attempt fails, we think it is an interesting direction to further explore whether we could utilize the implicit policy regularization induced by polyak averaging to further enhance the results.

Reviewer comment (Experiments):
Based on the analysis and evidence presented in the paper, I believe that PROTO's capabilities extend beyond improving methods that are already capable of offline to online finetuning. Can PROTO also cooperate with other algorithms that cannot solve the offline to online problem?

Author response:
>**W3. Can PROTO also cooperate with other algorithms that cannot solve the offline to online problem?**
- Yes. SAC and TD3 in our paper are online RL methods that cannot directly solve offline2online problems since no regularization is introduced to combat the initial performance drop during online finetuning. Figure 3, 4, 5 show that with PROTO stabilization, all PROTO+SAC and PROTO+TD3 variants obtain stable and near-optimal performances. This demonstrates the versatility of PROTO on diverse online RL methods that cannot directly obtain stable offline2online results.

Reviewer comment (Theory):
In this setting, the KL-regularized MDP is a moving target, this means the optimal policy per iteration is changing. This seems conflicts with the motivation of the approach. Could the author clarify the intuition here? I assume the \pi* is the optimal policy of original MDP (without KL-constraint).

Author response:
- Yes, the optimal policy per iteration is changing in the KL-regularized MDP, and we denote it as $\pi_k$ in our paper, which is different from the optimal policy $\pi^*$ in the original MDP. 
- However, this does not conflict with our motivation as $\pi_k$ will gradually converge to $\pi^*$ as stated in Eq.(6), which enjoys stable and near-optimal convergence compared to fixed regularization (Eq. (8)) and no regularization (Eq. (7)).

Reviewer comment (Reproducibility):
How many iterations of policies in total in practice? How many gradient steps we did one delayed updates?

Author response:
During finetuning,
- 1M policy iterations/gradient updates are performed for all PROTO+SAC-based methods, e.g. EQL+PROTO+SAC and BC+PROTO+SAC, the numbers of policy updates, value updates, and online samples for SAC are the same. 
- 0.5M policy iterations/gradient updates are performed for PROTO+TD3 since TD3 performs one policy update per two value updates.

Reviewer comment (Evaluation):
I think it is good to see that in Figure 7, there is a clear shift of distribution in terms of horizon. It might be more clear to see if there is state-action distribution shift. I am trying to understand if this iterative approach can reach OOD state-action pairs gradually, compared to the \pi_0. I think this is important because: 1) with this verification, we can then understand if the learning can be stable and gradually move to an area deviated from initial state-action coverage. 2) We are not just re-weighting the existing trajectories in the offline dataset.

Author response:
Good point! We thank the reviewer for this constructive comment. 

- We added quantitative comparisons on the distributional shift degrees of different policy regularizations in Figure 23, Appendix K in our revised paper. The results show that the iterative policy regularization method stays close to $\pi_0$ initially and can gradually deviate from the pretrained policy $\pi_0$ to explore more OOD regions. In contrast, fixed policy regularization always remains close to $\pi_0$ as it constrains w.r.t $\pi_0$. Also, without any regularization, the finetuned policy quickly shifts away from $\pi_0$, suffering potential instability.