Paper: CPPO: Continual Learning for Reinforcement Learning with Human Feedback (ICLR-2024)

Reviewer comment (Experiments):
In the main experiment to evaluate the continual learning ability of CPPO in NLP tasks, only one benchmark Reddit TL;DR summarization is considered. What’s more, the length of the continual task stream is only 2, which is not enough for a solid evaluation of continual learning ability compared to related work [4][5][6][7].

Author response:
# Comment 2.1: "The experiment part is weak.  In the main experiment to evaluate the continual learning ability of CPPO in NLP tasks, only one benchmark Reddit TL;DR summarization is considered. What’s more, the length of the continual task stream is only 2, which is not enough for a solid evaluation of continual learning ability compared to related work... "

* Response: To further validate the proposed method, we add a new benchmark for Domain-Incremental Learning (DIL) **on 18 domains, comprising 3 tasks**.
We build this benchmark based on the Stanford Human Preferences Dataset (SHP)[1] dataset. Our task design methodology can also be extended to sequences of 6, 9, or even 18 tasks. 
Here, we only present the final experimental results. The details of the SHP dataset, task design, and evaluation metrics can be referenced in the **new PDF Appendix Section H**. 


|                                   | Task 1                               | Task 2                               | Task 3                      |                                     |                                      |                                       |                                      |   |
|-----------------------------------|--------------------------------------|--------------------------------------|--------------------------------------|-------------------------------------|--------------------------------------|---------------------------------------|--------------------------------------|---|
| **Method** | **SteamSHP (↑)**                | **SteamSHP (↑)**                | **SteamSHP (↑)**                | **AA (↑)**                     | **AIA (↑)**                     | **BWT (↑)**                      | **FM (↓)**                      |
|       SFT (In order)    | 0.806 $(\pm0.0101)$          | 0.836 $(\pm0.0103)$          | 0.853 $(\pm0.0103)$          | 0.832 $(\pm0.0061)$         | 0.837 $(\pm0.0039)$          | -0.022 $(\pm0.0094)$          | 0.022 $(\pm0.0094)$          |
|    SFT (multi-tasks)      | 0.831 $(\pm0.0266)$          | 0.847 $(\pm0.0145)$          | 0.858 $(\pm0.0114)$          | 0.845 $(\pm0.0147)$         | 0.844 $(\pm0.0082)$          | -0.006 $(\pm0.0183)$          | 0.009$(\pm0.0160)$           |
|   Iterated RLHF   | 0.869 $(\pm0.0583)$          | 0.88 $(\pm0.0490)$           | 0.887 $(\pm0.0421)$          | 0.879 $(\pm0.0488)$         | 0.874 $(\pm0.0433)$          | **0.0004** $(\pm0.0186)$ | **0.003** $(\pm0.0162)$ |
|PPO| 0.853 $(\pm0.0573)$          | 0.879 $(\pm0.0428)$          | 0.889 $(\pm0.0369)$          | 0.874 $(\pm0.0433)$         | 0.877 $(\pm0.0378)$          | -0.017 $(\pm0.0351)$          | 0.020 $(\pm0.0327)$          |
|CPPO(Heuristic)| 0.864 $(\pm0.0557)$          | 0.89 $(\pm0.0448)$           | 0.894 $(\pm0.0350)$          | 0.883 $(\pm0.0429)$         | 0.887 $(\pm0.0391)$          | -0.015 $(\pm0.0321)$          | 0.018 $(\pm0.0300)$          |
| CPPO(Learn)                       | **0.872** $(\pm0.0544)$ | **0.898** $(\pm0.0450)$ | **0.899** $(\pm0.0342)$ | **0.89** $(\pm0.0424)$ | **0.894** $(\pm0.0389)$ | -0.013 $(\pm0.0298)$          | 0.016 $(\pm0.0281)$          |

Reviewer comment (Experiments):
I wonder about the performance of $M_{\pi2}$ on the Task-1 test set. The result of this experiment can show how much knowledge the model retains for the first task.

Author response:
Thank you for your helpful comments! We appreciate your comments about our main idea and the motivation behind it make sense. We address your questions below. 

# Comment:  "I wonder about the performance of 1) $M_{\pi_2}$ on the Task-1 test set and 2) $M_{\pi_1}$ on the Task-2 test set. The result of the first experiment can show how much knowledge the model retains for the first task. In addition, the second experiment is supposed to show a mediocre result to prove the two tasks have a clear difference and that the experiment setting is indeed a continual learning setting. The performance gap of $M_{\pi_1}$ and $M_{\pi_2}$ on Task-2 can reveal to what extent the two data distributions mismatch."

Response: The reference PM score of  $M_{\pi_2}$ on Task 1 can be computed by $rPM(M_{2}, \mathbb{D}_1^{test} )=rPMS_1 - SFR$. 

Here, we supplement the performance of $M_{\pi_1}$ on Task 2, comparing CPPO with PPO.


| Method   | $rPM(M_{1}, \mathbb{D}_1^{test} )$ | $rPM(M_{1}, \mathbb{D}_2^{test} )$ | $rPM(M_{2}, \mathbb{D}_1^{test} )$ | $rPM(M_{2}, \mathbb{D}_2^{test} )$ |
|----------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|
| PPO      | 2.631                              | 2.417                              | 2.550                              | 2.688                              |
| CPPO (H) | 3.021                              | 2.812                              | 3.187                              | 2.982                              |
| CPPO (L) | 3.174                              | 2.714                              | 3.337                              | 3.090                              |

Comparing the evaluations in columns 2 and 4 for $M_{\pi_1}$ and $M_{\pi_2}$ on Task-2, we observe that model $M_{\pi_1}$  significantly underperforms $M_{\pi_2}$. This indicates a substantial difference in the data distributions between the two tasks. 

Comparing the evaluations in columns 1 and 3 for $M_{\pi_1}$ and $M_{\pi_2}$ on Task-1, we observe that the policy model trained with PPO exhibits forgetting, indicating a risk of catastrophic forgetting during continually learning of Task-1 and Task-2. The model trained with CPPO demonstrates Backward Transfer (BWT) capability, specifically a negative SFR, suggesting that CPPO overcomes forgetting in this task and that learning new human preferences leads to a positive impact on retaining old preferences.

Reviewer comment (Experiments):
I wonder about the performance of $M_{\pi1}$ on the Task-2 test set. This experiment is supposed to show a mediocre result to prove the two tasks have a clear difference and that the experiment setting is indeed a continual learning setting.

Author response:
Thank you for your helpful comments! We appreciate your comments about our main idea and the motivation behind it make sense. We address your questions below. 

# Comment:  "I wonder about the performance of 1) $M_{\pi_2}$ on the Task-1 test set and 2) $M_{\pi_1}$ on the Task-2 test set. The result of the first experiment can show how much knowledge the model retains for the first task. In addition, the second experiment is supposed to show a mediocre result to prove the two tasks have a clear difference and that the experiment setting is indeed a continual learning setting. The performance gap of $M_{\pi_1}$ and $M_{\pi_2}$ on Task-2 can reveal to what extent the two data distributions mismatch."

Response: The reference PM score of  $M_{\pi_2}$ on Task 1 can be computed by $rPM(M_{2}, \mathbb{D}_1^{test} )=rPMS_1 - SFR$. 

Here, we supplement the performance of $M_{\pi_1}$ on Task 2, comparing CPPO with PPO.


| Method   | $rPM(M_{1}, \mathbb{D}_1^{test} )$ | $rPM(M_{1}, \mathbb{D}_2^{test} )$ | $rPM(M_{2}, \mathbb{D}_1^{test} )$ | $rPM(M_{2}, \mathbb{D}_2^{test} )$ |
|----------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|
| PPO      | 2.631                              | 2.417                              | 2.550                              | 2.688                              |
| CPPO (H) | 3.021                              | 2.812                              | 3.187                              | 2.982                              |
| CPPO (L) | 3.174                              | 2.714                              | 3.337                              | 3.090                              |

Comparing the evaluations in columns 2 and 4 for $M_{\pi_1}$ and $M_{\pi_2}$ on Task-2, we observe that model $M_{\pi_1}$  significantly underperforms $M_{\pi_2}$. This indicates a substantial difference in the data distributions between the two tasks. 

Comparing the evaluations in columns 1 and 3 for $M_{\pi_1}$ and $M_{\pi_2}$ on Task-1, we observe that the policy model trained with PPO exhibits forgetting, indicating a risk of catastrophic forgetting during continually learning of Task-1 and Task-2. The model trained with CPPO demonstrates Backward Transfer (BWT) capability, specifically a negative SFR, suggesting that CPPO overcomes forgetting in this task and that learning new human preferences leads to a positive impact on retaining old preferences.

Reviewer comment (Evaluation):
The performance gap of $M_{\pi1}$ and $M_{\pi2}$ on Task-2 can reveal to what extent the two data distributions mismatch.

Author response:
Thank you for your helpful comments! We appreciate your comments about our main idea and the motivation behind it make sense. We address your questions below. 

# Comment:  "I wonder about the performance of 1) $M_{\pi_2}$ on the Task-1 test set and 2) $M_{\pi_1}$ on the Task-2 test set. The result of the first experiment can show how much knowledge the model retains for the first task. In addition, the second experiment is supposed to show a mediocre result to prove the two tasks have a clear difference and that the experiment setting is indeed a continual learning setting. The performance gap of $M_{\pi_1}$ and $M_{\pi_2}$ on Task-2 can reveal to what extent the two data distributions mismatch."

Response: The reference PM score of  $M_{\pi_2}$ on Task 1 can be computed by $rPM(M_{2}, \mathbb{D}_1^{test} )=rPMS_1 - SFR$. 

Here, we supplement the performance of $M_{\pi_1}$ on Task 2, comparing CPPO with PPO.


| Method   | $rPM(M_{1}, \mathbb{D}_1^{test} )$ | $rPM(M_{1}, \mathbb{D}_2^{test} )$ | $rPM(M_{2}, \mathbb{D}_1^{test} )$ | $rPM(M_{2}, \mathbb{D}_2^{test} )$ |
|----------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|
| PPO      | 2.631                              | 2.417                              | 2.550                              | 2.688                              |
| CPPO (H) | 3.021                              | 2.812                              | 3.187                              | 2.982                              |
| CPPO (L) | 3.174                              | 2.714                              | 3.337                              | 3.090                              |

Comparing the evaluations in columns 2 and 4 for $M_{\pi_1}$ and $M_{\pi_2}$ on Task-2, we observe that model $M_{\pi_1}$  significantly underperforms $M_{\pi_2}$. This indicates a substantial difference in the data distributions between the two tasks. 

Comparing the evaluations in columns 1 and 3 for $M_{\pi_1}$ and $M_{\pi_2}$ on Task-1, we observe that the policy model trained with PPO exhibits forgetting, indicating a risk of catastrophic forgetting during continually learning of Task-1 and Task-2. The model trained with CPPO demonstrates Backward Transfer (BWT) capability, specifically a negative SFR, suggesting that CPPO overcomes forgetting in this task and that learning new human preferences leads to a positive impact on retaining old preferences.

Reviewer comment (Theory):
In general, section 3 prior to section 3.2.1 is very hard to follow. A number of things are referred to as theoretically justified or derived, but as far as can be determined, this is not true in any meaningful sense. Starting with the desiderata of section 3.2.1 and explaining the reason for each intuitively with some math (which is currently lacking) would make the contribution’s positioning stronger. The work may be better positioned as an exploration of the impact of various intuitions on continual RLHF, rather than as a theoretically justified approach for RL — which the current discourse does not deliver.

Author response:
# Comment 3:  "The key difference with generic RLHF seems to be that the optimization of the two terms is only over a subset of the data in equation 4. Why does it help us to essentially throw away data? Or is this motivated as part of a computational constraint from some implicitly considered problem formulation that is not spelled out?  "

* Response:

 Eq.4 derives from the learning objective of the Policy Consolidation[2] method, as described in Eq.3 (in the new PDF):  $ \max_{\theta} E_{s\sim {S_t}, a\sim \pi_{\theta}(\cdot \mid s)} \bigl[r_t(s,a)\bigr] - E_{s \in {S_{t-1}}} D_{KL}({P_{\pi_{\theta}}} (a|s)||{P_{\pi_{t-1}}}(a|s))$. 

The relationship of $D_1$ in Eq.4 and $S_t$ in Eq.3 is $D_1 \subset S_t$, analogously  $D_2 \subset S_{t-1}$.
Eq.4 is based on the intuitive objective: **to make the language model have a high probability of generating a response with a high reward.** Next, we clarify the reason that the objective Eq.4 is optimized on the subset of rollouts, not all of them. 

Due to the LM usually using the top-P or top-K sampling method to generate the response, namely responses with high generation probability are more potentially generated. Therefore, **the impact of low-probability samples on the final evaluation is weaker than the high-probability samples**.  Hence, we expect only to maximize the average reward on the rollout set with **high** generation probability in $S_t$, namely $D_1$.
 
To prevent forgetting, we focus more on the high-reward samples, because we expect to **keep the ability of the old policy to generate high-reward rather than low-reward samples**. 
In our learning strategy, those responses with **low rewards can be forgotten when learning new preferences**. Hence, we minimize the KL divergence on the responses with high rewards in $S_{t-1}$, namely $D_2$, which is essentially a form of function regularization[1]. 

The above intuition seeks a "compromise" between plasticity (learning new preference) and stability (retaining old preference).  It is similar to the EWC algorithm, where parameters with high importance are subjected to stronger regularization to keep the old knowledge, while parameters with low importance receive weaker regularization and have more capacity to learn new knowledge.

# References:

[1] Magdalena Biesialska, et al. Continual lifelong learning in natural language processing: A survey. ICCL, 2020.

[2] Christos Kaplanis, et al. Policy consolidation for continual reinforcement learning. PMLR, 2019.

Reviewer comment (Evaluation):
Improvements shown in Table 6, Fig 5, and Table 8 appear present, but it is unclear how significant these are or why they would generalize across domains. There is no analysis of statistical significance, which is especially concerning because PPO (and most RL methods) have high run-to-run variance, making RL results hard to take seriously without such analysis.

Author response:
# Comment 1: "What is the statistical significance of the reported results across random seeds?"
* Response: Due to time and computational constraints, we currently do not have the statistical significance results for all the baselines. In subsequent versions of the paper, we will include the corresponding experimental results.  In the table below, we present the mean and variance of the evaluation results for some methods across three random seeds (due to the randomness, there may be slight variations compared to the original PDF). 



| Method                             | **rPMS$_1$ ($\uparrow$)**    | **rouge ($\uparrow$)**       | **AT ($\downarrow$)**        | **rPMS$_2$ ($\uparrow$)**    | **rouge ($\uparrow$)**       | **SFR ($\downarrow$)**        | **rPMS ($\uparrow$)**        | **rouge ($\uparrow$)**       |
|------------------------------------|-----------------------------------|-----------------------------------|-----------------------------------|-----------------------------------|-----------------------------------|------------------------------------|-----------------------------------|-----------------------------------|
| **SFT (In order)**            | 1.499 $\pm$0.130          | **0.248** $\pm$0.006 | $-$                               | 1.543 $\pm$0.067          | **0.237** $\pm$0.007 | $-$                                | 1.498 $\pm$0.051          | **0.237** $\pm$0.009 |
| **SFT (multi-tasks)**         | 1.524 $\pm$0.041          | 0.254 $\pm$0.011          | $-$                               | 1.536 $\pm$0.092          | 0.234 $\pm$0.009          | $-$                                | 1.505 $\pm$0.011          | 0.236 $\pm$0.008          |
| **PPO (In order)**        | 2.629 $\pm$0.183          | 0.196 $\pm$0.050          | 0.052 $\pm$0.044      | 2.546 $\pm$0.201          | 0.151 $\pm$0.022          | 0.144 $\pm$0.024          | 2.502 $\pm$0.242          | 0.186 $\pm$0.016          |
| **Iterated RLHF**         | 2.629 $\pm$0.183          | 0.196 $\pm$0.050          | 0.052 $\pm$0.044           | 2.732 $\pm$0.163          | 0.211 $\pm$0.011          | 0.061 $\pm$0.018          | 2.666 $\pm$0.124          | 0.200 $\pm$0.010          |
| **PPO**             | 2.629 $\pm$0.183          | 0.196 $\pm$0.050          | 0.052 $\pm$0.044          | 2.687 $\pm$0.126          | 0.184 $\pm$0.017          | 0.080 $\pm$0.017          | 2.612 $\pm$0.191          | 0.188 $\pm$0.013          |
| **PPO+OnlineL2 Reg** | 2.758 $\pm$0.121          | 0.206 $\pm$0.042          | 0.042 $\pm$0.042         | 2.701 $\pm$0.205          | 0.180 $\pm$0.012          | 0.062 $\pm$0.013          | 2.700 $\pm$0.114          | 0.196 $\pm$0.011          |
|   **PPO+EWC**    | 2.833 $\pm$0.122          | 0.201 $\pm$0.043          | 0.047 $\pm$0.039   |     2.823 $\pm$0.192          | 0.175 $\pm$0.022          | 0.040 $\pm$0.015          | 2.801 $\pm$0.202          | 0.196 $\pm$0.023          |
| **HN-PPO**    | 2.859 $\pm$0.051          | 0.212 $\pm$0.034          | 0.036 $\pm$0.042       | 2.868 $\pm$0.132          | 0.171 $\pm$0.017          | 0.028 $\pm$0.029          | 2.846 $\pm$0.177          | 0.201 $\pm$0.011          |
|**CPPO (Heuristic)** | 3.020 $\pm$0.137          | 0.213 $\pm$0.024          | 0.035 $\pm$0.023          | 2.978 $\pm$0.113          | 0.174 $\pm$0.019          | -0.164 $\pm$0.009          | 3.099 $\pm$0.153          | 0.179 $\pm$0.016          |
| **CPPO (Learn)**     | **3.180** $\pm$0.154 | 0.220 $\pm$0.040          | **0.028** $\pm$0.042 | **3.085** $\pm$0.134 | 0.164 $\pm$0.024          | **-0.161** $\pm$0.008 | **3.207** $\pm$0.113 | 0.179 $\pm$0.008          |

Reviewer comment (Theory):
The paper claims improvement when stating “In the PPO method, the objective is to maximize the expectation of the advantage function instead of the reward value. Hence, we improve the above objective as” — but here the improvement seems either empirical or tied to bias/variance theory, and this lacks clear explanation.

Author response:
We noticed your concerns regarding Eq.5 in the **Weaknesses** part, and we guess that you may have questions about the derivation in Appendix B. Here, we will list out the derivation step by step, and if you have any specific doubts about a particular step, please feel free to point them out. And **we are eagerly waiting for your feedback on our response**. 

## from Eq.4 to Eq.5 (same with the new PDF Appendix B)

By introducing the actor-critic version, the clipped ratio, the objective Eq. 4. can be written as the objective Eq. 5. Next, we provide the derivation step by step:  


1. We utilize notations $I_{D_1}(x)$ and $I_{D_2}(x)$ to rewrite the Eq.4 as $\max_{\theta} \mathbb E_{x \sim \pi_{\theta}} I_{D_1}(x) \cdot R(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel P_{\pi_{t-1}}(x))$.
   
2. Then we introduce the importance sampling like PPO, the above objective can be written as $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {R}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel{P_{\pi_{t-1}}}(x))$.  


3. In the PPO method, the objective is to maximize the expectation of the advantage function instead of the reward value. By introducing the advantage function instead of the reward, the above objective can be written as $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {A}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel{P_{\pi_{t-1}}}(x))$.

4. By introducing the knowledge retention penalty instead of KL-divergence, the above objective is written as: $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {A}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot L^{KR}(x)$.


5. In the CL task, the new policy $\pi_{t}$ is generally initialized by the old policy $\pi_{t-1}$. In CPPO, we treat the $\pi_{t-1}$ and $\pi_{t}$ as the reference model and policy model respectively. Then, we consider the actor-critic version, the clipped ratio, and the entropy bonus used in PPO, the above objective can be written as ${J}({\theta})^{'} = L^{ {I_{D_1}} \cdot  CLIP+ {I_{D_2}} \cdot KR + VF + S}(\theta)$, namely the Eq. 5.

Reviewer comment (Theory):
The paper states “we introduce the knowledge retention penalty instead of the true KL divergence, we discuss the reason in lines 134-137 in our paper. Here, the above objective is improved as:” — but here the improvement appears based on the authors’ own empirical observations and computational justification, not rigorous theoretical grounding.

Author response:
We noticed your concerns regarding Eq.5 in the **Weaknesses** part, and we guess that you may have questions about the derivation in Appendix B. Here, we will list out the derivation step by step, and if you have any specific doubts about a particular step, please feel free to point them out. And **we are eagerly waiting for your feedback on our response**. 

## from Eq.4 to Eq.5 (same with the new PDF Appendix B)

By introducing the actor-critic version, the clipped ratio, the objective Eq. 4. can be written as the objective Eq. 5. Next, we provide the derivation step by step:  


1. We utilize notations $I_{D_1}(x)$ and $I_{D_2}(x)$ to rewrite the Eq.4 as $\max_{\theta} \mathbb E_{x \sim \pi_{\theta}} I_{D_1}(x) \cdot R(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel P_{\pi_{t-1}}(x))$.
   
2. Then we introduce the importance sampling like PPO, the above objective can be written as $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {R}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel{P_{\pi_{t-1}}}(x))$.  


3. In the PPO method, the objective is to maximize the expectation of the advantage function instead of the reward value. By introducing the advantage function instead of the reward, the above objective can be written as $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {A}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel{P_{\pi_{t-1}}}(x))$.

4. By introducing the knowledge retention penalty instead of KL-divergence, the above objective is written as: $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {A}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot L^{KR}(x)$.


5. In the CL task, the new policy $\pi_{t}$ is generally initialized by the old policy $\pi_{t-1}$. In CPPO, we treat the $\pi_{t-1}$ and $\pi_{t}$ as the reference model and policy model respectively. Then, we consider the actor-critic version, the clipped ratio, and the entropy bonus used in PPO, the above objective can be written as ${J}({\theta})^{'} = L^{ {I_{D_1}} \cdot  CLIP+ {I_{D_2}} \cdot KR + VF + S}(\theta)$, namely the Eq. 5.

Reviewer comment (Theory):
The authors write “Then we introduce the importance sampling like PPO, the above objective can be written as” — but importance sampling is a deep theoretical topic related to off-policy optimization with practical variance issues. The analysis and discussion of this component is entirely inadequate in the appendix and not discussed in the main text at all.

Author response:
We noticed your concerns regarding Eq.5 in the **Weaknesses** part, and we guess that you may have questions about the derivation in Appendix B. Here, we will list out the derivation step by step, and if you have any specific doubts about a particular step, please feel free to point them out. And **we are eagerly waiting for your feedback on our response**. 

## from Eq.4 to Eq.5 (same with the new PDF Appendix B)

By introducing the actor-critic version, the clipped ratio, the objective Eq. 4. can be written as the objective Eq. 5. Next, we provide the derivation step by step:  


1. We utilize notations $I_{D_1}(x)$ and $I_{D_2}(x)$ to rewrite the Eq.4 as $\max_{\theta} \mathbb E_{x \sim \pi_{\theta}} I_{D_1}(x) \cdot R(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel P_{\pi_{t-1}}(x))$.
   
2. Then we introduce the importance sampling like PPO, the above objective can be written as $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {R}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel{P_{\pi_{t-1}}}(x))$.  


3. In the PPO method, the objective is to maximize the expectation of the advantage function instead of the reward value. By introducing the advantage function instead of the reward, the above objective can be written as $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {A}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot D_{KL}({P_{\pi_{\theta}}} (x)\parallel{P_{\pi_{t-1}}}(x))$.

4. By introducing the knowledge retention penalty instead of KL-divergence, the above objective is written as: $\max_{\theta} \mathbb E_{x \sim \pi_{t-1}} I_{D_1}(x) \cdot \frac{{P_{\pi_{\theta}}} (x)}{{P_{\pi_{t-1}}} (x)} {A}(x) - \mathbb E_{x \sim \pi_{t-1}} I_{D_2}(x) \cdot L^{KR}(x)$.


5. In the CL task, the new policy $\pi_{t}$ is generally initialized by the old policy $\pi_{t-1}$. In CPPO, we treat the $\pi_{t-1}$ and $\pi_{t}$ as the reference model and policy model respectively. Then, we consider the actor-critic version, the clipped ratio, and the entropy bonus used in PPO, the above objective can be written as ${J}({\theta})^{'} = L^{ {I_{D_1}} \cdot  CLIP+ {I_{D_2}} \cdot KR + VF + S}(\theta)$, namely the Eq. 5.