Paper: Reflective Policy Optimization (ICLR-2024)

Reviewer comment (Experiments):
The theorems and proofs proposed in the paper are interesting, but the performance shown in the experiments does not seem to be much different from the performance of the baselines. In particular, there are many mentions of convergence speed, but the results shown do not show a significant improvement over existing methods. It is necessary to show the performance improvement in an environment where the subsequent state-action can be better considered.

Author response:
We are grateful for your query about the performance improvement and apologize for not articulating this clearly in our paper. In our experiments, we found that our method has a significant advantage over the original PPO. However, to showcase the robustness and broad applicability of our approach, we did not perform specific parameter tuning. The results post-tuning will be detailed in the paper's appendix to transparently exhibit RPO’s performance. In Fig. 3, our method is compared not only with the PPO algorithm but also with some algorithms that have modified PPO from an on-policy to an off-policy approach (GePPO and OTRPO) and it is found that our method is also advantageous. Figure 1 shows that the direct use of subsequent data plays a positive role and our approach can be seamlessly integrated into existing algorithm frameworks.

We once again apologize for any confusion caused by our lack of clarity in conveying that specific parameter tuning was not conducted to demonstrate the robustness and general adaptability of our method. Please forgive any inconvenience this may have caused during your review. Your guidance is invaluable, and we will take greater care in this regard in our subsequent paper submissions.

Reviewer comment (Theory):
In the introduction section of the paper, it is argued that existing policy optimization algorithms do not directly consider the impact of subsequent state-actions in the trajectories. Also, in the example, if the agent repeatedly visits the cliff state, it is dangerous because agent is likely to act to fall off the cliff. However, in policy optimization, since there is a discounted sum of reward term, it is possible to provide feedback on the cliff state and the falling action. Therefore, the motivation of the proposed method is difficult to understand, as it allows us to consider the impact of subsequent state action under the influence of rewards.

Author response:
I am very sorry that we did not make this example more straightforward. We will modify it in the introduction. With this example, we want to consider the impact of the subsequent data directly. The value function potentially contains information about the subsequent data. We need to consider the question: Is it the best way to optimize a policy using only value functions? The answer is definitely not. There is another way to optimize the current policy by directly using the subsequent data, unlike optimizing the policy using only the value function. Intuitively, the direct use of the subsequent data may speed up the algorithm's convergence  and improve sample efficiency. In the subsequent sections of this paper, we verify this intuition theoretically and experimentally, respectively.

Reviewer comment (Experiments):
The proposed algorithm increases the number of hyperparameters as the value of k increases. The authors claim that k=2,3 is suitable because of the stability of learning, but even with k=3, it requires 5 hyperparameters to adjust the clipping and learning rate. In addition, the experiments show the case of k=2, and the best performing hyperparameters are not consistent across experimental environments. Therefore, it can be said that the algorithm is sensitive to hyperparameters.

Author response:
Fine-tuning the hyperparameters of our algorithm indeed leads to improved performance. In the main experiment (as shown in Figure 3), we fixed the hyperparameters without fine-tuning  for a fair comparison. From Fig. 3, even with fixed hyperparameters, our approach achieves good results compared with PPO and the off-policy version of PPO.

Reviewer comment (Theory):
In Section 4, a new generalized lower bound is defined via Theorem 4.1. In this part, the authors introduce a new surrogate objective function, which has a slightly different semantics than the surrogate objective function in Theorem 3.1, just minus one. Also, they say that they directly optimize the information of current and future state-action pairs, can you explain what the difference is, or can you compare the difference experimentally?

Author response:
Consider $L_1(\pi, \hat{\pi})$ of the theorem 3.1 as an example. If the environment is unknown, it can only be optimized by sampling. If we sample a trajectory $\tau$, and $(s_0, a_0, s_1, a_1)\in\tau$, if $ A^{\hat{\pi}}(s_1, a_1)<0 $ and $ r_0-1 <0$, we know that $ (r_0-1)r_1 A^{\hat{\pi}}(s_1, a_1)>0 $. Optimizing the $ L_1(\pi, \hat{\pi}) $ (considering the extreme case, using one sample), the probability of $ a_1 $ is increased. However, $ A^{\hat{\pi}}(s_1, a_1)<0 $, we should decrease the probability of $ a_1 $. It's a contradiction. Thus this term "1" of $ r_0-1 $ may adversely affect policy optimization though the theory is sound. This situation exists when the environment is unknown. But for $\hat{L}_1(\pi, \hat{\pi})$ in the theorem 4.1, there is no such problem. We also experimentally compared these two methods, *i.e.*, RPO vs.* TayPO (with the term ”1"). The possibility of what we said earlier is also verified from the experiment.

Reviewer comment (Experiments):
The paper states that it applied multi-step RL directly from a policy optimization point of view, but it seems that Generalized Adversarial Estimation (GAE) was also used when training. Could you tell me if there was any correlation between the GAE parameters and multi-step k in experiments?

Author response:
In this paper, GAE is evaluated in the same way as PPO, *i.e.*, $\hat{A}^{GAE}_t=\delta^t+(\gamma \lambda)\delta^{t+1}+\cdots+(\gamma \lambda)^{T-t+1} \delta^{T-1}  $,

 where $ \delta^t=r_t+\gamma V(s_{t+1})-V(s_t)$.

Assume that the sampled trajectory is $\tau = (s_0,a_0,s_1,a_1, \cdots, s_t, a_t, \cdots)$, we can calculate the advantage functions ($\hat{A}^{GAE}_0, \hat{A}^{GAE}_1, \cdots, \hat{A}^{GAE}_t, \cdots$). 

When k=2, if we sample ($s_t, a_t, s_{t+1}, a_{t+1}$), the current policy is optimized by ($\hat{A}^{GAE}_t$, 

$\hat{A}^{GAE}_{t+1}$). It can be seen that the relationship between them is that when k is fixed, the number of advantage functions is also fixed.

Reviewer comment (Experiments):
The paper shows that the convergence speed increases with the value of K, but the performance is only shown in three environments. Can you explain whether the performance varies significantly depending on the difficulty of the environment or the dimensionality of the state and action?

Author response:
Previously, due to limited resources and time, we did not perform parameter tuning for k=3. We only ran some experiments in three environments and included these results in our paper. Now, we have supplemented this experiment in the Appendix.

Reviewer comment (Theory):
There can be some technical misstatement & errors in the main contributions. First, the following statement can be erroneous: “If you recombine the above equation $(r_0-1)r_1 A^{\hat{\pi}}(s_1, a_1)]\cdot r_1 >0$, optimizing it will be found to increase the probability of $a_1$ . However, $A_{\hat{\pi}}(s_1 , a_1) < 0$, we should decrease the probability of $a_1$ . This would present a contradiction.” Optimizing this objective can result in a bit more complicated consequence than what’s been stated here: since the policy in both $r_0$ and $r_1$ is parameterized with a same set of parameters, optimizing the objective may increase the probability of $a_0$ and thus tip over the sign of $r_0-1$. The problem is $a_0$ and $a_1$ are from the same parametrized policy.

Author response:
**Question 6** : (Weaknesses) about $ (r_0-1)r_1 A^{\hat{\pi}}(s_1, a_1) = [(r_0-1) A^{\hat{\pi}}(s_1, a_1)]\cdot r_1>0 $ ... present a contradiction. Optimizing this objective can result in a bit more complicated consequence than what’s been stated here. 

**Answer 6**: Your point is valid. From the perspective of parameter optimization, optimizing this objective be considered a more complex situation. In this paper, we consider this function intuitively without focusing on the specific form of the parameters. When the model is unknown, we optimize this objective function by sampling and give intuitively the possible problems which there is no way to avoid. 

**Re-Question 6**:  Rigorous and flawless descriptions should be given as an example.

**Re-Answer 6**:  I am very sorry for not making this example clear. I've added a more rigorous description in our manuscript, and we've explained it to you to make it easier for you to understand. 

Consider $L_1(\pi, \hat{\pi})$ in Eqn. (2) as an example.  We consider this function without focusing on the specific form of the parameters. When the environment is unknown, it can only be optimized by sampling. Considering the extreme case, the function  $L_1(\pi, \hat{\pi})$ is optimized by using a sample $(s_0, a_0, s_1, a_1)$, *i.e.*, $L_1(\pi, \hat{\pi})\approx(r_0-1)r_1 A^{\hat{\pi}}(s_1, a_1) $.

If $ A^{\hat{\pi}}(s_1, a_1)<0 $ and $ r_0-1 <0$, we know that $ (r_0-1)r_1 A^{\hat{\pi}}(s_1, a_1)=[(r_0-1)A^{\hat{\pi}}(s_1, a_1)]r_1>0 $. The probability of $ a_1 $ is increased. However, when $ A^{\hat{\pi}}(s_1, a_1)<0 $, we should  decrease the probability of $ a_1 $. It's a contradiction. Thus, this term "1" of $ r_0-1 $ may adversely affect policy optimization, though the theory is sound. This situation exists when the environment is unknown.

Reviewer comment (Theory):
The monotonic improvement statement of Theorem 4.1 can be erroneous. Consider a special case of $k=1$, which yields the TRPO bound as suggested by the paper (assume this statement is correct), then the lower bound will be $\alpha_0 \hat{L}_0(\pi, \hat{\pi}) - \hat{C}_1(\pi, \hat{\pi})$, which equals to $E_{s_0, a_0\sim\rho^{\hat{\pi}}} [r_0 A^{\hat{\pi}}(s_0, a_0)] - \frac{\gamma R_{max}}{(1-\gamma)^3}||\pi - \hat{\pi}||_1^2$. This is the surrogate objective implied by Theorem 4.1 and should be improved upon. However, this surrogate objective is strictly negative for $\gamma=0.995$ used by this paper: consider the first term $E_{s_0, a_0\sim\rho^{\hat{\pi}}} [r_0 A^{\hat{\pi}}(s_0, a_0)] = E_{s_0, a_0\sim\rho^{\hat{\pi}}} [ (r_0-1) A^{\hat{\pi}}(s_0, a_0)] $ (this is because the advantage is defined on $\hat{\pi}$ and the integral over $\hat{\pi}$ will be zero). Hence, $E_{s_0, a_0\sim\rho^{\hat{\pi}}} [ (r_0-1) A^{\hat{\pi}}(s_0, a_0)] \leq \frac{  ||\pi - \hat{\pi}||_1^2 Rmax }{1-\gamma} $ (this is a direct result from the proof of Corollary A.2 in appendix, i.e., $G_1$). Thus,  
$E_{s_0, a_0\sim\rho^{\hat{\pi}}} [r_0 A^{\hat{\pi}}(s_0, a_0)] - \frac{\gamma R_{max}}{(1-\gamma)^3}||\pi - \hat{\pi}||_1^2$  
$\leq \frac{  ||\pi - \hat{\pi}||_1^2 Rmax }{1-\gamma} - \frac{\gamma Rmax}{(1-\gamma)^3}||\pi - \hat{\pi}||_1^2$  
$\leq \frac{  ||\pi - \hat{\pi}||_1^2 Rmax }{1-\gamma} [ 1- \frac{\gamma}{(1-\gamma)^2}] < 0$ (for $\gamma =0.995)$.  
So, the maximum value of the lower bound is strictly negative. It turns out to be impossible to get "a monotonically improving sequence of policies $\{\pi_i\}_{i=1}^{\infty}$ satisfying $\eta(\pi_0) \leq \eta(\pi_1) \leq ...$" for this case $k=1$. Please correct me if my understanding is wrong.

Author response:
**Question 7** : (Weaknesses) about the monotonic improvement statement of Theorem 4.1.

**Answer 7**: Thank you for providing a proof, the approach of proof is correct. We have identified a flaw in one of the proof steps, *i.e.*, it is not true that $ E_{s_0, a_0\sim\rho^{\hat{\pi}}}[(r_0-1) A^{\hat{\pi}}(s_0, a_0)]\frac{\|\pi-\hat{\pi}\|_1^2R\max}{1-\gamma}$, 

but rather that $ E_{s_0, a_0 \sim \rho^{\hat{\pi}}}[(r_0-1) A^{\hat{\pi}}(s_0, a_0)] \leq \frac{\|\pi-\hat{\pi}\|_1 R\max}{1-\gamma}$.

We can see that $ G_1(\pi, \hat{\pi}) = E_{s_0, a_0 \sim \rho^{\hat{\pi}}(\cdot)}(r_0-1) E_{s_{1}, a_{1} \sim \rho^{\pi}(\cdot|s_{0}, a_{0})}A^{\hat{\pi}}(s_1, a_1) $ is different from $ E_{s_0, a_0 \sim \rho^{\hat{\pi}}}[(r_0-1) A^{\hat{\pi}}(s_0, a_0)]$. 

Therefore, the results of $ G_1 $ cannot be used directly. We give the complete proof below. 
$ E_{s_0, a_0 \sim \rho^{\hat{\pi}}}[(r_0-1) A^{\hat{\pi}}(s_0, a_0)] - \frac{ \gamma R\max}{(1-\gamma)^2}\|\pi-\hat{\pi}\|_1^2\leq \frac{ R\max}{1-\gamma}\|\pi-\hat{\pi}\|_1 -\frac{ \gamma R\max}{(1-\gamma)^2}\|\pi-\hat{\pi}\|_1^2 $. 

When $ 0<\|\pi-\hat{\pi}\|_1<\frac{1-\gamma}{\gamma} $, the upper bound is greater than zero. So the lower bound of performance has solutions.

**Re-Question 7** : (Weaknesses) About the monotonic improvement statement of Theorem 4.1.

**Re-Answer 7**:  Firstly, we define $\rho(\cdot|s, a)$ from another perspective.

Inspired by discount state visitation distribution $\rho(\cdot)$, *i.e.*, 
$$\rho^{\pi}(s) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\mathbb{P}(s_t=s|\rho_0, \pi),$$
we can similarly  give a definition of $\rho^{\pi}(s,a,s')$. It is defined as $$\rho^{\pi}(s,a,s') = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\mathbb{P}(s_t=s,a_t=a,s_{t+1}=s'|\rho_0, \pi),$$
where $\mathbb{P}(s_t=s,a_t=a,s_{t+1}=s'|\rho_0, \pi)=\mathbb{P}(s_t=s|\rho_0, \pi)\pi(a_t=a|s_t=s)P(s_{t+1}=s'|s_t=s,a_t=a),$ Further, we can give the definition of $\rho^{\pi}(\cdot|s, a)$, *i.e.*, $\rho^{\pi}(s'|s, a)=\frac{\rho^{\pi}(s, a,s')}{\rho^{\pi}(s, a)}$.

By this way, $\rho^{\pi}(s_2|s_1, a_1)=\frac{\rho^{\pi}(s_0,a_0,s_1,a_1,s_2)}{\rho^{\pi}(s_1,a_1|s_0,a_0)\rho^{\pi}(s_0,a_0)}$. Similarly, we can define $\rho^{\pi}(s_i|s_{i-1}, a_{i-1})$.

Secondly, thank you for re-providing proof. The approach of proof is correct. Moreover, we have identified a flaw in one of the proof steps. The penalty term in the lower bound is $\frac{ \gamma R\max }{(1-\gamma)^2}\|\pi-\hat{\pi}\|_1^2$ instead of $\frac{ \gamma R\max }{(1-\gamma)^3}\|\pi-\hat{\pi}\|_1^2$ (see theorem 4.1 of my paper or theorem 1 of the TRPO paper [1]). So, the quadratic form takes the maximal when $\|\pi-\hat{\pi}\|_1=\frac{1-\gamma}{2\gamma}$.

Lastly, we are well aware of your concern that $\|\pi-\hat{\pi}\|_1$ may be too small. By the analysis of the theorem, we are more interested in the existence of solutions. In practice, as the TRPO paper says, "the KL divergence is bounded at every point in the state space, ...,  is impractical to solve due to the large number of constraint".  This issue is familiar to our approach but rather a challenge universally encountered by all lower bound algorithms, *e.g.*, TRPO [1], CPO [2], GePPO [3], and so on. 

[1] Schulman J, Levine S, Moritz P,et al. Trust Region Policy Optimization. ICML, 2015.

[2] Achiam J , Held D, Tamar A,et al. Constrained Policy Optimization. ICML, 2017.  

[3] Queeney J , Paschalidis I C,et al. Generalized Proximal Policy Optimization with Sample Reuse. NeurIPS, 2021.

Reviewer comment (Writing):
I don’t see the point in the examples of “cliff” and “treasure” states in the second paragraph of Introduction Section. Would the value function just suffice whether a state is favourable?

Author response:
**Question 1**: About the examples of “cliff” and “treasure” stated in the second paragraph of the Introduction Section. Would the value function suffice whether a state is favorable?

**Answer 1**: I am very sorry for not making this example clear. We will modify it in the introduction. Let's briefly restate the example. Consider an environment with a "cliff." what would an agent do if it performed an action under a state and fell into a "cliff"? This action is dangerous. Meanwhile, this state might also be dangerous due to the fact that the next time the agent reaches this state again, it is likely to perform the same action. Hence, the agent also needs to avoid getting to this state again and keep out of this state as much as possible. The previous action, when reaching this state, the previous action also needs to directly avoid being performed because it is possible to fall into that state again. The same result is found for the "treasure" environment.  Hence, it is necessary to optimize the previous action directly with the subsequent state-action pairs information, not only through the value function, thereby improving sample efficiency. 

Yes, this way is feasible if the policy is optimized using the value function of the current state due to the fact that the value function potentially contains information about the subsequent states. We need to consider the question: Is it the best way to optimize a policy using only value functions? The answer is definitely not. What we are trying to convey with this example is that there is another, better way to optimize the current policy by directly using the subsequent data, unlike optimizing the policy using only the value function. Intuitively, the direct use of the subsequent data may speed up the convergence of the algorithm and improve sample efficiency. In the subsequent sections of this paper, we verify this intuition theoretically and experimentally, respectively.

Reviewer comment (Presentation):
Most of the citations are using a wrong format. It seems that the author misused the citep & citet.

Author response:
**Question 2** : About the citep \& citet.

**Answer 2**: Thank you very much for pointing this out, we revise it in the next version.

Reviewer comment (Writing):
There is a notation issue in the second line of $\eta(\pi) - \eta(\hat{\pi})$. The first term on RHS should be with (s_0, a_0).

Author response:
**Question 3** : In the second line of $\eta(\pi)-\eta(\hat{\pi})$, the first term on RHS should be with ($s_0, a_0$).

**Answer 3**: Thank you very much, we revise it in the next version.

Reviewer comment (Writing):
The $||\pi - \hat{\pi}||_1$ has not been defined in the paper.

Author response:
**Question 4** : The $\|\pi-\hat{\pi}\|_1 $ has not been defined in the paper.

**Answer 4**: Thank you very much, we revise it. $\|\pi-\hat{\pi}\|_1$  is defined as $\max_s\sum_a|\pi(a|s)-\hat{\pi}(a|s)|$.

Reviewer comment (Theory):
The definition of the conditional occupancy measure is a bit confusing. From the proofs in the appendix, this conditional occupancy measure is a distribution with the conditions on the initial state-action distribution, i.e., the stationary state distribution induced from a predefined initial state distribution. Then I’m not sure if $(s_0, a_0)\sim\rho$ and $(s_i, a_i)\sim\rho(\cdot|s_{i-1}, a_{i-1})$ can be defined coherently.

Author response:
**Question 5** : About $ (s_0, a_0)\sim \rho $, and $ (s_i, a_i)\sim \rho(\cdot|s_{i-1},a_{i-1}) $.

**Answer 5**: I'm very sorry, I didn't make that clear. Let us explain in detail below. From the definition of $ \rho(s) $ in Preliminaries Section, we see that $ \mathbb{P}(s_t=s|\rho_0, \pi) $ has Markov properties. So, $ \rho^{\pi}(s) $ has the same properties. From the proof of Theorem A.1 (see appendix), the link between the performance difference of $ \pi $ and $ \hat{\pi} $ and state visit distribution and conditional state visit distribution has been established. We know that $ \rho(s_i|s_{i-1}, a_{i-1})= \rho(\cdot|s_{i-1}, a_{i-1}, s_{i-2}, a_{i-2}, \cdots, s_0, a_0)$, which is the conditional distribution under $ s_{i-1} $ and $ a_{i-1} $, and $ s_{i-1}\sim\rho(\cdot|s_{i-2}, a_{i-2}) $ and $ a_{i-1}\sim \pi(\cdot|s_{i-1}) $. For example, given the distributions of $ \rho(s_0) $ and $ \pi(a_0|s_0) $, $ \rho(s_1|s_{0}, a_{0}) $ must exist according to Bayesian theory, but the exact form is difficult to give directly. Their relationship is $ \rho(\cdot) = \int \rho(\cdot|s_0, a_0) \rho(s_0) \pi(a_0|s_0) ds_0da_0 $. For the general case, we have $\rho(\cdot) = \int \rho(\cdot|s_{i-1}, a_{i-1})\rho(s_{i-1}|s_{i-2}, a_{i-2})\pi(a_{i-1}|s_{i-1})\cdots \rho(s_0) \pi(a_0|s_0) ds_{i-1}da_{i-1}\cdots ds_0da_0 $.