Paper: On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling (ICLR-2024)

Reviewer comment (Novelty):
The idea of using near on-policy update or constrained update to improve off-policy learning has been extensively studied in the RL literature, and as a result, the constrained update rule proposed in the paper is not very novel.

Author response:
We are slightly confused by some of these comments and hope we can clarify our novelty during the discussion period. We would like to emphasize that the core novelty of our work is how we demonstrate that *off-policy sampling* can reduce sampling error and improve the data efficiency of *on-policy* policy gradient algorithms. In other words, the core novelty is not only the PROPS update we use to adapt our behavior policy but moreover the fact that we are using a separate behavior policy for on-policy learning in the first place. So while KL penalties and clipping (as in PPO) are not novel on their own, our use of them to enable scalable adaptive sampling with the behavior policy is novel to the best of our knowledge.

Reviewer comment (Novelty):
The main idea presented in the paper is not fundamentally different from trust region policy updates and does not present additional insight beyond what is already in the existing literature.

Author response:
We are slightly confused by some of these comments and hope we can clarify our novelty during the discussion period. We would like to emphasize that the core novelty of our work is how we demonstrate that *off-policy sampling* can reduce sampling error and improve the data efficiency of *on-policy* policy gradient algorithms. In other words, the core novelty is not only the PROPS update we use to adapt our behavior policy but moreover the fact that we are using a separate behavior policy for on-policy learning in the first place. So while KL penalties and clipping (as in PPO) are not novel on their own, our use of them to enable scalable adaptive sampling with the behavior policy is novel to the best of our knowledge.

Reviewer comment (Theory):
The PPO objective appears similar to the CLIP objective in Eqn 4. In PPO, $\theta$ and $\theta_{\text{old}}$ are close by policy (one iteration apart), and when the two policies are very close, the update approximates the policy gradient update. For CLIP to be sensible, $\theta$ and $\phi$ also need to be close, which would be softly enforced by the KL loss. The reviewer asks whether this is correct and what would be the major difference between CLIP and PPO objectives, despite technical differences in clipping operation and KL regularization.

Author response:
The major difference between these objectives is that CLIP is an objective for *behavior policy* optimization and the PPO objective is for optimizing the *target policy.* In CLIP, we weight behavior policy updates by -1 to increase the probability of under-sampled actions, where as PPO weights policy changes by the advantage $A(s, a)$ to increase the probability of good actions with large advantages. Note that we use the PPO objective to optimize the target policy in both PROPS and vanilla PPO.

Reviewer comment (Reproducibility):
In Eqn 4 and 5, a loss is specified for both $\theta$ and $\phi$. The reviewer asks whether both are updated at the same time or if one is updated at a slower rate than the other.

Author response:
The behavior policy is updated more frequently than the target policy so that the behavior policy has a chance to reduce sampling error before each target update. In our experiments, the behavior policy is updated every 256 timesteps and the target policy is updated every 1024 - 8192 timesteps.

Reviewer comment (Theory):
If $\theta$ and $\phi$ are practically close to each other, the CLIP objective is effectively very similar to the idea of trust region policy updates.

Author response:
Your understanding is correct. At the start of each PROPS update, we set $\phi \gets \theta$ (line 3 in Algorithm 2), and then make a location adjustment to $\phi$ that increases the probability of under-sampled actions. You are also correct in saying that regularization helps to keep the behavior policy close to the target policy and that the PROPS update is similar to a trust-region update.

Reviewer comment (Experiments):
The reviewer acknowledges the insight that on-policy learning should match the learning distribution rather than the exact policy but notes that much of the community would define on-policy learning as learning from the same policy. In practice, it is difficult to maintain even on-policy samples or sub-sample them from the replay buffer due to policy drift.

Author response:
While we agree that the formulation of the on-policy update does indeed point to on-policy data as the key desiderata, unfortunately, we know of many sources that distinguish on-policy learning by its sampling distribution, including some prominent educational resources:

1. Sutton & Barto’s RL textbook [1] states, “the distinguishing feature of on-policy methods is that they estimate the value of a policy while using it for control.”
2. Lecture 5 of David Silver’s RL course [2] states that on-policy algorithms “learn about policy π from experience sampled from π.”
3. The taxonomy of RL algorithms provided in OpenAI’s Spinning Up package [3] states that with on-policy algorithms, “each update only uses data collected while acting according to the most recent version of the policy.”

Because the view that "on-policy refers to a sampling procedure" is so prominent, we wanted to elaborate on the distinction between on-policy data and on-policy sampling to emphasize how this popular perception is not a reality. We will clarify these points in our revisions and hope to further discuss them with reviewers.

Reviewer comment (Experiments):
In relation to Figure 5, the reviewer asks what PPO-buffer exactly is — whether it corresponds to running PPO on off-policy samples naively — and expresses surprise at the empirical gains of PROMPS. They feel PROMPS is a slightly more adaptive variant of PPO-buffer to make use of buffer data. They are surprised because PPO-buffer still applies clipping and enforces a trust region constraint, meaning the policy update should still maintain stability even when using very off-policy samples.

Author response:
Your understanding is correct; PPO-Buffer naively reuses historic off-policy data without any algorithmic changes, and PROPS is indeed an adaptive version of PPO-Buffer. 

PPO-Buffer outperforms PPO in Walker2d-v4 and Humanoid-v4, which indeed suggests that PPO’s clipping mechanism may be robust to stale data. In fact, OpenAI reused historic data in a similar fashion when using PPO to train the RL agents that ultimately defeated the world champions in the Dota 2 game [4]. However, we note that in most tasks we consider, PPO-Buffer performs comparably to PPO.

Reviewer comment (Theory):
There is a mismatch between the idea stated in the first half of the paper – adaptively correcting sampling error in previously collected data by increasing the probability of sampling under-sampled actions with respect to the current policy – and the practical method proposed later. The idea is to adapt the behavior policy so that the resultant data distribution is more on-policy (as illustrated in Figure 1), but the practical method instead adapts the behavior policy to fit the collected data (with the gradient $\nabla_{\phi}L=-\nabla_{\phi} \sum_{(s,a) \in D} \log \pi_{\phi}(a|s)$ and its improvement).

Author response:
We believe this concern arises from a simple misunderstanding of the direction of the ROS/PROPS update, and we will clarify this point in the main paper. The key point is that we are doing gradient *ascent* on the negative log-likelihood (NLL) rather than gradient *descent* as done in MLE learning. You correctly point out that MLE learning would have a mismatch with the goal of our work. However, since we use gradient *ascent* on the NLL, the update has the desired effect of increasing the probability of under-sampled actions.

It is important to note that the ROS and PROPS algorithms do not aim to fully optimize their objectives. Rather, at each iteration, the behavior policy is set equal to the target policy and then the gradient is used to make a local adjustment which decreases the probability of over-sampled actions, thereby increasing the probability of under-sampled actions.

Reviewer comment (Theory):
In the second paragraph of Section 5, the paper states that the gradient $\nabla_{\phi}L=-\nabla_{\phi} \sum_{(s,a) \in D} \log \pi_{\phi}(a|s)$ increases the probability of under-sampled actions with respect to $\pi_{\theta}$. However, since the expectation is over the collected data, the gradient appears to be maximum likelihood estimation based on collected data (or imitation learning) and is almost the same as Equation 7 in the appendix (differing only in max vs min). This point is unclear to the reviewer, who requests correction if they have misunderstood.

Author response:
We believe this concern arises from a simple misunderstanding of the direction of the ROS/PROPS update, and we will clarify this point in the main paper. The key point is that we are doing gradient *ascent* on the negative log-likelihood (NLL) rather than gradient *descent* as done in MLE learning. You correctly point out that MLE learning would have a mismatch with the goal of our work. However, since we use gradient *ascent* on the NLL, the update has the desired effect of increasing the probability of under-sampled actions.

It is important to note that the ROS and PROPS algorithms do not aim to fully optimize their objectives. Rather, at each iteration, the behavior policy is set equal to the target policy and then the gradient is used to make a local adjustment which decreases the probability of over-sampled actions, thereby increasing the probability of under-sampled actions.

Reviewer comment (Theory):
The example and discussion in Section 4 appear improper. The main idea and Figure 1 focus on the distribution mismatch of actions, but Section 4 discusses distribution mismatch of outcomes, which cannot be known in the practical learning process (since we cannot know whether a large reward exists). This feels off-track to the reviewer.

Author response:
You are correct in that PROPS focuses on correcting the state-conditioned distribution of actions as opposed to the distribution of trajectories (since we generally don’t know the trajectory distribution). To convey intuition for how sampling error could impact learning, our example in Section 4 focused on sampling error in the outcomes space. However, we see how this was confusing to multiple reviewers and so we can make a few minor modifications to make the example focus on action probabilities.

In particular, we can add the following action probabilities to Figure 2: 

$\pi(left | s_0) = 0.5$, 

$\pi(right | s_0) = 0.5$,

$\pi(left | s_L) = 0.02$,

$\pi(right | s_L) = 0.98$

Where, for brevity, we use $left$ = “attempt to move” and $right$ = “remain standing”. These action probabilities preserve the outcome probabilities originally listed.

After 100 visits to $s_0$, the agent may never sample the $left$ branch at $s_L$ and believe that the $left$ branch at $s_0$ always leads to a return of -1 (falling). Having observed a return of +1 (standing) in the $right$ branch at $s_0$, the agent reduces the probability of choosing the $left$ branch at $s_0$, ultimately making the agent more likely to converge to a suboptimal standing policy. We can help correct this sampling error by increasing the probability of sampling the under-sampled $left$ branch at $s_L$.

Reviewer comment (Experiments):
Important baselines such as ROS and GePPO are not included in Figure 5. Computational cost (e.g., wall-clock time) is not discussed in the main body of the paper. An ablation study on the KL term in learning performance is also expected.

Author response:
We considered these baselines but ultimately decided to exclude them from the main RL experiments shown in Figure 5.

We exclude ROS from our RL experiments for two reasons:
1. Since ROS fails to reduce sampling error even when the target policy is fixed (Fig. 3 in the main paper and Fig. 7 and 8 in Appendix C of the supplemental material), we expect ROS to also fail to reduce sampling error in the more difficult RL setting where the target policy is continually updated. 
2. ROS is too expensive to run in the RL setting. ROS updates the behavior policy *every timestep* using a gradient computed over *all data* in the agent’s buffer, whereas PROPS updates the target policy every 256 timesteps using minibatches sample from the agent’s buffer.

We can clarify both of these reasons in our revisions.

PROPS and GePPO are orthogonal approaches to improve the data efficiency of on-policy learning: PROPS uses adaptive data collection to reduce sampling error, while GePPO reweights data according to historic policies *after* data collection. We ultimately decided that the comparison distracted from our main focus of determining whether adaptive off-policy sampling could produce sufficiently on-policy data for effective learning. For the purpose of answering the main questions of our study, the empirical evaluation is complete as-is.

Reviewer comment (Experiments):
Since there could be under-sampled actions, there should also be over-sampled actions. How are these considered in the proposed method?

Author response:
It is important to note that the ROS and PROPS algorithms do not aim to fully optimize their objectives. Rather, at each iteration, the behavior policy is set equal to the target policy and then the gradient is used to make a local adjustment which decreases the probability of over-sampled actions, thereby increasing the probability of under-sampled actions.

Reviewer comment (Evaluation):
What is the computation cost of PROPS, especially in terms of wall-clock time?

Author response:
We can include wall-clock time comparisons of PROPS, PPO, and PPO-Buffer in our revisions. We will share our results in a follow-up comment once we have them. 

Intuitively, we expect PROPS will take roughly twice as much time as PPO-Buffer; both algorithms perform updates using the same amount of data, though PROPS learns two policies. 

---

The following table shows runtimes for PROPS, PPO-Buffer, and PPO averaged over 3 runs with standard deviations in parentheses. We have results for 4 environments, and we can add the two remaining environments (Ant-v4 and Humanoid-v4)  in a camera-ready version. We trained all agents on a MacBook Air with an M1 CPU, and we use the same tuned hyperparameters used in throughout the paper. 

| Environment | PROPS | PPO-Buffer | PPO |
| -------------- | -----------  | ----------- | ------ |
| Swimmer-v4 | 618 (1)  | 529 (30) | 446 (30) |
| Hopper-v4   | 1473 (40) | 795 (54) | 525 (30) |
| HalfCheetah-v4 | 1786 (44) | 921 (6) | 1228 (23) |
| Walker2d-v4 | 2963 (33) | 1500 (112) | 1203 (42) | 

PROPS takes at most twice as long as PPO-Buffer. These results agree with the intuition we shared in our initial response: Both PROPS and PPO-Buffer learn from the same amount of data but PROPS learns two policies.

We note that PPO-Buffer is faster than PPO is HalfCheetah-v4 because, with our tuned hyperparameters, PPO-Buffer performs fewer target policy updates than PPO. In particular, PPO-Buffer is updating its target policy every 4096 steps, whereas PPO is updating the target policy every 1024 steps.

Reviewer comment (Presentation):
The paper presents an inaccurate characterization of the history of on-policy vs off-policy algorithms, which detracts from the overall presentation. The core issue is the characterization that the field is uncertain if on-policy updates require on-policy data or on-policy sampling. The reviewer explains that from the trajectory of literature, it is clear that on-policy algorithms are designed and work well with data distributed according to the target policy, and that the easiest way to get this data is by sampling according to the behavior. The reviewer views the narrative and the dichotomy presented by the authors as misleading and ill-conceived, although they have no complaint about the proposed method itself.

Author response:
Thank you for the in-depth comment on the framing of our work. We completely agree with the historical framing you provide,  so we believe a minor change can clarify the narrative. However, before doing so, we would like to explain why we framed the paper as we did and discuss how we can combine your suggestions with our original motivation.

While we agree that the historical trajectory does indeed point to on-policy data as the key desiderata for on-policy policy gradient methods, unfortunately, we know of many cases where on-policy learning is defined by the sampling distribution, including some prominent educational resources:
1. Sutton & Barto’s RL textbook [1] states, “the distinguishing feature of on-policy methods is that they estimate the value of a policy while using it for control.”
2. Lecture 5 of David Silver’s RL course [2] states that on-policy algorithms “learn about policy π from experience sampled from π.”
3. The taxonomy of RL algorithms provided in OpenAI’s Spinning Up package [3] states that with on-policy algorithms, “each update only uses data collected while acting according to the most recent version of the policy.”

Because the view that "on-policy refers to a sampling procedure" is so prominent, we chose a more pointed narrative to emphasize that it is the empirical distribution of data that really matters for accurate gradient estimates. In a sense, our work is pointing back to the historical trajectory you give and saying that we really do want to have samples weighted according to their actual probability and not just generated according to a specific process.

Overall, we believe that there is not a large gap between our view and the reviewer’s view and that we can fine-tune the discussion in our paper to note the historical trajectory given. Specifically, it will be straightforward to 1) add discussion on the historical basis of these methods and say that, yes, clearly on-policy data is the goal, and 2) clarify that we aim to correct a popular misconception in how these algorithms are understood. We hope that we can discuss this point more with the reviewer over the next week to make the narrative as strong as possible.

[1] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.

[2] Silver, David. "RL Course by David Silver, Lecture 5: Model-Free Control." University College London, Computer Science Department, 2015. https://www.davidsilver.uk/wp-content/uploads/2020/03/control.pdf

[3] Achiam, Joshua. Spinning Up in Deep Reinforcement Learning. Github, 2018. https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html#a-taxonomy-of-rl-algorithms

Reviewer comment (Theory):
The example used in section 4 is confusing in the context of off-policy and on-policy policy optimization. The main confusion is the conflation of a trajectory and an action, which propagates throughout the paper. This results in correcting only the distribution according to the action distribution and not the state distribution.

Author response:
We greatly appreciate your response regarding our clarification of the motivating narrative! We'll provide another update on that soon. In the meantime, we wanted to use this separate post to respond to your other comments regarding the example in section 4 and a few experimental details. 

# **Example in section 4**

You are correct in that PROPS focuses on correcting the state-conditioned distribution of actions as opposed to the distribution of trajectories (since we generally don’t know the trajectory distribution). To convey intuition for how sampling error could impact learning, our example in Section 4 focused on sampling error in the outcomes space. However, we see how this was confusing to multiple reviewers and so we can make a few minor modifications to make the example focus on action probabilities.

In particular, we can add the following action probabilities to Figure 2: 

$\pi(left | s_0) = 0.5$, 

$\pi(right | s_0) = 0.5$,

$\pi(left | s_L) = 0.02$,

$\pi(right | s_L) = 0.98$

Where, for brevity, we use $left$ = “attempt to move” and $right$ = “remain standing”. These action probabilities preserve the outcome probabilities originally listed.

After 100 visits to $s_0$, the agent may never sample the $left$ branch at $s_L$ and believe that the $left$ branch at $s_0$ always leads to a return of -1 (falling). Having observed a return of +1 (standing) in the $right$ branch at $s_0$, the agent reduces the probability of choosing the $left$ branch at $s_0$, ultimately making the agent more likely to converge to a suboptimal standing policy. We can help correct this sampling error by increasing the probability of sampling the under-sampled $left$ branch at $s_L$.

Please let us know if further clarification is needed on this updated example!

Reviewer comment (Experiments):
The empirical section would benefit from the inclusion of an off-policy baseline to better assess how much progress is being made by the adaptive sampling method. Including SAC as a baseline could be beneficial for interpreting the results, even if the proposed method does not necessarily have to outperform it.

Author response:
Thank you again for the thoughtful comments! We've uploaded a revised submission that we believe address your comments about the narrative. We decided to take your suggestion and focus on the heart of our motivation for PROPS:
1. We've reframed the introduction to focus on finite sampling error in on-policy learning.
2. Instead of asking whether on-policy algorithms require on-policy sampling or on-policy data, we now *state* that on-policy algorithms require on-policy data and that on-policy sampling is simply one way to acquire it.

We're glad to see that you think our work would make a great contribution :) Please let us know if this any concerns remain about the narrative!

We've also included SAC as a baseline in Appendix H. In general, SAC is more data efficient than PROPS but 5-10x slower in terms of wall clock time. In Hopper-v4, PROPS is fairly competitive with SAC. Humanoid-v4 results are still pending; SAC agents take much longer to train on this task with our computing resources. We can include Humanoid results in a camera-ready version of this work.

Reviewer comment (Writing):
The introduction can be clarified to better distinguish between on-policy sampling and on-policy data. The focus on this distinction may be less helpful than highlighting that on-policy methods need on-policy data for updates, and that data collection should consider the eventual replay data distribution. The paper shows that on-policy sampling can result in the replay appearing quite off-policy.

Author response:
# **The distinction between on-policy sampling and on-policy data in the Introduction**

We framed the introduction in this way to emphasize that the focus with on-policy algorithms should be on the data, not the procedure for generating the data. In the literature, we find that this point is often overlooked. For instance,  Sutton & Barto’s RL textbook [1] states that “the distinguishing feature of on-policy methods is that they estimate the value of a policy while using it for control.”

Reviewer comment (Presentation):
Consider using PPO-buffer as an example to motivate the issue, with an explanation showing that while data is sampled on-policy, the replay buffer may still contain off-policy data from past policies, leading to potential failure in policy improvement with PPO updates. A diagram illustrating replay buffer data from multiple policies could help convey this.

Author response:
# **Example in Section 4**

While we agree that an example with a replay buffer would clearly illustrate how historic off-policy data can bias updates, we wanted to emphasize that sampling error can be a problem even when all data in our buffer comes from the current target policy. In fact, in Fig. 15 and 16 of Appendix E in the supplemental material, we show that PROPS can improve the data efficiency of on-policy learning even when the replay buffer contains *no historic data* (PROPS with b=1).

Reviewer comment (Writing):
The point at the beginning of section 4 could be made clearer by using an example with a replay buffer containing data from multiple policies, rather than arguing that a sample from a single policy may look off-policy, which is harder to establish and less of a practical issue.

Author response:
# **Example in Section 4**

While we agree that an example with a replay buffer would clearly illustrate how historic off-policy data can bias updates, we wanted to emphasize that sampling error can be a problem even when all data in our buffer comes from the current target policy. In fact, in Fig. 15 and 16 of Appendix E in the supplemental material, we show that PROPS can improve the data efficiency of on-policy learning even when the replay buffer contains *no historic data* (PROPS with b=1).

Reviewer comment (Experiments):
The KL penalty in Eq. 5 and the KL rule in line 9 of Alg. 2 are under-investigated. The KL regularizer likely contributes to making collected data more on-policy, and it would be helpful to understand its impact on the replay buffer's on-policiness.

Author response:
# **Ablations on KL regularizer and clipping mechanism**

We ablate PROPS’s KL regularizer and the clipping mechanism in Appendix C and D and find that both help reduce sampling error and improve agent performance:

* Appendix C, Fig. 9 shows how clipping and regularization affect PROPS’s ability to reduce sampling error with respect to a *fixed* target policy. In this setting, clipping generally has a stronger effect on sampling error reduction than regularization.

* Appendix D, Fig 12 shows how clipping and regularization affect PROPS’s ability to reduce sampling error in the RL setting with a *continually changing* target policy. In this setting, both regularization and clipping are crucial to reducing sampling error.

* Appendix D, Fig 13 shows IQM return achieved by PROPS during RL training with and without regularization or clipping. In nearly all tasks, PROPS performs best with both clipping and regularization.

Reviewer comment (Evaluation):
In Figure 3, the KL regularization increases the on-policiness of the data, unlike ROS, which was motivated to achieve the same goal. Additionally, making data more on-policy might also result from clipping and regularization preventing large deviations between updated and pre-update target policies.

Author response:
# **Ablations on KL regularizer and clipping mechanism**

We ablate PROPS’s KL regularizer and the clipping mechanism in Appendix C and D and find that both help reduce sampling error and improve agent performance:

* Appendix C, Fig. 9 shows how clipping and regularization affect PROPS’s ability to reduce sampling error with respect to a *fixed* target policy. In this setting, clipping generally has a stronger effect on sampling error reduction than regularization.

* Appendix D, Fig 12 shows how clipping and regularization affect PROPS’s ability to reduce sampling error in the RL setting with a *continually changing* target policy. In this setting, both regularization and clipping are crucial to reducing sampling error.

* Appendix D, Fig 13 shows IQM return achieved by PROPS during RL training with and without regularization or clipping. In nearly all tasks, PROPS performs best with both clipping and regularization.

Reviewer comment (Presentation):
Figure 5 should be placed earlier, closer to its discussion.

Author response:
* We will try to keep figures on the same page on which they are referenced.

Reviewer comment (Presentation):
References should be sorted alphabetically rather than by citation order to make them easier to refer to.

Author response:
* We will order references alphabetically.

Reviewer comment (Novelty):
The paper's broader contribution beyond the problem of on-policy methods with a replay is unclear. Specifically, how does it relate to the broader problem of increasing sample efficiency using a replay? Are adjusted on-policy methods with a replay a suitable alternative to off-policy methods?

Author response:
We want to begin by answering your question about the work’s broader impact. The core contribution of our work is that we demonstrate how *off-policy sampling* can reduce sampling error and improve the data efficiency of *on-policy* policy gradient algorithms. Our work also emphasizes that on-policy algorithms simply require on-policy data and are agnostic to how the data is generated – a point which is often overlooked in the literature. 

# **Should we prefer PROPS over other off-policy methods?**

We view on-policy learning with replay as a means to improve the data efficiency of on-policy learning rather than an alternative to off-policy learning. On-policy and off-policy algorithms have different advantages and disadvantages, and the choice of algorithm typically depends on what advantages are most relevant for a given task. For instance, PPO has played a role in several RL success stories [*e.g.*, 2, 3] in part because it is empirically stable and robust to hyperparameter choices.