Paper: Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
The claim about the synergies between DQD and PPO looks insufficiently backed-up. In particular, the main paper does not even mention the TD3GA algorithm, while the study of combining DQD with TD3 is crucial to understand these synergies. More generally, your central claim is that using on-policy RL better fits the DQD framework, so the comparison to TD3GA should be central.

Author response:
We appreciate the incredibly in-depth feedback. Points on writing brought up by the reviewer that require no discussion (including the reviewer's main concern on making the TD3GA comparison central) have been directly implemented in the paper, and we have updated the pdf accordingly.

Reviewer comment (Writing):
In the abstract, the phrase “and struggle with highly stochastic environments” does not clearly correspond to a solution proposal in the rest of the abstract. If you mention this issue, you should also mention how you will solve it.

Author response:
**the rest of the abstract makes it clear that you switch to on-policy RL, but it does not make clear that you will solve the highly stochastic environments issue.**

PPGA inserts policies into the archive based on their average performance and behavior over 10 parallel environments. We find empirical evidence that this indeed improves robustness to uncertainty. We have added this point to the end of section 3.2. Section 4.2 also investigates PPGA's robustness to uncertainty.

Reviewer comment (Writing):
In the abstract, “propose several changes” is awkwardly formulated; changes with respect to what? Since you have a new algorithm, you are proposing new things, and there is no need to phrase it as “changes.”

Author response:
**"propose several changes" -> changes wrt what?**

We wish to convey other changes, such as replacing CMA-ES with xNES, etc., that are subtle improvements over existing work that have large implications for our method (ex. CMA-ES destabilizing training). To improve clarity, we changed this to read "propose additional improvements over prior work".

Reviewer comment (Writing):
In introduction paragraph 2, the phrase “all QD-RL algorithms for locomotion to date use non-differentiable measure functions, which in most cases prevents direct RL optimization” is problematic because the term "measure function" is undefined for the standard ML reader, and because it is questionable whether non-differentiable measure functions prevent direct RL optimization.

Author response:
**At this point, the standard ML reader does not know what a measure function is...**

**I don't agree that having non-differentiable measure functions prevents direct RL optimization...**

Per the reviewer's suggestions, we have changed this to read "For example, all QD-RL algorithms for locomotion to date use non-markovian measures of behavioral diversity, which in many cases prevents direct RL-optimization."

Reviewer comment (Writing):
In the introduction, the phrase “can we better adapt DQD algorithms to challenging RL domains by exploiting its synergy with PPO?” repeats the “exploiting this synergy” idea for the third time in the same paragraph.

Author response:
**the third time you mention exploiting this synergy in the same paragraph...**

We changed this to read "It is through this high level view that we see the emergent synergy between PPO and DQD, in that PPO can be used to collect gradient estimates from online data when one or both of the objective and measure functions are markovian and non-differentiable."  We have also removed redundant uses of the word "synergy" from the manuscript.

Reviewer comment (Writing):
The term “non-stationary QDRL task” is unclear and should be clarified.

Author response:
**What a "non-stationary QDRL task"**

We have expanded on this definition at the beginning of section 3.2.

Reviewer comment (Theory):
The classification in the Deep RL background should make clear that PPO is on-policy; currently it is misplaced under “trust region.”

Author response:
**I think the point you want to make is that PPO is on-policy. In the current classification, it is not in the "on-policy" part, but in the "trust region" part**

We wish to make both points i.e. the on-policy classification is synergistic with DQD, and that the approximate trust region is a nice property to have when the reward function is non-stationary.

Reviewer comment (Theory):
The mention of actor-critic in the on-policy part is confusing, as many actor-critic algorithms (DDPG, TD3, SAC) are off-policy.

Author response:
**Mentioning actor-critic in the on-policy part the way you do brings confusion...**

We have updated the DeepRL background section according to the reviewer's suggestions.

Reviewer comment (Writing):
The term “some minimum threshold” is unclear; “threshold for what?” It may be better described as an exploration bonus.

Author response:
**"some minimum threshold". Why call this a minimum threshold? Threshold for what? I rather see it as an exploration bonus...**

In some cases, if the threshold value is set too low, the algorithm can initially fill the archive with very poor performing policies from which gradient arborescence cannot recover the true optimal policies. Intuitively, this can be thought of as seeding the archive with policies too far from the manifold and then attempting to perform search nowhere near the manifold. This min-threshold value prevents too poorly performing policies from entering the archive in the first place.

Reviewer comment (Writing):
The phrase “in the direction of the archive that is least” should clarify whether performance is also taken into account.

Author response:
**"in the direction of the archive that is least" -> doesn't this also take performance into account?** 

You are correct, although the ranking implicitly favors policies that land in new cells over those that improve on known ones. We have rephrased this to say "in the direction of greatest archive improvement"

Reviewer comment (Writing):
In section 2.4, the phrasing “In prior work (ref) ... In this work” reveals the authorship; self-references should follow anonymity rules to avoid desk rejection.

Author response:
**In 2.4: "In prior work (ref) ... In this work" -> You should not write this in a way that let us know who you are.** 

We have done a pass through the introduction and related works sections of the paper and re-balanced the citations.

Reviewer comment (Theory):
In section 3.2, the claim that trust region methods provide formal guarantees on the quality of gradient estimates is misleading, as TRPO’s proof relies on unrealistic assumptions violated in practice.

Author response:
**In 3.2, I'm afraid the claim that trust region methods provide formal guarantees on the quality of the gradient estimates is wrong.**

While we agree that papers often make unrealistic assumptions to formalize proofs, this theory-practice gap is ubiquitous in nature, and present in other areas of AI and robotics research such as in robot safety, multi-agent planning and collision avoidance, etc. Rather than saying the proof of TRPO formally holds in the QD-RL setting, we wish to convey the idea that constrained optimization of the policy, especially in settings with non-stationary objective functions, is a nice property to have rather than not. We have softened the language in the manuscript to better convey this. "Being an approximate trust region method, the constrained policy update step provides some robustness to noisy and non-stationary objectives."

Reviewer comment (Evaluation):
Clarify whether using MMPs as rewards implies seeking as much leg contact as possible, or leg contact diversity; this remains unclear.

Author response:
**You use the MMPs as rewards to optimize. So it means that your algorithm is looking for as much leg contact as possible. Shouldn't it be looking for as much "leg contact diversity" as possible instead? This is unclear. Could you clarify?**

That is correct, we estimate the gradient to increase foot contact time. However, during the branching phase, the gradient coefficients we sample are unconstrained. We could sample a range of positive and negative coefficients, resulting in a range of lower and higher foot contact times of varying degree. The onus is thus on the evolution strategy to update the search distribution such that we pick gradient coefficients that maximally improve the archive, which implies maximizing diversity.

Reviewer comment (Writing):
The description of modifying computation of the policy gradient into a batched method is unclear and could benefit from a diagram or equations.

Author response:
**"We additionally modify the computation of the policy gradient into a batched policy gradient method, where intermediate gradient estimates of each function w.r.t. policy params only flow back to the parameters corresponding to respective individual policies during minibatch gradient descent." -> This part is very unclear to me, a small diagram or equations would probably help.**

While estimating the objective-measure Jacobian, we use a single, vectorized policy $\pi_{\theta}$ s.t. $\theta = <\theta^1, ..., \theta^{k+1}>$, where each of the $k+1$ parameter vectors correspond to parameters for separate behavior policies optimizing separate objectives. Since they correspond to different policies, we must slice the total number of environments $|E|$ into $k+1$ chunks of size $\frac{|E|}{k+1}$. Each i'th chunk will generate a batch of data  $B^i = (s_{t}, a_{t}, r_{t}, s_{t+1}, ..., s_{t+K})^i$, where $K$ is the rollout length. During the update step, the advantages and importance sampling ratios are only computed within each batch i.e. we do not mix data across batches. While performing gradient descent, the losses across batches are summed and backpropagated through the computation graph of the vectorized policy. However, since only the $i'th$ sub-policy $\pi_{\theta^i}$ was used to create the i'th batch of data $B^i$, only the gradients corresponding to the i'th loss term will flow back to these parameters inside of the vectorized policy. This essentially allows us to jointly optimize $\pi_{\theta^1}, ..., \pi_{\theta^{k+1}}$ without spawning separate PPO instances for each policy and sequentially computing the gradients of $f, m_{1}, ..., m_{k+1}$.

Reviewer comment (Experiments):
The discussion on variance use in policies is insufficient; literature shows three options (constant variance, tunable constant variance, tunable state-dependent variance), and the approach of disabling gradient flow for variance depending on environment is unsatisfactory.

Author response:
**There are many considerations about using constant variance or not...**

To the best of our knowledge, we have not seen examples of PPO used with a fixed, unlearnable standard deviation parameter for the action distribution. We would appreciate if the reviewer could provide references to this in the literature, and we would be happy to update the paper accordingly. TRPO and PPO indeed have options for state-dependent and state-independent *learnable* standard deviation parameters. However, we are concerned with using a fixed, unlearnable standard deviation parameter as opposed to a learnable one. 

In the canonical training paradigm, the standard deviation parameter decreases as the policy approaches optimality such that actions are sampled quasi-deterministically at convergence (many RL libraries provide the option to directly use the action means instead of sampling at test time). In the DQD setting, however, this is undesirable behavior. Ideally, the parameter would adaptively decrease or increase depending on if we are converging to a peak in the objective landscape or moving away from one. In our experiments, we have found that no matter the choice of hyperparameter, the addition of exploration bonuses, etc., that enabling gradient flow to this parameter results in it monotonically decreasing to 0. As this parameter converges, it becomes increasingly difficult to sample new actions and have positive advantages, which essentially cripples the branching and walking phase of PPGA since it relies on gradient estimates from PPO. We acknowledge that making this parameter static in some environments is a nonideal solution and would like to have a more theoretically grounded approach in future iterations of this work. However, in the absent of such a tool, we instead opted for this choice and only include it in the paper to be fully transparent in our design decisions.

Reviewer comment (Presentation):
In section 3.3, the paragraph on xNES vs CMA-ES could be moved to the appendix, freeing space for the more important TD3GA vs PPGA comparison.

Author response:
**About Section 3.3, I think the first paragraph which establishes that xNES is a better option could be moved in an appendix**

We agree that section 3.3 contains unnecessary details and have simplified this to a single paragraph explaining the choice of switching from CMA-ES to xNES. We have also recovered space in other parts of the paper to move the TD3GA ablation to the main paper, as per the reviewer's suggestions.

Reviewer comment (Writing):
The “archive learning rate” is introduced without prior explanation.

Author response:
**The fact that there is such a learning rate is not explained before.**

The archive learning rate is $\alpha$ referred to in the last paragraph of section 2.3 describing CMA-MAE/MAEGA. We have made this more explicit in the revised manuscript.

Reviewer comment (Presentation):
Experimental figures and tables make poor use of space; reworking them could improve readability and save space.

Author response:
**All experimental figures and tables make a poor use of space.**

**In Fig. 4, using +/- one std as variance information is a bad practice in RL, where the variance is generally not Gaussian...**

We have revised all figures and updated all plots to use bootstrapped confidence intervals to represent the shaded regions in accordance to the papers provided by the reviewer.

Reviewer comment (Presentation):
In Figure 4, showing ± one standard deviation as variance information is bad practice in RL because the variance is often non-Gaussian; relevant literature offers better visualization approaches.

Author response:
**All experimental figures and tables make a poor use of space.**

**In Fig. 4, using +/- one std as variance information is a bad practice in RL, where the variance is generally not Gaussian...**

We have revised all figures and updated all plots to use bootstrapped confidence intervals to represent the shaded regions in accordance to the papers provided by the reviewer.

Reviewer comment (Novelty):
The statement “one of the first QD-RL methods to leverage on-policy RL” should be supported by naming the other methods.

Author response:
**"We present a new method, PPGA, which is one of the first QD-RL methods to leverage on-policy RL," -- one of the first, so what are the others?**

To the best of our knowledge, [1] and [2] are the only other concurrent works that leverage on-policy RL. However, [1] does not provide a reference implementation with PPO (only TD3) in their official repo, and [2] does not present any findings with on-policy RL, although it should, in theory, work with on-policy RL. 

[1] Wu S, Yao J, Fu H, et al. Quality-Similar Diversity via Population Based Reinforcement Learning. In The Eleventh International Conference on Learning Representations, (2023).

[2] Pierrot, Thomas, and Arthur Flajolet. "Evolving Populations of Diverse RL Agents with MAP-Elites." arXiv preprint arXiv:2303.12803 (2023).

Reviewer comment (Evaluation):
The statement “DQD algorithms and on-policy RL have emergent synergies” is not convincingly demonstrated; a dedicated paragraph is needed to explain investigations of these synergies.

Author response:
**"We show that DQD algorithms and on-policy RL have emergent synergies that make them work particularly well with each other." -- I'm sorry but this point does not emerge clearly from reading the paper. You should have somewhere a paragraph about the investigations of these synergies.**

In sections 2 and 3, we attempt to convey how on-policy RL is particularly well-suited for the DQD setting, and the specific design decisions made to exploit this synergy. In section 4, we show the resulting formulation yields state of the art results compared to baselines. The choice of ablations and baselines attempts to answer the counterfactuals "what if component X of PPGA were replaced with component Y?". For example, what if PPO were replaced with TD3 and all other components were held constant? To this end, we completely agree with the reviewer that certain comparisons such as PPGA vs TD3GA should be central to the paper, and have updated the manuscript accordingly. However, we believe these comparisons justify the claim about the synergies between on-policy RL and DQD.

Reviewer comment (Presentation):
The main paper slightly exceeds the 9-page limit; this could have led to desk rejection, so rules should be followed.

Author response:
**Your paper needs slightly more than 9 pages. Again, it could have been desk-rejected for that. Please follow the rules.**

All content of our paper fits within the 9 pages --- the section after our paper content is a reproducibility statement that does not count towards the page limit (author guidelines: https://iclr.cc/Conferences/2024/AuthorGuide). We apologize for any confusion.

Reviewer comment (Experiments):
The paper could benefit from a more detailed comparison with other QD-RL methods and an explanation of why PPO was specifically chosen over other RL algorithms.

Author response:
**Can you provide more insight into why PPO was specifically chosen over other RL algorithms for this work?**

Regarding other QD-RL methods, we selected recent methods that have demonstrated high performance and have accessible open-source implementations. In particular, we chose PGA-ME, the current state-of-the-art, as well as QD-PG, separable CMA-MAE, and CMA-MAEGA (TD3, ES), all of which have shown competitive performance in prior work. Appendix H also includes a comparison to PBT-ME (SAC), which we only ran on Humanoid due to computational expense. We believe these baselines are representative of the current state of the QD-RL field. 

Regarding PPO, we believe PPO is an ideal choice for integration into QD-RL algorithms for several reasons. First, PPO can leverage the high parallelization of GPU-accelerated simulators such as Brax to significantly accelerate learning. Current QD-RL methods rely on off-policy RL algorithms such as TD3 and SAC, which are designed to be sample-efficient and benefit from faster sequential execution. In other words, these off-policy methods operate with minimal amounts of data. We are interested in the other end of this spectrum -- that is, when large amounts of data are available, what kind of methods benefit the most? There is a large body of evidence that shows PPO continues to be competitive in these regimes even on contemporary robot learning problems [1, 2, 3]. This was a significant factor when choosing which RL algorithm to investigate. 

Finally, regarding our choice of PPO over other on-policy methods such as IMPALA, PPO's use of an approximate trust region to constrain the policy update empirically provides robustness to noisy and even non-stationary objectives. This is especially important in the QD-RL setting, where the branching phase and walking the search policy require optimizing non-stationary objectives. 

[1] Rudin, Nikita, et al. "Learning to walk in minutes using massively parallel deep reinforcement learning." Conference on Robot Learning. PMLR, 2022.

[2] Handa, Ankur, et al. "DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality." arXiv preprint arXiv:2210.13702 (2022).

[3] Batra, Sumeet, et al. "Decentralized control of quadrotor swarms with end-to-end deep reinforcement learning." Conference on Robot Learning. PMLR, 2022.

Reviewer comment (Experiments):
The generalizability of the proposed algorithm to other domains and tasks beyond locomotion is not discussed.

Author response:
**How does the proposed PPGA algorithm perform in other domains and tasks beyond locomotion?**

For this paper, we focused on comparing the algorithmic design choices of PPGA and the resulting performance differences against prior SOTA QD-RL algorithms. As such, we ran PPGA on benchmarks that were introduced in [1] and have become standard in the field for a majority of QD-RL methods.
However, we are interested in applying PPGA to non-standard QD tasks with discrete action spaces such as Atari. 
Given the prior success of PPO in a wide variety of tasks with discrete and continuous actions spaces, we believe PPGA has a good chance of succeeding in these tasks.

[1] Olle Nilsson and Antoine Cully. 2021. Policy gradient assisted MAP-Elites. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '21). Association for Computing Machinery, New York, NY, USA, 866–875. https://doi.org/10.1145/3449639.3459304

Reviewer comment (Experiments):
It would be better to see some discussion about the scalability of the proposed algorithm, especially in comparison to existing methods.

Author response:
**Can you discuss the scalability of the proposed algorithm, especially in comparison to existing methods?**

We found empirical evidence of scalability of our method from two perspectives: scaling to larger archives and learning from more data.
We present results in scaling to larger 50x50 archives and compare to baselines in Appendix G. 
PPGA scales well and maintains state of the art performance over baselines when archives of thousands of policies are concerned.
We believe this is a direct result of the fact that, like prior DQD-based algorithms, PPGA *explicitly* estimates the gradient of greatest archive improvement via gradient arborescence. 
That is, the information on what new cells were discovered during the branching phase is utilized by xNES to compute the natural gradient w.r.t. archive improvement. 
Critically, the higher resolution the archive i.e. the more cells there are, the better this gradient estimate is, and so PPGA may in fact favor larger archives. 

Finally, we demonstrate PPGA's ability to continually learn with more data by running an additional experiment where PPGA and the current state of the art baseline PGA-ME were run for 1.2 million evaluations, or twice as long as the experiments in the main paper, on larger 50x50 archives. The results have been added to Appendix G. We notice that the average performance of the archive, i.e., the average performance across all policies in the archive, continues to increase, suggesting that PPGA continuously optimizes for performance after the initial discovery phase. We observe this effect to a much lesser extent with PGA-ME.

Reviewer comment (Experiments):
Can you provide more insight into why PPO was specifically chosen over other RL algorithms for this work?

Author response:
**Can you provide more insight into why PPO was specifically chosen over other RL algorithms for this work?**

Regarding other QD-RL methods, we selected recent methods that have demonstrated high performance and have accessible open-source implementations. In particular, we chose PGA-ME, the current state-of-the-art, as well as QD-PG, separable CMA-MAE, and CMA-MAEGA (TD3, ES), all of which have shown competitive performance in prior work. Appendix H also includes a comparison to PBT-ME (SAC), which we only ran on Humanoid due to computational expense. We believe these baselines are representative of the current state of the QD-RL field. 

Regarding PPO, we believe PPO is an ideal choice for integration into QD-RL algorithms for several reasons. First, PPO can leverage the high parallelization of GPU-accelerated simulators such as Brax to significantly accelerate learning. Current QD-RL methods rely on off-policy RL algorithms such as TD3 and SAC, which are designed to be sample-efficient and benefit from faster sequential execution. In other words, these off-policy methods operate with minimal amounts of data. We are interested in the other end of this spectrum -- that is, when large amounts of data are available, what kind of methods benefit the most? There is a large body of evidence that shows PPO continues to be competitive in these regimes even on contemporary robot learning problems [1, 2, 3]. This was a significant factor when choosing which RL algorithm to investigate. 

Finally, regarding our choice of PPO over other on-policy methods such as IMPALA, PPO's use of an approximate trust region to constrain the policy update empirically provides robustness to noisy and even non-stationary objectives. This is especially important in the QD-RL setting, where the branching phase and walking the search policy require optimizing non-stationary objectives. 

[1] Rudin, Nikita, et al. "Learning to walk in minutes using massively parallel deep reinforcement learning." Conference on Robot Learning. PMLR, 2022.

[2] Handa, Ankur, et al. "DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality." arXiv preprint arXiv:2210.13702 (2022).

[3] Batra, Sumeet, et al. "Decentralized control of quadrotor swarms with end-to-end deep reinforcement learning." Conference on Robot Learning. PMLR, 2022.

Reviewer comment (Experiments):
How does the proposed PPGA algorithm perform in other domains and tasks beyond locomotion?

Author response:
**How does the proposed PPGA algorithm perform in other domains and tasks beyond locomotion?**

For this paper, we focused on comparing the algorithmic design choices of PPGA and the resulting performance differences against prior SOTA QD-RL algorithms. As such, we ran PPGA on benchmarks that were introduced in [1] and have become standard in the field for a majority of QD-RL methods.
However, we are interested in applying PPGA to non-standard QD tasks with discrete action spaces such as Atari. 
Given the prior success of PPO in a wide variety of tasks with discrete and continuous actions spaces, we believe PPGA has a good chance of succeeding in these tasks.

[1] Olle Nilsson and Antoine Cully. 2021. Policy gradient assisted MAP-Elites. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '21). Association for Computing Machinery, New York, NY, USA, 866–875. https://doi.org/10.1145/3449639.3459304

Reviewer comment (Experiments):
Can you discuss the scalability of the proposed algorithm, especially in comparison to existing methods?

Author response:
**Can you discuss the scalability of the proposed algorithm, especially in comparison to existing methods?**

We found empirical evidence of scalability of our method from two perspectives: scaling to larger archives and learning from more data.
We present results in scaling to larger 50x50 archives and compare to baselines in Appendix G. 
PPGA scales well and maintains state of the art performance over baselines when archives of thousands of policies are concerned.
We believe this is a direct result of the fact that, like prior DQD-based algorithms, PPGA *explicitly* estimates the gradient of greatest archive improvement via gradient arborescence. 
That is, the information on what new cells were discovered during the branching phase is utilized by xNES to compute the natural gradient w.r.t. archive improvement. 
Critically, the higher resolution the archive i.e. the more cells there are, the better this gradient estimate is, and so PPGA may in fact favor larger archives. 

Finally, we demonstrate PPGA's ability to continually learn with more data by running an additional experiment where PPGA and the current state of the art baseline PGA-ME were run for 1.2 million evaluations, or twice as long as the experiments in the main paper, on larger 50x50 archives. The results have been added to Appendix G. We notice that the average performance of the archive, i.e., the average performance across all policies in the archive, continues to increase, suggesting that PPGA continuously optimizes for performance after the initial discovery phase. We observe this effect to a much lesser extent with PGA-ME.

Reviewer comment (Experiments):
I have a slight concern that all the ablation studies show a dramatic decrease in performance. Could finding a better set of hyperparameters instead of inheriting the PPGA hyperparameters help?

Author response:
In the xNES vs CMA-ES ablation, we made two attempts using two separate, highly-tuned and optimized reference implementations of CMA-ES inside PPGA provided by [1] and [2]. We noticed the training divergence in both instances across multiple seeds.

Regarding the TD3GA ablation, we use a reference TD3 implementation used by other off-policy QD-RL papers such as PGA-ME for QD-RL tasks. Admittedly, certain hyperparameters such as the optimal size and update schedule of the replay buffer in high-throughput regimes remains an open question that could be further optimized. However, given that using TD3 as a drop-in replacement for PPO in PPGA leads to a fundamentally different learning paradigm, we argue that this algorithm and an involved hyperparameter search warrants its own proper investigation as a separate line of work. 

[1] Tjanaka, Bryon, et al. "pyribs: A bare-bones Python library for quality diversity optimization." arXiv preprint arXiv:2303.00191 (2023).

[2] Nikolaus Hansen, Youhei Akimoto, and Petr Baudis. CMA-ES/pycma on Github. Zenodo, DOI:10.5281/zenodo.2559634, February 2019.