Paper: Behind the Myth of Exploration in Policy Gradients (ICLR-2024)

Reviewer comment (Theory):
The contribution is entirely empirical (there are some nice theory concepts, but no proofs or useful properties are shown except empirically), and the empirical results are based entirely on extremely simple toy problems. In my opinion, this does not invalidate their interesting perspectives on the issues that they illustrate with these results, but it does make the contribution less substantial.

Author response:
We acknowledge that the paper was rather vague on this point, but the criteria also introduce theoretical guarantees.  
Following your comment, Sections 3.1 and 4.1 have been reworked to better highlight the assumptions and results they imply.  
In particular, if our two criteria in Section 3 are met (note that we have changed the name of the second to better fit in the literature), it is possible to compute a policy by stochastic gradient ascent that is suboptimal by at most epsilon.  
Under the assumptions clarified in Section 4.1, the probability of improvement defines the probability of having a good solution after optimization.  
We agree, however, that there is future work to establish additional theoretical convergence guarantees and have introduced some avenues of reflection in conclusion.  

Regarding generalization, we acknowledge that our illustrations were conducted on relatively simple use cases.  
The low-dimensional spaces of states, actions, and parameters were chosen for clarity in visual representation.  
In addition, the experimental results were computed through exhaustive search in the parameter space combined with Monte-Carlo estimations, constraining the parameter space to low dimensions.  
Moreover, most exploration methods rely on function approximators for dealing with high dimensions, introducing complexities in learning dynamics that prevent isolating the sole effect of exploration.  
Nevertheless, we believe that the strongly non-linear dynamics in the first environment and the sparse reward function in the second reflects sufficient complexity from an optimization perspective.

Reviewer comment (Writing):
The paper overall is a bit challenging to read. For an example of a well-written bit that does *not* have this issue, the conclusion says “First, it modifies the learning objective in order to remove local extrema. Second, it modifies the gradient estimates and increases the likelihood that the update steps lead to optimal policies.” This is great; more “high-level summary” passages like this in sections 3-4 would’ve made it much easier to read. For an example of a hard-to-understand part, see the last question below.

Author response:
We acknowledge your remarks concerning the clarity and have modified the paper substantially, including the two particular points you mentioned.  
In particular, the revised introduction aims to provide a clearer high-level view of the paper, related works, and contributions.

Reviewer comment (Theory):
In 4.1, J^d seems less like an exploration term, and more like a “reward-engineering” term that simply makes the problem easier. Am I missing a perspective on this? This leads to a larger concern, in that much of the contribution of 4.1 hinges on this term, and I have doubts about whether this term can be legitimately thought of as an exploration bonus that is superior to the entropy bonus.

Author response:
Firstly, we have clarified in the introduction the distinction between exploration in policy gradient as intended in theory, i.e. optimizing a sufficiently stochastic policy, and how it is implemented in practice, i.e. involving reward shaping and optimizing a surrogate learning objective to increase stochasticity.  
In our paper, we analyse the second aspect by studying the exploration-based learning objective functions.  
Nevertheless, no assumption is made on the structure of the intrinsic rewards that may thus be built with any reward-shaping strategy.  
The analyses and discussions thus hold true for the expert-knowledge reward shaping in Section 4.1, which illustrates the potential achievements of exploration when well-designed for the problem at hand.  
We furthermore emphasize that all discussions are also validated on the standard entropy regularized objective.

Reviewer comment (Reproducibility):
My concern lies in practicality. While intuitively, the proposed four criterion all make sense, however, for a general RL question, how to effectively compute these four measures is somehow challenging.

Author response:
The purpose of the paper is to provide a framework facilitating the interpretation of policy gradients and to identify relevant criteria for obtaining guarantees in policy gradients.
As is often the case when theoretical criteria are proposed, their empirical verification is as complex as solving the problem at hand.
If such criteria were easy to verify empirically, the problem would certainly be much more easy to solve too.
However, we still believe that these criteria provide a good account of the effect of current exploration techniques, and their trade-offs.
Moreover, we think that they provide some valuable, and often lacking, intuition for the design of new exploration techniques.
We have added the development of more problem-specific bounds to our future work.

Reviewer comment (Experiments):
Question on Practically: current simulations are good for readers to get the idea of the paper, but insufficient to demonstrate the capacity. It would be nice if the authors could take a concrete example, say a maze, with a few popular regularized objectives, compute (estimate) the proposed four criterion of these objectives, see which one is more effective based on these criterion, and then validate such prediction on the trained agent's behaviors.

Author response:
The examples primarily serve to illustrate the criteria, which are derived from optimization theory.
The experimental results were computed through exhaustive search in the parameter space combined with Monte-Carlo estimations, constraining the parameter space to low dimensions.
Moreover, most exploration methods rely on function approximators for dealing with high dimensions, introducing complexities in learning dynamics that prevent isolating the sole effect of exploration.
Nevertheless, we believe that the strongly non-linear dynamics in the first environment and the sparse reward function in the second reflects sufficient complexity from an optimization perspective.
We have also clarified in the introduction that finding good exploration strategies is known to be problem specific and that we thus introduce a general framework for the study and interpretation of exploration in policy gradient methods instead of trying to find the best exploration method for a given task.