Paper:
As reinforcement learning continues to drive machine intelligence beyond its conventional boundary, unsubstantial practices in sparse reward environment severely limit further applications in a broader range of advanced fields. Motivated by the demand for an effective deep reinforcement learning algorithm that accommodates sparse reward environment, this paper presents Hindsight Trust Region Policy Optimization (HTRPO), a method that efficiently utilizes interactions in sparse reward conditions to optimize policies within trust region and, in the meantime, maintains learning stability. Firstly, we theoretically adapt the TRPO objective function, in the form of the expected return of the policy, to the distribution of hindsight data generated from the alternative goals. Then, we apply Monte Carlo with importance sampling to estimate KL-divergence between two policies, taking the hindsight data as input. Under the condition that the distributions are sufficiently close, the KL-divergence is approximated by another f -divergence. Such approximation results in the decrease of variance and alleviates the instability during policy update. Experimental results on both discrete and continuous benchmark tasks demonstrate that HTRPO converges significantly faster than previous policy gradient methods. It achieves effective performances and high data-efficiency for training policies in sparse reward environments.

Review:
*Summary* : This paper augments the TRPO policy optimization objective with hindsight data, where the hindsight data is generated from goals based on trajectories. The key contribution of the paper is based on deriving an on-policy adaptation of hindsight based TRPO, that can be useful for sparse reward environments. The paper draws ideas from existing papers such as HPG and considers the IS based variant of HPG for on-policy, similar to TRPO, that can achieve monotonic performance improvements. Furthermore, the authors introduce a logarithmic form of constraint, by re-deriving the KL constraint and leading to a f-divergence based constraint, which is argued to have useful effects in terms of lowering the variance. Experimental results are compared with baselines including HPG and its variants on standard sparse reward benchmark tasks. 
*Comments* : 
	- The core contribution of the paper is to introduce hindsight based TRPO where end states from the trajectory treated as goals can be useful for generating pseudo-rewards, such that existing TRPO based approaches can be better suited for sparse reward environments. The claim is that existing policy gradient methods cannot sufficiently explore in sparse reward environments. 
	- Since incorporating hindsight data can make the approach off-policy in nature, leading to higher variance and instability, the authors propose to augment hindsight approach into on-policy based methods such as TRPO. The key contribution is to develop a theoretically sound approach for hindsight based policy optimization. 
	- The hindsight TRPO objective is derived based on goal-conditioned policies with TRPO, based on existing recent work on Hindsight Policy Gradient (HPG) (Rauber et al., 2019). The key difficulty that needs to be tackled is when introducing hindsight experience makes the approach off-policy in nature. The authors derive the HTRPO objective based on an importance sampling approach that can also incorporate the hindsight data. 
	- Equation 7 writes out the key quantity, an IS based approach considering the current goal and alternative goals to derive a similar TRPO objective based on IS based Monte-Carlo estimators, while maintaining the trust region guarantees. Equation 8 further shows the gradient of the objective. Theorem 3.1 and 3.2 follows naturally where the key trick in Thm 3.1 is going from equation 19 to 20 to derive the IS based estimator with goal conditioned trajectories. I am not sure why Thm 3.2 needs to be written out explicitly, given that it follows naturally for gradient of the expectation? Is there any key takeaway from it? 
	- In TRPO, the expectation is based on states sampled from a state distribution. The authors argue that for hindsight based data, this state distribution in fact can change due to changes in the generated goals (ending states), and hence the KL needs to be in expectation w.r.t to the state occupancy measure. Furthermore, the authors change the KL based constraint into a logarithmic form of a constraint such as to reduce variance and introduce more stability. To achieve this, the paper uses an approximation to the logarithmic form of the constraint, by using an expectation of the square instead of plain differences between the log of policies. The key is that instead of using the KL divergence, the authors introduce f-divergence where the function is convex allowing smooth optimization. 
	- The overall contribution of the paper can be summarized from equation 15 - introducing a IS based correction, while remaining on-policy for hindsight based TRPO objective. And since hindsight can change the underlying state distribution, leading to more instability, the paper introduces a different form of constraint (based on the f-divergence) which can have lower variance than the KL form of constraint. 
	- Experimental results are demonstrated for few sparse reward benchmarks, comparing to the standard HPG, TRPO and several variants of the proposed HTRPO approach with weighted IS and with the KL constraint. The advantages of HTRPO on these tasks seems clear, mainly in the Fetch Reach and Fetch Push tasks, significantly outperforming the baselines. Even in the continuous tasks, HTRPO seems to outperform the baselines consistently. 
*Feedback/Questions* : 
	- I am not sure of the significance of Theorem 3.2 - it seems obvious that the gradient of the objective spans out naturally from equation 7? 
	- The authors mention about the state occupancy measure instead of considering the state distribution for the KL term. However, the discussion of state occupancy measure seemed to have faded away? What was the significance of mentioning it, or why should it be considered even? There are no assumptions being made on whether the state distribution should be the discounted state occupancy measure or the stationary distribution (if infinite horizon is considered). 
	- The introduction of the logarithmic form of constraint, even though shows theoretically to reduce variance, is not well motivated or demonstrated from the suite of experimental results? From the results, it is not obvious whether this form of constraint is indeed having useful effects in terms of reduced variance? 
	- The paper seems to adapt from the HPG objective, and does indeed a great job comparing to HPG throughout the suite of experiments. However, in the results, the paper mainly compares to other off-policy based methods including HPG and its variants (official and re-implemented one). I find the comparison of results a bit odd in that sense, since it is comparing the on-policy adaptation of HPG (ie, HTRPO) and the off-policy variants? If run for long enough, does all converge to the same results? If so, then the benefits is mainly in faster learning (assumably due to better exploration in the sparse reward tasks). But then again, these benefits may be because of the on-policy approach compared to the off-policy one? 
	- I would encourage the authors to compare to more standard goal-conditioned or reward shaping based baselines for TRPO. For example, does the proposed HTRPO approach perform better compared to other goal-conditioned approaches of TRPO, or for example if a form of reward shaping (based on goals) are used in TRPO? It would then be a more likewise comparison? The current results seem to show benefits of HTRPO, but I think there is a need for stronger baselines where TRPO + exploraton (reward shaping or goal conditioning) performs better?
	- I am not convinced about the arguments with sensitivity of HTRPO with network architecture and parameters. How is this demonstrated from the suite of results?

*Summary of Review and Score* : 
Overall, I think the paper has merits, mainly in terms of deriving the on-policy adaptation with hindsight data. The key objectives are derived nicely in the write-up and easy to follow, although there are some confusions that need to be clarified (example : the discussion on state occupancy measure and the significance of it). The paper motivates exploration for TRPO in sparse reward tasks, and considers the hindsight adaptation of existing TRPO. But related works such as HPG have already taken a similar approach for the off-policy case, and this paper's key contribution is in terms of theoretically deriving the objectives for on-policy adaptation. However, I am not convinced about the overall merits and contributions of the paper, especially in terms of demonstrated results and proper comparisons with baselines. I think while the objectives and derivations follow naturally, the contributions of the paper is somewhat marginal. 
I would therefore vote to marginally reject this paper - mainly in light of the core novel contribution and due to lack of sufficient results demonstrating the usefulness of the approach. The paper combines several things together - especially discussions of the logarithmic form of the constraint. I doubt whether all these introduced together led to the improvements shown in the experimental results or not. It would be useful to clarify the contributions from each of the components.