Paper: PEAR: Primitive enabled Adaptive Relabeling for boosting Hierarchical Reinforcement Learning (ICLR-2024)

Reviewer comment (Presentation):
The learning curve figures are too small and very difficult to see. It would be much appreciated if the authors could fix this.

Author response:
**1. "The learning curve figures are too small and very difficult to see. It would be much appreciated if the authors could fix this."**

Response:

We apologize for the figure size and clarity issues. We have now fixed the issues in the paper. We hope this improves the overall clarity.

Reviewer comment (Presentation):
In Algorithm 1 for Lines 5-13 it could improve clarity if the authors included comments on what each line is doing. I found this part a bit difficult to understand.

Author response:
**2. "In Algorithm 1 for Lines 5-13 it could improve clarity if the authors included comments on what each line is doing. I found this part a bit difficult to understand."**

Response:

We thank the reviewer for pointing this out. We have now added the comments in Algorithm 1 wherever applicable. We hope that this improves the clarity of the algorithm psuedo-code.

Reviewer comment (Experiments):
It seems it is necessary to manually set $Q_{\text{thresh}}$ for each environment.

Author response:
**3. "It seems it is necessary to manually set $Q_{thresh}$ for each environment."**

Response:

The proposed approach does require setting $Q_{thresh}$ hyper-parameter but we empirically found that the performance is not unstable with varying hyperparameter value. Intuitively, this is a reasonable trade-off that renders the higher level subgoal generation a self-supervised process, thereby eliminating the need for expert guided annotations for segmenting subgoals. Additionally, please note that we empirically found that after we normalize $Q_{\pi^L}$ values of a trajectory, the $Q_{thresh}$ hyper-parameter value of $0$ works consistently well across all environments, which exhibits that setting $Q_{thresh}$ value does not cause significant overhead. We have added the hyper-parameter ablation analysis in Appendix A.4 Figure 10 for your consideration. Additionally, in real robotic experiments, we used the default $Q_{thresh}$ value of $0$ and PEAR was able to outperform the baselines in all experiments.

Reviewer comment (Evaluation):
I'm curious why PEARL-IRL outperforms PEAR_BC on Maze, Pick Place, Bin, and Hollow, but PEAR-BC outperforms PEARL-IRL on Kitchen. Do the authors have any ideas why this is?

Author response:
**4. "I'm curious why PEARL-IRL outperforms PEAR_BC on Maze, Pick Place, Bin, and Hollow, but PEAR-BC outperforms PEARL-IRL on Kitchen. Do the authors have any ideas why this is?"**

Response:

Our motivation for using inverse reinforcement learning (IRL) is that although behavior cloning (BC) works efficiently in many scenarios, it sometimes fails to perform well for complex tasks that require long term planning. However, the advantages of IRL are sometimes counterbalanced by the fact that they are really difficult to train. We believe that since it is a really difficult environment to train on, PEAR-BC slightly outperforms PEAR-IRL in Franka kitchen environment:

Success Rates:
| Method    | Success Rate |
| -------- | ------- |
| PEAR-IRL  | **0.89 $\pm$ 0.06**   |
| PEAR-BC | **1.0 $\pm$ 0.0**    |

Thus, although in general, PEAR-IRL works better than PEAR-BC in long horizon complex tasks, PEAR-BC might work better in some cases where it is hard to train an IRL objective. However, the performance does not differ significantly and we found that in real world tasks, PEAR-IRL consistently outperforms PEAR-BC in the experiments.

Reviewer comment (Experiments):
Have the authors experimented with different numbers of expert demonstrations? I'm curious how the method would perform with more/less demonstrations.

Author response:
**5. "Have the authors experimented with different numbers of expert demonstrations? I'm curious how the method would perform with more/less demonstrations."**

Response:

Yes, we have performed ablations to analyze the effect of varying the number of expert demonstrations for each task (Please refer to Appendix A.4 Figure 9). Although it is subject to availability, we increase the number of expert data until there is no significant performance boost.

Reviewer comment (Experiments):
How is $Q_{\text{thresh}}$ chosen for each environment?

Author response:
**6. "How is $Q_{thresh}$ chosen for each environment?"**

Response:

Firstly, we select $100$ randomly generated environments for training, testing and validation. Using this, for calculating $Q_{thresh}$, we compute success rate plots for various $Q_{thresh}$ values to select the best $Q_{thresh}$ parameter (the ablation analysis is shown in Appendix A.4 Figure 10). 

We hope that the response addresses the reviewer’s concern. Please let us know, and we will be happy to address additional concerns if any.

Reviewer comment (Theory):
**[Deal with the non-stationarity]** I don't agree that this work ameliorates non-stationarity. Since PEAR can access expert samples when training the high-level policy by BC or IRL, the high-level dynamics remain consistent with the dynamics that generate the expert sample trajectories without any change. This is inconsistent with the claim of non-stationarity made by the authors, which raises concerns about the contribution in this regard.

Author response:
We understand the reviewer's confusion and explain the claim regarding non-stationarity as follows: as explained in detail in Section 4, PEAR jointly optimizes HRL agents by employing reinforcement learning (RL) with imitation learning (IL) regularization. Since the higher level policy does not have direct access to expert subgoal dataset, PEAR using primitive enabled adaptive relabeling to generate efficient subgoal supervision for the higher level policy. While PEAR leverages imitation learning regularization using expert demonstrations to bootstrap learning, it also employs reinforcement learning to explore the environment and come up with efficient actions to execute in complex task environments. Due to this exploration and joint optimization, PEAR significantly outperforms the imitation learning baselines, as stated in the Experiment Section 5 and Table 1. Moreover, PEAR mitigates non-stationarity by periodically re-populating subgoal transition dataset $D_g$ after every $p$ timesteps according to the goal achieving capability of the current lower primitive. This generates a natural curriculum of achievable subgoals for the lower primitive, thereby ameliorating non-stationarity. We hope that this elucidates the motivation behind how and why PEAR deals with non-stationarity. Please let us know and we would be happy to clarify any other doubts and concerns regarding the motivation.

Reviewer comment (Experiments):
**[Performance limitation from expert]** Since the high-level policy is optimized only through BC or IRL using expert samples and does not participate in the environment exploration process, the performance of expert samples significantly limits the upper-performance limit of PEAR. Therefore, the authors need to analyze the impact of expert performance on the upper-performance limit of PEAR and conduct experiments with expert samples of varying performance.

Author response:
As clarified in the previous answer, PEAR employs reinforcement learning along with imitation learning regularization to explore the environment and devise efficient actions to execute in the complex task environments. Due to this, PEAR significantly outperforms various hierarchical and non-hierarchical baselines that may or may not employ imitation learning. Additionally, we perform ablations to analyze the effect of varying the number of expert demonstrations for each task (Please refer to Appendix A.4 Figure 9). Although it is subject to availability, we increase the number of expert data until there is no significant performance boost. We also conduct experiments where we vary the quality of demonstrations by adding in some g number dummy/garbage demonstrations with the N expert demonstrations, but since the algorithm is unable to learn from such garbage demonstrations, empirically the method performs as if it is being trained using (N-g) demonstrations, which is similar to the ablations analyzing the effect of varying the number of expert demonstrations for each task, and hence we omit the ablation results from the paper.

Reviewer comment (Reproducibility):
**[How to get $Q_{thresh}$]** When performing adaptive relabeling, the authors require additional environment-specific Q values, which are not available in standard experimental settings and may limit the practicality of PEAR.

Author response:
The proposed approach does require setting $Q_{thresh}$ hyper-parameter but we empirically found that the performance is not unstable with varying hyperparameter value. Intuitively, this is a reasonable trade-off that renders the higher level subgoal generation a self-supervised process, thereby eliminating the need for expert guided annotations for segmenting subgoals. Additionally, please note that we empirically found that after we normalize $Q_{\pi^L}$ values of a trajectory, the $Q_{thresh}$ hyper-parameter value of $0$ works consistently well across all environments, which exhibits that setting $Q_{thresh}$ value does not cause significant overhead. We have added the hyper-parameter ablation analysis in Appendix A.4 Figure 10 for your consideration. Additionally, in real robotic experiments, we used the default $Q_{thresh}$ value of $0$ and PEAR was able to outperform the baselines in all experiments.

Reviewer comment (Theory):
Many of the algorithm designs are not well supported: Why does the additional margin classification objective in low-level policy optimization prevent over-estimation? In equation (2), why does the low-level policy need the BC regularization objective? In my view, the goal of the low-level policy is to reach sub-goals more effectively, and this regularization is confusing. Why is the joint value function formulated as a summation of $J$ and $J_{BC}$?

Author response:
As explained in Section 4.1, if the expert states are outside the training distribution, $Q_{\pi_L}$ might erroneously over-estimate the values on out-of-distribution states, which might result in poor subgoal selection. This surrogate objective prevents over-estimation of $Q_{\pi_{L}}$ by penalizing states that are out of the expert state distribution. Please note that the additional margin classification objective is only used with the higher level policy optimization and not with the lower level policy optimization objective. The ablation results demonstrating the importance of margin classification objective is shown in Figure 5.

In complex tasks, specially in sparse reward scenarios, it is hard for the flat single level policy to achieve the goal, and using expert demonstrations to bootstrap learning using behavior cloning has been shown to improve performance [https://arxiv.org/abs/1709.10089]. Notably, we also found this to be the case in our empirical analysis (Please refer to Table 1). The higher level policy expert dataset provides subgoal targets and the lower level expert dataset provides primitive actions targets for behavior cloning training. Thus, the BC regularization objective allows the lower level to reach the goals more effectively by making use of expert primitive actions from the demonstration trajectories using behavior cloning (BC) or inverse reinforcement learning IRL) regularization.

The joint optimization objective for learning hierarchical policies is formulated as a sum of $J$ and $J_{BC}$. Here, $J$ denotes the off-policy reinforcement learning objective, and $J_{BC} denotes the behavior cloning or inverse reinforcement learning regularization objective. This joint optimization allows the agent to explore the environment and collect experience for the replay buffer, while leveraging expert dataset supervision using IRL or BC. Thus, this joint optimization objective comprising of RL and BC/IRL, combined with hierarchical training provide our method with the tools to solve complex long horizon tasks and outperform the baselines.

Reviewer comment (Presentation):
The clarity of the figures in the experiments is poor, especially in Figure 5.

Author response:
We apologize for the figure size and clarity issues. We have now fixed the issues in the paper. We hope this improves the overall clarity.

Reviewer comment (Novelty):
I recommend that the authors survey and include some of the most recent works in the related works section.

   [1] Lee S, Kim J, Jang I, et al. DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 13668-13678.

   [2] Chane-Sane E, Schmid C, Laptev I. Goal-conditioned reinforcement learning with imagined subgoals[C]//International Conference on Machine Learning. PMLR, 2021: 1430-1440.

   [3] Zhang T, Guo S, Tan T, et al. Generating adjacency-constrained subgoals in hierarchical reinforcement learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 21579-21590.

   [4] C-Learning: Learning to Achieve Goals via Recursive Classification

Author response:
We thank the reviewer with the survey suggestions. We have surveyed the suggested papers and improved the Related works section by adding their discussion.

Reviewer comment (Experiments):
In the experimental setup, since PEAR requires training with expert samples (possibly high-performance expert samples), while other baselines like HAC do not depend on expert data, please clarify the fairness of the experimental comparisons.

Author response:
The reviewer is correct to point out that PEAR leverages expert demonstration samples whereas HAC does not. HAC deals with non-stationarity by relabeling transitions and assuming an optimal lower primitive. Whereas, PEAR deals with non-stationarity by periodic adaptive relabeling and IL regularization. Notably, expert demonstrations can not be directly leveraged in raw trajectory form to train the higher level, and therefore require data relabeling to segment the them into meaningful subgoals. Thus, although we agree that the comparisons with HAC baseline are not exactly fair, comparion with HAC demonstrate that PEAR can better ameliorate non-stationarity using a handful of expert demonstrations by generating expert subgoal dataset using adaptive relabeling.

We hope that the response addresses the reviewer's concern. Please let us know, and we will be happy to address additional concerns if any.

Reviewer comment (Writing):
Some parts of the paper are difficult to understand or seem contradictory. For example, in section three it is stated that no actions are part of the demonstrations:  
> Notably, we only assume access to demonstration states and not actions.  
However, in section 4.2 expert data $(s^f, a^f, s^f_\text{next})$ is used for imitation learning on the lower level, where $a^f$ is explicitly used as an expert action.

Author response:
When we train the higher level policies using IRL or BC, the higher level joint optimization procedures (Eqn 1 and Eqn 3) do not require expert actions $a^f$, which is what we initially wanted to convey through the statement. However, expert actions are indeed required when lower level policies are trained, using IRL or BC joint optimization. We understand that our previous statement may lead to confusion for the reader, and thank the reviewer for pointing this out. We have removed the statement and clarified in the paper draft wherever applicable (Section 3 and otherwise).

Reviewer comment (Presentation):
In Algorithm 1 line 12, triplets $(s_j, s_w, s_k)$ are added to the expert subgoal dataset $D_g^e$ where $k$ runs from the initial to the current goal index. Adding $s_k$ seems unnecessary as in section 4.2 only the first two entries in the triplet are used for imitation learning. This suggests that Algorithm 1 could be significantly simplified.

Author response:
Considering the triplets $(s_j,s_w,s_k)$, the reviewer is absolutely right that $s_k$ is not used in the joint optimization objectives. Nevertheless, we had added the triplets to the dataset $D^e_g$ so that the dataset transitions are consistent with the replay buffer transition storage format used by Soft Actor critic for off policy reinforcement learning. We could remove the $s_k$ entry from the transitions, which simplifies the notation and improves space complexity, but honestly we are not sure how this would simplify Algorithm 1 significantly. Please feel free to let us know if we missed something and we would be happy to add the improvements.

Reviewer comment (Writing):
In the second to last paragraph of section 4.1, triplets $(s_0^e, s_i^e, a_i)$ are sampled from $D_g$ to train the lower level, which contradicts what was actually added to the dataset $D_g$ earlier.

Author response:
In the second to last paragraph of section 4.1, in $Q_{\pi^{L}}(s^e_0,s^e_i,a_i)$, $s^e_0$ and $s^e_i$ are sampled from dataset $D_g$, whereas $a_i = \pi^{L}(s^e_{0},s^e_i)$, which is the action generated by the lower primitive policy. We have also shown this explicitly in Algorithm 1 when computing $Q_{\pi^{L}}(s^e_{init},s^e_i,a_i)$ where $a_i = \pi^{L}(s^e_{i-1},s^e_i)$.

Reviewer comment (Theory):
The theoretical analysis in section 4.3 is very difficult to follow. For example, $\Pi^H_D$ and $\Pi^L_D$ are defined as higher- and lower-level probability distributions generating datasets $D_H$ and $D_L$, while $\pi^H_D$ and $\pi^L_D$ are policies from those datasets. It is unclear how $\pi^H_D$ can be "from" $D_H$ if $D_H$ is an expert dataset and not a set of policies, unless $\pi^H_D$ is an empirical policy defined only on the data, but this is not explained.

Author response:
$\Pi_{D}^{H}$ represents some unknown distribution over policies from which we can sample a policy $\pi_{D}^{H}$. The expert dataset $D_H$ does not consist of policies; rather, it consists of state action transitions. Let us assume that these transitions are generated from an unknown empirical policy $\pi_{D_H}^{H}$. Thus, if we know $\pi_{D_H}^{H}$, we can use it to predict actions for the state transitions present in the dataset $D_H$. We hope that this clarifies the confusion.

Reviewer comment (Theory):
The meaning of the distribution $\kappa$ in the theoretical analysis is unclear. It seems it might be either the expert data distribution or the distribution induced by the current policy. This would make the factor $\lVert \frac{d_c^{\pi*}}{\kappa} \rVert_\infty$ almost certainly infinite or extremely large, rendering the bound vacuous.

Author response:
$\kappa$ represents some unknown distribution over states from which we can sample a state $s$. In our sub-optimality bounds, we use it for importance sampling when we sample from arbitrary state distribution $\kappa$ over unknown distribution $d_{c}^{\pi^*}$. The factor $\| \frac{d_c^{\pi^{*}}}{\kappa} \|_{\infty}$ represents the infinite norm ratio between these distributions. We have kept the notations consistent with the theoretical analysis in the paper OPAL[https://arxiv.org/pdf/2010.13611.pdf]. The reviewer is encouraged to refer to this paper for further clarifications.

Reviewer comment (Theory):
There are claims that PEAR mitigates the non-stationarity issue of HRL (in the introduction and section 4.1). However, the transitions in the replay buffer of the SAC algorithm used at the higher level become outdated when the lower level changes, which PEAR does not address. This casts doubt on whether the algorithm really mitigates non-stationarity.

Author response:
We propose to mitigate non-stationarity by periodically re-populating subgoal transition dataset $D_g$ after every $p$ timesteps according to the goal achieving capability of the current lower primitive. Since the lower primitive continuously improves with training and gets better at achieving harder subgoals, $Q_{\pi^L}$ always picks reachable subgoals of appropriate difficulty, which generates a natural curriculum of subgoals for lower primitive. Intuitively, $D_g$ always contains achievable subgoals for the current lower primitive, thereby mitigating non-stationarity caused by outdated lower level transitions. We have also already explained this in detail in Section 4.1 (Primitive enabled adaptive relabeling) and Algorithm 2.

Reviewer comment (Experiments):
While many environments are considered in the experiments, the baselines may not be suitable for these tasks. In more complex environments, rewards seem to be extremely sparse, making learning progress very difficult without demonstrations. Strong IL baselines should be incorporated into the experiments. It is also unclear how the hyperparameters for PEAR and the baselines were tuned.

Author response:
Sparse rewards are relevant for tasks such as the robotic tasks considered in this paper where it is hard to come across dense rewards. Since learning in sparse reward scenarios is extremely difficult, we chose to deal with such challenging scenarios for this work. Moreover, dense reward may sometimes cause bias in learning task when the the rewards are not optimal, and thus sparse reward scenarios lead to generalizable and effective policies. We would like to also point out that multiple hierarchical and non-hierarchical baselines which may/may not use expert denostrations have been considered in the paper. Relay Policy Learning (RPL) and Discriminator Actor Critic(DAC) are imitation learning based multi-level and single-level baselines that leverage expert demonstrations. We have also added behavior cloning baseline in success rate performance comparison results.

We agree with the reviewer that it is hard to learn in the absence of expert demonstrations, but in complex scenarios, the Flat single level policies are not sufficient and therefore recent approaches employ hierarchical learning and leverage expert demonstrations. We hope that this clarifies the motivation of working in complex sparse reward scenarios and reasons for employing the beaslines in the paper. 
For selecting the hyper-parameters, firstly we select $100$ randomly generated environments for training, testing and validation. Using these, we compute success rate plots for various hyper-parameter values for selecting the best hyper-parameter values (the ablation analysis are shown in Appendix A.4).

Reviewer comment (Experiments):
In Figure 5, there are no shaded regions in the second and third row. This could imply that only one seed was used for these experiments, which would make the results potentially too noisy for proper interpretation.

Author response:
The results were averaged over 5 seeds and for success rate performance comparisons in all the environments. For the ablations analysis, we averaged over 3 seeds and found that the results were consistent across seeds, but unfortunately due to resource and time constraints we could not compute the results over 5 seeds for all the ablation experiments. We promise to add the comparisons over multiple seeds before submitting the final draft.

Reviewer comment (Writing):
The overall writing seems unfinished in some parts. Articles are frequently missing, and verb forms are sometimes incorrect (singular vs. plural). These issues appear from the first paragraph of the introduction and persist throughout the paper. Plots in figures 3 and 5 and some in the appendix are too small to be legible when printed.

Author response:
Thank you for pointing out the errors. We have improved the writing in the recent draft wherever we could find the errors. We hope that this improves the overall clarity of the paper. Kindly let us know if you find any improvements and we will be happy to implement them.

Reviewer comment (Writing):
The use of notation is inconsistent. For example, the threshold for the Q-function during relabeling is introduced as $Q_\text{thresh}$ in section 4.1 but appears as $D_\text{thresh}$ in section 5 (“Ablative analysis”)—it is unclear if these refer to the same thing. There are also smaller issues, such as $G$ vs. $\mathcal{G}$ for the goal space, which appears identical to the state space (since states and goals are subtracted in section 3), but this is not explicitly stated.

Author response:
$Q_{thresh}$ and $D_{thresh}$ indeed refer to the same threshold parameter. We apologize for the typo and we have made changes to the draft to keep the notation consistent throughout.

Reviewer comment (Presentation):
Figure 2 is difficult to interpret as the subgoals appear somewhat arbitrary. A video might better demonstrate the improvement of the subgoals over training.

Author response:
We understand that the subgoals improvement depiction in Figure 2 might not seem very clear, especially since the figures are presented in 2D. However, we indeed found that the subgoals complexity consistently improves with training. We are working to compile videos to demonstrate better depiction of improving subgoal and will add links to the video in the final draft.