Paper: Stochastic Subgoal Representation for Hierarchical Reinforcement Learning (ICLR-2024)

Reviewer comment (Evaluation):
I am not very familiar with GPs, and am interested in the computation complexity of HSSR. How much computation does it task compared to HESS?

Author response:
A3: In comparing the average inference time across 10,000 episodes using 5 different random seeds, HSSR exhibits an approximately 3% longer inference time than HESS.

Reviewer comment (Writing):
I am not very familiar with GPs, and am a bit confused about $f$ and $z$. From Fig 1 it seems that $f$ is an encoding of $s$, and is passed through GP to get $z$. However, Eq 1 seems to say that $f$ depends on $z$. Is it that HSSR first acquires $f$, and then denoises $f$ to get $z$?

Author response:
A2: Thank you for your question regarding the role of $f$ and $z$ in the context of this paper. You are correct in your understanding that $f$ represents an intermediate encoding of the state space $s$, and $z$ is the latent subgoal embedding that we are ultimately interested in. The process flow involves mapping the original state space to an intermediate space $f$ through encoding, which is then used as an input to the GP to obtain $z$, i.e., the subgoal representation. The GP enables denoising the representation, taking into account the uncertainty and correlations in the input space. Eq. 1 provided defines a two-step process in a GP based model:

1. $z_i$, the subgoal representation of $s_i$, is governed by a Gaussian Process. This GP is characterized by a mean function, which is assumed to be zero, and a covariance function defined by the kernel $\kappa(s_i, s_j)$. The kernel function dictates the covariance or degree of similarity between any two points in the state space, thereby determining the relationship between the values of $z_i$ at different states $s_i$ and $s_j$.

2. $f_i$ is then defined as $z_i$ with an added Gaussian noise $\epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma)$. This introduces a stochastic element to the latent variable, creating the observed representation $f_i$ that encompasses the “stochasticity” not captured by the GP.

The dependence of $f$ on $z$ indicates that $f$ is not just a direct observation from the state space but is a smoothed representation informed by the GP's interpretation of the state space. The addition of noise $\epsilon$ to form $f$ acknowledges that real-world observations may deviate from the GP's idealized $z$, reflecting the inherent uncertainty of real-world environments, capturing the unpredictability and randomness that are typical in practical scenarios. This addition not only improves the model's generalization capabilities to new or less frequent states but also facilitates exploration in the learning process, which is vital for effective policy development in HRL.

Reviewer comment (Theory):
The explanation on how to learn the subgoal representation and perform GP inference is good. However, more theoretical analysis on why the idea of using GP in subgoal representation learning could work is appreciated.

Author response:
Thank you for your question regarding the use of Gaussian Processes (GPs) in subgoal representation learning. We agree that providing more theoretical evidence, such as meaningful bounds, would strengthen our argument for using GPs in this context. We plan to include additional theoretical analysis in our future work to more rigorously demonstrate the advantages of GPs in modeling the latent subgoal space. Compared to existing approaches in learning subgoal representations for HRL, such as predefined subgoal spaces or deterministic representations learned via neural networks, our method introduces a significant advancement. We formulate a posterior distribution over the latent subgoal space, employing GPs to account for stochastic uncertainties in the learned representation. This approach allows for a more flexible and robust representation, adapting to the complex and uncertain environments typically encountered in HRL. We hope this response provides a clearer understanding of our approach and the future direction of our research in this area. Thank you again for your valuable feedback.

Reviewer comment (Theory):
Can the authors provide more evidence (e.g. show some meaningful bounds) on why GP could help with subgoal representation learning?

Author response:
Thank you for your question regarding the use of Gaussian Processes (GPs) in subgoal representation learning. We agree that providing more theoretical evidence, such as meaningful bounds, would strengthen our argument for using GPs in this context. We plan to include additional theoretical analysis in our future work to more rigorously demonstrate the advantages of GPs in modeling the latent subgoal space. Compared to existing approaches in learning subgoal representations for HRL, such as predefined subgoal spaces or deterministic representations learned via neural networks, our method introduces a significant advancement. We formulate a posterior distribution over the latent subgoal space, employing GPs to account for stochastic uncertainties in the learned representation. This approach allows for a more flexible and robust representation, adapting to the complex and uncertain environments typically encountered in HRL. We hope this response provides a clearer understanding of our approach and the future direction of our research in this area. Thank you again for your valuable feedback.

Reviewer comment (Experiments):
Is it possible to test the proposed method in environments other than MuJoCo simulations? Some subgoal-sensitive tasks, such as NetHack or MineDojo could be interesting for experimenting.

Author response:
Thank you for recommending that we extend our method's testing to environments beyond MuJoCo, such as NetHack and MineDojo. The advantage of using learnable subgoal representation is its adaptability to various settings without necessitating manual specification of subgoal environments. This flexibility allows our method to be effectively utilized in diverse environments, including NetHack and MineDojo. 

We chose MuJoCo for its established benchmarks in HRL research, following the lead of studies like HIRO, HRAC, and LESSON. While integrating new environments would greatly enhance our research, the limited timeframe of the rebuttal period makes this challenging. Comprehensive testing in additional environments requires significant time and resources for proper integration and analysis. We value your recommendation and plan to explore diverse environments in our future work to further validate and strengthen our method.

Reviewer comment (Experiments):
Although the motivation of dealing with stochasticity and uncertainty is interesting, I find the experimental setup of stochastic environments a bit artificial: the observations are just added Gaussian noises with \sigma=0.1 to the (x, y) position only. This seems to be an unrealistic setting that might be too advantageous for the method.

Author response:
**Q1: setup of stochastic environments being a bit artificial; why not perturbing the entire state space?**

A1: Thank you for your valuable feedback regarding our experimental setup, particularly our approach to simulating environmental stochasticity. Our study follows the precedent set by recent works such as **HIGL (Kim et al., 2021)** and **HRAC (Zhang et al., 2020)**. Specifically, HIGL added Gaussian noise at $\sigma=0.05$ to the $(x, y)$ position to model stochastic environments. Similarly, HRAC experimented with adding Gaussian noise levels of $\sigma=0.01, 0.05,$ and $0.1$ to the $(x, y)$ position, describing $\sigma=0.1$ as the "most noisy scenario."

In our experiments, we consistently applied Gaussian noise at $\sigma=0.1$ across all tests, adhering to these established benchmarks. We further extended our investigation to include a $\sigma=0.15$ setting, as shown in Fig. 6 (Left). This decision aimed to thoroughly test our method's performance in highly stochastic settings, which are arguably more demanding than those used in previous studies.

Your suggestion to perturb the entire state space in future work is insightful, yet it requires careful consideration. While we recognize that adding Gaussian noise to the $(x, y)$ positions as a proxy for environmental stochasticity is a simplification, these coordinates are the primary dimensions that are directly influenced by such stochasticity. Other dimensions, like torso contact forces, joint angles, and velocity, pose challenges in quantifying their respective impacts from the same level of stochasticity. Emphasizing the need for extensive experimentation with hyperparameters and design choices, such an investigation goes beyond the scope of this paper. It would require a dedicated research effort to adequately address these complexities. We hope this clarification provides a better understanding of our experiment settings and their context within the broader scope of HRL research. We remain open to further discussion and are grateful for the opportunity to enhance our work based on your insights.

Reviewer comment (Experiments):
It seems that the experiment settings can only deal with local, independent Gaussian noises rather than the inherent uncertainty that is a result of lack of exploration (i.e. unexplored state space).

Author response:
**Q2: the experiment settings can only deal with local, independent Gaussian noises rather than the inherent uncertainty that is a result of lack of exploration (i.e. unexplored state space).**

A2: Our approach, which learns a posterior distribution over the subgoal representation, is specifically designed to address the limitations you've highlighted. Traditional deterministic subgoal representation methods often struggle in novel state regions due to insufficient historical information to form accurate subgoal representations. This can lead to an underfitting of the learning objective and an inability to accurately capture the underlying dynamics in these new areas of the state space. In contrast, our method explores the intrinsic structure in the state space through learnable kernels, and forms a posterior distribution over the subgoal representation. This approach is particularly effective in underpinning representations in novel state regions, offering a significant degree of resilience against environmental stochasticity, including those inherent uncertainties induced by unexplored spaces. 


To empirically validate our approach, we conducted comprehensive experiments in deterministic environments, as detailed in Fig. 8 (Appendix Section) of our paper. The results clearly demonstrate that our stochastic subgoal representation method offers performance improvements in handling the inherent uncertainty induced by unexplored state spaces, compared to the deterministic subgoal representation method LESSON. We hope this explanation clarifies how our method effectively deals with both types of uncertainties – the local, independent Gaussian noises and the inherent uncertainty from unexplored state regions.

Reviewer comment (Presentation):
Plots (especially Figure 6) are difficult to read, the plot should introduce more smoothing and post-processing. Regarding Figure 6(Left), it's very difficult to examine the effect of the stochasticity (different \sigma values).

Author response:
**Q2: Figure plotting**

We are grateful for your suggestion to enhance the quality of our figures. In the camera ready version, we will ensure consistent smoothing, use distinctive colors, and provide clear breakdowns in our figures for improved visual clarity and consistency.

Reviewer comment (Theory):
The learning objective given in Equation (3) seems a bit arbitrary. Can the authors shed more light on why it has to be such a particular form? For instance, why are the \Delta_f terms given in the fractions? Why not simply have the form like \Delta_f^1  - \Delta_f^k + \Delta_z^1 - \Delta_z^k with some weight coefficients?

Author response:
**Q4: learning objective formulation**

A4: Our logarithmic term is based on the softplus function, as introduced in Dugas et al. [2001]. This choice offers two advantages over the more commonly used hinge loss function for contrastive learning:
1. **No Hyperparameter Requirement**: Unlike the hinge loss, the softplus function does not require setting a margin as a hyperparameter.
2. **Smooth Behavior**: The hinge loss exhibits discontinuity around margin planes, which can be problematic in GP regression. In contrast, the softplus function decays exponentially, avoiding the abrupt cut-off characteristic of the hinge function. This design ensures that even triplets meeting the margin criteria contribute to the loss, enabling continuous pushing/pulling samples as close or as far as possible. The goal is to regularize the absolute distance in $\mathbf{Z}$ space as the subgoal space.

For $\mathbf{F}$, we use the ratio (fraction) as a relative distance measure to avoid two potential issues: (a) the need for an extra hyperparameter to balance absolute distances in both $\mathbf{Z}$ and $\mathbf{F}$ spaces, which would arise if the softplus function were used for both; and (b) the undue emphasis on the absolute distance in $\mathbf{F}$, a factor that is less crucial in our context. Our primary concern in $\mathbf{F}$ is the ratio between high-level and low-level transitions, effectively captured by a 'relaxed' relative measure rather than the absolute distances. This 'relaxed' approach provides more adaptable control over the transitions in $\mathbf{F}$.

These design choices are driven by the distinct roles and requirements of the subgoal space and intermediate latent space, ensuring that each space is optimized in a way that best serves its purpose in the overall model. We hope the above explanation clarifies our choice regarding the two variables.

A straightforward formulation like $\Delta_\mathbf{f}^1 - \Delta_\mathbf{f}^k + \Delta_\mathbf{z}^1 - \Delta_\mathbf{z}^k$ with weight coefficients would linearly combine absolute distances in two spaces. This approach might not capture the intricate distances in either space effectively and could lead to excessive hyperparameter tuning and unstable training.

**Reference:**

Dugas C, Bengio Y, Bélisle F, Nadeau C, Garcia R. "Incorporating second-order functional knowledge for better option pricing." Advances in Neural Information Processing Systems. 2000;13.

Reviewer comment (Experiments):
Figure 6(Left) should also include HESS which is the strongest baseline other than LESSON. Why was HESS missing here?

Author response:
**Q3: Figure 6(Left) suggest that LESSON with \sigma=0.1 doesn't degrade as much compared to the fully deterministic case -- \sigma=0.0, whereas 0.15 can significantly hurt LESSON; so I doubt that sigma=0.1 is a noise that is high enough that would make existing baselines. Figure 6(Left) should also include HESS which is the strongest baseline other than LESSON. HESS is missing in Figure 6(Left)**

A3: Indeed, LESSON with $\sigma=0.1$ doesn't degrade as much compared to the $\sigma=0.0$ case in this dense reward setting. However, as illustrated in Fig. 6 (Left) of the revision, such degradation is more pronounced in the sparse external reward setting, which is the primary setting also investigated in Fig. 5. Due to the lack of dense learning signals and the increasing presence of environmental stochasticity, HESS and LESSON start to show deteriorating degradations compared with HSSR.

Our initial comparison didn’t include HESS as it is an extension of LESSON, incorporating an active exploration strategy on top of the deterministic subgoal representation method proposed by LESSON. Our objective was to isolate the impact of the stochastic vs. deterministic nature of subgoal representation methods in the face of environmental uncertainties, without the additional complexity introduced by active exploration strategies like those in HESS. In the revised version, we have now incorporated HESS into Figure 6 (Left) to ensure a more comprehensive comparison.

Reviewer comment (Writing):
This paper is hard to understand without introducing their algorithm.

Author response:
A1: While all other three reviewers appreciated the technical clarity of the paper **(presentation: good, good, good)**, we acknowledge the importance of clarity in presenting and the reviewer’s suggestions. We have improved the paper and answer below each question in detail. Hope this satisfies the reviewer. We are more than willing to improve the paper writing and presentation further if needed.

Reviewer comment (Writing):
The reviewer felt confused about the poorly notations such as $\phi(s)$, $Z=[z_1 z_2 ... z_N]$, $\Delta_f$, $\Delta_z$, etc.

Author response:
A3: In light of your comments, we wish to emphasize that our notation is broadly in line with established conventions in HRL and GP literatures. To clarify:
- The notation $\phi(s)$ aligns with established conventions in HRL literature, as used in HESS [Li et al., 2022] (3rd line, last paragraph on page 2) and LESSON [Li et al., 2021] (4th line, 4th paragraph on page 2).
- $\mathbf{Z} = \left( \mathbf{z}_1, \mathbf{z}_2, \cdots, \mathbf{z}_N \right)$ is similarly defined in [1] (3rd line, first paragraph on page 5).
- $\Delta_\mathbf{f}$ and $\Delta_\mathbf{z}$ are shorthand notations representing distances, used for brevity. These forms are consistent with conventions found in algebra literature, such as those detailed in Section 3.2 of [2].

We hope this clarification helps, and we are open to further suggestions to improve our manuscript.

**References:**
- [1] Lee, J., Bahri, Y., Novak, R., Schoenholz, S.S., Pennington, J. and Sohl-Dickstein, J., 2017. Deep neural networks as Gaussian processes. ICLR 2018.
- [2] Berg, T., 2020. Intermediate Algebra. Kwantlen Polytechnic University.

Reviewer comment (Writing):
The disturbing technical communication is below the standard of ICLR.

Author response:
A1: While all other three reviewers appreciated the technical clarity of the paper **(presentation: good, good, good)**, we acknowledge the importance of clarity in presenting and the reviewer’s suggestions. We have improved the paper and answer below each question in detail. Hope this satisfies the reviewer. We are more than willing to improve the paper writing and presentation further if needed.

Reviewer comment (Reproducibility):
What is the proposed complete algorithm?

Author response:
A2: Due to the limited space in the main manuscript the detailed algorithm is provided in Appendix Section A.1. In the main paper, in Section 3.1, we introduce our two-level HRL framework, with Soft Actor Critic (SAC) being utilized at each HRL level. We have included the SAC RL training objective in the updated manuscript (Sec. 3.1) to improve clarity. We hope this clarifies your concern.

Reviewer comment (Reproducibility):
Is the number of subgoals fixed? What is this number?

Author response:
A4: Thank you for inquiring about the number of subgoals. In our experiments, the high-level policy is designed to generate a subgoal every $k$ timesteps, with $k$ set to 50 for all MuJoCo tasks, as detailed in A.4.1 in the Appendix section.

Reviewer comment (Writing):
What is the relationship between $g$ and $z$?

Author response:
A5: Your question brings up an important aspect regarding the relationship between $g$ and $z$. In our framework, the high-level policy samples a subgoal $g_t$ from the latent subgoal space abstracted by the representation function $\phi(s)$. The low-level controller then strives to achieve these subgoals, with $z_t = \phi(s_t)$ serving as the mapping from the state space to the subgoal space. Essentially, $g$ represents the action taken by the high-level policy within the subgoal space, while $z$ is the corresponding subgoal space representation of the state $s$, as illustrated in Fig. 1.