A CTIVE H IERARCHICAL E XPLORATION WITH S TABLE S UBGOAL R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Goal-conditioned hierarchical reinforcement learning (GCHRL) provides a promising approach to solving long-horizon tasks.
Recently, its success has been extended to more general settings by concurrently learning hierarchical policies and subgoal representations.
Although GCHRL possesses superior exploration ability by decomposing tasks via subgoals, existing GCHRL methods struggle in temporally extended tasks with sparse external rewards, since the high-level policy learning relies on external rewards.
As the high-level policy selects subgoals in an online learned representation space, the dynamic change of the subgoal space severely hinders effective high-level exploration.
In this paper, we propose a novel regularization that contributes to both stable and efﬁcient subgoal representation learning.
Building upon the stable representation, we design measures of novelty and potential for subgoals, and develop an active hierarchical exploration strategy that seeks out new promising subgoals and states without intrinsic rewards.
Experimental results show that our approach signiﬁcantly outperforms state-of-the-art baselines in continuous control tasks with sparse rewards.
I NTRODUCTION
Goal-conditioned hierarchical reinforcement learning (GCHRL) has long demonstrated great potential to solve temporally extended tasks (Dayan & Hinton, 1993; Schmidhuber & Wahnsiedler, 1993;
Vezhnevets et al., 2017; P´er´e et al., 2018; Nair & Finn, 2019), where a high-level policy periodically sets subgoals to a low-level policy, and the low-level policy is intrinsically rewarded for reaching those subgoals.
Early GCHRL studies used a hand-designed subgoal space, such as positions of robots (Nachum et al., 2018; Levy et al., 2019) or objects in images (Kulkarni et al., 2016a).
To alleviate the dependency on domain-speciﬁc knowledge, recent works investigate learning subgoal representations along with hierarchical policies (Nachum et al., 2019a; Dilokthanakul et al., 2019;
Li et al., 2021), which have shown promise in a more general setting.
Although the exploration ability of HRL is boosted via decomposing tasks and reusing low-level policies, existing GCHRL methods struggle in long-horizon tasks with sparse external rewards.
The high-level policy learning in GCHRL relies on external rewards, and those tasks are hard to solve with naive high-level exploration strategies.
Furthermore, as the high-level policy makes decisions in a latent subgoal space learned simultaneously with the hierarchical policy, the instability of the online-learned subgoal space results in severe non-stationarity of the high-level state transition function and hinders effective high-level exploration.
To address these challenges, we develop an active Hierarchical Exploration approach with Stable
Subgoal representation learning (HESS).
Our approach adopts a contrastive objective for subgoal representation learning inspired by Li et al.
To learn a subgoal space that could effectively guide hierarchical exploration, we propose a novel state-speciﬁc regularization that contributes to both stable and efﬁcient subgoal representation learning.
The proposed regularization constrains the changes for those embeddings that have already well satisﬁed the representation learning objective.
As the hierarchical agent gradually expands its exploration to new state regions, the regularization allows for updates of the embeddings that underﬁt the learning objective.
Beneﬁting from the proposed regularization, we could improve the representation learning efﬁciency without hurting its stability via the prioritized sampling technique (Hinton, 2007), which prioritizes training samples with larger losses.
Building upon our stable subgoal representation learning, we design an active exploration strategy for high-level policy learning.
As shown in previous work (Strehl & Littman, 2008; Bellemare et al., 2016; Ostrovski et al., 2017), the visit count provides a simple and effective novelty measure for exploration.
However, with online subgoal representation learning, the visit count for subgoals is deﬁned in a changing space. Therefore, such a novelty measure alone is not sufﬁcient for efﬁcient high-level exploration, although the stability regularization could improve its accuracy.
Our insight is that desirable novel subgoals should be reachable and effectively guide the agent to unexplored areas.
Thus we design a novel potential measure to regularize the novelty measure, which indicates the reachability to the neighbor regions of the visited state embeddings.
With the regularized novelty measure, our proposed active exploration strategy directly selects state embeddings with high potential and novelty as subgoals without introducing intrinsic rewards.
It is more efﬁcient than the reactive exploration methods that need to learn how to maximize the intrinsic rewards before performing exploratory behaviors (Tang et al., 2017; Pathak et al., 2017; Burda et al., 2018), since in the active exploration approach, the subgoal measures have direct effects on the behavior policy.
Furthermore, the active exploration strategy naturally avoid introducing additional non-stationarity (i.e., induced by dynamically changing intrinsic rewards) into HRL.
We compare the proposed method HESS with state-of-the-art baselines in a number of difﬁcult control tasks with sparse rewards.
Note that the environment setting in this paper is much more challenging to exploration than the multi-task and deceptive dense-reward ones in the baselines.
Experimental results demonstrate that HESS signiﬁcantly outperforms existing baselines.
In addition, we perform multiple ablations illustrating the importance of the various components of HESS.
B ACKGROUND
We consider a Markov Decision Process (MDP) deﬁned as a tuple ( S, A, P, r, γ ) , where S is a state space, A is an action space, P ( s (cid:48) | s, a ) is an unknown transition function, r : S ×
A → R is a reward function, and γ ∈ [0 , 1) is a discount factor.
Let π ( a | s ) denote a stochastic policy over actions given states.
The objective of reinforcement learning (RL) is to learn a policy that maximizes the expected cumulative discounted rewards: max π E P,π [ (cid:80)
Tt =
γ t r ( s t , a t )] , where T denotes the horizon length.
[Equation]
[Figure]
In GCHRL, a two-level hierarchical policy π hier is composed of a high-level policy π h ( g t | s t ) and a low-level policy π l ( a t | s lt , g t ) , as illustrated in Figure 1.
A latent subgoal space G is abstracted by a representation function φ ( s ) :
S → R d .
The high-level policy π h samples a subgoal g t from a neighborhood ∆( φ ( s t )) of the current latent state φ ( s t ) every c steps, i.e., when t ≡ 0 (mod c ): ∆( φ ( s t )) =
{ g t ∈ G (cid:12)(cid:12) D ( g t , φ ( s t ))
≤ r g } , where c is a ﬁxed constant, D is a distance function, and r g is a radius of the neighborhood.
The sampled subgoal is repeated, g t =
g t − 1 , when t (cid:54)≡(mod c ).
π h is trained to optimize the expected extrinsic rewards.
The low-level controller takes a primitive action a t ∈
A every step, and is intrinsically rewarded to reach subgoals set by the high level, r l t = − D ( g t , φ ( s t )) .
To provide dense non-zero rewards for low-level policy learning, we employ L2 distance as D. Furthermore, to keep the low-level reward function r l stationary while learning φ , we concatenate states and state embeddings together as the low-level states, s l = s || φ ( s ) .
Previous work (Li et al., 2021) minimizes a contrastive triplet loss L tri to learn φ ( s ) , where positive pairs are adjacent states in trajectories, and negative pairs are states c steps apart.
[Equation]
where δ is a margin parameter.
In this work, we also adopt the contrastive loss in Equation 1.
M ETHOD
In GCHRL, the low-level policy is optimized with intrinsic rewards generated by subgoals, but external rewards for the high level are still sparse and hard to explore.
Furthermore, the changing subgoal space makes it even more challenging for high-level exploration.
In this section, we ﬁrst present a novel regularization that contributes to stable and efﬁcient subgoal representation learning, and then introduce two measures for subgoals, novelty and potential.
Finally, we design an active hierarchical exploration strategy based on these two measures.
3. S
Both stability and efﬁciency are crucial to subgoal representation learning.
The stability of subgoal representation contributes to the stationarity of both the high-level transition function and the low-level reward function.
Meanwhile, a fast learned subgoal representation could provide effective guidance to exploration.
However, stability seems at odds with efﬁciency, e.g., training neural networks with smaller learning rates leads to better stability but is slower (Goodfellow et al., 2016).
In this subsection, we develop a novel state-speciﬁc regularization that resolves the stability-efﬁciency dilemma in subgoal representation learning.
To achieve stable representation learning, we propose a regularization L s which restricts the representation change during each update.
L s works by anchoring the current representation φ ( s ) to the old subgoal representation φ old ( s ) before the update as follows:
[Equation]
where B is a replay buffer, and λ ( s ) is a function that controls the weight of regularization for different states.
States with smaller representation losses ﬁt the learning objective well, so λ ( s ) should be larger for those states.
In practice, before each representation update, we rank the triplets in the buffer with L tri ( s t , s t +1 , s t + c ) .
For the top k % of the triplets with the minimum representation losses, we set λ ( s ) =
λ 0 > 0 for the anchor states ( s t ) in these triplets, and for the other states, λ ( s ) = 0 .
The stability regularization enables us to use prioritized sampling (Hinton, 2007) to improve representation learning efﬁciency without hurting its stability.
The overall loss for subgoal representation learning is:
[Equation]
where B p is a prioritized replay buffer, and states with larger L tri have higher probabilities of being sampled for training.
3. M EASURES FOR SUBGOALS
Novelty Measure: Inspired by count-based exploration methods (Strehl & Littman, 2008; Bellemare et al., 2016; Ostrovski et al., 2017;
Tang et al., 2017), we formulate the novelty of subgoals with visit counts.
As desirable subgoals should incentivize the agent to explore faraway novel states, we consider both immediate counts n ( φ ( s i )) and expected cumulative counts of future states, and the novelty measure N ( φ ( s i )) is deﬁned as follows:
[Equation]
In practice, we partition the low-dimensional continuous latent space into discrete cells, i.e., the state embeddings are mapped into cells containing them.
By maintaining how many times each cell is visited, we could estimate the visit count n ( φ ( s )) .
In practice, the novelty measure is approximated with the data from the replay buffer, and the implementation detail is described in Appendix B.
Potential Measure: With online representation learning, the novelty measure is a mixture of counts in the past and current representation spaces, so it might mislead the exploration, as demonstrated in Figure 3.
Our insight is that desirable novel subgoals should be reachable and effectively guide the agent to unexplored areas.
Therefore, we design a potential measure for subgoals to regularize the novelty measure.
In the following, we ﬁrst introduce a subgoal generation mechanism, which is involved in the deﬁnition of the potential measure.
To guide the low-level controller to reach unexplored states, the subgoals pursued by the low level had better be in unexplored areas as well.
Therefore, we propose to add some perturbations to the subgoal g t selected from the replay buffer and obtain an imagined subgoal g e , and then pass g e to the low-level policy.
To enable g e in an unexplored or less explored area, the perturbation is conducted as extending g t in the direction of g t − φ ( s t ) , as illustrated in Figure 2.
[Figure]
The imagined subgoal is g e =
g t + d e ( g t − φ ( s t )) /
||
g t − φ ( s t )
|| 2 , where d e denotes an extended distance.
As g e is imagined, and may have not been visited before, it could be inherently unreachable due to the transition function of the MDP or online representation learning, e.g., there may be obstacles in navigation tasks.
To encourage the agent to explore in promising and reachable directions, we deﬁne a measure of potential U ( g t ) for the selected subgoal g t as the expected negative distance between the ending state φ ( s t + c ) and g e :
[Equation] [Equation] [Equation]
g e )] , where
[Equation]
[Equation]
ρ h is the high-level state distribution under the hierarchical policy π hier .
Building on (Li et al., 2021), the representation learned with the contrastive objective preserves temporal coherence, and the distances between nearby features approximately represent the number of transitions between them (Oord et al., 2018).
Hence, with higher potential U ( g t ) , g e is more reachable, and thus exploring the direction of g t is more promising to expand the explored areas.
The potential U ( g t ) in
Equation 5 is estimated from the data in buffer as well.
To calculate the novelty, we partition the continuous representation space into discrete cells.
Similarly, we maintain the potential of a cell by averaging the potential of features in that cell, and use the potential of the cell to represent that of the states inside it.
[Figure]
Illustrative example: In Figure 3(a) and 3(b), we visualize the visitation density at 0.2M steps in the Ant Maze task in Figure 5.
The visitation density is the counts normalized by the total number of transitions in the buffer.
Comparing Figure 3(a) to 3(b), we found a mismatch between the density in the x, y coordinate space and the representation space, especially in the red box, since the updated representation at 0.2M steps has projected the state embeddings to somewhere new, and the feature changes are noted by the black arrows in Figure 3(c).
Furthermore, the counts in frontiers of the latent explored areas would not increase with more exploration, as the online learned representation keeps changing.
With the potential measure, we could distinguish the promising novel areas from the unpromising ones, illustrated in Figure 3(d).
3. A CTIVE E XPLORATION S TRATEGY
In the reactive exploration methods with intrinsic rewards, the agent needs to learn how to maximize cumulative intrinsic rewards before performing exploratory behaviors.
Therefore, the effects of intrinsic rewards on behavior policy is indirect.
Furthermore, the changing intrinsic rewards would introduce additional non-stationarity into HRL.
To make the proposed measures directly inﬂuence the behavior policy and avoid the non-stationary issue, we develop an active exploration strategy without intrinsic rewards for high-level policy learning.
Based on the two measures in Section 3.2, the proposed exploration strategy selects and explores a novel subgoal g t with high potential, speciﬁed as follows:
[Equation]
subject to
[Equation]
where s t is the current state and α is a scaling factor.
To balance these two measures more easily, we normalize future count N ( φ ( s )) by the total number of transitions in the buffer, and denote it with (cid:101) N ( φ ( s )) .
We visualize the proposed exploration incentive in Figure 3(e).
Taking the state embeddings with darker colors as subgoals could lead the agent to expand the explored areas.
Algorithm 1 provides the full procedure of the proposed approach, HESS.
To balance exploration and exploitation, HESS probabilistically utilizes the exploration strategy in Equation 6 or explores with the learned policy π h (Line:5-6), and the probability p is decayed over the course of learning.
The representation φ is updated every I episodes (Line:13-15) so that φ is stable during the update interval, and the proposed regularization is to maintain the stability before and after the update. Algorithm
1: Initialize: π h ( g | s ) , π l ( a | s l , g ) and φ ( s ) .
2: for i = 1 ..num episodes do 3: for t = 0 ..
T − 1 do 4: if t ≡ 0 (mod c ) then 5:
With a probability of p , explore with the strategy in Eq. 6.
With a probability of 1 − p , sample subgoal g t with learned policy π h .
Update π h with an off-policy RL method.
8: else 9: g t =
g t −10: end if 11: r t , s t +1 ← execute a t ∼ π l ( ·| s lt , g t ) , and update π l with an off-policy RL method.
12: end for 13: if i ≡ 0 (mod I ) then 14:
Update φ with Eq.
3 using m minibatches.
15: end if 16: end for 17: Return: π h , π l and φ .
R ELATED W ORK
Learning effective subgoal representations has been an important and challenging problem in
GCHRL (Dwiel et al., 2019).
Previous works have proposed to learn the subgoal representation in an end-to-end manner with policies (Vezhnevets et al., 2017; Dilokthanakul et al., 2019), or by bounding the sub-optimality of the hierarchical policy (Nachum et al., 2019a), or with a learning objective of slow dynamics (Li et al., 2021).
Nevertheless, those methods have not considered the stability of the subgoal representation learning, which results in a non-stationary high-level learning environment.
Other prior methods utilize a predeﬁned or pretrained subgoal space (P´er´e et al., 2018;
Nair & Finn, 2019; Zhang et al., 2020; Ghosh et al., 2018) to keep the stability of the subgoal representation.
However, those methods require task-speciﬁc human knowledge or extra training data.
In this work, we propose a novel regularization to stabilize the subgoal representation learning.
Specifically, the subgoal representation is learned with a contrastive triplet loss.
The contrastive objective is used as an auxiliary representation loss in other RL literature as well (Oord et al., 2018;
Laskin et al., 2020; Fortunato et al., 2019).
Beneﬁting from temporally extended exploration, HRL methods have shown better exploration abilities (Nachum et al., 2019b).
Bottom-up HRL works learn a set of diverse skills or options in a self-supervised manner, and use those semantically meaningful low-level skills to explore in downstream tasks (Jinnai et al., 2020; Co-Reyes et al., 2018;
Eysenbach et al., 2018; Sharma et al., 2019;
Li et al., 2019).
Nevertheless, the skills produced by those methods may not be required by the downstream tasks, since when learning the skills, the agent knows nothing about the downstream task rewards.
In GCHRL, Zhang et al.
(2020) restricted the high-level action space to a k -step adjacent region of the current state to achieve better sample efﬁciency.
R¨oder et al.
(2020) aimed at improving high-level exploration via curiosity-driven intrinsic rewards.
However, those methods require oracle subgoal spaces designed with prior knowledge.
In contrast, HESS learns subgoal representations online.
Li et al.
(2021) proposed an HRL approach, LESSON, with online subgoal representation learning, and provided theoretical analysis that the learned representation could support efﬁcient exploration.
However, without intrinsic rewards at the high level, LESSON could hardly solve long-horizon tasks with extremely sparse rewards.
This paper improves exploration from an orthogonal perspective from LESSON, and develops an active exploration strategy for high-level policy learning.
We propose two measures for subgoals: novelty and potential.
The cumulative count in the novelty measure is related to successor features (Kulkarni et al., 2016b).
Previous exploration methods use successor features to propagate the uncertainty of value function (Janz et al., 2019) or whether the features occur (Machado et al., 2020).
In contrast, our method propagates an explicit estimation of the number of feature occurrence through trajectories, which is conducted without function approximation.
The potential measure of reachability shares some similarities with empowerment (Salge et al., 2014), which is deﬁned as the agent’s control over its environment.
By maximizing empowerment (Campos et al., 2020; Gregor et al., 2016; Mohamed & Rezende, 2015), an agent could reach more diverse states.
Unlike empowerment measured by mutual information, the potential measure is formalized with the distances between desired subgoals and achieved latent states.
In Appendix C, we discuss how HESS is related to goal selection strategies in the multi-goal RL domain (Schaul et al., 2015).
E XPERIMENTS
We evaluate HESS on a set of challenging sparse-reward environments that require a combination of locomotion and object manipulation skills, aiming at answering the following questions: (1)
Can HESS outperform state-of-the-art exploration strategies in sample efﬁciency and overall performance?
(2) Can HESS learn a stable subgoal representation space efﬁciently?
(3) How important are the various components of the HESS agent?
(4) What do the subgoals selected by HESS look like?
(5) How much would the choice of hyper-parameters inﬂuence the experimental performance?
For better sample efﬁciency, we utilize an off-policy algorithm, Soft Actor-Critic (SAC) (Haarnoja et al., 2018), to learn both-level policies.
To mitigate the inﬂuence of the entropy term of SAC on HESS, we use SAC with automatic entropy tuning.
5. E NVIRONMENT S ETUP
We evaluate on a suite of MuJoCo (Todorov et al., 2012) tasks that are widely used in the HRL community, including Ant (or Point) Maze, Ant Push, Ant FourRooms, Cheetah Hurdle, Cheetah
Ascending, and two variants with low-resolution image observations.
The experiments with image input are labeled ‘Images’.
To demonstrate the exploration ability of HESS, we adapt those tasks to make them more challenging.
Different from the settings of random start or random goal with dense external rewards (Nachum et al., 2018; 2019a; Li et al., 2021; Zhang et al., 2020), in the tasks used in this work, a simulated robot starts from a ﬁxed position and needs to reach a faraway target position with sparse external rewards.
Furthermore, there is no predeﬁned subgoal space provided.
More details about the environments and implementation are available in Appendix A and B, respectively.
5. C OMPARATIVE A NALYSIS
We conduct experiments comparing the proposed hierarchical exploration approach to the state-ofthe-art baselines:
(1) H-ICM
(Pathak et al., 2017): prediction error in the latent subgoal space as intrinsic rewards to the high level.
(2) H-SR (Machado et al., 2020): count-based exploration bonus in the high level, and the counts are estimated by the norm of the successor representation.
(3) DADS (Sharma et al., 2019): an unsupervised skill discovery method via predicting dynamics.
(4) LESSON (Li et al., 2021): an GCHRL method learning the subgoal representation online using triplet loss, and no intrinsic rewards in the high level.
(5) SAC (Haarnoja et al., 2018): the non-hierarchical base
RL algorithm used in this work.
As shown in Figure 4, the proposed method substantially outperforms the baselines in all the tasks.
The outperformance in the Ant FourRooms task is more signiﬁcant since the maze scale of this task is larger, and the exploration problem is harder.
Both H-ICM and H-SR underperform our method.
The dynamic model of the simulated physics engine is relatively easy to ﬁt by neural networks, so the intrinsic rewards of H-ICM may vanish before the policies have been learned well.
The HSR method estimates visit counts using the (cid:96) 1 -norm of the successor representation, which is the expected discounted future state occupancies in trajectories.
This idea shares some similarities with the cumulative counts in the novelty measure.
Nevertheless, the successor representation estimates the expected future state occupancy starting from a given state (Dayan, 1993), but not the visitation number of the given state, which is less helpful to promote exploration.
When predicting dynamics in the whole observation space, the intrinsic rewards of DADS could hardly help the agent learn gait skills that make the robots move.
The original paper (Sharma et al., 2019) has also demonstrated that without x, y prior, the trajectories generated by a primitive skill have a large variance.
We provide a visualization of those skills in Appendix D. As a result, the success rates of DADS on all the tasks are zero.
Hence, we omit them from Figure 4.
LESSON uses no intrinsic reward to promote exploration in the high level.
Even with a good subgoal representation space, it still could not solve those tasks with sparse rewards.
The non-hierarchical method SAC performs poorly in all the tasks, demonstrating the strength of the hierarchical structures in solving long-horizon tasks with sparse rewards.
5. V ISUALIZATION OF R EPRESENTATION L EARNING
[Figure]
We visualize state embeddings of 5 trajectories from the beginning to the end of the ⊃ -shape corridor in the Ant Maze (Images) task in Figure 6, comparing the representations learned by HESS and HESS without stability regularization.
Those representations are all learned from top-down image observations.
HESS is able to learn an effective subgoal representation at an early stage.
By optimizing the contrastive objective, the Euclidean distances in the latent space approximately correspond to the number of transitions between states in local areas, which provides subgoal-reaching rewards for low-level policies.
But globally, the distance could not represent the number of transitions.
For example, it requires lots of transitions to move from the start
[Figure]
to the goal in the ⊃ -shaped maze, but they have not been pushed much away in the latent space, since there is no explicit constraint in the triplet loss for the distance between features more than c timesteps apart.
Comparing Figure 6(a) to Figure 6(b), we ﬁnd that with the stability regularization, when the features have already ﬁtted the learning objective well, their changes are minor afterward.
There is almost no change for the red features from 0.25M steps to the end in Figure 6(a).
Without the stability regularization, all the features are changing dramatically.
The non-stationary issue makes high-level policy learning very hard.
In Figure 6(b), at 0.25M steps (the second subﬁgure), the high-level agent at the red start should select a subgoal to move upwards.
However, at 0.65M steps (the fourth subﬁgure), selecting a subgoal in the right of the start state is better.
5. A BLATIVE A NALYSIS OF V ARIOUS C OMPONENTS
To understand the importance of various design choices of HESS, we conduct ablation studies on the subgoal representation learning and the proposed exploration strategy separately.
In the ablation studies of representation learning, we compare HESS, HESS without stability regularization, and
HESS without both stability regularization and prioritized sampling, shown by the curves without markers in Figure 7.
In the tasks with image input, the advantage of stability regularization and prioritized sampling is more substantial, since the representation of image observations is harder to learn.
In the tasks with vector input, the subgoal representation has a good generalization ability, so the effect of stability regularization and prioritized sampling becomes less signiﬁcant.
In the ablation studies of the exploration strategy, we compare HESS, HESS without the potential measure, replacing cumulative counts N ( φ ( s )) with immediate counts n ( φ ( s )) , and reactive exploration with high-level intrinsic rewards
r
i ( s ) =
η 1 / (cid:112) N ( φ ( s ))
+ η 2 / (cid:112) − U
( φ ( s )) , shown by the curves with markers in Figure 7.
The potential measure is extremely important, since without it, the subgoals are concentrated on the areas with low counts, but not extendable, like the areas near the corners in Figure 8(b).
Therefore, the success rates without the potential measure are much lower.
With immediate counts n ( φ ( s )) as the novelty measure, the agent loses a long-term vision to seek out novel subgoals, and only focuses on novel states in its neighborhood.
Hence, the states near the walls are selected as subgoals more frequently in Figure 8(c) than in Figure 8(a), but those subgoals are still better for exploration than the subgoals selected without the potential measure.
In the ablation study of active exploration, we can see that by actively selecting favorable subgoals,
HESS has achieved better sample efﬁciency than the reactive exploration method with intrinsic rewards, as the reactive method could not immediately seek the novel and promising subgoals even when it has detected them.
[Figure]
5. A BLATION S TUDIES ON H YPER -P ARAMETER S ELECTION
In this section
[Figure]
we set up a set of ablation tests on several hyper-parameters of HESS in the Ant Push (Images) task.
From Figure
[Figure]
we can see that most parameters work in a large range.
Furthermore, we use a single suite of parameters in all the tasks in this paper, which also indicates that the proposed approach is robust to hyper-parameter selection
[Figure]
Scaling factor α balances novelty and potential.
Smaller α encourages HESS to choose more novel state embeddings as subgoals.
Results in Figure 9 (a) indicate that HESS is robust against α , since the proposed method could work well, when the potential measure mitigates the negative inﬂuence of novelty inaccuracy on exploration.
For all tasks in Section 5.2, we set α = 0 .
Extended distance d e is the distance between the subgoal selected from the replay buffer and the imagined subgoal g e .
To keep g e still near the current latent state, d e should be no larger than the radius r g of the neighborhood for selecting subgoals.
The performance of HESS is reasonable as long as d e is not too large, as an excessively large d e may cause the subgoal g e pursued by the low level too far away from the current latent state.
For all tasks in Section 5.2, we set d e = 5 . 0 .
Stable ratio k % : The stability regularization constrains the changes of the embeddings for k % of the states with the minimum triplet loss in the buffer.
When k % is too large, the representation learning efﬁciency may be slightly affected, which is harmful to the hierarchical policy learning, as indicated in Figure 9 (c).
For all tasks in Section 5.2, we set k % = 0 .
3 , i.e., the regularization is applied to 30% of the data in buffer.
Low-level policy length c is an important and common hyper-parameter in HRL.
With a larger c , the burden of high-level decision-making is lighter, but low-level policy learning becomes harder.
The learning performance is better when c is 20 or 50 with an episode length of 500 .
Among all the hyper-parameters, c seems to inﬂuence the performance most.
For all tasks and baselines in Section 5.2, we set c = 50 .
C ONCLUSION
To solve long-horizon sparse-reward tasks with GCHRL, we design novelty and potential measures for subgoals upon stable subgoal representation learning, and develop a hierarchical exploration strategy that actively seeks out new promising subgoals and states.
As the dimension of the subgoal space in this work is low, we employ a naive count estimation method in the representation space.
When the dimension of the subgoal space is higher, it would be better to learn a density model (Papamakarios et al., 2019), or utilize more advanced count estimation methods, such as kernelbased methods (Davis et al., 2011).
For the future work, we believe that the idea of stability is general and could be used beyond HRL.
For example, maybe many existing techniques in non-deep
RL could be applied to representations learned with stability as well.
R EPRODUCIBILITY S TATEMENT
For reproducibility, we include an anonymous downloadable source code in the supplementary material.
Beyond that, we describe the details about the environments and the implementation in Appendix A and B.
A E NVIRONMENT D ETAILS
The environments of Point Maze, Ant Maze, Ant Push, Ant FourRooms, Cheetah Hurdle, and Cheetah Ascending are shown in Figure 10.
The environment rewards are sparse and binary.
We deﬁne a “success” as being within an L2 distance of 1.5 from the goal.
If the agent successfully reaches the goal, the reward is 1 , and else the reward is 0 .
The observation space includes the current position and velocity of the simulated robot and the environment goal.
For the ‘Images’ versions of these environments, we zero-out the x, y coordinates in the observation and append a low-resolution 5 × 5 × 3 top-down view of the environment, as described in (Nachum et al., 2019a; Li et al., 2021).
Each episode ends at 500 timesteps for the Ant Maze, Point Maze, and Ant Push task and ends at 1000 timesteps for the Ant FourRooms, Cheetah Hurdle, and Cheetah Ascending task.
[Figure]
Ant Maze.
The size of the ⊃ -shape maze is 12 × 12 .
For both training and evaluation, the start position is ﬁxed to (0, 0), and the environment goal is ﬁxed to (0, 8).
Point Maze.
The maze structure is the same as the Ant Maze task.
The simulated robot is a point mass, which is controlled by actions of dimension two, rotation on the pivot of the point mass and movement in the direction of the pivot.
Ant Push.
In this task, a movable block is placed at (0, 4).
The agent is initialized at position (0, 0), and the goal position is ﬁxed to (0, 8).
Therefore, the agent must ﬁrst move to the left, then push the movable block to the right, and then navigate to the goal unimpeded.
Ant FourRooms.
In this environment
, the maze structure is larger, 18 × 18 .
The start position is (0, 0), and the goal is (14, 14).
Cheetah Hurdle.
The simulated Half-Cheetah robot is required to stride over a hurdle with a height of 0.35 and walk to the goal, which is 3.2 meters from the start position.
Cheetah Ascending.
The simulated Half-Cheetah robot has to climb up to a hill to reach the goal.
The height of the goal position is 1.
B I MPLEMENTATION D ETAILS
Except for the hyper-parameters discussed in Section 5.5, we list other hyper-parameters and their ranges considered in the experiments in Table 1.
The probability p of actively selecting subgoals with the proposed exploration strategy is initialized as 0 . 7 , and linearly annealed to 0 at half of the total training steps.
For subgoal selection, our method ﬁrstly samples M states satisfying the constraint from the replay buffer B , { s i | D ( φ ( s i ) , φ ( s t )) <
r g } Mi =1 ∼ B , and then selects a favorable subgoal with Equation 6 from the candidate subgoal set { φ ( s i ) }
Mi =1 .
When calculating the expectation in the novelty and potential measures (Eq. 4 and 5), to deal with the off-policyness of the data in the buffer, we decrease the effects of the older rollouts with a discount factor γ = 0 .
995 , taking inspiration of the Exponential Moving Average (EMA) method.
For the neural network structures, the actor network for each level is a Multi-Layer Perceptron (MLP) with two hidden layers of dimension 256 using ReLU activations.
The critic network structure for each level is identical to that of the actor network.
We scale the outputs of the actor networks of both levels to the range of corresponding action space with tanh nonlinearities.
The representation function φ ( s ) is parameterized by an MLP with one hidden layer of dimension 100 using ReLU activations.
We use PyTorch and Adam optimizer to train all the networks.
Experiments are carried out on NVIDIA GTX
Ti GPU.
The learned hierarchical policies are evaluated everytimesteps by averaging performance over 10 episodes.
[Table]
C R ELATED W ORK
With the proposed novelty and potential measures, we develop a subgoal selection strategy to improve exploration in HRL.
There are numerous goal selection strategies in the multi-goal RL domain (Schaul et al., 2015) as well, including sampling diverse goals uniformly from a buffer (WardeFarley et al., 2018;
Pong et al., 2019), sampling goals from the achieved goal distribution (Nair et al., 2018), sampling goals of intermediate difﬁculty from a generative model (Florensa et al., 2018), and selecting goals in sparsely explored areas (Pitis et al., 2020).
However, those methods use predeﬁned or pretrained goal spaces.
In addition, unlike HESS, those ﬂat goal-conditioned methods pursue only one goal in an episode and thus may be less able to decompose complex tasks.
D B ASELINES
H-ICM represents the baseline method with prediction error as intrinsic rewards in the high level (Pathak et al., 2017; Burda et al., 2018).
Speciﬁcally, the intrinsic rewards are set as r it =
η 2 || f
( φ ( s t ) , a t ) − φ ( s t +1 )
|| 2 , where f is a forward model, and φ is the latent representation learned by the proposed method in this work.
We have tried the hyper-parameter η 2 in the range of { 10 , 100 , 1000 } and ﬁnd that η 2 = 100 works the best.
As the simulated physics engine is not much complex, the dynamic model f ( φ ( s t ) , a t ) may ﬁt the learning objective well before a successful policy has been learned.
Therefore, the intrinsic rewards of the prediction error are not so useful to promote exploration.
[Figure] H-SR is the baseline method using count-based intrinsic rewards in the high level, where the counts are estimated with the (cid:96) 1 -norm of the successor representation (Machado et al., 2020).
Since the (cid:96) 1 -norm of the successor representation only counts for whether the features occur in trajectories, but not the times of feature occurrence.
Therefore, the intrinsic rewards of H-SR vanish at an early learning stage.
We show how the variance of the intrinsic rewards changes with the environment timesteps in the Ant Maze task in Figure 12.
Since there is little difference in the intrinsic rewards among different states, the H-SR intrinsic rewards could hardly facilitate efﬁcient exploration in a late learning stage.
DADS learns a set of primitive skills by predicting the skill dynamics and then uses the learned skills in downstream tasks (Sharma et al., 2019).
The intrinsic rewards of skill learning encourage a skill to produce transitions (a) predictable under the dynamic model and (b) different from the transitions produced under other skills.
However, when the dynamic model is deﬁned on the whole observation space without the x, y prior, the trajectories of the learned skills suffer from a large variance, as shown in Figure 13.
Those high variance skills offer limited utility for hierarchical control.
FeUdal Networks (FuN) (Vezhnevets et al., 2017) also utilizes directional subgoal vectors.
We have run this method in the Ant Maze task for 70 million steps, and the success rate of FuN is still zero.
We analyze the reason for the underperformance of FuN, and ﬁnd that the subgoal representation learned by FuN is much worse than ours, as shown in Figure 14, since the representation learning in FuN lacks a strong inductive bias.
In our approach, we use the contrastive objective to learn the subgoal representation.
[Figure]
E A DDITIONAL E XPERIMENTAL R ESULTS E. A BLATION S TUDIES ON A DDITIONAL H YPER -P ARAMETERS
We conduct ablation studies on additional hyper-parameters in the Ant Maze and Ant Maze (Images) task, and ﬁnd that the representation update interval I which inﬂuences the representation stability as well, is also important to learning performance.
[Figure]
Representation update interval I (episodes) balances the stability and efﬁciency of the representation learning.
The task performance is robust when I is in a proper range ( 50 ≤ I ≤ 150 ).
With a short update interval ( I = 50 ), the learning performance is slightly hurt since the subgoal representation is less stable.
When I is too large ( I = 500 ), a successful policy could hardly be learned in the
Ant Maze (Images) task, as the large update interval hinders the representation learning efﬁciency.
Buffer size bs for both levels.
Figure 15 demonstrates that HESS is indeed robust against this hyper-parameter when 2 e 5 ≤ bs ≤ 2 e 6 .
Initialization p 0 for probability p .
When p 0 is set to a relatively large value ( p 0 ≥ 0 . 5 ), HESS achieves stable performance, as shown in Figure 15.
Small p 0 (0.3) makes the proposed exploration strategy seldom used, which leads to poor sample efﬁciency.
E. A BLATION S TUDY ON I MAGINED S UBGOALS
To guide the low-level controller to reach unexplored areas, we ﬁrstly sample state embedding g t from the neighborhood of the current latent state with a radius of r g , and then extend g t with an extended distance d e to obtain an imagined subgoal g e .
In this section, we compare the performance of directly increasing r g and perturbing g t in subgoal selection.
[Figure]
Results in Figure 16 demonstrate that when d e > 0 , the sample efﬁciency is better.
g e is obtained via perturbing an achieved subgoal, which might be in the unexplored areas, and thus more helpful to exploration.
Only increasing r g , the sampled subgoal g t is still be in explored areas, which is less useful to expand the explored areas. E. N OVELTY E STIMATION
We estimate the novelty measure N ( φ ( s )) with the data in the replay buffer.
To make the novelty estimation more accurate, we need to deal with the off-policyness of the data in the buffer, so we adopt the EMA method to decrease the effect of old rollouts, as described in Appendix B.
An alternative way to deal with the off-policyness is the importance sampling (IS) technique, i.e.,
N
[Equation]
φ
[Equation]
s
[Equation]
γ j ρ is n ( φ ( s i + jc ))] , where ρ is = (cid:81) ( i + jc ) /c
t = π hnew ( g | s tc )
/π hold ( g | s tc ) , π hnew is the current high-level policy, π hold is the old high-level sampling policy at state s tc , B is the replay buffer.
We compare these two implementations in the Ant Maze task, and those two methods empirically demonstrate comparable performance, as shown in Figure 17.
[Figure]
E. I MPLEMENTATION OF THE S TABILITY R EGULARIZATION
In this section, we ablate different implementations of the weighting function λ ( s )
in the stability regularization: - Binary λ ( s ) : For the top k % of the triplets with the minimum representation losses, we set λ ( s ) =
λ 0 for the anchor states ( s t ) in these triplets, and for the other states, λ ( s ) = 0 .
- Continuous λ ( s ) :
Weighting λ ( s ) continuously based on the representation loss, i.e., λ ( s t ) =
λ 0 / (cid:112) 1 + L tri ( s t , · , · ) , where the triplet ( s t , · , · ) is uniformly sampled from the replay buffer.
Compared to the continuous λ ( s ) implementation, binary λ ( s ) sharply distinguishes the welloptimized embeddings from the underﬁtting ones, and thus has a slightly better performance, as demonstrated in Figure 18.
[Figure]
E. G ENERALIZATION A BILITY OF D IFFERENT O BSERVATION S PACE
To study the generalization ability of different observation space, we set the start state in a less explored area, and compare the trajectories of the policies with image input and non-image input, when their replay buffers have similar coverage of the state space.
From Figure 19, we can see that the policy with image input seems to avoid the walls better, maybe because the image observations contains the information where the walls are, leading to better generalization.
[Figure]
E. A BLATION ON L OW -L EVEL S TATE S PACE
We study the effects of including the latent state embeddings in the low-level states, and compare the learning performance using s lt = s t and s lt = s t
|| φ ( s t ) in the Ant Push (Images) task.
When s is an image, practically, the latent embedding φ ( s ) is concatenated to the hidden layer in the low-level policy network.
As shown in Figure 20, including the learned state embeddings in the low-level states leads to faster convergence, since the changes of the representation space are known to the low-level policy, which reduces the non-stationarity of the low-level rewards.
[Figure]
Ablation on
[Figure]
E. I NTERACTION B ETWEEN N OVELTY AND P OTENTIAL M EASURES
In this section, we analyze the effects of the interaction between novelty and potentials measures by visualizing the subgoal selection w/ and w/o potential in the Ant Push task.
Figure 21 contains (a) the subgoal selection w/o the potential measure, i
[Figure]
e., only using the novelty measure to select subgoals; (b) potential in the latent representation space; and (c) subgoal selection using the combined measure of novelty and potential.
In Figure 21(a), the subgoals are concentrated in latent areas with low visitation density, but there are some latent states close to the walls selected as subgoals (See the subgoals in the red box).
For those less reachable states near to the walls, their potential is lower, shown in Figure 21(b).
Therefore, the combination of the novelty and potential measure could better guide the agent to unexplored areas in Figure 21(c)
[Figure]