L EARNING S UBGOAL R EPRESENTATIONS WITH S LOW D YNAMICS
Anonymous authors
Paper under double-blind review
A BSTRACT
In goal-conditioned Hierarchical Reinforcement Learning (HRL), a high-level policy periodically sets subgoals for a low-level policy, and the low-level policy is trained to reach those subgoals.
A proper subgoal representation function, which abstracts a state space to a latent subgoal space, is crucial for effective goal-conditioned HRL, since different low-level behaviors are induced by reaching subgoals in the compressed representation space.
Observing that the high-level agent operates at an abstract temporal scale, we propose a slowness objective to effectively learn the subgoal representation (i.e., the high-level action space).
We provide a theoretical grounding for the slowness objective.
That is, selecting slow features as the subgoal space can achieve efﬁcient hierarchical exploration.
As a result of better exploration ability, our approach signiﬁcantly outperforms stateof-the-art HRL and exploration methods on a number of benchmark continuouscontrol tasks 1 .
Thanks to the generality of the proposed subgoal representation learning method, empirical results also demonstrate that the learned representation and corresponding low-level policies can be transferred between distinct tasks.
I NTRODUCTION
Deep Reinforcement Learning (RL) has demonstrated increasing capabilities in a wide range of domains, including playing games (Mnih et al., 2015; Silver et al., 2016), controlling robots (Schulman et al., 2015;
Gu et al., 2017) and navigation in complex environments (Mirowski et al., 2016; Zhu et al., 2017).
Solving temporally extended tasks with sparse or deceptive rewards is one of the major challenges for RL.
Hierarchical Reinforcement Learning (HRL), which enables control at multiple time scales via a hierarchical structure, provides a promising way to solve those challenging tasks.
Goal-conditioned methods have long been recognized as an effective paradigm in HRL (Dayan &
Hinton, 1993; Schmidhuber & Wahnsiedler, 1993; Nachum et al., 2019).
In goal-conditioned HRL, higher-level policies set subgoals for lower-level ones periodically, and lower-level policies are incentivized to reach these selected subgoals.
A proper subgoal representation function, abstracting a state space to a latent subgoal space, is crucial for effective goal-conditioned HRL, because the abstract subgoal space, i.e., high-level action space, simpliﬁes the high-level policy learning, and explorative low-level behaviors can be induced by setting different subgoals in this compressed space as well.
Recent works in goal-conditioned HRL have been concentrated on implicitly learning the subgoal representation in an end-to-end manner with hierarchical policies (Vezhnevets et al., 2017; Dilokthanakul et al., 2019), e.g., using a variational autoencoder (P´er´e et al., 2018; Nair & Finn, 2019;
Nasiriany et al., 2019), directly utilizing the state space (Levy et al., 2019) or a handcrafted space (Nachum et al., 2018) as a subgoal space.
Sukhbaatar et al.
(2018) proposed to learn subgoal embeddings via self-play, and Ghosh et al.
(2018) designed a representation learning objective using an actionable distance metric, but both of the methods need a pretraining process.
Near-Optimal
Representation (NOR) for HRL (Nachum et al., 2019) learns an abstract space concurrently with hierarchical policies by bounding the sub-optimality.
However, the NOR subgoal space could not support efﬁcient exploration in challenging deceptive reward tasks.
In this paper, we develop a novel method, which LEarns the Subgoal representation with SlOw dyNamics (LESSON) along with the hierarchical policies.
Subgoal representation in HRL is not only a state space abstraction, but also a form of high-level action abstraction.
Since the high-level agent makes decisions at a low temporal resolution, our method extracts features with slow dynamics from observations as the subgoal space to enable temporal coherence.
LESSON minimizes feature changes between adjacent low-level timesteps, in order for the learned feature representation to have the slowness property.
To capture dynamic features and prevent the collapse of the learned representation space, we also introduce an additional contrastive objective that maximizes feature changes between high-level temporal intervals.
We provide a theoretical motivation for the slowness objective.
That is, selecting slow features as the subgoal space can achieve the most efﬁcient hierarchical exploration when the subgoal space dimension is low and ﬁxed.
We illustrate on a didactic example that our method LESSON accomplishes the most efﬁcient state coverage among all the compared subgoal representation functions.
We also compare LESSON with state-of-theart HRL and exploration methods on complex MuJoCo tasks (Todorov et al., 2012).
Experimental results demonstrate that (1) LESSON dramatically outperforms previous algorithms and learns hierarchical policies more efﬁciently; (2) our learned representation with slow dynamics can provide interpretability for the hierarchical policy; and (3) our subgoal representation and low-level policies can be transferred between different tasks.
P RELIMINARIES
In reinforcement learning, an agent interacts with an environment modeled as an MDP M =
( S, A, P, R, γ ) , where S is a state space,
A is an action space.
P : S ×
A × S → [0 , 1] is an unknown dynamics model, which speciﬁes the probability
P
s (cid:48) | s, a ) of transitioning to next state s (cid:48) from current state s by taking action a .
R : S ×
A → R is a reward function, and γ ∈ [0 , 1) is a discount factor.
We optimize a stochastic policy π ( a | s ) , which outputs a distribution over the action space for a given state s .
The objective is to maximize the expected cumulative discounted reward E π [ (cid:80) ∞ t =
γ t r t ] under policy π .
M ETHOD
In this section, we present the proposed method for LEarning Subgoal representations with SlOw dyNamics (LESSON).
First, we describe a two-layered goal-conditioned HRL framework.
We then introduce a core component of LESSON, the slowness objective for learning the subgoal representation of HRL.
Finally, we summarize the whole learning procedure.
3. F RAMEWORK
Following previous work (Nachum et al., 2018; 2019), we model a policy π ( a | s ) as a two-level hierarchical policy composed of a high-level policy π h ( g | s ) and a low-level policy π l ( a | s, g ) .
The high-level policy π h ( g | s ) selects a subgoal g in state s every c timesteps.
The subgoal g is in a low dimensional space abstracted by representation function φ ( s ) :
S → R k .
The low-level policy π l ( a | s, g ) takes the high-level action g as input and interacts with the environment every timestep.
Figure 1 depicts the execution process of the hierarchical policy.
[Equation]
[Figure]
LESSON iteratively learns the subgoal representation function φ ( s ) with the hierarchical policy.
To encourage policy π l to reach the subgoal g , we train π l with an intrinsic reward function based on the negative Euclidean distance in the latent space, r l ( s t , a t , s t +1 , g )
=
−|| φ ( s t +1 )
− g
|| 2 .
Policy π h is trained to optimize the expected extrinsic rewards r envt .
We use the off-policy algorithm SAC (Haarnoja et al., 2018) as our base RL optimizer.
In fact, our framework is compatible with any standard RL algorithm.
Apparently, a proper subgoal representation φ ( s ) is critical not only for learning an effective lowlevel goal-conditioned policy but also for efﬁciently learning an optimal high-level policy to solve a given task.
As the feature dimension k is low, φ ( s ) has a compression property, which is necessary to make the hierarchical policy learning easier.
If φ ( s ) is exactly an identity function without any abstraction, the high-level policy π h still needs to explore in a large space and the complicated subgoal g for the low-level policy is hard to reach as well.
In this circumstance, the hierarchical structure cannot simplify the MDP and has no advantage over a ﬂat structure.
3. L EARNING S UBGOAL R EPRESENTATIONS
Inspired by physics-based priors, features with slow dynamics preserve higher temporal coherence and less noise (Wiskott & Sejnowski, 2002).
As the high-level policy acts at a lower temporal resolution compared to the low-level policy, it is sensible to learn a subgoal representation function with a slowness objective.
To solve large-scale problems, we parameterize the representation function φ ( s ) with a neural network to extract slow features.
One natural way of learning φ ( s ) is to minimize the squared difference between feature values at times t and t + 1 , min
[Equation]
E (
[Equation]
(1) where D is a replay buffer.
This loss function eliminates fast features, but can be trivially optimized if we allow lossy representation function φ
(e.g., if φ ( s ) =
for ∀ s ∈ S ).
To avoid such trivial solutions and capture dynamic features, we propose a contrastive loss to maximize the distance between high-level state transitions in the latent subgoal space, i.e., min φ E ( s t ,s t + c ) ∼ D [ −|| φ ( s t ) − φ ( s t + c ) || 2 ] .
To trade off these two loss functions, we adopt the technique of triplet loss (Chopra et al., 2005), i.e., imposing the latent distance between high-level transitions larger than a margin parameter m , as shown by Eq. 2.
If we remove the margin parameter m and the max operator, Eq.will be dominated by the maximizing distance part.
Margin m deﬁnes a unit of distance in the latent space, which prevents trivial solutions as well.
[Equation]
The above learning objective abstracts the state space to a latent subgoal space with slow dynamics.
As Eq.
2 optimizes the squared difference between feature values, the learned representation can preserve the spatial locality property of the state space, so a subgoal g can be selected in the neighborhood of φ ( s ) .
In the next section, we give a theoretical motivation for the slowness objective.
That is, selecting slow features as the subgoal space can promote efﬁcient exploration.
Algorithm 1 shows the learning procedure of our method.
We update φ ( s ) and π l at the same frequency so that the low-level reward function varies in a stationary way.
The high-level policy is updated less frequently, as the high-level transitions are less.
: Compute
Update buffer
Optimize by task with D c timesteps.
Optimize by maximizing cumulative low-level with timestep.
Sample a batch of state transitions from update
Eq every
E FFICIENT E XPLORATION WITH SLOW SUBGOAL REPRESENTATION
In this section, we provide a theoretical motivation for subgoal representation learning with slow dynamics from a statistical view.
To support a formal analysis, we consider selecting a subset of features from the state space as a subgoal space.
We prove that, given a ﬁxed subgoal space dimension, selecting slow features as the subgoal space can achieve the most efﬁcient hierarchical exploration.
We ﬁrst deﬁne a measure for exploration and describe assumptions of our analysis.
Then, we present a theorem about the optimality property and corresponding implications.
4. D EFINITIONS AND A SSUMPTIONS
To develop a theoretical analysis, we give a deﬁnition of slow features and a measure of exploration.
Then, we formulate the exploration process in goal-conditioned HRL as a random walk in the state space as follows.
As our theoretical analysis is broadly applicable to arbitrary feature space, we denote a state s t =
[ s 1 t , ..., s It ] T as a vector containing I features 2 .
State s t can be factored into slow features s slow and fast features s fast with a one-step feature change metric ∆ s it = | s it − s it +
| (1 ≤ i ≤ I ) .
Without loss of generality, we assume that E π r [∆ s it ] <
E π r [∆ s i +1 t ] , where π r is a random policy.
The expected one-step feature change of slow features is relatively small.
With a limited slow feature dimension k , s slow =
[ s 1 , ..., s k ] T , and the rest are fast features.
For example, the movements of a robot are slow, but the changes of noisy sensory observations are fast.
Deﬁnition 1 (Measure of Exploration) .
In goal-conditioned HRL, an effectiveness measure of hierarchical exploration is deﬁned as the Kullback–Leibler (KL) divergence from the distribution of explored states q ( x ) to a desired state distribution p ( x ) :
[Equation]
In this deﬁnition, the desired state distribution p ( x ) is a prior state distribution while q ( x ) is the distribution of the states explored by the agent.
According to Deﬁnition 1, when the state distribution of exploration q ( x ) is closer to the target state distribution p ( x ) , the exploration is more effective.
Deﬁnition 2 (Random Walk) .
In goal-conditioned HRL, the exploration process in the state space is an I -dimensional random walk when there is no extrinsic reward for the high-level policy and the low-level policy is optimal.
Deﬁne s 0 as the origin of the state space: s 0 = 0 , and the unit step of the random walk is X ct =
s t − s t − c , t = c, 2 c, · · · , which is i.i.d.
Denote a sequence of random variables Y n = (cid:80) ni
=1 X cic , then the asymptotic distribution of Y n is q ( x ) : Y n D → q ( x ) .
To analyze the exploration, we consider the circumstance with optimal low-level policy and the absence of extrinsic rewards for the high-level, so the high-level policy selects subgoals randomly. The agent can move independently and identically in the state space, leading to X ct is i.i.d.
In fact, q ( x ) can be seen as the steady state distribution of the Markov chain induced by the policy.
To facilitate the analysis of different subgoal representations, we make the following assumptions throughout this section:
(a) The transition function P ( s (cid:48) | s, a ) is deterministic.
(b) The features are all independent.
(c) X ct is bounded in the state space: {| x i | ≤ r i , i = 1 , · · · , I } , where x i is the i -th element of X ct , and r i is a ﬁxed upper bound of | x i | .
(d) The subgoal g selected by the high-level policy at time t is constrained in the neighbourhood of s t : (cid:110) | g j − s jt | ≤ r g , j = 1 , · · · , k (cid:111) , where g j and s jt are the j -th elements of corresponding vectors, and r g is a ﬁxed bound of subgoals in all dimensions.
Assumption (a) is a general technique to simplify theoretical analysis in RL (Krishnamurthy et al., 2016; Boyan & Moore, 1995).
Assumption (b) makes it possible to analyze the exploration of each feature dimension separately.
Assumption (c) means that every c timesteps, the agent can move in dimension i with a step size | x i | ≤ r i , and slower features have a smaller bound: ∀
i < i (cid:48) , r i < r i (cid:48) .
Taking advantage of the spatial continuity of the state space, subgoals are set in the neighborhood of the current state in the selected subgoal feature dimensions, speciﬁed as Assumption (d).
4. O PTIMALITY AND I MPLICATIONS
Theorem 1.
Assume p ( x ) is a multivariate Gaussian distribution: p ( x ) ∼ N I ( x ; 0 , R ) , where R is a diagonal matrix diag( r 2 ) and r is large enough.
Given a ﬁxed subgoal space dimension k , selecting the k slowest features for the subgoal space leads to the optimal hierarchical exploration.
Denote the distribution of the explored states in this case as q slow , we have:
[Equation]
where Q is the sets of all distributions of explored states brought by different subgoal space selection.
Without any prior knowledge, we assume p ( x ) is an isotropic Gaussian distribution with zero mean.
When r is large enough, q ( x ) approximates a uniform distribution.
Proof sketch.
The exploration process is decided by the coverage area of the random walk, as shown in Deﬁnition 2, and a larger coverage area leads to better exploration (see Deﬁnition 1).
We analyze the coverage scale in each dimension separately.
The exploration ability varies in different dimensions since the slow-feature dimension has a smaller coverage scale.
Notice that the exploration ability in dimension i changes if we select the i -th feature for the subgoal space.
Concretely, if we choose slow features for the subgoal space, the coverage area in these dimensions will expand.
In contrast, selecting fast features decreases the ability of exploration.
We prove that with a ﬁxed subgoal space dimension k , if and only if we select the k slowest features as the subgoal space, D KL ( p (cid:107) q ) is minimized, i.e., achieves the optimal hierarchical exploration deﬁned in Deﬁnition 1.
See the detailed proof in Appendix A.
Selecting slow features as the subgoal space can achieve superior exploration shown in Theorem 1.
This property indicates that using the slowness objective to learn the subgoal representation can promote more efﬁcient exploration.
As real-world tasks are often on a large scale, utilizing neural networks to extract slow features as the subgoal space is more general.
To conclude, Theorem 1 is a theoretical grounding for the learning objective of LESSON.
R ELATED W ORK
Learning subgoal representations is a challenging problem in HRL (Dwiel et al., 2019).
Nachum et al.
(2018) predeﬁned a subspace of observations as a subgoal space with domain knowledge.
Li et al.
(2019) sought for an alternative way of setting advantage-based auxiliary rewards to the low level policy to avoid this difﬁcult problem.
Levy et al.
(2019) directly used the whole observation space, which is unscalable to high-dimensional tasks.
A variational autoencoder (VAE) (Kingma &
Welling, 2013) can compress the high-dimensional observations in an unsupervised way, and it has been utilized to learn a subgoal space in (P´er´e et al., 2018; Nair & Finn, 2019; Nasiriany et al., 2019).
However, the features extracted by VAE can hardly capture the transitional relationship in MDPs.
In Vezhnevets et al.
(2017) and Dilokthanakul et al.
(2019), a subgoal representation is learned in an end-to-end way with hierarchical policies.
Since the resulting representation is under-deﬁned, those methods often underperformed (see Nachum et al.
Ghosh et al.
(2018) proposed to learn representations using an actionable distance metric, assuming that goal-conditioned policies are given.
Sukhbaatar et al.
(2018) developed a method called HSP to learn subgoal representations via self-play, but HSP requires a pretraining process, and thus it may be inefﬁcient.
Near-Optimal
Representation (NOR) for HRL (Nachum et al., 2019) outperforms the previous methods by learning representations bounding the sub-optimality of hierarchical policies.
However, the optimization of NOR is complicated, and the abstraction of the NOR space does not aim for efﬁcient exploration.
In contrast, we develop a simple subgoal space learning method with a slowness objective.
Furthermore, we formally show that the slowness objective has a theoretical grounding for better exploration ability.
Slowness or temporal coherence has been an important prior for learning state representations in continuous control tasks (Bengio et al., 2013; Jonschkowski & Brock, 2015; Lesort et al., 2018).
Standard Slow Feature Analysis (SFA) methods learn slow features by solving an optimization problem with constraints (Wiskott, 1999; Wiskott & Sejnowski, 2002).
However, their expressivity tends to scale unfavorably in high-dimensional problems.
To increase the expressivity, hierarchical
SFA (Franzius et al., 2007; 2011; Escalante-B & Wiskott, 2013) composes multiple SFA modules in a layer-wise way.
More recent works use neural networks to extract slow features using a slowness loss function.
To avoid trivial solutions, another term, such as reconstruction loss (Goroshin et al., 2015a; Finn et al., 2016) or prediction error (Goroshin et al., 2015b), is also included in the loss function.
In similarity metric learning, contrastive or triplet loss is investigated to capture slow features in video and audio datasets as well (Jayaraman & Grauman, 2016; Jansen et al., 2018).
In reinforcement learning, several approaches exploit the slowly changing bias to extract useful features so that policy learning can be accelerated (Legenstein et al., 2010; Oord et al., 2018).
To the best of our knowledge, we are the ﬁrst to utilize the slowness objective in HRL, and our proposed method signiﬁcantly outperforms state-of-the-art HRL methods on benchmark environments.
E XPERIMENTS
We conduct experiments to compare our approach to existing state-of-the-art methods in HRL and in efﬁcient exploration.
Firstly, we show on a didactic example that LESSON can achieve the most efﬁcient state coverage among all the compared subgoal reprentations.
To demonstrate our strengths in high-dimensional tasks, we then compare with several baselines on a number of benchmark continuous-control tasks.
After that, we analyze the dynamic property of the learned subgoal representation and provide an interpretation by visualization.
Lastly, we show that both the subgoal representation and the low-level policies learned by our method are transferable.
6. D IDACTIC E XAMPLE : NC HAIN
[Figure]
The NChain environment was designed hard to explore by Osband et al.
(2016), as shown in Figure 2(a).
Starting from state 0, an agent can move forward (blue arrow) to the next state in the chain or backward (green arrow) to the previous state.
The state representation is encoded in binary, so the low bits are features with fast dynamics.
To make the problem harder, the effect of each action is randomly swapped with a probability of 0 . 1 .
In this near-deterministic environment, we compare the exploration ability of our method to HRL methods using other subgoal spaces with a dimension k = 1 .
Baselines include the NOR subgoal space (Nachum et al., 2019), a randomly selected bit of the state representation and the lowest bit (fast features).
For pure exploration comparison, we consider the circumstance of no external rewards, the same as the setting of our theoretical analysis in
Section 4.
Figure 2(b) illustrates that using slow features as the subgoal space can achieve the most efﬁcient exploration with goal-conditioned HRL.
As expected, the performance of fast features as the subgoal space is the worst.
Randomly selected features perform better than fast features.
Although NOR aims to bound the sub-optimality of the value function, our method outperforms NOR in terms of exploration.
6. M UJOCO T ASKS
We evaluate our proposed subgoal representation learning objective on a set of challenging MuJoCo tasks that require a combination of locomotion and object manipulation.
The details of our full implementation and environments are available in Appendix B.
We conduct experiments comparing to the following methods in hierarchical learning and exploration, and all the learning curves in this section are averaged over 10 runs.
[Figure] • NOR : HRL with a learned subgoal space, which is optimized to bound the sub-optimality of the hierarchical policy (Nachum et al., 2019).
Oracle : HRL with the oracle subgoal space ( x , y position of the agent) in navigation tasks.
• DCO : A hierarchical exploration method with deep covering options (Jinnai et al., 2020) 3 .
• EMI :
A ﬂat exploration method by predicting dynamics in a latent space (Kim et al., 2019).
• SAC : The base RL algorithm used in our method (Haarnoja et al., 2018).
Beneﬁting from a better exploration ability, our method with a temporally-coherent subgoal space signiﬁcantly outperforms baseline methods in terms of speed and quality of convergence.
Even when the raw observation is given by using top-down images, our method can approach the performance of the oracle ( x , y ) position subgoal space, presented in Figure 3(e), (f).
In the Ant Maze task, our method reaches a success rate of 100% at only 1 . 5 million training steps, which is more than two times faster than the NOR algorithm.
In the Point Maze task, the ﬂat exploration method EMI shows an equal performance with our approach.
However, when the dynamic model is more complex (e.g., for the Ant robot), predicting dynamics becomes much harder, and the performance of EMI degrades dramatically.
We evaluate
NOR with its published code 4 , and results show its ineffectiveness of exploration in challenging tasks.
The online DCO method can hardly learn successful policies in those tasks, partly because the pretraining of the second eigenfunction in their method is necessary.
6. A NALYSIS OF LEARNED REPRESENTATIONS Point Maze
[Table] Table 1 demonstrates that the dynamics of the features learned by our method are slow.
NOR has a relatively good performance in Ant Maze and Ant Maze (Images), since the NOR features in these [Table] two tasks are slower than those in other tasks.
The state space of the Point robot is low-dimensional and contains little information other than the ( x, y ) position, so the slow features (positions) are easy to be selected by a random strategy.
But NOR projects the state space of the Point robot to a latent space with fast dynamics, which results in unsatisfactory performance.
[Figure]
We visualize the subgoal representation and learned hierarchical policies of our method in the Ant
Push task in Figure 4.
The learned subgoal space highly resembles the oracle ( x , y ) position space.
By setting subgoals in the learned latent space, the high-level policy guides the agent to jump out of the local optimum of moving towards the goal.
The Ant robot under the hierarchical policy ﬁrstly [-
-]moves to the left, then pushes the block to the right, and ﬁnally reaches the goal.
In contrast, the SAC agent without a hierarchical structure easily gets stuck into the local optimum of moving directly to the goal, since the immediate extrinsic reward is given as the negative L2 distance to the environment goal.
We show the subgoal representation learning process in the Ant Push (Images) task in this section. Figure 5 (a) ∼ (f) visualize a trajectory to a hard goal in the representation space at different learning stages. Along with the representation visualization, we evaluate an easy goal as midpoint of the trajectory to the hard goal.
[Figure]
At the early stage (less than 0 . 25 million steps), the hierarchical policy and the representation learning is a short-term co-adaptation.
The hierarchical policy can approximately learn to reach the easy goal by using an inaccurate representation, which is updated in a slowly-changed manner. The data collected by the policy beneﬁts the learning of representation, and the improved representation function boosts the policy learning.
In the long term, as the representation function abstracts useful information of the state space and can be generalized to the less explored areas, it facilitates the exploration of the hierarchical policy to reach further goals.
6. T RANSFERABILITY OF R EPRESENTATIONS
Because of the generality of our representation learning objective with slow dynamics, the learned subgoal space is transferable between different tasks of the same robot.
The low-level policy induced by the learned subgoal representation is transferable as well.
To verify this transferability, we initialize the representation network and the low-level policy network in a target task with those weights learned in a source task and further ﬁnetune them in the target task.
The high-level policy for the target task is randomly initialized.
From Figure 6, we can see that transfer learning helps the agent learn more efﬁciently and achieve better asymptotic performance. [Figure]
C ONCLUSION
In this work, we propose a self-supervised subgoal representation learning method, LESSON.
Our approach is motivated by the slowness prior and supports iterative learning of the representation function and hierarchical policies.
In addition, we provide a theoretical grounding for the slowness prior in hierarchical exploration.
We test our method on a suite of high-dimensional, continuous control tasks, and it signiﬁcantly outperforms state-of-the-art HRL and exploration methods.
Furthermore, the subgoal representation and low-level policies learned by LESSON are transferable between different tasks.
Since the low-level policy learning may result in a non-stationary high-level transition function, combining LESSON with off-policy correction methods to reduce the variance of off-policy learning might be a promising future direction.
B E XPERIMENTAL D ETAILS B. ENVIRONMENTS
The environments of Point Maze, Ant Maze, Ant Push, and Ant Fall are as described in Nachum et al.
(2019), shown in Figure 7.
In each navigation task, we create an environment composed of 4 × 4 × 4 blocks, some movable and some with ﬁxed position.
During training, the target locations ( x, y ) are randomly selected by the environment from all possible points.
Final results are evaluated on a single challenging goal denoted by a small green block.
For the ‘Images’ versions of these environments, we zero-out the x , y coordinates in the observation and append a low-resolution 5 × 5 × 3 top-down view of the environment, equal to that used in Nachum et al.
[Figure]
The Ant FourRooms task has a much larger maze structure, which is four times as large as the Ant
Maze task.
So the maximal episode length for Ant FourRooms is also larger, which equals 1000 .
The maximal episode lengths of the other tasks are 500 .
B. N ETWORK S TRUCTURE
The actor network for each level is a Multi-Layer Perceptron (MLP) with two hidden layers of dimension 256 using ReLU activations.
The critic network structure for each level is identical to that of the actor network.
We scale the outputs of the actor networks of both levels to the range of corresponding action space with tanh nonlinearities.
The representation function φ ( s ) is parameterized by an MLP with one hidden layer of dimension 100 using ReLU activations.
B. T RAINING P ARAMETERS
• Discount factor γ = 0 .
99 for both levels.
• Adam optimizer; learning rate 0 . 0002 .
• Soft update targets τ = 0 .
005 for both levels.
• Replay buffer of size 1 e 6 for both levels.
• Reward scaling of 0 . 1 for high-level; 0 . 01 for low-level.
• Entropy coefﬁcient of SAC α = 0 .
2 for both levels.
• Low-level policy length c = 10 for the point robot and c = 20 for the ant robot since the ant robot moves more slowly than the point robot.
• Subgoal dimension of size 2 .
We train the high-level policy to output actions in [ − 50 , 50]in all the tasks except Ant FourRooms 5 .
In Ant FourRooms, we use [ − 200 , 200] 2 , as the maze is larger.
These actions correspond to absolute subgoals in the representation space.
• We early stop the training of representations and low-level policies at 1.5 million steps to make the hierarchical policy learning more stable.
B. E VALUATION
Learned hierarchical policies are evaluated every 25000 timesteps by averaging performance over 10 random episodes.
C A BLATION OF THE H IGH -L EVEL A CTION S PACE
In the experiments above, the high-level policy samples subgoals in an area [ − 50 , 50] 2 in the latent space.
In this section, we compare this sampling method with selecting subgoals in a neighborhood [ − 10 , 10] 2 of φ ( s t ) .
Figure 8 demonstrates that the size of the high-level action space has almost no effect on the learning performance.
It is easy for the high-level policy to learn to sample subgoals nearby φ ( s t ) because the faraway subgoals are difﬁcult to reach.
So the performance of these two implementations is nearly the same.
[Figure]
cc
[Figure]
ss
[Figure]
D A BLATION OF E ARLY S TOPPING
We have early stopped the training of the subgoal representation and low-level policies in the MuJoCo tasks, since we have observed that the hierarchical agent is able to learn a good subgoal representation and useful low-level primitive policies in a relatively early stage.
Furthermore, early stopping can save the training time, shown in Figure 9(b).
In fact, early stopping or not has little inﬂuence on the experimental performance.
[Figure]
E T HE S LOW U PDATE OF THE R EPRESENTATION F UNCTION
To make the hierarchical policy learning happen in a stable way, we would like to learn the subgoal representation smoothly, so we update the representation function φ ( s ) every timestep with only one minibatch data.
In this section, we compare this updating method with other updating methods at a slower time scale.
As all of these methods update the representation in a relatively stable way, there are no signiﬁcant difference of their experimental performance in the Ant Maze task, as demonstrated by Figure 10.
• 1 batch 1 step: Sample one batch of transitions and update φ ( s ) every timestep.
• 1 batch c steps : Sample one batch of transitions and update φ ( s ) every c timesteps.
• c batches c steps : Sample c batches of transitions and update φ ( s ) every c timesteps.
[Figure]