DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Abstract
Hierarchical Reinforcement Learning (HRL) has made notable progress in complex control tasks by leveraging temporal abstraction.
However, previous HRL algorithms often suffer from serious data inefficiency as environments get large.
The extended components, i.e. , goal space and length of episodes, impose a burden on either one or both high-level and low-level policies since both levels share the total horizon of the episode.
In this paper, we present a method of D ecoupling H orizons
Using a Graph in Hierarchical R einforcement L earning (DHRL) which can alleviate this problem by decoupling the horizons of high-level and low-level policies and bridging the gap between the length of both horizons using a graph.
DHRL provides a freely stretchable high-level action interval, which facilitates longer temporal abstraction and faster training in complex tasks.
Our method outperforms state-of-the-art HRL algorithms in typical HRL environments.
Moreover, DHRL achieves long and complex locomotion and manipulation tasks.
Introduction
[Figure]
Reinforcement Learning (RL) has been successfully applied to a range of robot systems, such as locomotion tasks [23, 7], learning to control aerial robots [9, 12] and robot manipulation [13, 21].
Goalconditioned RL, which augments state with the goal to train an agent for various goals [22, 18], further raised the applicability of RL in robot systems allowing the agent to achieve diverse tasks.
Hierarchical Reinforcement Learning (HRL), which trains multiple levels of goal-conditioned RL, has improved the performance of RL in complex and sparse tasks with long horizons using temporally extended policy [24, 25, 15].
On the back of these strengths, HRL was adopted to solve various complex robotics tasks [19, 10, 17].
However, HRL often has difficulty in complex or large environments because of training inefficiency.
Previous studies speculated that the cause of this problem is the large goal space, and restricted the high-level action space to alleviate this phenomenon [27, 11].
Nevertheless, this approach performs well only in limited length and complexity and still suffers from the same trouble in larger environments.
[Figure]
We show that this practical limitation of HRL can be mitigated by breaking down the coupled horizons of HRL.
In previous HRL frameworks, the horizons of the low level and high level are related to each other structurally because they share the total length of the episode.
This relation causes a tradeoff between the training burden of both levels; if the intervals between highlevel actions increase (x-axis in Figure 2), the low-level policy has to cover a wider range, and in the opposite case, the high-level policy takes charge of the extended burden alone in large environments (y-axis in Figure 2).
This is the reason why the previous HRL algorithms cannot cope with extended components of large environments (see Table 1 for the performance of the previous HRL method at various intervals).
We break down the coupled horizons of HRL:
To break the relation between the horizons of both levels, we adopt a graph structure.
In our method, the high-level policy can use a longer temporal abstraction while the lower one only takes charge of smaller coverage by decomposing the subgoal into several waypoints with a graph.
In this way, the HRL algorithm obtains the capability to stretch the interval of high-level action freely and achieves complex and large tasks, thanks to the enlarged strength of the HRL.
In summary, our main contributions are:
• We show that the previous HRL structures are not scalable in large environments, and that this limitation can be mitigated by removing coupled traits of high level and low level.
• To break down the coupled traits of HRL, we propose DHRL which decouples the horizons of high-level and low-level policies and bridges the gap using a graph. • Our algorithm outperforms state-of-the-art algorithms in typical HRL environments and achieves complex and long tasks.
Preliminaries
We consider a finite-horizon Universal Markov Decision Process (UMDP) which can be represented as a tuple ( S , G , A , T , R , γ ) where S , A and G are state space, goal space and action space respectively.
The environment is defined by the transition distribution T ( s t +1 | s t , a t ) and reward function R :
S × A × G → R , where s t ∈ S and a t ∈ A are the state and action at timestep t respectively.
Also, total return of a trajectory τ = ( s 0 , a 0 , ..., s H , a H ) is R ( τ, g )
= (cid:80) H − 1 t =
γ t r ( s t +1 , g ) where r ( s t +1 , g ) (or r ( s t , a t , g ) ) is a goal conditioned reward function and γ is a discount factor.
Subgoal sg , waypoint wp and goal g are defined in goal space G and we consider G that is a subspace of S with a mapping ψ : S → G .
HRL framework typically has high-level policy
π hi and low-level policy π lo , each maintaining a separated replay buffer B hi and B lo .
Our method also follows the general HRL framework and employs both buffers that store high-level ( s t , g t , sg t , r t , s t
+ c h ) ∈
B hi and low-level transition data ( s t , wp t , a t , r t , s t +1 )
∈ B lo , where c h is the interval between high-level action.
One of the key problems in learning HRL is that the low-level policy π lo is non-stationary and thus old data from past policies may contain different next-states s t + c h even though the high-level policy provides the same subgoal in the identical state.
To bypass this off-policy discrepancy, HRL models use off-policy correction which relabels the old subgoal of high-level policy to be the most ‘plausible subgoal’ that will result in a similar transition in old data with the current low-level agent [16, 27, 11] (HIRO-style off-policy correction).
Other approaches propose to relabel high-level action sg t to be achieved state s t + c h (HAC-style hindsight action relabelling), reducing the computation cost to find a ‘plausible subgoal’ [14].
Graph-guided RL methods, which combine the strength of RL and planning by decomposing a long-horizon task into multi-step sub-problems, estimate the temporal distance between states and goals to construct a graph G =
( V , E ) on goal space G without additional prior knowledge about environments.
Previous studies proposed various methods to recover distance from Q-value [3, 8, 26].
If the agent gets -1 reward at every step except when it is in a goal area where the agent gets 0 reward, then, Q lo ( s, a | g ) can reveal the temporal distance between s to g as: (Refer to the Appendix B for the detailed derivation.)
[Equation]
[Equation]
However, it is known that recovering temporal distance correctly from the vanilla Q-network in this setting is challenging.
For that reason, the previous methods use an additional Value function approximator [26] or distributional Q-networks [3].
Related work
[Figure]
Graph-guided RL.
Graphs have recently been used as a non-parametric model in reinforcement learning (RL) to combine the advantages of RL and planning [3, 8, 26, 2, 5].
By decomposing a long-horizon task into multi-step planning problems, these studies have shown better performance and data efficiency.
Search on the Replay Buffer (SORB) [3] constructs a directed graph based on the states randomly extracted from a replay buffer and Q-functionbased edge cost estimation.
The follow-up studies further improved the performance of earlier graph-guided RLs [3, 8] by combining additional methods such as graph search on latent space[26] or model predictive control [2].
However, previous papers sidestepped the exploration problems in complex tasks through ‘uniform initial state distribution’ [3, 8, 26, 5], or work only on the dense reward settings [2].
We emphasize that the ‘uniform initial state distribution’ accesses privileged information about the environment during training by generating the agent uniformly within the feasible area of the map.
This greatly reduces the scope of application of these algorithms.
Unlike prior methods, ours can train from sparse reward settings and a ‘fixed initial state distribution’ without knowledge of the agent’s surroundings, which makes it practical for physical settings.
For detailed examples and comparison of various initial state distributions, see Table 2 and Figure 11 in Appendix C.
Constrained-subgoal HRL.
To mitigate the training inefficiency issue of HRL, several researchers proposed methods that restrict the action of high-level agent to be placed in adjacent areas.
Hierarchical Reinforcement Learning with k-step Adjacency Constraint (HRAC) [27] limits the subgoal to be in the adjacency space of the current state.
Hierarchical reinforcement learning Guided by Landmarks (HIGL) [11] improved the data efficiency by adding novelty-based landmarks to adjacency-constrained HRL.
However, these improvements only work on the limited length and complexity of the environment.
The previous work closest to our method is HIGL.
However, there are three key differences between the previous work and our approach.
First, our model explicitly includes the whole graph structure while HIGL needs to train high-level action to imitate the graph by using an additional loss term corresponding to the Euclidean distance from the nodes of the graph.
Second, we use a graph to decouple the horizons of high level and low level, unlike the previous method which uses the graph only for guidance.
Most importantly, ours can achieve goals in long and complex environments.
To the best of our knowledge, there is no prior HRL research to train a model that has decoupled the horizons of the two levels.
𝜋 ℎ𝑖 (𝑠𝑔 𝑡 |𝑠 𝑡 ,𝑔 𝑡 ) 𝜋 𝑙𝑜 (𝑎 𝑡:𝑡+𝑐 𝑙 |𝑠 𝑡 ,𝑤𝑝 𝑡,1 )
𝑠𝑔 𝑡 𝑠𝑔 𝑡+𝑐 ℎ 𝑔 𝑡 𝑔 𝑡+𝑐 ℎ … 𝜋 𝑙𝑜 (𝑎 𝑡+𝑐 𝑙 :𝑡+2𝑐 𝑙 |𝑠 𝑡+𝑐 𝑙 ,𝑤𝑝 𝑡,2 ) 𝜋 ℎ𝑖 (𝑠𝑔 𝑡+𝑐 ℎ |𝑠 𝑡+𝑐 ℎ ,𝑔 𝑡 ) 𝜋 𝑙𝑜 (𝑎 𝑡+𝑐 ℎ +𝑐 𝑙 :𝑡+𝑐 ℎ +2𝑐 𝑙 |𝑠 𝑡+𝑐 ℎ +𝑐 𝑙 ,𝑤𝑝 𝑡+𝑐 ℎ ,2 ) 𝜋 𝑙𝑜 (𝑎 𝑡+𝑐 ℎ :𝑡+𝑐 ℎ +𝑐 𝑙 |𝑠 𝑡+𝑐 ℎ ,𝑤𝑝 𝑡+𝑐 ℎ ,1 ) 𝑤𝑝 𝑡, 𝑤𝑝 𝑡, 𝑤𝑝 𝑡+𝑐 ℎ , 𝑤𝑝 𝑡+𝑐 ℎ ,2 𝑠 𝑡 𝑠 𝑡+𝑐 ℎ high low 𝑠𝑔 𝑤𝑝 [ Planning Over the Graph] ℬ 𝑙𝑜 𝑉 [Graph Construction] 𝑄 𝑔𝑟𝑎𝑝ℎ 𝑙𝑜 𝐸 𝐺(𝑉,𝐸) 𝐺(𝑉,𝐸) … … …
Methods
We introduce Decoupling Horizons Using a Graph in Hierarchical Reinforcement Learning (DHRL), which can separate the time horizons of high-level and low-level policies and bridge the gap between both horizons using a graph.
Our framework consists of high-level policy π hi ( sg | s, g ) , low-level policy π lo ( a | s, wp ) , and a graph G .
Given a goal g t in the environment, the high-level policy outputs a subgoal sg t (see Figure 4).
Then, the shortest path from the current state s t to sg t is found on the graph.
To do so, s t and sg t are added to the existing graph structure, then a sequence of waypoints ( s t , wp t, 1 , wp t, 2 , ..., sg t ) is returned using a graph search algorithm.
Finally, the low-level policy tries to achieve wp t,i during c l,i steps.
The key point of our method is that the low-level horizon h low =
c l is unrelated to the high-level horizon h high .
Since the high level generates one subgoal every c h steps, the H-step task is a H/c hstep task for a high-level agent ( h high
= H/c h ) where c h is the interval between high-level action ( c h > c l ).
In other words, unlike the previous HRL methods which have the relationship of
[Equation]
our algorithm does not have such relations, removing an obstacle toward a scalable-RL algorithm.
Since the low-level horizon c l is determined by the edge cost between waypoints, we can also express the c l between the i - 1 th waypoint and i th waypoint as c l,i , but we omit the letter i in the later statements that do not need to specify the waypoint.
In section 4.1, we explain how to construct a graph over states and find a path on the graph level.
In section 4.2, we present a low-level policy which can recover temporal distance between states without overestimation.
In section 4.3, We introduce a strategy to train our method through graph-agnostic off-policy learning.
Finally, in section 4.4, we propose additional techniques for better data efficiency in large environments.
4.1 Graph level: planning over the graph
This section details the graph search part in DHRL planning.
We emphasize that every distance in the DHRL model is based on temporal distance Dist ( · → · ) , obtained through Eq.
Thus, our algorithm requires no further information about the environment ( e.g. Euclidean distance between states) than general HRL settings.
To find the shortest path, we adopt Dijkstra’s Algorithm, as in the previous study [3].
The differences from the previous methods [3, 26] are the existence of the high-level policy, and whether the secondary path is considered.
Let ψ : S → G be the projection of states on the goal space.
In the graph initialization phase, we samples n nodes (also called landmarks) using FPS algorithm (Algorithm 2 in Appendix A) from ψ ( s ) where s ∈ S is state sampled from B lo .
Then, we connect a directed edge s 1 → s 2 if the temporal distance from s 1 to s 2 is less than the cutoff-threshold.
In the planning phase, the graph G ( V , E ) gets the subgoal sg t from the high-level policy and adds the projection of current state ψ ( s t ) and sg t to V , so that the number of nodes in G becomes n + 2 .
Then, we connect the edges with costs less than the cutoff-threshold between the newly added nodes and existing nodes.
Next, we find the sequence of waypoints W : ( wp t, 0 = ψ ( s t ) , wp t, 1 , wp t, 2 , ..., wp t,k − 1 , wp t,k = sg t ) that connects from ψ ( s t ) to sg t using a graph search algorithm.
At this time, if there is no path from ψ ( s t ) to sg t , we adopt a secondary path from ψ ( s t ) to sg † t , where sg † t is the closest node to sg t among the nodes connected from ψ ( s t ) .
After finding the waypoint sequence, the graph level provides wp t, 1 to the low-level policy and instructs it to reach wp t, 1 .
If it has been Dist ( wp t,i − 1 → wp t,i ) steps since the agent started tracking wp t,i or it achieves wp t,i , then we update the current tracking waypoint to the next waypoint
wp t,i +1 .
In order to reflect the newly discovered state, we update graph nodes at every N episodes.
However, it is challenging to train Q-network which recovers reliable temporal distance.
We explain the reason and a solution in the next section.
4.2 Low level: Separate Q-networks for Graph and Critic (SQGC)
Unlike previous graph-guided RL methods that leverage a graph only after the actor and critic networks have been learned, we construct and utilize a graph during training.
To do so, the low-level policy Q lo is evaluated during training to assign edge costs.
However, when π lo is not yet competent in achieving some goals or encounters difficult goals, the accumulation of failure experiences in the replay buffer causes underestimation of Q-values.
This leads to an overestimation of temporal distance reconstructed from Eq. (1) and spoils the graph near the overestimated region by making the node-selection algorithm (Algorithm 2 in Appendix A) select more and more graph nodes around the overestimated area.
Therefore, a temporal-distance reconstruction method is needed even when the policy is not sufficiently trained.
For this reason, we propose Separate Q-networks for Graph and Critic (SQGC) to prevent temporal distance overestimation.
SQGC is composed of two identical Q-networks using different proportions of hindsight goal relabeling (HER) [1].
The SQGC includes Q locritic and Q lograph where Q locritic is for training π lo , just like a typical application, and Q lograph is for recovering temporal distance between nodes.
We substitute wp t with ˆ wp t :=
ag t + t ftr in the sequential transition of a single episode ( s t , wp t , a t , r ( s t +1 , wp t ) , s t +1 ) t
=1: H − 1 where ag means the achieved goal and t ftr is a random integer drawn from the uniform distribution between 0 and H − t .
To train Q lograph , we relabel 100% of wp t in ( s t , wp t , a t , r ( s t +1 , wp t ) , s t +1 ) t =1: H − 1 as ˆ wp t , while we replace only 80% of wp t for
Q locritic .
Our method can prevent overestimation in the distance recovery by relabelling the goals in all transitions for training Q lograph because the experiences of failure are replaced with the successful trajectory (see section 5.3 for the ablation study).
Also, by maintaining the original Q criticlo to train π lo , there is no degradation in the performance of the agent who might otherwise not be able to get negative feedback from failure since the failure will be relabeled as a desired goal.
However, it is still challenging to train HRL including a graph level using off-policy RL algorithms.
We describe our approach to train DHRL using an off-policy algorithm in the next section.
4.3 High level: hindsight transitions for graph-agnostic off-policy learning
Thanks to decoupling the time horizons of both levels in HRL, the high-level policy in our method can look further without any additional burden on the low-level policy.
In other words, We can stretch c h , which represents how long each high-level action operates for.
However, because of the extended interval and non-stationarity of the high-level MDP, it is challenging to train DHRL with an off-policy algorithm, which is important for data efficiency in that the off-policy algorithm can use the previous data from the replay buffer.
This non-stationarity of the high-level MDP is caused by the presence of a graph and low-level policy.
Since the graph is gradually updated, it is challenging to train a model using an off-policy method from the data given by the previous graph.
Furthermore, as c h gets longer, predicting the similarity of the trajectory of c h steps with only the first step of π lo ( a i | s i , g i ) gets more difficult when we replace subgoal using HIRO-style off-policy correction [16] which is a popular method adopted by SOTA
HRL algorithms [27, 11].
To facilitate the off-policy learning for longer c h and the changing graph in DHRL, we adopt a well-known hindsight action relabelling method proposed in Hierarchical Actor-Critic (HAC) [14] in the high-level replay buffer data ( s t , g t , sg t , r t , s t
+ c h ) ∈
B hi .
Alongside the original transitions data, we copy the transition data and replace the subgoal ( sg t = π high β ( sg t | s t , g t ) ) with the achieved goal ag t + c h after c h steps.
Thus, we use both transitions ( s t , g t , sg t , r t , s t + c h ) and ( s t , g t , ag t + c h , r t , s t + c h ) to train the high-level policy.
Our method utilizes the optimality of the graph and low-level policy to include a graph in the learning process, while the HAC-style hindsight action relabelling method assumes the optimal low-level policy π lo ∗ only.
By replacing the previous subgoal with the achieved goal, we can assume that this transition was obtained from a stationary graph and π lo ∗ with an error below a bound which is set to be a function of the density of the graph.
We provide a theoretical analysis of the possibility of replacing the old off-policy graph with a virtual stationary graph in this section and
Appendix B.
Definition 4.1.
Given a compact state space S , G ( V , E ) is an ϵ − resolution graph if ∀ s ∈ S , ∃ v ∈ V s.t. max( Dist ( ψ ( s ) → v ) , Dist ( v → ψ ( s ))) <
ϵ , where V ⊂ ψ ( S ) and E = { ( v i , v j ) | v i , v j ∈
V , Dist ( v i → v j ) <
c l } .
Theorem 4.2.
Let G be an arbitrary ϵ − resolution graph ( ϵ <
c l / 2 ).
Also, let π lo β
and G β be the low-level policy and graph at the time of collecting the data.
Off-policy error rate ρ ( G ) is the normalized distance error with respect to the total traversal distance according to the change of π lo β and G β to π lo ∗ and G .
If there is a path from s to g , the upper bound of off-policy error rate ρ ( G ) using a path obtained from graph search over G is 2 ϵ/c l .
Theorem 4.2 shows that if we replace the previous subgoal with the achieved goal, and assume that these transitions are obtained from a stationary ϵ − resolution graph G , then the off-policy error ρ is less than 2 ϵ/c l .
Thus, we can train DHRL through a graph-agnostic off-policy RL algorithm using the substituted transition.
4.4 Optional techniques: gradual penalty and frontier-based goal-shifting
[Figure]
In this section, we propose two additional techniques, gradual penalty and frontier-based goalshifting (FGS).
These optional techniques can boost the performance of DHRL on some long tasks as shown in Figure 9(a) and (c).
We note that these are not essential to train DHRL and ours outperforms previous HRL frameworks without these techniques (see section 5.3 for the ablation study).
Gradual penalty.
Similar to the subgoal testing in the previous method [14], we propose gradual penalty (algorithm 4 in Appendix A), which can impose the penalty more delicately when the action space of the high-level agent gets large.
We evaluate sg t of the original transition data ( s t , g t , sg t , r t , s t + c h ) by categorizing the following three cases; (a) close to the graph and low level actually achieved the subgoal ∈ L 1 , (b) close to the graph but the low-level policy could not achieve the subgoal ∈ L 2 and (c) far from the graph ∈ L 3 .
In this way, we can impose the more detailed penalties to respond to the expansion of the high-level action space as the high-level interval c h stretches.
Frontier-based goal-shifting.
This optional technique has been devised to accelerate learning in a complex environment (algorithm 5 in Appendix A).
FGS moves the final goal g to the frontier area when g comes into the place where the graph is already laid out during training.
To check whether the goal is in the graph area, we examine whether min v ∈ V ( Dist ( v → g )) is smaller than cut-off threshold, where V is the set of the graph nodes.
Alternative goals are the addition of random noise to nodes sampled from v ∈ V proportional to − Q ( s 0 , π ( s 0 , v ) | v ) .
FGS is similar to the previous goal-directed exploration research [20] in that both use weighted samples, but ours does not maintain a generative model and samples the goals from the graph level in DHRL.
Note that we do not use FGS when we compare DHRL with the previous state-of-the-art, since this FGS is beyond the main contribution, which is about HRL structure.
PointMaze
AntMazeSmall
AntMazeBottleneck
Reacher3D
AntMazeComplex
[Figure] k)
[Figure] (k)
T e s t s u cc e ss r a t e
AntMaze (OURS)
T e s t s u cc e ss r a t e
Timesteps(k)
UR3Obstacle
Timesteps(k)  Experiments 5.1 Environment description
We evaluate DHRL on robot environments based on the MuJoCo simulator including some sparse and long-horizon tasks.
Firstly, various locomotion environments with ‘fixed initial state distributions’ are used to validate the temporal abstraction capability of DHRL in long-horizon and cluttered environments.
In test episodes of the locomotion environments, the agent gets one of the most challenging goals ( i.e. , the end of the maze).
We also evaluate our algorithm in a robot arm environment in which the agent aims to make the end-effector touch the goal, to evaluate our method in more complex dynamics.
• PointMaze / AntMazeSmall : The point / ant achieves the goal if it comes within 2.5 distance (success threshold) from the target point in 12 × 12 maze.
• AntMaze : 24 × 24 maze with success threshold• AntMazeBottleneck : Bottleneck exists at the middle of the maze.
The ant can barely pass through bottleneck.
• AntMazeComplex : 56 × 56 maze with success threshold• Reacher3D : 7-dof robot arm aims to reach a goal.
• UR3Obstacle : 6-dof robot arm aims to reach a goal in an environment with several boardshaped obstacles.
We adopt TD3 algorithm [4] for high-level and low-level networks and use Dijkstra algorithm to find the shortest path in the graph level.
Also, we note that only a few hyperparameters have been changed across various locomotion experiments: the number of nodes, penalty, and c h .
The results of DHRL in this paper are obtained using only sparse reward.
5.2 Experiment result
[Figure]
Baselines.
We compare our method with stateof-the-art algorithms with and without a graph respectively; HIGL [11] and HRAC [27].
For more comparison with shallow RL (SAC) [6] and vanilla HRL (HIRO) [16], see Table 3 in Appendix C.
Comparison to state-of-the-art algorithms.
Results are shown in Figure 6.
Thanks to decoupling the horizons of the levels, DHRL could stretch the high-level interval and shows high data efficiency and success rate in various locomotion [Figure] [Table] and goal-reaching tasks.
Moreover, DHRL is the only algorithm that can succeed in complex environments (AntMazeComplex and AntMazeBottleneck).
When measuring the performance of the baselines, we selected c h with the best performance among the values c h =10, 20, ..., and 80.
As shown in Figure [Table] previous HRL methods cannot solve long-horizon tasks.
This is likely due to the coupling ( c h = c l ) of the high-level horizon ( h high = H/c h ) and the low-level horizon ( h low =
c l ) resulting in an increased burden on either the high-level or the low-level policy.
For long-horizon tasks (large H ), if c h (= c l ) is fixed, then h high increases and the high-level performance plunges (Table 1).
On the other hand, if c h (= c l ) is increased, the low level has to manage a wider area and the performance plunges as shown in Figure 7 and Table 1.
5.3 Ablation study
Decoupling horizons.
In this section, we examine how decoupling horizons affects the performance of long-horizon HRL and evaluate whether DHRL can stretch the interval of high-level policy.
As shown in Figure 9(b), we tested various values of spacing of the high-level action, c h .
The result shows that even in long intervals of the high-level policy, DHRL shows consistent or improved performance (Figure 9(b)) without major degradation unlike previous HRL methods (Figure 7).
In particular, DHRL agent in long-horizon environments such as AntMazeComplex shows that extended high-level interval is crucial for exploration and performance (see the rightmost figure in Figure 9(b)).
Considering that the baselines could not solve long-horizon tasks even with various values of c h , and the performance degrades with increasing c h , we conclude that our method successfully decoupled the time steps in both levels and stretched the interval between high-level actions.
This trait allows the high-level policy to look further and take advantage of the extended temporal abstraction.
Separate Q-network for Graph and Critic (SQGC).
In this section, we evaluate the effects of the
Separate Q-network for Graph and Critic (SQGC) in our method.
The red line in Figure 9(a), which is a variant of DHRL without an SQGC, clearly shows that the SQGC is critical to the performance.
We empirically found that the overestimation of the temporal distance between states occurs especially near the obstacles ( e.g. corners at the maze), and it is difficult for an agent to pass by without a separate Q network for graph construction.
This is consistent with our analysis that the experiences of failure spoil the ability to recover the temporal distance from Q-network.
Ablate high-level policy.
Graph-guided RL, which maintains graph level and low-level policy, is also a variant of HRL in that the graph level is a non-parametric version of the high-level policy (see
Figure 3).
From this point of view, the main difference between graph-guided RL and our method is the existence of the high-level policy.
Then, why do we need additional high-level policy above the graph level?
Most graph-guided RL algorithms use ‘uniform initial state distribution’ to train the agent in complex environments and such assumption could be expensive, especially in the physical world where it is challenging to start from different positions each time.
Figure 9(d) shows the result of the graphguided RL algorithm, L3P
[26], with a uniform and fixed initial state distributions.
This indicates that without ‘uniform initial state distribution’, the performance and data efficiency of the previous graph-guided RL method drops seriously even in the smallest environment we experimented with.
In contrast, DHRL can explore without ‘uniform initial state distribution’ even in long environments thanks to the enlarged temporal abstraction and exploration performance.
Considering that the highlevel policy suggests a subgoal to graph level, we conclude that the high level facilitates better exploration of the graph-guided RL.
Thus, DHRL that has exploration capability also can be seen as the improved version of graph-guided RL.
Optional techniques.
The result in Figure 9(c) shows that the gradual penalty can improve data efficiency in long tasks.
By imposing a penalty on subgoals that are far from the current graph, the gradual penalty encourages the high-level agent to output a subgoal near the graph.
Since the low-level agent can hardly achieve subgoals far from the explored area, the gradual penalty accelerates the training by providing more valid goals.
FGS also can improve training efficiency in complex environments (Figure 9(a)).
This means that it is important to provide goals near the unexplored areas which help the agent explore the unseen states in complex environments.
With FGS, DHRL efficiently guides the agent by sampling the goals located on the rim of the graph.
Conclusion
We proposed a Decoupling Horizons Using a Graph in Hierarchical Reinforcement Learning (DHRL), which is a data-efficient HRL algorithm leveraging a graph to expand the range of problems that
HRL can solve, by decoupling horizons and allowing both levels to operate at their suitable horizons.
Our experimental results show that our method successfully separates the horizons of the levels and outperforms prior state-of-the-art methods.
We believe that our method can present a direction towards scalable HRL allowing the hierarchical policy to take advantage of the extended temporal abstraction and have a lower dependency on the horizon of the environment.
In this paper, we adopted a vanilla graph construction algorithm and sparse reward settings.
We expect that if a novelty-based node selection is added to the graph construction, the performance could be further improved.
Limitation.
While our algorithm shows successful results in complex and long-horizon tasks, it might be difficult to construct a graph in some tasks that require complex interactions with the environment because of the higher-dimensional state space.
We expect that this limitation can be solved by extending our algorithms into latent state space or by incorporating a node sparsification algorithm, which we leave for future works.  Acknowledgement This work was supported by AI based Flight Control Research Laboratory funded by Defense Acquisition Program Administration under Grant UD200045CD.
Also, Seungjae Lee is grateful for financial support from Hyundai Motor Chung Mong-Koo Foundation.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[No] This work inherits the potential negative social impact of reinforcement learning in robotics.
We do not anticipate that there will be any additional negative impact of this paper.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section B in Appendix (b) Did you include complete proofs of all theoretical results?
[Yes] See Section B in Appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Figure(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section C in Appendix 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.
(b) Did you mention the license of the assets?
[Yes] See Section 5.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A
Algorithms.
1: sample D lo i =
( s t , wp t , a t , r ( s t +1 , wp t ) , s t +1 )
i ∈ B lo 2: relabel wp t ← ˆ wp t =
ag t + t ftr to make ˆ D lo
i 3: update Q lo critic ,θ 1 and π lo ϕ 1 using D lo i ∪ ˆ D lo i 4: update Q lo graph ,θ 2 using ˆ D lo i 5:
if t mod d then 6: sample D hi i = ( s t , g t , sg t , r t , s t + c h )
i ∈ B hi 7:
relabel sg t ← ag t + c h to make ˆ D
hi i 8: for ( s t , g t , sg t , r t , s t + c h ) in D
hi i do 9: if r ( s t + c h , sg t )
=
then 10: r t ← r t ◁ subgoal ∈ L11: else if use GradualPenalty then 12: r t ← GradualPenalty (graph G , sg t , Q lograph ) 13: else 14: r t ← penalty p15: end if 16: end for 17: update Q hi θ 3 and π hi ϕ 2 using D
hi i ∪ ˆ D
hi i 18: end if
4: for do 5: ) 6: add SelectedNode 7: FarthestNode
FarthestNode DistList
ElementwiseMin ) 9: end for 10: return SelectedNode
Algorithm 3 Planning with DHRL 1: while not done do 2: if t mod graph _ construct _ freq then 3: construct a graph G ( V , E ) : sample V = ψ ( s ) where s ∈ D lo through FPS algorithm and get edge cost E by Eq.
(1) 4: end if 5: sg t
= π hi ( s t , g t ) 6: get W : ( wp t, 0 = ψ ( s t ) , wp t, 1 , wp t, 2 , ..., wp t,k − 1 , wp t,k = sg t ) 7: previous waypoint index idp ← 0 ; tracking waypoint index idt ← 1 ; tracking time t tr ←8: for τ = 1 to c h do 9: get low-level action a τ from π lo ( a τ | s τ , wp t,idt ) 10: act a τ in the environment and get s τ + 11: t tr += 1 ; t +=12: if agent achieve wp t,idt or t tr > Dist ( wp t,idp → wp t,idt ) then 13: idp += 1 ; idt += 1 ; t tr ←14: end if 15: end for 16: end while
1: Input: graph G ( V , E ) , subgoal sg t , Q lograph ,θ 2 , gradual penalty threshold ζ 1 , penalty p 1 , penalty p2: if min( Q lograph ,θ 2 ( v ∈ V , sg t )) <
ζ 1 then 3: r t ← penalty p◁ subgoal ∈ L4: else 5: r t ← penalty p◁ subgoal ∈ L6: end if
1: Input: s t , graph G ( V , E ) , goal g , Q lograph ,θ 2 , cut-off threshold ζ2: Dist ( s, g ) :
= log γ (1 + (1 − γ ) Q lograph ,θ 2 ( s, π ( s, g ) | g ))
3: if min v ∈ V ( Dist ( v → g )) <
ζ 2 then 4:
V candidate ← V + noise 5: g t ← random .
choice ( V candidate , weight = − Q lograph ,θ 2 ( s t , π ( s t , V candidate ) | V candidate )) 6: end if 7: return g t
1: Input: initial random steps τ randomwalk , initial steps without planning τ w / o graph , total training step τ total , Env, low-level agent Q lo critic ,θ 1 , Q lo graph ,θ 2 and π lo ϕ 1 , high-level agent Q hi θ 3 and π hi ϕ2: Dist ( s, g ) :
= log γ (1 + (1 − γ ) Q lograph ,θ 2 ( s, π ( s, g ) | g )) 3: for τ = 1 to τ total do 4: if Env.done then 5:
Env.reset (episode step resets to 0) 6: if Use FGS then 7: g ← FGS( G , g, Q lo graph ,θ 2 ) 8: end if 9: end if 10: if τ < τ randomwalk then 11: a t ← random.uniform(high = action.high, low = action.low) ◁ random action 12: else if τ < τ w / o graph then 13: a t ← vanilla HRL ( sg t = π hi ϕ 2 ( s t , g ) and π lo ϕ 1 ( s t , sg t ))
◁ act without planning 14: else 15: if Graph G is not initialized then 16:
Create a graph G ( V , E ) using FPS algorithm ◁ initialize graph 17: end if 18: if episode step(the step of the environment) %
c l = 0 then 19: sg t ← π hi ϕ 2 ( s t , g ) ◁ get subgoal 20: { wp t, 1 , wp t, 2 , · · · wp t,k } ← Dijkstra ′ salgorithm ( s t , sg t )
◁ get waypoints 21: current waypoint index n =22: end if 23: if achieved wp t,n or tried more than Dist ( wp t,n − 1 , wp t,n ) to achieve wp t,n then 24: current waypoint index +=25: end if 26: a t ← π lo ϕ 1 ( s t , wp t,n +1 )
◁ get low-level action 27: end if 28:
Env.step( a t ) 29:
Train low-level agent Q lo critic ,θ 1 , Q lo graph ,θ 2 and π lo ϕ 1 , high-level agent Q hi θ 3 and π hi ϕ30: if τ % graph update freq = 0 then 31:
Update Graph G ( V , E ) using FPS algorithm 32: end if 33: end for
B Proofs of Theorems.
Derivation of equation 1.
If a given policy π lo requires n steps to get from current s to a goal g , the γ -discounted return is Q lo ( s, π ( s, g ) | g ) = ( −
[Equation]
+ ( − [Equation] γ + ( −
[Equation]
γ 2 · · ·
[Equation]
γ n
[Equation]
−
[Equation]
n
[Equation]
γ . Thus, the temporal distance between s to g (= n ) is derived from γ n − 1 =
(1 − γ ) Q lo ( s, π ( s, g ) | g ) as n = log γ (1 + (1 − γ ) Q lo ( s,
[Equation]
( s,
[Equation]
| g
[Equation]
Definition B.1.
W G ( s t , sg t ) =
( wp t, 0 , wp t, 1 , ..., wp t,k ) is a sequence of waypoint obtained by the graph search algorithm and w ( W
G , τ ) =
wp t,i ∈ W G ( s t , sg t ) is the waypoint that is given to low-level policy at τ .
Given the transition distribution of the environment
T ( s τ +1 | s τ , a τ ) , the transition data ( s t , g t , sg t , r ( s t + c h , g t ) , s t + c h ) from the high-level policy’s replay buffer has been obtained as
[Equation]
(4) where π lo β and G β are the previous low-level policy and graph respectively.
Also, by using a different graph G and an optimal policy π lo ∗ , we get a new transition data ( s t , g t , sg t , r ( s ′ t
+ c h , g t ) , s ′ t
+ c h ) , where s
[Equation]
t + c h = t + c h − 1 (cid:89) τ
= t
T ( s τ +1 | s τ , a τ [Equation] For given s t and s t + c h , we define the off-policy error rate, which is the normalized distance error with respect to the total traversal distance according to the change of π lo β and G β to π lo ∗ and G , as
[Equation]
Lemma B.2.
Suppose that Dist ( · → · ) in Eq. (1) is Lipschitz continuous.
Then, there exists a constant L > 0 such that ∀ x and y , max( Dist ( x → y ) , Dist ( y → x ))
≤ L || x − y || , where || · || is the Euclidean norm, since
Dist ( x → x ) = 0 .
Then, any ϵ/L − resolution graph w.r.t the Euclidean norm, whose existence is trivial, is an ϵ − resolution graph w.r.t Dist ( · → · ) .
Proof
of Theorem 4.
Proof.
Let C s → g be one of the shortest paths from s to g and T be the distance of C s → g .
Also let p 1 ∈ C s → g be a point that Dist ( ψ ( s )
→ p 1 )
=
c l − ϵ .
Then, ∃ wp 1 ∈ V s.t. max( Dist ( p 1 → wp 1 ) , Dist ( wp 1 → p 1 )) <
ϵ , because G is an ϵ − resolution graph.
Since Dist ( · → · ) is a temporal distance, it satisfies the triangular inequality and then, Dist ( ψ ( s ) → wp 1 ) ≤ Dist ( ψ ( s ) → p 1 ) + Dist ( p 1 → wp 1 ) <
( c l − ϵ )
+ ϵ = c l and Dist ( wp 1 → g ) ≤ Dist ( wp 1 → p 1 ) + Dist ( p 1 → g ) <
ϵ + ( T − c l + ϵ )
=
T − c l + 2 ϵ .
Repeating the above procedure, let p i +
∈
C
wp i
→ g be a point that Dist ( wp i → p i +1 )
=
c l − ϵ .
Then, ∃ wp i +1 ∈ V s.t. max( Dist ( p i +
→ wp i +1 ) , Dist ( wp i +
→ p i +1 )) <
ϵ .
Then, Dist ( wp i → wp i +1 )
<
c l and Dist ( wp i +1 → g ) <
T − ( i + 1) c
l +
( i + 1) ϵ .
Consequently, the agent after T time-step will be closer than the ⌊ T/c l ⌋ th waypoint from g .
The remaining distance is less than
[Equation]
[Equation]
Thus, if an agent follows the sequence of waypoints { s, wp 1 , wp 2 , ..., g } , which is generated from a graph search algorithm over G and π lo ∗ , the error rate over this path satisfies
[Equation]
Thus the off-policy error rate ρ is equal or less than 2 ϵ/c l during T .
Since all path from s to g takes at least T time-steps, this upper-bound of error rate is also satisfied in all path from s to g .
C Additional Results
PointMaze
AntMazeSmall AntMazeBottleneck
Reacher3D
AntMazeComplex
Timesteps(k)
Timesteps(k)
T e s t s u cc e ss r a t e
AntMaze
DHRL(OURS)
T e s t s u cc e ss r a t e
Timesteps(k)
Timesteps(k)
Timesteps(k) Timesteps(k) PointMaze
AntMazeSmall
AntMazeBottleneck
Reacher3D
AntMazeComplex
Timesteps(k)
Timesteps(k)
T e s t s u cc e ss r a t e
AntMaze
T e s t s u cc e ss r a t e
Timesteps(k)
Timesteps(k)
Timesteps(k) Timesteps(k)
HIRO(Dense)
HIRO(Sparse)
SAC
UR3Obstacle UR3Obstacle
Timesteps(k) Timesteps(k)
[Table]
As shown in the table above, the wider the initial distribution, the easier it is for the agent to explore the map.
In other words, the ‘fixed initial state distribution’ condition we experimented with in this paper is a more difficult condition than the ‘uniform initial state distribution’ that previous graph-guided RL algorithms utilize.
Of course, ‘fixed initial state distribution’ requires less prior information about the state space.
We further experimented with ours (DHRL) under various types of reset conditions as shown in Table 2.
As expected, our algorithm shows faster exploration at the uniform reset point.
[Table]
N E C
K 0 .
M 0 .% 
 M 0 .%
C O M
P LE
X 2 .
M 0 .% 
 M 0 .%
R E A
C H E
RD 0 .
M 49 .% - 66 .% - 26 .% - 78 .% 95 .% 0 .
M 47 .% - 67 .% - 44 .% - 47 .% 90 .%
U R
O B S T
A C
LE 0 .% 2 .% 1 .% 0 .% 0 .% 11 .% 0 .% 69 .%
[Table]
[Table] [Table]