DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Hierarchical Reinforcement Learning (HRL) has made notable progress in complexcontrol tasks by leveraging temporal abstraction.
However, previous HRL algorithms often suffer from serious data inefficiency as environments get large.
Theextended components, i.e. , goal space and length of episodes, impose a burden oneither one or both high-level and low-level policies since both levels share the totalhorizon of the episode.
In this paper, we present a method of D ecoupling H orizons
Using a Graph in Hierarchical R einforcement L earning (DHRL) which can alleviate this problem by decoupling the horizons of high-level and low-level policiesand bridging the gap between the length of both horizons using a graph.
DHRLprovides a freely stretchable high-level action interval, which facilitates longertemporal abstraction and faster training in complex tasks.
Our method outperformsstate-of-the-art HRL algorithms in typical HRL environments.
Moreover, DHRLachieves long and complex locomotion and manipulation tasks.
Introduction
[Figure]
Reinforcement Learning (RL) has been successfully applied to a range of robot systems, suchas locomotion tasks [23, 7], learning to control aerial robots [9, 12] and robot manipulation[13, 21].
Goal-conditioned RL, which augmentsstate with the goal to train an agent for variousgoals [22, 18], further raised the applicabilityof RL in robot systems allowing the agent toachieve diverse tasks.
Hierarchical Reinforcement Learning (HRL),which trains multiple levels of goal-conditioned
RL, has improved the performance of RL incomplex and sparse tasks with long horizons using temporally extended policy [24, 25, 15].
Onthe back of these strengths, HRL was adopted to solve various complex robotics tasks [19, 10, 17].
However, HRL often has difficulty in complex or large environments because of training inefficiency.
Previous studies speculated that the cause of this problem is the large goal space, and restrictedthe high-level action space to alleviate this phenomenon [27, 11].
Nevertheless, this approachperforms well only in limited length and complexity and still suffers from the same trouble in largerenvironments.
[Figure]
We show that this practical limitation of HRLcan be mitigated by breaking down the coupledhorizons of HRL.
In previous HRL frameworks,the horizons of the low level and high level arerelated to each other structurally because theyshare the total length of the episode.
This relation causes a tradeoff between the trainingburden of both levels; if the intervals betweenhigh-level actions increase (x-axis in Figure 2),the low-level policy has to cover a wider range,and in the opposite case, the high-level policytakes charge of the extended burden alone inlarge environments (y-axis in Figure 2).
This isthe reason why the previous HRL algorithmscannot cope with extended components of largeenvironments (see Table 1 for the performanceof the previous HRL method at various intervals).
We break down the coupled horizons of HRL:
To break the relation between the horizons of bothlevels, we adopt a graph structure.
In our method, the high-level policy can use a longer temporalabstraction while the lower one only takes charge of smaller coverage by decomposing the subgoalinto several waypoints with a graph.
In this way, the HRL algorithm obtains the capability to stretchthe interval of high-level action freely and achieves complex and large tasks, thanks to the enlargedstrength of the HRL.
In summary, our main contributions are:
• We show that the previous HRL structures are not scalable in large environments, and that
this limitation can be mitigated by removing coupled traits of high level and low level.
• To break down the coupled traits of HRL, we propose DHRL which decouples the horizonsof high-level and low-level policies and bridges the gap using a graph.• Our algorithm outperforms state-of-the-art algorithms in typical HRL environments andachieves complex and long tasks.
Preliminaries
We consider a finite-horizon Universal Markov Decision Process (UMDP) which can be represented as a tuple ( S , G , A , T , R , γ ) where S , A and G are state space, goal space and action spacerespectively.
The environment is defined by the transition distribution T ( s t +1 | s t , a t ) and rewardfunction R : S ×
A × G → R .
Also, total return of a trajectory τ =
( s 0 , a 0 , ..., s H , a H ) is
R ( τ, g ) =
(cid:80) H − 1 t =
γ t r ( s t +1 , g ) where r ( s t +1 , g ) (or r ( s t , a t , g ) ) is a goal conditioned reward function and γ is a discount factor.
Subgoal sg , waypoint wp and goal g are defined in goal space G andwe consider G that is a subspace of S with a mapping ψ : S → G .
HRL framework typically has high-level policy
π hi and low-level policy π lo , each maintaining aseparated replay buffer B hi and B lo .
Our method also follows the general HRL framework andemploys both buffers that store high-level ( s t , g t , sg t , r t , s t
+ c h ) ∈
B hi and low-level transition data( s t , wp t , a t , r t , s t +1 )
∈ B lo where c h is the interval between high-level action.
One of the key problems in learning HRL is that the low-level policy π lo is non-stationary andthus old data from past policies may contain different next-states s t + c even though the highlevel policy provides the same subgoal in the identical state.
To bypass this off-policy discrepancy, HRL models use off-policy correction which relabels the subgoal of high-level policy to be
[Equation]
)) 2 , where π lo β is the previous policy used to collect data [16].
It can be interpreted as correcting the old subgoal to the most plausiblesubgoal that will result in the same transition with the current agent.
In practical use of this off-policycorrection, we choose k = 1 because of restrictions on access to the environment.
Graph-guided RL methods, which combine the strength of RL and planning by decomposing along-horizon task into multi-step sub-problems, estimate the temporal distance between states and goals to construct a graph G =
( V , E ) on goal space G without additional prior knowledge aboutenvironments.
Previous studies proposed various methods to recover distance from Q-value [3, 8, 26].
If the agent gets -1 reward at every step except when it is in a goal area where the agent gets 0 reward,then, Q lo ( s, a | g ) can reveal the temporal distance between s to g as
Dist ( s → g ) =
log γ (1 + (1 − γ ) Q lo ( s, π ( s, g ) | g ))
However, it is known that recovering temporal distance correctly from the vanilla Q-network inthis setting is challenging.
For that reason, the previous methods use an additional Value functionapproximator [26] or distributional Q-networks [3].
Related work
[Figure]
Graph-guided RL.
Graphs have recentlybeen used as a non-parametric model in reinforcement learning (RL) to combine the advantages of RL and planning [3, 8, 26, 2, 5].
By decomposing a long-horizon task into multi-stepplanning problems, these studies have shownbetter performance and data efficiency.
Searchon the Replay Buffer (SORB)
[3] constructsa directed graph based on the states randomlyextracted from a replay buffer and Q-functionbased edge cost estimation.
The follow-up studies further improved the performance of earliergraph-guided RLs [3, 8] by combining additional methods such as graph search on latentspace[26] or model predictive control [2].
However, previous papers sidestepped the exploration problems in complex tasks through ‘uniforminitial state distribution’ [3, 8, 26, 5], or work only on the dense reward settings [2].
We emphasizethat the ‘uniform initial state distribution’ accesses privileged information about the environmentduring training by generating the agent uniformly within the feasible area of the map.
This greatlyreduces the scope of application of these algorithms.
Unlike prior methods, ours can train from sparsereward settings and a fixed initial state distribution without knowledge of the agent’s surroundings,which makes it practical for physical settings.
For detailed examples and comparison of variousdistributions, see Table 2 and Figure 11 in Appendix C.
Constrained-subgoal HRL.
To mitigate the training inefficiency issue of HRL, several researchersproposed methods that restrict the action of high-level agent to be placed in adjacent areas.
Hierarchical Reinforcement Learning with k-step Adjacency Constraint (HRAC) [27] limits the subgoal to be in the adjacency space of the current state.
Hierarchical reinforcement learning Guidedby Landmarks (HIGL) [11] improved the data efficiency by adding novelty-based landmarks toadjacency-constrained HRL.
However, these improvements only work on the limited length andcomplexity of the environment.
The previous work closest to our method is HIGL.
However, there are three key differences betweenthe previous work and our approach.
First, our model explicitly includes the whole graph structurewhile HIGL needs to train high-level action to imitate the graph by using an additional loss termcorresponding to the Euclidean distance from landmark or pseudo-landmark.
Second, we use a graphto decouple the horizons of high level and low level, unlike the previous method which uses the graphonly for guidance.
Most importantly, ours can achieve goals in long and complex environments.
Tothe best of our knowledge, there is no prior HRL research to train a model that has decoupled thehorizons of the two levels.
Methods
We introduce Decoupling Horizons Using a Graph in Hierarchical Reinforcement Learning (DHRL),which can separate the time horizons of high-level and low-level policies and bridge the gap betweenboth horizons using a graph.
Our framework consists of high-level policy π hi ( sg | s, g ) , low-level 𝜋 ℎ𝑖 (𝑠𝑔 𝑡 |𝑠 𝑡 ,𝑔 𝑡 ) 𝜋 𝑙𝑜 (𝑎 𝑡:𝑡+𝑐 𝑙 |𝑠 𝑡 ,𝑤𝑝 𝑡,1 )
𝑠𝑔 𝑡 𝑠𝑔 𝑡+𝑐 ℎ 𝑔 𝑡 𝑔 𝑡+𝑐 ℎ … 𝜋 𝑙𝑜 (𝑎 𝑡+𝑐 𝑙 :𝑡+2𝑐 𝑙 |𝑠 𝑡+𝑐 𝑙 ,𝑤𝑝 𝑡,2 ) 𝜋 ℎ𝑖 (𝑠𝑔 𝑡+𝑐 ℎ |𝑠 𝑡+𝑐 ℎ ,𝑔 𝑡 ) 𝜋 𝑙𝑜 (𝑎 𝑡+𝑐 ℎ +𝑐 𝑙 :𝑡+𝑐 ℎ +2𝑐 𝑙 |𝑠 𝑡+𝑐 ℎ +𝑐 𝑙 ,𝑤𝑝 𝑡+𝑐 ℎ ,2 ) 𝜋 𝑙𝑜 (𝑎 𝑡+𝑐 ℎ :𝑡+𝑐 ℎ +𝑐 𝑙 |𝑠 𝑡+𝑐 ℎ ,𝑤𝑝 𝑡+𝑐 ℎ ,1 ) 𝑤𝑝 𝑡, 𝑤𝑝 𝑡, 𝑤𝑝 𝑡+𝑐 ℎ , 𝑤𝑝 𝑡+𝑐 ℎ ,2 𝑠 𝑡 𝑠 𝑡+𝑐 ℎ high low 𝑠𝑔 𝑤𝑝 [ Planning Over the Graph] ℬ 𝑙𝑜 𝑉 [Graph Construction] 𝑄 𝑔𝑟𝑎𝑝ℎ 𝑙𝑜 𝐸 𝐺(𝑉,𝐸) 𝐺(𝑉,𝐸) … … … policy π lo ( a | s, wp ) , and a graph G .
Given a goal g t in the environment, the high-level policy outputsa subgoal sg t (see Figure 4).
Then, the shortest path from the current state s t to sg t is found on thegraph.
To do so, s t and sg t are added to the existing graph structure, then a sequence of waypoints( s t , wp t, 1 , wp t, 2 , ..., sg t ) is returned using a graph search algorithm.
Finally, the low-level policytries to achieve wp t,i during c l,i steps.
The key point of our method is that the low-level horizon h low =
c l is unrelated to the high-levelhorizon h high .
Since the high level generates one subgoal every c h steps, the H-step task is a H/c hstep task for a high-level agent ( h high
= H/c h ) where c h is the interval between high-level action( c h > c l ).
In other words, unlike the previous HRL methods which have the relationship of
[Equation] [Equation]
our algorithm does not have such relations, removing an obstacle toward a scalable-RL algorithm.
Since the low-level horizon c l is determined by the edge cost between waypoints, we can also expressthe c l between the i - 1 th waypoint and i th waypoint as c l,i , but we omit the letter i in the laterstatements that do not need to specify the waypoint.
In section 4.1, we explain how to construct a graph over states and find a path on the graph level.
Insection 4.2, we present a low-level policy which can recover temporal distance between states withoutoverestimation.
In section 4.3, We introduce a strategy to train our method through graph-agnosticoff-policy learning.
Finally, in section 4.4, we propose additional techniques for better data efficiencyin large environments.
4.1 Graph level: planning over the graph
This section details the graph search part in DHRL planning.
We emphasize that every distance in the
DHRL model is based on temporal distance Dist ( · → · ) , obtained through (1).
Thus, our algorithmrequires no further information about the environment ( e.g. L2 norm distance between states) thangeneral HRL settings.
To find the shortest path, we adopt Dijkstra’s Algorithm, as in the previous study [3].
The differencesfrom the previous methods [3, 26] are the frequency of the graph search, the target node of searching,and whether the secondary path is considered.
Let ψ : S → G be the projection of states on thegoal space.
In the graph initialization phase, we samples n nodes (also called landmarks) using FPSalgorithm (Algorithm 2 in Appendix A) from ψ ( s ) where s ∈ S is state sampled from B lo .
Then, weconnect a directed edge s 1 → s 2 if the temporal distance from s 1 to s 2 is less than the cutoff-threshold.
In the planning phase, the graph G ( V , E ) gets the subgoal sg t from the high-level policy and addsthe projection of current state ψ ( s t ) and sg t to V , so that the number of nodes in G becomes n + 2 .
Then, we connect the edges with costs less than the cutoff-threshold between the newly added nodesand existing nodes.
Next, we find the sequence of waypoints W : ( wp t, 0 = ψ ( s t ) , wp t, 1 , wp t, 2 ,..., wp t,k − 1 , wp t,k = sg t ) that connects from ψ ( s t ) to sg t using a graph search algorithm.
At thistime, if there is no path from ψ ( s t ) to sg t , we adopt a secondary path from ψ ( s t ) to sg † t , wheresg † t is the closest node to sg t among the nodes connected from ψ ( s t ) .
After finding the waypointsequence, the graph level provides wp t, 1 to the low-level policy and instructs it to reach wp t, 1 .
Ifit has been Dist ( wp t,i − 1 → wp t,i ) steps since the agent started tracking wp t,i or it achieves wp t,i , then we update the current tracking waypoint to the next waypoint
wp t,i +1 .
In order to reflect thenewly discovered state, we update graph nodes at every N episodes.
However, it is challenging to train Q-network which recovers reliable temporal distance.
We explainthe reason and a solution in the next section.
4.2 Low level: Separate Q-networks for Graph and Critic (SQGC)
Unlike previous graph-guided RL methods that leverage a graph only after the actor and criticnetworks have been learned, we construct and utilize a graph during training.
To do so, the low-levelpolicy Q lo is evaluated during training to assign edge costs.
However, when π lo is not yet competentin achieving some goals or encounters difficult goals, the accumulation of failure experiences inthe replay buffer causes underestimation of Q-values.
This leads to an overestimation of temporaldistance reconstructed from (1) and spoils the graph near the overestimated region by making thenode-selection algorithm (Algorithm 2 in Appendix A) select more and more graph nodes around theoverestimated area.
Therefore, a temporal-distance reconstruction method is needed even when thepolicy is not sufficiently trained.
For this reason, we propose Separate Q-networks for Graph and Critic (SQGC) to prevent temporaldistance overestimation.
SQGC is composed of two identical Q-networks using different proportionsof hindsight goal relabeling (HER) [1].
The SQGC includes Q locritic and Q lograph where Q locritic is fortraining π lo , just like a typical application, and Q lograph is for recovering temporal distance betweennodes.
We substitute wp t with ˆ wp t :=
ag t + t ftr in the sequential transition of a single episode( s t , wp t , a t , r ( s t +1 , wp t ) , s t +1 ) t
=1: H − 1 where ag means the achieved goal and t ftr is a randominteger drawn from the uniform distribution between 0 and H − t .
To train Q lograph , we relabel 100%of
wp t in ( s t , wp t , a t , r ( s t +1 , wp t ) , s t +1 ) t
=1: H − 1 as ˆ wp t , while we replace only 80% of wp t for
Q locritic .
Our method can prevent overestimation in the distance recovery by relabelling the goals in alltransitions for training Q lograph because the experiences of failure are replaced with the successfultrajectory (see section 5.3 for the ablation study).
Also, by maintaining the original Q criticlo to trainπ lo , there is no degradation in the performance of the agent who might otherwise not be able to getnegative feedback from failure since the failure will be relabeled as a desired goal.
However, it is still challenging to train HRL including a graph level using off-policy RL algorithms.
We describe our approach to train DHRL using an off-policy algorithm in the next section.
4.3 High level: hindsight transitions for graph-agnostic off-policy learning
Thanks to decoupling the time horizons of both levels in HRL, the high-level policy in our methodcan look further without any additional burden on the low-level policy.
In other words, We can stretchc h , which represents how long each high-level action operates for.
However, because of the extendedinterval and non-stationarity of the high-level MDP, it is challenging to train DHRL with an off-policyalgorithm, which is important for data efficiency in that the off-policy algorithm can use the previousdata from the replay buffer.
This non-stationarity of the high-level MDP is caused by the presence of a graph and low-level policy.
Since the graph is gradually updated, it is challenging to train a model using an off-policy methodfrom the data given by the previous graph.
Furthermore, as c h gets longer, predicting the similarity ofthe trajectory of c h steps with only the first step of π lo ( a i | s i , g i ) gets more difficult when we replacesubgoal using HIRO-style off-policy correction [16] which is a popular method adopted by SOTA
HRL algorithms [27, 11].
To facilitate the off-policy learning for longer c h and the changing graph in DHRL, we adopta well-known hindsight action relabelling method proposed in Hierarchical Actor-Critic (HAC)[14] in the high-level replay buffer data ( s t , g t , sg t , r t , s t
+ c h ) ∈
B hi .
Alongside the original transitions data, we copy the transition data and replace the subgoal ( sg t = π high β ( sg t | s t , g t ) ) withthe achieved goal ag t + c h after c h steps.
Thus, we use both transitions ( s t , g t , sg t , r t , s t + c h ) and 222 ( s t , g t , ag t + c h , r t , s t + c h ) to train the high-level policy.
Our method utilizes the optimality of thegraph and low-level policy to include a graph in the learning process, while HAC-style hindsightaction relabelling method assumes the optimal low-level policy π lo ∗ only.
By replacing the previous subgoal with the achieved goal, we can assume that this transition wasobtained from a stationary graph and π lo ∗ with an error below a bound which is set to be a functionof the density of the graph.
We provide a theoretical analysis of the possibility of replacing the oldoff-policy graph with a virtual stationary graph in this section and
Appendix B.
Definition 4.1.
Given a compact state space S , G ( V , E ) is an ϵ − resolution graph if ∀ s ∈ S , ∃ v ∈ Vs.t.
max( Dist ( ψ ( s ) → v ) , Dist ( v → ψ ( s ))) <
ϵ , where V ⊂ ψ ( S ) and E = { ( v i , v j ) | v i , v j ∈
V , Dist ( v i → v j ) <
c l } .
Theorem 4.2.
Let G be an arbitrary ϵ − resolution graph ( ϵ <
c l / 2 ).
Also, let π lo β
and G β be thelow-level policy and graph at the time of collecting the data.
Off-policy error rate ρ ( G ) is thenormalized distance error with respect to the total traversal distance according to the change of π lo βand G β to π lo ∗ and G .
If there is a path from s to g , the upper bound of off-policy error rate ρ ( G )using a path obtained from graph search over G is 2 ϵ/c l .
Theorem 4.2 shows that if we replace the previous subgoal with the achieved goal, and assume thatthese transitions are obtained from a stationary ϵ − resolution graph G , then the off-policy error ρ isless than 2 ϵ/c l .
Thus, we can train DHRL through a graph-agnostic off-policy RL algorithm usingthe substituted transition.
4.4 Optional techniques: gradual penalty and frontier-based goal-shifting
[Figure]
- tioned in the rim of the graph.
In this section, we propose two additional techniques, gradual penalty and frontier-based goalshifting (FGS).
These optional techniques canboost the performance of DHRL on some longtasks as shown in Figure 9(a) and (c).
We notethat these are not essential to train DHRL andours outperforms previous HRL frameworkswithout these techniques (see section 5.3 forthe ablation study).
Gradual penalty.
Similar to the subgoal testing in the previous method [14], we proposegradual penalty (algorithm 4 in Appendix A),which can impose the penalty more delicatelywhen the action space of the high-level agentgets large.
We evaluate sg t of the original transition data ( s t , g t , sg t , r t , s t + c h ) by categorizing thefollowing three cases; (a) close to the graph and low level actually achieved the subgoal ∈ L 1 , (b)close to the graph but the low-level policy could not achieve the subgoal ∈ L 2 and (c) far from thegraph ∈ L 3 .
In this way, we can impose the more detailed penalties to respond to the expansion ofthe high-level action space as the high-level interval c h stretches.
Frontier-based goal-shifting.
This optional technique has been devised to accelerate learning ina complex environment (algorithm 5 in Appendix A).
FGS moves the final goal g to the frontierarea when g comes into the place where the graph is already laid out during training.
To checkwhether the goal is in the graph area, we examine whether min v ∈ V ( Dist ( v → g )) is smaller thancut-off threshold.
Alternative goals are the addition of random noise to nodes sampled from v ∈ Vproportional to − Q ( s 0 , π ( s 0 , v ) | v ) .
FGS is similar to the previous goal-directed exploration research[20] in that both use weighted samples, but ours does not maintain a generative model and samplesthe goals from graph level in DHRL.
Note that we do not use FGS when we compare DHRL withthe previous state-of-the-art, since this FGS is beyond the main contribution, which is about HRLstructure.
Experiments
5.1 Environment description
We evaluate DHRL on robot environments based on the MuJoCo simulator including some sparseand long-horizon tasks.
Locomotion environments have fixed initial state distributions and random
PointMaze
AntMazeSmall
AntMazeBottleneck
Reacher3D
AntMazeComplex
Timesteps(k)
Timesteps(k)
T e s t s u cc e ss r a t e
AntMaze (OURS)
T e s t s u cc e ss r a t e
Timesteps(k)
UR3Obstacle
Timesteps(k) goal points during training.
In test episodes, they give the agent one of the most challenging goals( i.e. , the end of the maze).
We also evaluate our algorithm in a robot arm environment in which theagent aims to make the end-effector touch the goal.
• PointMaze / AntMazeSmall : The point / ant achieves the goal if it comes within 2.5 distance
(success threshold) from the target point in 12 × 12 maze.
• AntMaze : 24 × 24 maze with success threshold
• AntMazeBottleneck : Bottleneck exists at the middle of the maze.
The ant can barely passthrough bottleneck.
• AntMazeComplex : 56 × 56 maze with success threshold
• Reacher3D : 7-dof robot arm aims to reach a goal.
• UR3Obstacle : 6-dof robot arm aims to reach a goal in an environment with several board-
shaped obstacles.
We adopt TD3 algorithm [4] for high-level and low-level networks and use Dijkstra algorithm tofind the shortest path in the graph level.
To train the separate Q-networks for graph and critic in
DHRL, We use 80% and 100% of HER transition for Q criticlo and Q graphlo respectively.
Our algorithmautomatically determines c l based on the temporal distance to the nearest waypoint.
Only a fewhyperparameters have been changed across various locomotion experiments: the number of nodes,penalty, and c h .
The results of DHRL in this paper are obtained using only sparse reward.
5.2 Experiment result
Baselines.
We compare our method with state-of-the-art algorithms with and without a graphrespectively; HIGL [11] and HRAC [27].
For more comparison with shallow RL (SAC) [6] andvanilla HRL (HIRO) [16], see Table 3 in Appendix C.
[Figure]
Comparison to state-of-the-art algorithms.
Results are shown in Figure 6.
Thanks to decouplingthe horizons of the levels, DHRL could stretch thehigh-level interval and shows high data efficiency andsuccess rate in locomotion and goal-reaching tasksin simulation environments.
Moreover, DHRL is theonly algorithm that can succeed in complex environments (AntMazeComplex and AntMazeBottleneck).
When measuring the performance of the baselines, we selected c h with the best performance amongthe values c h =10, 20, ..., and 50 (80 in AntMazeComplex).
As shown in Figure 6, previous HRL methods cannot solve long-horizon tasks.
This is likely due to thecoupling ( c h =
c l ) of the high-level horizon ( h high = H/c h ) and the low-level horizon ( h low =
c l )resulting in an increased burden on either the high-level or the low-level policy.
For long-horizontasks (large H ), if c h (= c l ) is fixed, then h high increases and the high-level performance plunges
[Figure]
(Table 1 in Appendix C).
On the other hand, if c h (= c l ) is increased, the low level has to manage a wider area and the performance plunges as shown in Figure 7 and Table 1.
5.3 Ablation study
Decoupling horizons.
In this section, we examine how decoupling horizons affects the performanceof long-horizon HRL and evaluate whether DHRL can stretch the interval of high-level policy.
Asshown in Figure 9(b), we tested various values of spacing of the high-level action, c h .
The resultshows that even in long intervals of the high-level policy, DHRL shows consistent or improvedperformance (Figure 9(b)) without major degradation unlike previous HRL methods (Figure 7).
Inparticular, DHRL agent in long-horizon environments such as AntMazeComplex shows that extendedhigh-level interval is crucial for exploration and performance (see the rightmost figure in Figure9(b)).
Considering that the baselines could not solve long-horizon tasks even with various valuesof c h ∈ [10 , 80] , and the performance degrades with increasing c h , we conclude that our method successfully decoupled the time steps in both levels and stretched the interval between high-levelactions.
This trait allows the high-level policy to look further and take advantage of the extendedtemporal abstraction.
Separate Q-network for Graph and Critic (SQGC).
In this section, we evaluate the effects of the
Separate Q-network for Graph and Critic (SQGC) in our method.
The red line in Figure 9(a), which isa variant of DHRL without a SQGC, clearly shows that the SQGC is critical to the performance.
Weempirically found that the overestimation of the temporal distance between states occurs especiallynear the obstacles ( e.g. corners at the maze), and it is difficult for an agent to pass by without aseparate Q network for graph construction.
This is consistent with our analysis that the experiencesof failure spoil the ability to recover the temporal distance from Q-network.
Ablate high-level policy.
Graph-guided RL, which maintains graph level and low-level policy, isalso a variant of HRL in that the graph level is a non-parametric version of the high-level policy (see
Figure 3).
From this point of view, the main difference between graph-guided RL and our method isthe existence of the high-level policy.
Then, why do we need additional high-level policy above thegraph level?
Most graph-guided RL algorithms use ‘uniform initial state distribution’ to train the agent in complexenvironments and such assumption could be expensive, especially in the physical world where it ischallenging to start from different positions each time.
Figure 9(d) shows the result of the graphguided RL algorithm, L3P
[26], with a uniform and fixed initial state distributions.
This indicatesthat without ‘uniform initial state distribution’, the performance and data efficiency of the previousgraph-guided RL method drops seriously even in the smallest environment we experimented with.
Incontrast, DHRL can explore without ‘uniform initial state distribution’ even in long environmentsthanks to the enlarged temporal abstraction and exploration performance.
Considering that the highlevel policy suggests a subgoal to graph level, we conclude that the high level facilitates betterexploration of the graph-guided RL.
Thus, DHRL that has exploration capability also can be seen asthe improved version of graph-guided RL.
Optional techniques.
The result in Figure 9(c) shows that the gradual penalty can improve dataefficiency in long tasks.
By imposing a penalty on subgoals that are far from the current graph, thegradual penalty encourages the high-level agent to output a subgoal near the graph.
Since the low-levelagent can hardly achieve subgoals far from the explored area, the gradual penalty accelerates thetraining by providing more valid goals.
FGS also shows better training efficiency in environments.
The results in Figure 9(a) show the wider improvements as the training progresses.
This means thatas the more area the agent has explored in the environment, it becomes more difficult to get goalsthat help exploration and the data efficiency decreases.
With FGS, DHRL learns efficiently usingrandomly generated goals by replacing easy goals with new goals located on the rim of the graph.
Conclusion
We proposed a Decoupling Horizons Using a Graph in Hierarchical Reinforcement Learning (DHRL),which is a data-efficient HRL algorithm leveraging a graph to expand the range of problems that
HRL can solve, by decoupling horizons and allowing both levels to operate at their suitable horizons.
Our experimental results show that our method successfully separates the horizons of the levelsand outperforms prior state-of-the-art methods.
We believe that our method can present a directiontowards scalable RL rather than constraining the high-level policy or stacking three or more levelsof policies like the previous methods in that ours has a lower dependency on the horizon of theenvironment.
In this paper, we adopted a vanilla graph construction algorithm and sparse rewardsettings.
We expect that if a novelty-based node selection is added to the graph construction, theperformance will be further improved.
Limitation.
While our algorithm shows successful results in complex and long-horizon tasks, itmight be difficult to construct a graph in some tasks that require complex interactions with theenvironment because of the higher-dimensional state space.
We expect that this limitation can besolved by extending our algorithms into latent state space, or by
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[No] This work inherits
the potential negative social impact of reinforcement learning in robotics.
We do not anticipate
that there will be any additional negative impact of this paper.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section B inAppendix(b)
Did you include complete proofs of all theoretical results?
[Yes] See Section B in Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[Yes] See supplemental material
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section ??
in Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[Yes] See Figure
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs,
internal cluster, or cloud provider)?
[Yes] See Section C in Appendix
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.
(b) Did you mention the license of the assets?
[Yes] See Section 5.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
informa-
tion or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N
[Table] [Table]
b) Did you describe any potential participant risks, with
[Table]
to Institutional
[Table]
(IRB)
approvals, if applicable?
[N/A
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on
[Table]
compensation?
[N/A]
A
Algorithms.
1: sample D lo i =
( s t , wp t , a t , r ( s t +1 , wp t ) , s t +1 )
i ∈ B lo 2: relabel wp t ← ˆ wp t =
ag t + t ftr to make ˆ D lo
i 3: update Q lo critic ,θ 1 and π lo ϕ 1 using D lo i ∪ ˆ D lo i 4: update Q lo graph ,θ 2 using ˆ D lo i 5:
if t mod d then 6: sample D hi i = ( s t , g t , sg t , r t , s t + c h )
i ∈ B hi 7:
relabel sg t ← ag t + c h to make ˆ D
hi i 8: for ( s t , g t , sg t , r t , s t + c h ) in D
hi i do 9: if r ( s t + c h , sg t )
=
then 10: r t ← r t ◁ subgoal ∈ L11: else if use GradualPenalty then 12: r t ← GradualPenalty (graph G , sg t , Q lograph ) 13: else 14: r t ← penalty p15: end if 16: end for 17: update Q hi θ and π
2 using D hi i ∪ ˆ D
hi i 18: end if
W wp waypoint waypoint
←
9: get low
10: act a τ the environment and get s
+ 11:
12: if agent achieve wp t idt or t tr
Dist ( wp t idp wp t idt ) then 13: idp
1 ; idt tr ←14: end
15: end
16: end while
1: Input: graph G ( V , E ) , subgoal sg t , Q lograph ,θ 2 , gradual penalty threshold ζ 1 , penalty p 1 , penalty p2: if min( Q lograph ,θ 2 ( v ∈ V , sg t )) <
ζ 1 then 3: r t ← penalty p◁ subgoal ∈ L4: else 5: r t ← penalty p◁ subgoal ∈ L6: end if
1: Input: s t , graph G ( V , E ) , goal g , Q lograph ,θ 2 , cut-off threshold ζ2: Dist ( s, g ) :
= log γ (1 + (1 − γ ) Q lograph ,θ 2 ( s, π ( s, g ) | g ))
3: if min v ∈ V ( Dist ( v → g )) <
ζ 2 then 4:
V candidate ← V + noise 5: g t ← random .
choice ( V candidate , weight = − Q lograph ,θ 2 ( s t , π ( s t , V candidate ) | V candidate )) 6: end if 7: return g t
1: Input: initial random steps τ randomwalk , initial steps without planning τ w / o graph , total training step τ total , Env, low-level agent Q lo critic ,θ 1 , Q lo graph ,θ 2 and π lo ϕ 1 , high-level agent Q hi θ 3 and π hi ϕ2: Dist ( s, g ) : = log γ (1 + (1 − γ ) Q lograph ,θ 2 ( s, π ( s, g ) | g )) 3: for τ = 1 to τ total do 4: if Env.done then 5:
Env.reset (episode step resets to 0) 6: if Use FGS then 7: g ← FGS( G , g, Q lo graph ,θ 2 ) 8: end if 9: end if 10: if τ < τ randomwalk then 11: a t ← random.uniform(high = action.high, low = action.low) ◁ random action 12: else if τ < τ w / o graph then 13: a t ← vanilla HRL ( sg t = π hi ϕ 2 ( s t , g ) and π lo ϕ 1 ( s t , sg t )) ◁ act without planning 14: else 15: if Graph G is not initialized then 16:
Create a graph G ( V , E ) using FPS algorithm ◁ initialize graph 17: end if 18: if episode step(the step of the environment) %
c l = 0 then 19: sg t ← π hi ϕ 2 ( s t , g ) ◁ get subgoal 20: { wp t, 1 , wp t, 2 , · · · wp t,k } ← Dijkstra ′ salgorithm ( s t , sg t )
◁ get waypoints 21: current waypoint index n =22: end if 23: if achieved wp t,n or tried more than Dist ( wp t,n − 1 , wp t,n ) to achieve wp t,n then 24: current waypoint index + =25: end if 26: a t ← π lo ϕ 1 ( s t , wp t,n +1 )
◁ get low-level action 27: end if 28:
Env.step( a t ) 29:
Train low-level agent Q lo critic ,θ 1 , Q lo graph ,θ 2 and π lo ϕ 1 , high-level agent Q hi θ 3 and π hi ϕ30: if τ % graph update freq = 0 then 31:
Update Graph G ( V , E ) using FPS algorithm 32: end if 33: end for
B Proofs of Theorems.
Derivation of equation 1.
If a given policy π lo requires n steps to get from current s to a goal g , theγ -discounted return is Q lo ( s, π ( s, g ) | g
[Equation]
= ( − 1) + ( − 1) γ + ( −
[Equation]
γ 2 · · ·
[Equation]
−
[Equation]
γ n
[Equation]
Thus, the temporal distance between s to g (= n ) is derived from γ n − 1 =
(1 − γ ) Q lo ( s, π ( s, g ) | g ) asn = log γ (1 + (
[Equation]
( s
[Equation]
Definition B.1.
W G ( s t , sg t ) =
( wp t, 0 , wp t, 1 , ..., wp t,k ) is a sequence of waypoint obtained by the graph 485 search algorithm and w ( W
G , τ ) =
wp t,i ∈ W G ( s t , sg t ) is the waypoint that is given to low-level policy at τ .
Given the transition distribution of the environment
T ( s τ +1 | s τ , a τ ) , the transition data 487 ( s t , g t , sg t , r ( s t + c h , g t ) , s t + c h ) from the high-level policy’s replay buffer has been obtained as
[Equation]
(4) where π lo β and G β are the previous low-level policy and graph respectively.
Also, by using a different graph Gand an optimal policy π lo ∗ , we get a new transition data ( s t , g t , sg t , r ( s ′ t
+ c h , g t ) , s ′ t
+ c h ) ,
wheres ′ t + c h = t + c h − 1 (cid:89)
[Equation]
For given s t and s t + c h , we define the off-policy error rate, which is the normalized distance error with respectto the total traversal distance according to the change of π lo β and G β to π lo ∗ and G , as
[Equation]
Lemma B.2.
Suppose that Dist ( · → · ) in eqn (1) is Lipschitz continuous.
Then, there exists a constant L > 0 493 such that ∀ x and y , max( Dist ( x → y ) , Dist ( y → x ))
≤ L || x − y || , where || · || is the Euclidean norm, since
Dist ( x → x ) = 0 .
Then, any ϵ/L − resolution graph w.r.t the Euclidean norm, whose existence is trivial, is an 495 ϵ − resolution graph w.r.t Dist ( · → · ) .
Proof of Theorem
Proof.
Let C s → g be one of the shortest paths from s to g and T be the distance of C s → g .
Also let p 1 ∈ C s → g be apoint that Dist ( ψ ( s ) →
p 1 )
=
c l − ϵ .
Then, ∃ wp 1 ∈ V s.t. max( Dist ( p 1 → wp 1 ) , Dist ( wp 1 → p 1 )) <
ϵ , 499 because G is an ϵ − resolution graph.
Since Dist ( · → · ) is a temporal distance, it satisfies the triangularinequality and then, Dist ( ψ ( s ) → wp 1 ) ≤ Dist ( ψ ( s ) → p 1 ) + Dist ( p 1 → wp 1 ) <
( c l − ϵ )
+ ϵ = c l and 501 Dist ( wp 1 → g )
≤ Dist ( wp 1 → p 1 ) + Dist ( p 1 → g ) <
ϵ + ( T − c l + ϵ )
=
T − c l + 2 ϵ .
Repeating the above procedure, let p i +
∈
C
wp i
→ g be a point that Dist ( wp i → p i +1 )
=
c l − ϵ .
Then, 503 ∃ wp i +1 ∈ V s.t. max( Dist ( p i +
→ wp i +1 ) , Dist ( wp i +
→ p i +1 )) <
ϵ .
Then, Dist ( wp i → wp i +1 )
<
c l 504 and Dist ( wp i +1 → g ) <
T − ( i + 1) c
l +
( i + 1) ϵ .
Consequently, the agent after T time-step will be closerthan the ⌊ T/c
l ⌋ th waypoint from g .
The remaining distance is less than
[Equation]
Thus, if an agent follows the sequence of waypoints { s, wp 1 , wp 2 , ..., g } , which is generated from a graphsearch algorithm over G and π lo ∗ , the error rate over this path satisfies
[Equation]
Thus the off-policy error rate ρ is equal or less than 2 ϵ/c l during T .
Since all path from s to g takes at least T 509 time-steps, this upper-bound of error rate is also satisfied in all path from s to g .
C Additional Results
[Table]
AntMazeSmall AntMazeBottleneck
Reacher3D
AntMazeComplex
Timesteps(k)
Timesteps(k)
T e s t s u cc e ss r a t e
AntMaze
DHRL(OURS)
T e s t s u cc e ss r a t e
Timesteps(k)
Timesteps(k)
Timesteps(k) Timesteps(k) PointMaze
AntMazeSmall
AntMazeBottleneck
Reacher3D
AntMazeComplex
Timesteps(k)
Timesteps(k)
T e s t s u cc e ss r a t e
AntMaze
T e s t s u cc e ss r a t e
Timesteps(k)
Timesteps(k)
Timesteps(k) Timesteps(k)
HIRO(Dense)
HIRO(Sparse)
SAC
[Table]
k)
Timesteps(k)
S
[Table]
R ATE
E ASY (U NIFORM )
M EDIUM (
[Table]
POINT )
H ARD (
[Table]
A NT M AZE 0.3M 80.4% 28.5% 12.2%
A NT M
[Table] [Table]
N E C
K 0 .
M 0 .% 
 M 0 .%
C O M
P LE
X 2 .
M 0 .% 
 M 0 .%
R E A
C H E
RD 0 .
M 49 .% - 66 .% - 26 .% - 78 .% 95 .% 0 .
M 47 .% - 67 .% - 44 .% - 47 .% 90 .%
U R
O B S T
A C
LE 0 .% 2 .% 1 .% 0 .% 0 .% 11 .% 0 .% 69 .%
[Table]
[Table] [Table]