H IERARCHICAL R EINFORCEMENT L EARNING WITH L IMITED P OLICIES AND H INDSIGHT
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce a new hierarchical reinforcement learning framework that can accelerate learning in tasks involving long time horizons and sparse rewards.
Our approach improves sample efﬁciency by enabling agents to learn a hierarchy of short policies that operate at different time scales.
The policy hierarchies can support an arbitrary number of levels, and all policies within the hierarchy are trained in parallel and end-to-end.
Our framework is the ﬁrst hierarchical reinforcement learning approach that can learn hierarchies with more than two levels of policies in continuous tasks.
We demonstrate experimentally in both grid world and simulated robotics domains that our approach can significantly boost sample efﬁciency.
A video illustrating our results is available at https://www.youtube.com/watch
v
= i04QF7Yi50Y
I NTRODUCTION
Reinforcement learning (RL) algorithms struggle in tasks that involve long time horizons and sparse rewards.
A major reason for this outcome is that credit assignment, or the process of estimating the long-term value of executing an action in a certain state, is slower.
For tasks that require a long sequence of decisions, it takes longer to backup the higher value estimates of the actions that produce the sparse reward to the lower value estimates of the preceding actions that produce no immediate reward.
The ability to learn a hierarchy of short policies that operate at different time scales can help alleviate this problem.
This idea is illustrated in Figure 1 for a four-rooms grid world task.
An agent that could only learn at the time scale of atomic actions would need to learn a policy of around 20 actions to solve this task.
However, an agent that learns at two time scales could learn this task using policies consisting of at most ﬁve actions.
Speciﬁcally, the high-level agent could learn to break down the task into the sequence of four subgoal states (i.e., a → b → c → d ).
The lowlevel agent could then learn the at most 5 atomic actions required to achieve each of these subgoals.
Although the ability to learn at different time scales has created more tasks to learn, each of these tasks requires a shorter sequence of actions.
If the agent could learn these subtasks in parallel, the agent could potentially learn the full task more quickly.
We introduce a framework that enables agents to learn a hierarchy of short policies operating at different time scales in parallel and end-to-end.
The framework contains two components: (i) a particular policy hierarchy to be learned and (ii) a method to train the policy hierarchy.
The policy hierarchy to be learned consists of a stack of nested and goal-parameterized policies.
Each policy receives as input the current state and goal state and outputs a subgoal state for the next level to achieve, except for the lowest level in the hierarchy, which outputs an atomic action.
Another critical feature of the policy hierarchy is a hyperparameter that enforces a limit on the length of the sequence of actions that each policy learns.
This policy limit hyperparameter not only ensures that the agent learns short policies, but enables the agent to learn the set of short policies in parallel and end-to-end using Hindsight Experience Replay ( Andrychowicz et al.
There are other automated hierarchical RL approaches that can work in tasks with continuous state and action spaces, such as Skill-Chaining ( Konidaris & Barto (2009)), the Option-Critic Architecture ( Bacon et al.
and FeUdal Networks (FUN) ( Vezhnevets et al.
The key advantage that our framework has over these other methods is that our approach can use policy hierarchies with an arbitrary number of levels, whereas the other methods are limited to just two levels.
The additional levels of hierarchy are important because they enable agents to divide the original
→ →
No Hierarchy 2-Level Hierarchy ↑ ↑ ↑ →→ ↓ ↓ ↑↑ →→→ ↓↓↓ ↓↓→→ a b c dd ↑↑↑↑↑→→→→→↓↓↓↓↓↓↓→→
Task Breakdown with 2 -Level Hierarchy a
High-Level b c d → → →
Low-Level ↑ ↑ → → ↓ ↓ ↑↑ →→
↓↓↓ ↓↓→ a b c dd → a b c task into even smaller sequences of actions, which can help agents learn the task more quickly.
To the best of our knowledge, our framework is the ﬁrst to enable agents to learn a policy hierarchy with more than two levels in continuous tasks.
We evaluated our approach on both grid world tasks and more complex simulated robotics environments.
For each task, we evaluated agents with 1, 2, and 3 levels of hierarchy.
In all tasks, agents using 2 and 3 levels of hierarchy signiﬁcantly outperformed agents that learned a ﬂat policy.
Further, in many of the challenging tasks, agents using 3 levels of hierarchy signiﬁcantly outperformed agents using 2 levels of hierarchy.
B ACKGROUND
A Markov Decision Process (MDP) is a tuple M =
( S , A , T, R )
( Sutton & Barto (1998)).
S denotes a set of states.
A represents a discrete set of actions.
T : S ×
A × S denotes the transition probability function in which T ( s, a, s (cid:48) ) is the probability of transitioning to state s (cid:48) when action a is taken from state s .
R : S ×
A → R is the expected reward from executing action a in state s .
The solution to an MDP is a control policy π : S
→ A that maximizes the expected sum of future discounted rewards when acting under the policy.
The Universal Value Function Approximator (UVFA)
( Schaul et al. (2015)) extends the MDP framework for the episodic case by introducing a set of goals G and a goal-dependent reward function
R g : S ×
A → R , in which g ∈ G .
A goal is typically a state or a set of states.
In a standard MDP, the reward function is the same on each episode.
In the UVFA scenario, each episode uses a potentially different reward function.
Speciﬁcally, at the beginning of each episode, we initialize both the state of the agent and the goal state.
The selected goal state g remains constant for the duration of the episode and deﬁnes the reward function r g in use for that episode.
In this framework, a policy maps from both state and goal to an action, π : S × G → A .
As a consequence, the Q-function is also now a function of the goal, Q ( s, a, g ) .
UVFAs are helpful because they enable agents to generalize their policy to new goals based on experience achieving similar goals.
Hindsight experience replay (HER) ( Andrychowicz et al.
(2017)) is a mechanism for improving sample efﬁciency when learning in a UVFA framework with sparse rewards.
During training, experience transitions are created, storing state s t , action a t , next state s t +1 , and reward r t +1 in a replay buffer.
However, the UVFA also requires knowledge of the relevant goal state in order to train Q ( s, a, g ) .
The most immediate solution is to augment s t , a t , s t +1 , r t +1 with the goal state in effect at the time the experience was encountered.
HER supplements this information by storing the same experience multiple times and replacing the original goal with a goal state the agent came across during an episode.
This way, a single experience can help the agent learn policies for multiple goals, not just the particular goal in use when the experience was initially stored.
A PPROACH
We introduce a new framework that enables agents to learn a hierarchy of limited policies that operate at different time scales in parallel and in an automated manner.
The framework includes two main components: (i) a policy hierarchy with some certain architectural and operating properties, and (ii) a method for training the policies within the hierarchy.
3. P OLICY H IERARCHY - A RCHITECTURE
To understand the architecture of the policies within the hierarchy, we ﬁrst formally deﬁne the hierarchy of Universal MDPs (UMDP) from which the policies emerge.
3.1. U NIVERSAL MDP S
The UVFA framework ( Schaul et al.
(2015)) can be viewed as an extension of the MDP framework.
We will use the following deﬁnition:
Deﬁnition 3.
A Universal MDP (UMDP) is a tuple U =
( S , A , T, R, G ) where S , A , and T are deﬁned exactly as they are in the MDP case; G is a goal set; and R : S × G → R is the reward function where R ( s, g ) denotes the reward obtained after arriving in state s for a goal g .
At the beginning of each episode in a UMDP a ﬁxed goal g ∈ G is assigned that deﬁnes the reward function for the duration of that episode.
The aim of this paper is to develop an efﬁcient method of solving an arbitrary UMDP U original =
( S , A , T, R, G ) in the standard reinforcement learning scenario where it is assumed that the algorithm can execute actions and observe the results.
We will assume that S , A , and G are known while
T and R are unknown.
3.1. C ONSTRUCTION OF THE H IERARCHY
Given an original UMDP U original
=
( S , A , T, R, G ) to be solved, our approach is to learn a hierarchical policy.
We generate this policy by deﬁning and solving a hierarchy of k UMDPs,
U 0 , . .
, U k − 1 .
The hyperparameter k is provided by the user.
The hierarchy is constructed as follows:
U
This is the lowest level of the hierarchy.
It has the same state set and action set as the original
UMDP: S
= S and A
= A .
At this level, every state is potentially a goal.
Therefore, the goal set G 0 =
S is equal to the state set.
We will use a shortest path reward function for all levels of the hierarchy, in which a reward of 0 is granted if next state, s (cid:48) , maps to the goal: R 0 ( s (cid:48) , g )
=
if s (cid:48) ∈ g .
Otherwise, a reward of -1 is given to the agent.
However, the framework is ﬂexible to any type of reward function.
U i , 0 < i < k − 1 : This is an intermediate level of the hierarchy.
It also has the same state set as the original UMDP: S i = S .
It has an action set equal to the state set: A i = S .
As on the ﬁrst level, its goal set is equal to the state set, G i = S , and its reward function is deﬁned similarly: R i ( s (cid:48) , g )
=if s (cid:48)
∈ g
and
R i ( s (cid:48) , g ) = − 1 otherwise.
U k − 1 : This is the top level of the hierarchy.
It has the same state set, goal set, reward function as the original UMDP:
S k − 1 =
S , G k − 1 = G , and R k − 1 = R .
Its action set is equal to the state space
A k − 1 = S .
To summarize, the goal of our approach is to learn a hierarchy of universal, or goal-parameterized policies.
Each policy will thus try to learn the sequence of actions that bring the agent from the current state to the goal state.
The key differences among the policies lie in the time scale of the actions, which is described in the next section, and the action space for each policy.
At all levels except for the bottom most level, policies will output subgoal states for the next layer to achieve.
The lowest level will output atomic actions like a regular non-hierarchical policy.
Figure 2 (Left) shows an example policy architecture of a 2-layered agent that operates in a continuous domain.
[Figure]
Figure 2 (Right) shows the policy, or actor networks, together with the critic networks used to train each policy.
3. P OLICY H IERARCHY - A CTION T IMING
The combination of the nested structure of the policy hierarchy and the policy limit hyperparameter,
T ∈ N , helps the agent learn short policies that operate at different time scales by affecting the timing of actions.
The nested structure of the policies causes action selection for each policy to work in a top-down manner.
Let π
i : S i × G i
→ A i denote the current policy at level i .
At the beginning of an episode, a start state s ∈ S and a goal g ∈ G are initialized and a top level action a k − 1 ∈
A k − 1 is selected by evaluating π k − 1 for s and g .
This action, a k − 1 , instantiates a goal g k − 2 ∈ G k − 2 at level k − 2 and enables the agent to select an action at level k − 2 by evaluating π k − 2 for s and g k − 2 .
This process repeats for each level all the way to the bottom hierarchy layer where π 0 is evaluated for s and goal g
= a 1 .
The actual actions that are executed by each policy work bottom up.
At the lowest level, once an action a 0 is selected for execution, the policy at that level π 0 continues until a goal state is reached (i.e., until current state s ∈ g 0 ) or until T a 0 actions have been executed.
At this point, control returns to the next level up (level 1 in this case).
If π 0 was able to achieve the goal a 1 = g 0 from π 1 , then we deﬁne π 1 as executing action a 1 .
However, if the base policy π 0 was not able to achieve goal g 0 in T steps, we will deﬁne π 1 as executing the subgoal state action, s 0 , in which s 0 is the state that was reached after T attempts by π 0 .
As the next section will show, replacing the original subgoal action with the subgoal state reached in hindsight if the original subgoal was not achieved will enable the use of Hindsight Experience Replay, which will help all policies in the hierarchy learn simultaneously.
After the π 1 action has been recorded, a new level 1 action a 1 is selected by π 1 and passed down to π 0 .
Execution continues in this way at level 1 until a goal state is reached (i.e. until current state s = g 1 ) or until the maximum number of level 1 actions, T , have been executed.
This process continues up until the top level.
The key idea here is that the policy hierarchy described above has the same qualities as the hierarchical policy we seek. The goal is to learn a hierarchy of short policies that act over different time scales in order to accelerate credit assignment. In the policy hierarchy we have introduced, the hyperparameter T ensures that each policy only needs to learn short sequences of actions. Further, the combination of the nested structure and T enable each policy to act over different time scales.
Each π 0 action will have the timescale of an atomic action.
Each π 1 action will have the time scale of at most T atomic actions.
Each π 2 action will have the time scale of at most T level 1 actions or T 2 level 0 actions and so on.
3. L EARNING THE P OLICY H IERARCHY
Agents learn this hierarchy of limited policies in parallel and end-to-end using two different types of transitions: (i) hindsight transitions and (ii) subgoal testing transitions.
Agents are able to learn goal-parameterized policies that operate at different time scales primarily as a result of Hindsight Experience Replay ( Andrychowicz et al.
In our approach, HER is implemented at each level of the hierarchy in nearly the same way as it was originally deﬁned.
Copies are made of the transitions that were produced by the original sequence of actions.
The original goal state in each transition is replaced by a state that was actually reached during that action sequence.
The original reward is also replaced with the appropriate value given the updated goal state.
The key difference lies in the action component used for all subgoal layer (i.e., levels i > 0 ) hindsight transitions.
As described in the previous section, if level i − 1 is able to achieve the subgoal state
a i proposed by level i , then the action contained in the hindsight transition for level i is also a i .
However, if a i is not achieved within T level i − 1 actions, then in the hindisght transition, the action component a
i is replaced with the subgoal state s that was actually achieved by level i − 1 after T steps.
These hindsight transitons are critical for learning in an end-to-end manner because they provide the higher level policies with examples of subgoal state actions that both have the appropriate time scale and are helpful in achieving more distant goal states.
The hindsight transitions also help all policies learn in parallel because a policy does not need to wait for the lower level policies to learn how to reach a subgoal state before it can evaluate the effectiveness of that subgoal state.
The example provided in the next section should provide more clarity.
Subgoal testing transitions play an important role in ensuring that subgoal policies do not propose subgoals that are too ambitious (i.e., subgoal states that cannot be achieved by the succeeding layer in T attempts).
A key issue caused by the nested structure of the hierarchy and the policy limit parameter is that the agent can only move so far in T attempts.
As a result, hindsight transitions will only expose a subgoal layer to relatively nearby subgoal state actions.
Yet, the action space for a subgoal layer is the full state space, and the agent needs to be able to assign a likely low value to these too distant subgoal state actions.
Otherwise, problems can arise, particularly in continuous domains when function approximation is used.
It is possible that the critic function approximator will assign a relatively high value to these distant subgoal actions because it has received no transitions indicating otherwise.
This may cause the actor function to output an unrealistic subgoal, which is more problematic for a hierarchical agent using our framework than for a ﬂat agent because the next lower level of the hierarchical agent is only trained to reach relatively close goals.
In order to overcome this issue, we perform a process we refer to as subgoal testing, which works as follows.
After policy π i proposes a subgoal state a i , a certain percentage of the time we will turn off all exploration noise that is added to the actions from the policies below level i .
Then if it turns out that policy π i − 1 is unable to achieve a i
=
g
i − 1 in T attempts, layer i will be penalized for proposing that subgoal via a transition that contains a low reward.
The framework is ﬂexible to the value of the penalty reward.
However, the penalty should be lower than the reward for proposing a subgoal action that is achieved but that does not achieve the higher level goal in order to incent the agent to not propose too ambitious subgoals in the future.
For instance, in our implementation, penalty transitions included a reward of − T , which is lower than the -1 reward for proposing a subgoal that is reached but does not achieve the goal provided by the layer above.
Also, it is important to note that at the beginning of training this will cause the agent to penalize useful subgoal states that can be reached with T actions.
In the short term, we have seen that the UVFA will average the subgoal penalty with the higher rewards granted by the hindsight transitions for these useful subgoal states.
In the longer term as the lower level policies learn how to achieve reachable subgoal states, the penalty transitions will be diluted away.
Each level of the hierarchy will use the transitions described above and the generalization capability of the UVFA to learn from experience.
For the lowest level layer, the hindsight transitions will show which sequences of atomic actions are and are not capable of achieving which subgoal states.
For all subgoal layers, the hindsight and subgoal testing transitions will show subgoal actions that (i) belong to the appropriate time scale and are useful towards achieving more distant goal states but also (ii) achievable within T steps by the level below in the hierarchy. 3. H IERARCHICAL A CTOR -C RITIC A LGORITHM
Algorithm 1 shows the procedure used to train the policy hierarchy in continuous domains, which we refer to as Hierarchical Actor − Critic ( HAC ) . For simplicity, the pseudocode assumes an agent that uses 2 levels of hierarchy. However, the algorithm can support a hierarchy with an arbitrary number of levels by simply adding more inner for loops. Further, the algorithm can be easily adapted to the discrete domain setting by replacing the off-policy actor-critic algorithm with
Q-learning, for instance.
Input:
• Key agent parameters: number of levels in hierarchy k , policy limit T , subgoal testing frequency and penalty.
• Task design: S init ← Initial state space, G k − 1 ← task goal space
• k trained actor and critic functions π 0 , ..., π k − 1 , Q 0 , ..., Q k −
for M episodes do (cid:46) Train for M episodes s ← S init , g ← G k −(cid:46) Sample initial state and task goal train − level ( k − 1 , s , g ) (cid:46) Begin training Update all actor and critic networks end for function
TRAIN - LEVEL ( i, s, g ) s i ← s , g i ← g (cid:46) Set current state and goal for level i for T attempts or until g n , i ≤ n < k achieved do a i ← π i ( s i , g [-
-]i )
+ noise (if not testing) (cid:46) Sample (noisy) action from policy if i > 0 then Determine whether to test subgoal a i s (cid:48) i ← train − level ( i − 1 , s i , a i ) (cid:46) Train level i − 1 using subgoal a i else
Execute atomic action a 0 and observe next state s (cid:48)end if (cid:46)
Create replay transitions if i > 0 and a i missed then if a i was tested then (cid:46) Penalize subgoal a
i Replay Buffer i ← [ s = s i , a = a i , r = Penalty, s (cid:48)
=
s (cid:48) i , g = g i ] end if a i ← s (cid:48)
[-Temporarily store copy of transition t 0 for Level 
-]i (cid:46) Replace original action with action executed in hindsight end if (cid:46) Evaluate executed action on current goal and hindsight goals
Replay Buffer i ← [ s = s i , a = a i , r = {− 1 , 0 [-← Perform Level 
-]} , s (cid:48)
=
s (cid:48) i , g = g i ] HER Storage i ← [ s = s i , a = a i , r = TBD, s (cid:48)
=
s (cid:48) i , g = TBD ]
s i ← s (cid:48)
i end for
Replay Buffer i ← Perform HER using HER Storage i transitions return s (cid:48)
i (cid:46)
Output current state end
function 3. E XAMPLE
This section walks through a simple toy example showing how hindsight and subgoal transitions are generated and how they help agents learn multiple limited policies in parallel.
The agent in this task gssss 3 gggs is a robot and its goal is to reach the yellow ﬂag.
The agent has two levels in its hierarchy and a policy limit parameter T = 5 .
Thus, the agent’s goal is to learn both a low and high-level policy that focus on action sequences that contain no more than 5 actions.
Figure 3 shows an example episode trajectory.
Please note that all subscript numbers in the ﬁgure refer to the iteration number of the high-level policy.
The episode begins with the robot in state swhen the high-level policy π 1 proposes its ﬁrst subgoal state g 0 .
The robot decides that during this iteration, subgoal g 0 will not be tested.
As a result, the low-level policy, π 0 , is free to add exploration noise to the atomic actions it proposes.
Level 0 then has at most T = 5 atomic actions (i.e., joint torques in this case) to achieve g 0 .
After each action, a transition will be created showing how the action from π 0 performed given the goal state it was provided.
For instance, the ﬁrst transition in this example may look like [ s = s 0 , a = Joint Torques, r = − 1 , s (cid:48)
= first tick mark, g = g 0 ] .
In addition, a separate transition will be created and temporarily stored in order to be used for level 0’s Hindsight Experience Replay.
This transition will be of the form [ s = s 0 , a = Joint
Torques, r = TBD, s (cid:48)
= first tick mark, g = TBD ] .
Per the example in Figure 3, the robot fails to achieve state g 0 after 5 attempts and lands in state s 1 instead.
After this sequence ofatomic actions, HER is performed for layer 0.
The state s 1 can replace the TBD goal component in the extra hindsight transitions that were created and the reward would then be replaced with its appropriate value.
The completion of level 0’s for loop represents a single action by level 1’s policy π 1 .
Because its original subgoal g 0 was not achieved, π 1 will then record the hindsight state s 1 as the actual action that was taken.
The transition [ s = s 0 , a = s 1 , r = − 1 , s (cid:48)
=
s 1 , g = Y ellow Flag ] will be added to level 1’s replay buffer.
The additional transition [ s = s 0 , a = s 1 , r = TBD, s (cid:48)
=
s 1 , g = TBD ] will be stored until HER is performed for level 1.
Skipping ahead to when the robot is in state s 2 , the high-level policy proposes subgoal g 2 and then decides to test the subgoal.
The low-level policy then has T = 5 attempts without exploration noise to try to achieve g 2 .
In this scenario, the robot fails to achieve g 2 .
Indeed, state g 2 is too far away from s 2 to ever be achieved in 5 attempts.
Because the subgoal was tested and the robot failed to achieve it, a penalty transition is created and passed to the replay buffer for level 1.
Assuming the penalty is − T = − 5 , the transition would be of the form [ s = s 2 , a = g 2 , r
=
− 5 , s (cid:48)
=
s 3 , g = Y ellow
Flag ] .
The robot will also create the regular and hindsight transitions showing that state s 3 is a possible subgoal state from s 2 .
The regular transition is again of the form [ s = s 2 , a = s 3 , r = − 1 , s (cid:48)
=
s 3 , g = Y ellow Flag ] .
The episode then ﬁnishes with the agent achieving its goal.
After the episode, HER is performed for level 1 and then all actor-critic networks are updated using an off-policy RL algorithm, such as DDPG ( Lillicrap et al.
From this single episode, both levels have learned important information.
Using hindsight, levelhas learned the joint torques that can achieve a variety of states.
Similarly, level 1 has learned a sequence of subgoal states ( s 1 → s 2 → s 3 → s 4 ) that ﬁts its time scale and can lead the robot to a certain area in the goal space.
The sequences of actions that each level is provided is short and thus can potentially be learned more quickly than a ﬂat policy trying to learn a long sequence of atomic actions.
Moreover, through subgoal testing, level 1 has also learned about subgoals that may be too ambitious.
Using its UVFA, each level should be able to use this new information to better generalize to different goal states.
E XPERIMENTS 4. E NVIRONMENTS
We evaluated our framework in a variety of discrete and continuous tasks.
The discrete tasks consisted of grid world environments.
The continuous tasks consisted of simulated robotics environments developed in Mujoco (Todorov et al.
Figure 4 shows sample episode sequences from a four rooms grid world environment (Top) and the inverted pendulum environment (Bottom).
A video showing our experiments can is available at https://www.youtube.com/watch?v= i04QF7Yi50Y .
4. S TATES , A CTIONS , AND R EWARDS
For the grid world tasks, the state space was an integer value representing which square the blue square agent was in.
The atomic actions available to the agent were a 0 = {↑ , ↓ , ← , →} .
Both the subgoal and end goal spaces were the same as the state space.
The rewards at all levels of the hierarchy were either { -1,0 } depending on whether a policy’s action achieved its goal.
For the grid world tasks, instead of testing subgoals, we simply used pessimistic Q-value initializations so subgoal state actions that are too far will always have poor Q-values.
For the simulated robotics tasks, the state space was the concatenation of all joint angles and joint velocities.
The atomic actions available to the agent were vectors of joint torques.
The subgoal action space was equal to the state space.
The end goal space was either equal to the state space or a lower dimensional projection of the state space.
The rewards granted at all levels were either {− T ,-1,0 } .
4. R ESULTS
For each task we compared the performance of agents using 1 and 2 subgoal layers with the ﬂat agent.
The ﬂat agents used Q-learning with HER in the discrete tasks and DDPG with HER in the continuous tasks.
[Figure]
Number Training Episodes
A v g .
S u cc e ss
R a t e
UR 5 Reacher
Number Training Episodes
Number Training Episodes Number Training Episodes
Test Period Test Period
A v g .
S u cc e ss
R a t e
A v g .
S u cc e ss
R a t e
A v g .
S u cc e ss
R a t e
A v g .
S u cc e ss
R a t e
A v g .
S u cc e ss
R a t e
Four Rooms
Inverted Pendulum
Cartpole
Pick-and-Place [Figure] UR5 Reacher [Figure]
In all tasks, our approach signiﬁcantly outperformed the ﬂat agent.
Figure 6 shows the performance graphs for each agent in each task.
Each chart plots the average episode success rate for a given level of training for the 2 subgoal layer agent (Red), 1 subgoal layer agent (Blue), and the ﬂat agent (Green).
For the discrete tasks, inverted pendulum, and UR Reacher task, the level of training is expressed in terms of the actual number of training episodes that have taken place.
For the remaining tasks, the level of training is expressed in terms of test periods, in which each test period is separated by around 250 training episodes.
For the discrete tasks, the charts averages data from 50 trial runs.
The continuous tasks use data from 5-10 trial runs.
Each chart also plots the areas that are 1 standard deviation from the mean success rate.
Our empirical results also support our claim that additional layers of hierarchy can improve sample efﬁciency because they can shorten the action sequence length that each policy needs to learn.
In all of the discrete tasks and in the more challenging continuous tasks, such as UR5 Reacher and
Cartpole, the agent using 2 subgoal layers outperformed the agent using 1 subgoal layer.
R ELATED W ORK
There have been a wide range of hierarchical reinforcement learning approaches that have been implemented.
Most of these approaches either only work in discrete domains, require low-level controllers, or require a model of the environment [ Suttonet al.
Dietterich (1998) , McGovern & Barto (2001) , Kulkarniet al. (2016) ,
Menacheet al. (2002) , Simseket al.
There are a few other automated hierarchical RL techniques that can work in continuous domains.
Konidaris & Barto (2009) proposed Skill-Chaining, a method that iteratively chains options from the end goal state to the start state.
A key advantage that our framework has relative to Skill-Chaining is that our approach learns the subgoal options needed to reach a more distant goal state in parallel rather than incrementally, which can lead to improved sample efﬁciency. For instance, in the toy robot example provided above, agents using Skill-Chaining may use the experience to only learn an option to move from state s 3 to s 4 . On the other hand, our approach can immediately begin to learn all of the {+low-level subgoal policies (i.e., s 
 → s 1 , s 1 → s 2 , s 2 → s 3 , s 3 → s 4 )+} in parallel.
The Option-Critic architecture ( Bacon et al.
and FeUdal Networks (FUN) ( Vezhnevets et al.
(2017)) do learn the policies within the hierarchy in parallel.
However, these methods are limited to hierarchies with only two levels, whereas our framework can support an arbitrary number of levels.
This enables agents using our approach to divide the original task into a parallel set of even shorter action sequences.
The approach taken by Feudal Reinforcement Learning ( Dayan & Hinton (1993)) is probably the hierarchical RL algorithm most similar to ours.
The method learns a top-down hierarchical policy, in which a ”manager” proposes subgoals for a ”worker” to achieve.
Like our approach, Feudal RL thus divides the original RL problem into a parallel set of simpler RL problems.
One key difference is how each method enforces policy length limits.
Our framework uses an extra hyperparameter that limits the number of actions a policy can take to achieve a goal.
Feudal RL chooses to break down the state space into increasingly ﬁne regions and have each level in the hierarchy become an expert at that granularity of the state space.
It is unclear how this strategy can be scaled to continuous state spaces that have several dimensions.
Feudal RL also does not take advantage of learning from hindsight to boost sample efﬁciency.
There are also non-hierarchical methods that can accelerate backwards credit assignment, such as n-step TD or eligility traces ( Sutton & Barto (1998)).
The advantage that our algorithm has over these techniques is that they do not shorten the length of the policies that need to be learned.
Further, our method does not exclude the use of these value estimation techniques.
We only presented the 1-step TD version of our method, but it can be combined with n-step TD or eligibility traces.
C ONCLUSION
We propose a new framework for effectively scaling reinforcement learning to long time horizon tasks involving sparse rewards.
Instead of learning a single lengthy policy, our approach learns a hierarchy of limited policies that operate at different time scales.
The policies are trained in parallel and end-to-end using Hindsight Experience Replay.
Our results show that our approach can signiﬁcantly improve sample efﬁciency in both discrete and continuous domains.