O PTION D ISCOVERY USING D EEP S KILL C HAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Autonomously discovering temporally extended actions, or skills , is a longstanding goal of hierarchical reinforcement learning.
We propose a new algorithm that combines skill chaining with deep neural networks to autonomously discover skills in high-dimensional, continuous domains.
The resulting algorithm, deep skill chaining , constructs skills with the property that executing one enables the agent to execute another.
We demonstrate that deep skill chaining signiﬁcantly outperforms both non-hierarchical agents and other state-of-the-art skill discovery techniques in challenging continuous control tasks.
I NTRODUCTION
Hierarchical reinforcement learning (Barto & Mahadevan, 2003) is a promising approach for solving long-horizon sequential decision making problems.
Hierarchical methods lower the decision making burden on the agent through the use of problem speciﬁc action abstractions (Konidaris, 2019).
While the use of temporally extended actions, or options (Sutton et al., 1999), has been shown to accelerate learning (McGovern & Sutton, 1998), there remains the question of skill discovery: how can agents autonomously construct useful skills via interaction with the environment?
While a large body of work has sought to answer this question in small discrete domains, skill discovery in high-dimensional continuous spaces remains an open problem.
An early approach to skill discovery in continuous-state environments was skill chaining (Konidaris & Barto, 2009b), where an agent constructs a sequence of options that target a salient event in the
MDP (for example, the goal state).
The skills are constructed so that successful execution of each option in the chain allows the agent to execute another option, which brings it closer still to its eventual goal.
While skill chaining was capable of discovering skills in continuous state spaces, it could only be applied to relatively low-dimensional state-spaces with discrete actions.
We introduce a new algorithm that combines the core insights of skill chaining with recent advances in using non-linear function approximation in reinforcement learning.
The new algorithm, deep skill chaining , scales to high-dimensional problems with continuous state and action spaces.
Through a series of experiments on ﬁve challenging domains in the MuJoCo physics simulator (Todorov et al., 2012), we show that deep skill chaining can solve tasks that otherwise cannot be solved by nonhierarchical agents in a reasonable amount of time.
Furthermore, the new algorithm outperforms state-of-the-art deep skill discovery algorithms (Bacon et al., 2017; Levy et al., 2019) in these tasks.
B ACKGROUND AND R ELATED W ORK
Sequential decision making problems can be formalized as Markov Decision Processes (MDPs).
We consider goal-oriented episodic MDPs, where S denotes the state space, A is the action space, R is the reward function, T is the transition function, γ is the discount factor and g ∈ S is the terminating goal state (Sutton & Barto, 2018).
Unlike goal-conditioned algorithms (Sutton et al., 2011; Schaul et al., 2015), we do not require that g be known; instead we assume access to an indicator functiong : s ∈ S −→ { 0 , 1 } which the agent can query to determine if it has reached the MDP’s goal.
One way to learn a policy in an MDP is to ﬁrst learn an action-value function.
The action-value function Q π ( s t , a t ) is deﬁned as the expected sum of discounted future rewards if the agent takes action a t from s t and then follows policy π thereafter:
Q π ( s t , a t ) =
E π [ r t + γ max a t +1 Q π ( s t +1 , a t +1 )] .
Q-learning (Watkins & Dayan, 1992) is a commonly used off-policy algorithm that uses the actionvalue function for control through a greedy policy
π ( s t ) =
arg max a t Q ( s t , a t ) .
Inspired by recent success in scaling Q-learning to high-dimensional spaces (Mnih et al., 2015; Van Hasselt et al., 2016;
Lillicrap et al., 2015; Tesauro, 1994), we learn the action-value function Q πφ ( s t , a t ) using non-linear function approximators parameterized by φ , by minimizing the loss L ( φ ) =
E π
[( Q φ ( s t , a t ) − y t ) 2 ] where the Q-learning target y t is given by the following equation (Van Hasselt et al., 2016):
[Equation]
Deep Q-Learning (DQN) (Mnih et al., 2015) casts minimizing L ( φ ) as a standard regression problem by using target networks (parameterized by φ (cid:48) ) and experience replay (Lin, 1993).
2. T HE O PTIONS F RAMEWORK
The options framework (Sutton et al., 1999) models skills as options .
An option o consists of three components: (a) its initiation condition, I o ( s ) , which determines whether o can be executed in state s , (b) its termination condition, β o ( s ) , which determines whether option execution must terminate in state s and (c) its closed-loop control policy, π o ( s ) , which maps state s to a low level action a ∈ A .
Augmenting the set of available actions with options results in a Semi-Markov Decision Process (SMDP) (Sutton et al., 1999) where the next state depends on the current state, action and time.
2. S KILL D ISCOVERY A LGORITHMS
Skill discovery has been studied extensively in small discrete domains (McGovern & Sutton, 1998;
S¸ims¸ek & Barto, 2004; S¸ims¸ek et al., 2005; Bakker & Schmidhuber, 2004; Schmidhuber, 1991;
Pickett & Barto, 2002; Dietterich, 2000).
Recently however, there has been a signiﬁcant body of work aimed at discovering skills in continuous spaces.
Option-critic methods : Option-Critic (Bacon et al., 2017) uses an end-to-end gradient based algorithm to learn options in high-dimensional continuous spaces.
Option-Critic was a substantial step forward in skill discovery and led to a family of related methods (Klissarov et al., 2017; Tiwari &
Thomas, 2019; Riemer et al., 2018; Liu et al., 2017; Jain et al., 2018).
Proximal Policy Option Critic (PPOC) (Klissarov et al., 2017) extends Option-Critic to continuous action spaces and is the version of Option-Critic that we compare against in this paper.
Our method bypasses two fundamental shortcomings of the Option-Critic framework: (a) unlike Option-Critic, we explicitly learn initiation sets of options and thus do not assume that all options are executable from everywhere, and (b) we do not treat the number of skills required to solve a task as a ﬁxed and costly hyperparameter.
Instead, our algorithm ﬂexibly discovers as many skills as it needs to solve the given problem.
Feudal methods : An alternative to the options framework is Feudal Reinforcement Learning (Dayan & Hinton, 1993), which creates a hierarchy in which managers learn to assign subgoals to workers ; workers take a subgoal state as input and learn to reach it.
Feudal Networks (FuN) (Vezhnevets et al., 2017) used neural networks to scale the Feudal-RL framework to high-dimensional continuous spaces; it was extended and outperformed by HIRO (Nachum et al., 2018) in a series of control tasks in the MuJoCo simulator.
More recently, Hierarchical Actor-Critic (HAC) (Levy et al., 2019) outperformed HIRO in a similar suite of continuous control tasks.
HAC learns a deep multilevel feudal-style hierarchy with the help of Universal Value Function Approximators (Schaul et al., 2015) and Hindsight Experience Replay (Andrychowicz et al., 2017).
We compare our method to HAC as a representative feudal method, given its superior performance in high-dimensional continuous control problems with sparse rewards.
Learning backward from the goal : The idea of sequencing locally applicable controllers is well established in robotics and control theory in the form of pre-image backchaining (Kaelbling & LozanoP´erez, 2017) and LQR-Trees (Tedrake, 2009).
Such methods either require individually engineered control loops or a model of the system dynamics.
Our work ﬁts in the model-free RL setting and thus requires neither.
More recently, reverse curriculum learning (Florensa et al., 2017) also learns backward from the goal.
However, they deﬁne a curriculum of start states to learn a single policy, rather than learning skills.
Relay Networks (Kumar et al., 2018) segment the value function backward from the goal using a thresholding scheme, which makes their method reliant on the accurate estimation of the value function.
By contrast, our algorithm is agnostic to errors in value estimation, which are unavoidable when using function approximation in high-dimensional spaces.
Planning with learned skills : Options have been shown to empirically speed up planning in several domains (Silver & Ciosek, 2012; Jinnai et al., 2019a; James et al., 2018; Francis & Ram, 1993;
Konidaris, 2016; Sharma et al., 2019).
However, Konidaris et al.
(2018) show that for resulting plans to be provably feasible, skills must be executable sequentially.
While they assume that such skills are given, we show that they can be autonomously discovered in high-dimensional spaces.
D EEP S KILL C HAINING
Deep skill chaining (DSC) is based on the intuition that it is easier to solve a long-horizon task from states in the local neighborhood of the goal.
This intuition informs the ﬁrst step of the algorithm: create an option that initiates near the goal and reliably takes the agent to the goal.
Once such an option is learned, we create another option whose goal is to take the agent to a state from which it can successfully execute the ﬁrst option.
Skills are chained backward in this fashion until the start state of the MDP lies inside the initiation set of some option.
The inductive bias of creating sequentially executable skills guarantees that as long as the agent successfully executes each skill in its chain, it will solve the original task.
More formally, skill chaining amounts to learning options such that the termination condition
β
o i ( s t ) of an option
o i is the initiation condition
I o i − 1 ( s t ) of the option that precedes it in its chain.
Our algorithm proceeds as follows: at time t , the policy over options
π O : s t ∈ S −→ o ∈
O determines which option to execute (Section 3.2).
Control is then handed over to the selected option o i ’s internal policy
π
o i : s ∈ S −→ a t ∈ R | A | .
π o i outputs joint torques until it either reaches its goal ( β o i :=
I o i − 1 ) or times out at its predetermined budget T (Section 3.1).
At this point, π O chooses another option to execute.
If at any point the agent reaches the goal state of the MDP or the initiation condition of a previously learned option, it creates a new option to target such a salient event.
The machinery for learning the initiation condition of this new option is described in Section 3.3.
We now detail the components of our architecture and how they are learned.
Readers may also refer to
Figures 4 & 7 and the pseudo-code in Appendix A.4 to gain greater intuition about our algorithm.
3. I NTRA -O PTION P OLICY
Each option o maintains its own policy π o : s −→ a t ∈ R | A | , which is parameterized by its own neural networks θ o .
To train π o ( s ; θ o ) , we must deﬁne o ’s internal reward function.
In sparse reward problems, o is given a subgoal reward when it triggers β o ; otherwise it is given a step penalty.
In the dense reward setting, we can compute the distance to the parent option’s initiation set classiﬁer and use that to deﬁne o ’s internal reward function.
We can now treat learning the intra-option policy ( π o ) as a standard RL problem and use an off-the-shelf algorithm to learn this policy.
Since in this work we solve tasks with continuous action spaces, we use Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) to learn option policies over real-valued actions.
3. P OLICY O VER O PTIONS
Initially, the policy over options ( π O ) only possesses one option that operates over a single time step ( T = 1) .
We call this option the global option ( o G ) since its initiation condition is true everywhere in the state space and its termination condition is true only at the goal state of the MDP (i.e, I o G ( s )
= 1 ∀ s and β o G = 1 g ) .
Using o G , π O can select primitive actions.
At ﬁrst the agent continually calls upon o G , which uses its internal option policy π o G to output exactly one primitive action.
Once o G triggers the MDP’s goal state N times, DSC creates its ﬁrst temporally extended option, the goal option ( o g ), whose termination condition is also set to be the goal state of the MDP, i.e, β
o g = 1 g .
As the agent discovers new skills, it adds them to its option repertoire and relies on π O to determine which option (including o G ) it must execute at each state.
Unlike o G , learned options will be temporally extended, i.e, they will operate over T > 1 time steps.
If in state s t the agent chooses to execute option
o i , then o i will execute its own closed-loop control policy (for τ steps) until its termination condition is met ( τ < T ) or it has timed out at τ = T time steps.
At this point, control is handed back to π O , which must now choose a new option at state s t + τ .
Option selection : To select an option in state s t , π O ﬁrst constructs a set of admissible options given by Equation 2.
π O then chooses the admissible option that maximizes its option-value function, as shown in Equation 3.
Since the agent must choose from a discrete set of options at any time, we learn its option-value function using Deep Q-learning (DQN) (Mnih et al., 2015).
[Equation]
Learning the option-value function : Given an SMDP transition ( s t , o t , r t : t + τ , s t + τ ) , we update the value of taking option o t in state s t according to SMDP Q-learning update (Bradtke & Duff, 1995).
Since the agent learns Q-values for different state-option pairs, it may choose to ignore learned options in favor of primitive actions in certain parts of the state-space (in the interest of maximizing its expected future sum of discounted rewards).
The Q-value target for learning the weights φ of the DQN is given by:
[Equation]
Adding new options to the policy over options : Equations 2, 3 and 4 show how we can learn the option-value function and use it for selecting options.
However, we must still incrementally add new skills to the network during the agent’s lifetime.
After the agent has learned a new option
o ’s initiation set classiﬁer
I o
(we will discuss how this happens in Section 3.3), it performs the following steps before it can add o to its option repertoire:
• To initialize o ’s internal policy π o , the parameters of its DDPG ( θ o ) are set to the parameters of the global agent’s DDPG ( θ o G ) .
Subsequently, their neural networks are trained independently.
This provides a good starting point for optimizing π o , while allowing it to learn sub-problem speciﬁc abstractions.
• To begin predicting Q-values for o , we add a new output node to ﬁnal layer of the DQN parameterizing π O .
• We must assign appropriate initial values to Q φ ( s, o ) .
We follow Konidaris & Barto (2009b) and collect all the transitions that triggered β o and use the max over these Q-values to optimistically initialize the new output node of our DQN.
This is done by setting the bias of this new node, which ensures that the Q-value predictions corresponding to the other options remain unchanged.
3. I NITIATION S ET C LASSIFIER
Central to the idea of learning skills is the ability to learn the set of states from which they can be executed.
First, we must learn the initiation set classiﬁer for o g , the option used to trigger the
MDP’s goal state.
While acting in the environment, the agent’s global DDPG will trigger the goal state N times (also referred to as the gestation period of the option by Konidaris & Barto (2009b) and Niekum & Barto (2011)).
We collect these N successful trajectories, segment the last K states from each trajectory and learn a one-class classiﬁer around the segmented states.
Once initialized, it may be necessary to reﬁne the option’s initiation set based on its policy.
We do so by executing the option and collecting data to train a two-class classiﬁer.
States from which option execution was successful are labeled as positive examples.
States from which option execution timed out are labeled as negative examples.
We continue this process of reﬁning the option’s initiation set classiﬁer for a ﬁxed number of episodes, which we call the initiation period of the option.
At the end of the initiation period, we ﬁx the option’s initiation set classiﬁer and add it to the list of salient events in the MDP.
We then construct a new option whose termination condition is the initiation classiﬁer of the option we just learned.
We continue adding to our chain of options in this fashion until a learned initiation set classiﬁer contains the start state of the MDP.
3. G ENERALIZING TO S KILL T REES
Our discussion so far has been focused on learning skill chains that extend from the goal to the start state of the MDP.
However, such a chain is not sufﬁcient if the agent has multiple start states or if we want the agent to learn multiple ways of solving the same problem.
To permit such behavior, our algorithm can be used to learn skills that organize more generally in the form of trees (Konidaris & Barto, 2009b; Konidaris et al., 2012).
This generalization requires some additional care while learning initiation set classiﬁers, the details of which can be found in Section A.1 of the Appendix.
To demonstrate our ability to construct such skill trees (and their usefulness), we consider a maze navigation task, E-Maze, with distinct start states in Section 4.
3. O PTIMALITY OF D ISCOVERED S OLUTIONS
Each option o ’s internal policy π o is is given a subgoal reward only when it triggers its termination condition
β o .
As a result, π o is trained to ﬁnd the optimal trajectory for entering its own goal region.
Naively executing learned skills would thus yield a recursively optimal solution to the MDP (Barto & Mahadevan, 2003).
However, since the policy over options π O does not see subgoal rewards and is trained using extrinsic rewards only, it can combine learned skills and primitive actions to discover a ﬂat optimal solution π ∗ to the MDP (Barto & Mahadevan, 2003).
Indeed, our algorithm allows π O to employ discovered skills to quickly and reliably ﬁnd feasible paths to the goal, which over time can be reﬁned into optimal solutions.
It is worth noting that our ability to recover π ∗ in the limit is in contrast to feudal methods such as HAC (Levy et al., 2019) in which higher levels of the hierarchy are rewarded for choosing feasible subgoals, not optimal ones.
To summarize, our algorithm proceeds as follows: (1) Collect trajectories that trigger new option o k ’s termination condition
β o k .
(2) Train o k ’s option policy π o k .
(3) Learn o k ’s initiation set classiﬁer
I o k .
(4) Add o k to the agent’s option repertoire.
(5) Create a new option o k +1 such that β
o k
+
=
I
o k .
(6) Train policy over options
π O .
Steps 1, 3, 4 and 5 continue until the MDP
’s start state is inside some option’s initiation set.
Continue steps 2 and 6 indeﬁnitely.
E XPERIMENTS
We test our algorithm in ﬁve tasks that exhibit a strong hierarchical structure: (1) Point-Maze (Duan et al., 2016), (2) Four Rooms with Lock and Key, (3)
Reacher (Brockman et al., 2016), (4) Point
E-Maze and (5) Ant-Maze (Duan et al., 2016; Brockman et al., 2016).
Since tasks 1, 3 and 5 appear frequently in the literature, details of their setup can be found in Appendix A.2.
Four Rooms with Lock and Key :
In this task, a point agent (Duan et al., 2016) is placed in the Four
Rooms environment (Sutton et al., 1999).
It must pick up the key (blue sphere in the top-right room in Figure 1(c), row 2) and then navigate to the lock (red sphere in the top-left room).
The agent’s state space consists of its position, orientation, linear velocity, rotational velocity and a has key indicator variable.
If it reaches the lock with the key in its possession, its episode terminates with a sparse reward of 0 ; otherwise it gets a step penalty of − 1 .
If we wish to autonomously discover the importance of the key, (i.e, without any corresponding extrinsic rewards) a distance-based dense reward such as that used in related work (Nachum et al., 2018) would be infeasible.
Point E-Maze : This task extends the benchmark U-shaped Point-Maze task (Duan et al., 2016) so that the agent has two possible start locations - on the top and bottom rungs of the E-shaped maze respectively.
We include this task to demonstrate our algorithm’s ability to construct skill trees.
4. C OMPARATIVE A NALYSES
We compared the performance of our algorithm to DDPG, Option-Critic and Hierarchical ActorCritic (HAC), in the conditions most similar to those in which they were originally evaluated.
For instance, in the Ant-Maze task we compare against Option-Critic under a dense-reward formulation of the problem while comparing to HAC under a sparse-reward version of the same task.
As a result, we show the learning curves comparing against them on different plots (columns (a) and (b) in Figure 1 respectively) to emphasize the difference between the algorithms, the settings in which they are applicable, and the way they are evaluated.
[Figure]
[Figure]
Comparison with DDPG and Option-Critic : Figure 1(a) shows the results of comparing our proposed algorithm (DSC) with a ﬂat RL agent (DDPG) and the version of Option-Critic designed for continuous action spaces (PPOC).
3 Deep skill chaining comfortably outperforms both baselines.
Both DSC and DDPG use the same exploration strategy in which a t = π θ ( s t ) + η t where η t ∼ N (0 , (cid:15) t ) .
Option-Critic, on the other hand, learns a stochastic policy π θ ( a t | s t ) and thus has baked-in exploration (Sutton & Barto, 2018, Ch. 13), precluding the need for additive noise during action selection.
We hypothesize that this difference in exploration strategies is the reason OptionCritic initially performs better than both DDPG and DSC in the Reacher and Point E-Maze tasks.
Comparison with Hierarchical Actor-Critic : We compare our algorithm to Hierarchical ActorCritic (HAC) (Levy et al., 2019), which has recently outperformed other hierarchical reinforcement learning methods (Nachum et al., 2018;
Vezhnevets et al., 2017) on a wide variety of tasks.
A noteworthy property of the HAC agent is that it may prematurely terminate its training episodes to prevent ﬂooding its replay buffer with uninformative transitions.
The length of each training episode in DSC however, is ﬁxed and determined by the test environment.
Unless the agent reaches the goal state, its episode lasts for the entirety of its episodic budget (e.g, this would be 1000 timesteps in the
Point-Maze environment).
Thus, to compare the two algorithms, we perform periodic test rollouts wherein all networks are frozen and both algorithms have the same time budget to solve the given task.
Furthermore, since both DSC and HAC learn deterministic policies, we set (cid:15) t = 0 during these test rollouts.
When comparing to HAC, we perform 1 test rollout after each training episode in all tasks except for Ant-Maze, where we average performance over 5 test rollouts every 10 episodes.
Figure 1(b) shows that DSC outperforms HAC in all environments except for Four Rooms with a Lock and Key, where their performance is similar, even though DSC does not use Hindsight
Experience Replay (Andrychowicz et al., 2017) to deal with the sparse reward nature of this task.
4. I NTERPRETING L EARNED S KILLS
Figure 2 visualizes the initiation set classiﬁers of options discovered by DSC in Four Rooms with a Lock and Key.
Despite not getting any extrinsic reward for picking up the key, DSC discovers the following skill chain: the options shown in Figure 2 columns (c) and (d) bring the agent to the room with the key.
The option shown in column (b) then picks up the key (top row) and then takes the agent to the room with the lock (bottom row).
Finally, the option in column (a) solves the overall problem by navigating to the lock with the key.
Similar visualizations of learned initiation set classiﬁers in the E-Maze task can be found in the Figure 6 in the Appendix.
[Figure]
Figure 3 shows that DSC is able to learn options that induce simple, efﬁcient policies along different segments of the state-space.
Furthermore, it illustrates that in some states, the policy over options prefers primitive actions (shown in black) over learned skills.
This suggests that DSC is robust to situations in which it constructs poor options or is unable to learn a good option policy in certain portions of the state-space.
In particular, Figure 3 (d) shows how DSC constructs a skill tree to solve a problem with two distinct start states.
It learns a common option near the goal (shown in blue), which then branches off into two different chains leading to its two different start states respectively.
D ISCUSSION AND C ONCLUSION
Deep skill chaining breaks complex long-horizon problems into a series of sub-problems and learns policies that solve those sub-problems.
By doing so, it provides a signiﬁcant performance boost when compared to a ﬂat learning agent in all of the tasks considered in Section 4.
We show superior performance when compared to Option-Critic, the leading framework for option discovery in continuous domains.
A signiﬁcant drawback of Option-Critic is that it assumes that all options are executable from everywhere in the state-space.
By contrast, deep skill chaining explicitly learns initiation set classiﬁers.
As a result, learned skills specialize in different regions of the statespace and do not have to bear the burden of learning representations for states that lie far outside of their initiation region.
Furthermore, each option in the Option-Critic architecture leverages the same state-abstraction to learn option-speciﬁc value functions and policies, while deep skill chaining permits each skill to construct its own skill-speciﬁc state-abstraction (Konidaris & Barto, 2009a).
An advantage of using Option-Critic over DSC is that it is not conﬁned to goal-oriented tasks and can work in tasks which require continually maximizing non-sparse rewards.
Section 4 also shows that deep skill chaining outperforms HAC in four out of ﬁve domains, while achieving comparable performance in one.
We note that even though HAC was designed to work in the multi-goal setting, we test it here in the more constrained single-goal setting.
Consequently, we argue that in problems which permit a stationary set of target events (like the ones considered here), deep skill chaining provides a favorable alternative to HAC.
Furthermore, HAC depends on Hindsight Experience Replay (HER) to train the different layers of their hierarchy.
Deep skill chaining shows the beneﬁts of using hierarchies even in the absence of such data augmentation techniques but including them should yield additional performance beneﬁts in sparse-reward tasks.
A drawback of deep skill chaining is that, because it builds skills backward from the goal, its performance in large state-spaces is dependent on a good exploration algorithm.
We used the naive exploration strategy of adding Gaussian noise to chosen actions (Lillicrap et al., 2015; Fujimoto et al., 2018) since the exploration question is orthogonal to the ideas presented here.
The lack of a sophisticated exploration algorithm also explains the higher variance in performance in the PointMaze task in Figure 1.
Combining effective exploration (Machado et al., 2018; Jinnai et al., 2019b) with DSC’s high reliability of triggering target events is a promising avenue for future work.
We presented a new skill discovery algorithm that can solve high-dimensional goal-oriented tasks far more reliably than ﬂat RL agents and other popular hierarchical methods.
To our knowledge,
DSC is the ﬁrst deep option discovery algorithm that does not treat the number of options as a ﬁxed and costly hyperparameter.
Furthermore, where other deep option discovery techniques have struggled to show consistent improvements over baseline ﬂat agents in the single task setting (Zhang & Whiteson, 2019; Smith et al., 2018;
Harb et al., 2018; Klissarov et al., 2017), we unequivocally show the necessity for hierarchies for solving challenging problems.
A A PPENDIX A. C REATING S KILL T REES
In Section 3.4, we introduced the idea of generalizing skill chains to skill trees to incorporate qualitatively different solution trajectories.
In this section, we provide some of the implementation details required to learn initiation set classiﬁers that organize in the form of trees.
When creating skill chains, the goal of each option is to trigger the initiation condition of the option that precedes it in its chain (i.e, its parent option).
When creating a skill tree of branching factor
B , we allow at most B options to target each salient event in the MDP (i.e, the goal state and the initiation set classiﬁers of preexisting options).
To further control the branching factor of the skill tree, we impose two more conditions on option creation:
1. Consider an option o 1 which already has one child option o 2 targeting it.
Now suppose that we want to learn another option o 3 that also targets o 1 .
We only consider state s t to be a positive example for training I o
if I o 2 ( s t ) = 0 .
To prevent signiﬁcant overlap between options that target the same event, we treat the positive examples used to train the initiation set classiﬁer of one as negative training examples of all its sibling options.
This allows for multiple options that trigger the same target event, while encouraging them to
specialize in different parts of the state-space.
In the Point E-Maze task considered in Section 4, we learn a skill tree with B = 2 .
A. T EST E NVIRONMENTS
A description of the Four Rooms and the Point E-Maze tasks was provided in Section 4.
Here we describe the remaining tasks considered in this paper:
Point Maze :
In this task, the same point agent as in the four rooms task must navigate around a
U-shaped maze to reach its goal.
The agent receives a reward of − 1 for every step it lives, and a sparse terminating reward of 0 when it reaches its goal location.
This is an interesting task for hierarchical agents because in order to reach the goal, the agent must ﬁrst move away from it.
It is clear that a dense distance-based reward formulation of this problem would only serve to deceive non-hierarchical agents such as DDPG.
Ant Maze : The ant (Duan et al., 2016) is a challenging agent to control due to its non-linear and highly unstable dynamics.
In this task, the ant must now navigate around the same U-shaped maze
[Figure]
as in the Point Maze task.
Getting the ant to cover signiﬁcant distances along the x, y plane without falling over, is a benchmark control task itself (Brockman et al., 2016).
As a result, constructing options backward from the goal could require prohibitively large training episodes or the use of a sophisticated exploration algorithms (Burda et al., 2019;
Bellemare et al., 2016; Tang et al., 2017).
To avoid conﬂating our results with the orthogonal investigation of effective exploration in RL, we follow the experimental design of other state-of-the- hierarchical reinforcement learning algorithms (Levy et al., 2019; Nachum et al., 2018) and sample the initial state of the ant uniformly across the maze for the ﬁrst 30 episodes.
For fair comparison, all baseline algorithms use this exploration strategy.
Fixed Reacher : We use the Reacher task (Brockman et al., 2016) with two modiﬁcations.
First, rather than randomly sampling a new goal at the start of each episode, we ﬁx the target across all episodes.
We do this because if the goal moves, following a learned skill chain will no longer solve the MDP.
Second, to increase the difﬁculty of the resulting task, we use a sparse reward function rather than the dense distance-based one used in the original formulation.
[Figure]
experimental domains
A. A BLATION S TUDY A.3. P ERFORMANCE AS A F UNCTION OF N UMBER OF S KILLS
Deep skill chaining generally discovers and learns as many skills as it needs to solve a given problem.
In this experiment however, we restrict the number of skills DSC can learn to examine its impact on overall agent performance (as measured by cumulative reward during training).
Figure 5 shows that the performance of the agent increases monotonically (with diminishing marginal improvements) as it is allowed to learn more skills
. A.3. N UMBER OF S KILLS OVER T IME
Figures 7 (a) and 7 (b) illustrate how deep skill chaining incrementally discovers options and adds it to the agent’s option repertoire.
Figure 7(c) shows how the number of skills empirically increases over time, plateaus and has low variance between runs.
Since the agent has to learn the importance of the key in the Four Rooms task, learning initiation set classiﬁers takes longer than in the Point-Maze task.
A.3. H YPERPARAMETER S ENSITIVITY
In this section, we analyze DSC’s sensitivity to some of the hyperparameters speciﬁc to the algorithm.
In Figure 8, we show that even under a fairly large range of values for the buffer length K and the gestation period N , DSC is able to retain its strong performance.
[Figure]
A.
A LGORITHM P SEUDO -C ODE
Algorithm s 0 is the start state of the MDPg ( s ) :
=
if s is a target state in the MDP, 0 otherwise
Given hyperparameter T 0 , the time budget for discovered, temporally extended options
Global option: o G =
( I o G , π o G , β o G = 1 g , T = 1)
Goal option: o g =
( I o g , π o g , β o g = 1 g , T = T 0 )
Agent’s option repertoire: O = { o G }
Untrained Option: o U
= o g // option whose initiation classiﬁer is yet unlearned
Policy over options: π O : s t −→ o t s t = swhile not s t .is terminal() do
// Collect trajectories that trigger termination region unless have ﬁnished chaining if β o U ( s t + τ ) & ( s 0 (
cid:26) ∈
I o i
∀ o
i ∈ O )
then o
U .learn
O .append
( o U )
o
U
= create child option ( o U ) end end endFunction create child option( o ) : ””” Create new β
””” o ∗ = Option() //
Create a new option
I o ∗ =
None β o ∗ =
I o return o ∗
Function execute option( o t ) : ””” t 0 =
t
T is the option’s episodic time budget π o t is the option’s internal policy while not β o t ( s t ) & t < T do a t =
π o t ( s t ; θ o t ) r t , s t +1 = env.step( a t ) s t
=
s t +1 t
= t +end τ = t // duration of option execution return r t 0 :
t 0 + τ , s t
+ τ
A. L EARNING I NITIATION S ET C LASSIFIERS
To learn initiation set classiﬁers as described in Section 3.3, we used scikit-learn’s
One-Class SVM and Two-Class SVM packages (Pedregosa et al., 2011).
Initiation set classiﬁers were learned on a subset of the state variables available in the domain.
For instance, in the Lock and Key domain, the initiation set classiﬁer was learned over the x, y position and the has key indicator variable.
This is similar to other methods like HAC (Levy et al., 2019) which require the user to specify the dimensions of the state variable necessary to achieve the overall goal of the MDP.
Incorporating the entire state variable to learn initiation set classiﬁers or using neural networks for automatic feature extraction is left as future work.
A. H YPERPARAMETER S ETTINGS
We divide the full set of hyperparameters that our algorithm depends on into two groups: those that are common to all algorithms that use DDPG (Table 2), and those that are speciﬁc to skill chaining (Table 3).
We did not try to optimize over the space of DDPG hyperparameters, and used the ones used in previous work (Lillicrap et al., 2015; Fujimoto et al., 2018).
Table 3 shows the hyperparameters that we chose on the different tasks considered in this paper.
Most of them are concerned with learning initiation set classiﬁers, the difﬁculty of which varies based on domain.
To determine the correct setting of these parameters, we usually visualized the learned initiation set classiﬁers during the course of training (like Figures 2 and 6), and made adjustments accordingly.
[Table] [Table]
A. C OMPUTE I NFRASTRUCTURE
We used 1 NVIDIA GeForce 2080 Ti, 2 NVIDIA GeForce 2070 Ti and 2 Tesla K80s on the Google
Cloud compute infrastructure to perform all experiments reported in this paper
. A. N OTE ON C OMPUTATION T IME
Each option is parameterized by its own neural networks, which are only updated when the agent is inside that option’s initiation set.
For a given transition, this leads to at most two or three updates.
In
Point-Maze, updating all options on a transition took 0 .
004 ± 0 .
0003 s more than just updating the global DDPG agent (averaged over 300 episodes using 1 NVIDIA
Ti GPU) - a trivial amount of extra computation time.