Skill Discovery for Exploration and Planning using Deep Skill Graphs
Abstract
Temporal abstraction provides an opportunity to drastically lower the decision making burden facing reinforcement learning agents in rich sensorimotor spaces.
Well constructed hierarchies induce state and action abstractions that can reduce large continuous MDPs into small discrete ones, in which planning with a learned model is feasible.
We propose a novel algorithm, Deep Skill Graphs , for acquiring such a minimal representation of an environment.
Our algorithm seamlessly interleaves discovering skills and planning using them to gain unsupervised mastery over ever increasing portions of the state-space.
The constructed skill graph can be used to drive the agent to novel goals at test time, requiring little-to-no additional learning.
We test our algorithm on a series of continuous control tasks where it outperforms baseline ﬂat and hierarchical RL methods alike.
1. Introduction
Model-free reinforcement learning agents can acquire goaldirected behaviors in unknown environments (Sutton &
Barto, 2018; Mnih et al., 2015; Lillicrap et al., 2015).
These behaviors however, often tend to be reactive and must be reacquired when the agent is tasked with solving a different, albeit related task (Farebrother et al., 2018;
Witty et al., 2018).
Planning, on the other hand, can generate longhorizon behaviors (Campbell et al., 2002; Silver et al., 2017) but assumes access to a model of the environment, which can often be hard to specify.
Our goal is to learn representations that enable an agent to transition from trial-and-error learning to solving long-horizon tasks using high-level planning.
One way to generate long-horizon behaviors is to plan using models learned from raw observations (Sutton, 1991; Levine Brown , Prov- et al., 2016; Ha & Schmidhuber, 2018; Hafner et al., 2018;
Kaiser et al., 2020).
However, model-based methods are plagued by the difﬁculty of learning effective models in highdimensional spaces.
Learning abstractions (Giunchiglia &
Walsh, 1992; Konidaris, 2019; Abel & Littman, 2020) afforded by the hierarchical reinforcement learning (HRL) framework (Barto & Mahadevan, 2003) could facilitate planning without having to confront the problem of learning a detailed model of the world.
The majority of research in
HRL has examined the beneﬁts of discovering and using abstract actions, or skills (Sutton et al., 1999; Mcgovern & Barto, 2002; Precup, 2001; Konidaris & Barto, 2011;
Ravindran & Barto, 2004; Bacon, 2018).
Less studied is the complimentary question of how discovered skills can inform useful state-abstraction.
We build on the key insight of Konidaris et al.
(2018) that abstract actions create an opportunity to build an accompanying state abstraction that supports planning.
They proved that to compress an SMDP (Sutton et al., 1999) into a discrete representation suitable for planning, the agent’s skills must be sequentially executable—successful execution of one should permit the execution of another.
Furthermore,
Bagaria & Konidaris (2020) recently showed how skills that have this property can be autonomously discovered in high-dimensional continuous spaces.
However, for skill discovery to proceed in their work, Bagaria & Konidaris (2020) assume access to a pre-speciﬁed goal state and an exploration policy that can generate a small number of trajectories that reach that goal state.
Such extrinsically deﬁned goals are often unavailable, and even when they are, random exploration may be insufﬁcient for the agent to ever reach them.
So that skill-acquisition may proceed in the absence of such extrinsic goals, we discover them using principles of intrinsic motivation (Barto et al., 2004; Chentanez et al., 2005).
We call such discovered target states salient events , and construct a collection of skills capable of moving the agent between such salient events, resulting in a skill graph .
This graph is a compressed discrete representation of the original MDP where vertices correspond to states in which skills operate and edges correspond to skill policies.
Because the skill graph was constructed to meet the speciﬁcations laid out by Konidaris et al.
, it is provably sound—meaning that plans constructed in the abstract MDP correspond to feasible solutions in the original ground MDP.
We test our algorithm on three challenging maze-navigation tasks in MuJoCo (Todorov et al., 2012; Fu et al., 2020), demonstrate its key properties, and ﬁnd that it can signiﬁcantly out-performs baseline ﬂat and hierarchical RL methods alike.
2. Background
We consider the class of MDPs M = ( S, A, R , T , γ, ρ ) .
S denotes the state space, A denotes the action space and γ is the discount factor (Sutton & Barto, 2018).
R is the set of reward functions and ρ is the set of start state distributions implied by the different tasks in the multi-task setting (Sutton et al., 2007; Tanaka & Yamamura, 2003; Brunskill &
Li, 2013; Wilson et al., 2007).
The transition dynamics T is assumed to be ﬁxed among the different tasks.
Importantly, we do not assume access to R during training time—skill discovery proceeds without extrinsic rewards.
Furthermore, ρ remains ﬁxed at training time, but can change arbitrarily at test-time.
This ensures that the agent cannot “teleport” to states that might otherwise be difﬁcult to reach and thus must confront the exploration problem in its entirety (Even-Dar et al., 2005; Kakade &
Langford, 2002).
2.1. The Options Framework
We model abstract actions (or skills) as options (Sutton et al., 1999).
Each option o in the agent’s option repertoire
O is deﬁned as a three element tuple ( I o , π o , β o ) .
The initiation set I o : s → { 0 , 1 } describes the set of states from which option o can be executed.
The termination set β o : s → { 0 , 1 } describes the set of states in which option execution is deemed successful.
Finally, the option policy π o : s → a is a closed-loop controller that drives the agent from states in I o to those in β o .
Augmenting the set of available actions with options results in a Semi-Markov
Decision Process (SMDP) (Sutton et al., 1999) where the next state depends on the current state, action, and time .
In addition to the three elements described above, Konidaris et al.
(2018) deﬁne the effect set E o of option o as the set of states in which the agent might ﬁnd itself after successfully executing π o from anywhere in I o .
Note that E
o ⊆
β
o since there may be states in β o not reachable from states in I o .
2.2. Skill Chaining
The skill chaining algorithm (Konidaris & Barto, 2009;
Bagaria & Konidaris, 2020) incrementally constructs options that extend backward from a goal state to the start states of the MDP.
Discovered skills have the property that the initiation condition of an option o i is the termination condition of the option o i − 1 that precedes it in its chain, i.e, β
o i −
=
I o i .
Option policies in the skill chaining framework have local support: they specialize in different regions of the state-space and do not have to learn representations for states far outside their initiation region.
This allows them to learn effective solutions to their own sub-problems which are then combined by a policy over options.
Each skill learns to initiate only from those states from which it can reliably solve its own sub-problem.
As a result, the skill chaining algorithm adaptively discovers as many skills as it needs, with whatever granularity it needs, to most reliably get from its start state to its goal state.
By deconstructing the solution to goal-oriented MDPs into sequential option executions, deep skill chaining (DSC) can learn overall policies that reach the goal far more effectively than ﬂat reinforcement learning techniques.
Given a single start state and a goal state, skill chaining organizes skills in the form of a chain.
When there are multiple start states, skill chaining organizes skills in the form of a tree rooted at the goal state.
While effective in solving many goal-oriented MDPs, this topology is insufﬁcient in the multi-task setting where the agent could start anywhere and be asked to go anywhere.
Consequently, we propose to organize skills in the form of a graph to handle the multi-task setting.
2.3. Covering Options
Recent work has focused on learning task agnostic representations by exploiting the spectral properties of the environment’s transition dynamics (Mahadevan & Maggioni, 2007;
Machado et al., 2017; Liu et al., 2017; Machado et al., 2018).
Most recently, Jinnai et al.
noticed that the eigenvector corresponding to the second smallest eigenvalue of the graph
Laplacian (Chung & Graham, 1997) (also known as the
Fiedler vector (Fiedler, 1973)) captures information about states that are furthest apart in the topology of an MDP.
In an effort to make the MDP easier to explore, they developed
Covering Options (Jinnai et al.)
to connect the two states in a discrete MDP that are furthest apart according to this
Fiedler vector.
Deep covering options (DCO) (Jinnai et al., 2020) used Wu et al.
(2019)’s approximation of the graph
Laplacian to extend covering options to large continuous spaces.
Because of their ability to identify states in the frontier and their demonstrated superiority to other methods that discover options for exploration (Eysenbach et al., 2019a;
Machado et al., 2017), we use it to boost exploration in our algorithm.
2.4. Rapidly Exploring Random Trees (RRT)
RRT is a motion planning algorithm used to ﬁnd a collisionfree path between two points in a robot’s conﬁguration space (LaValle, 1998).
The RRT algorithm, which builds a tree rooted at the start state, proceeds in three steps: (a) randomly sample a state from the conﬁguration space, (b) identify its nearest neighbor node in the tree and ﬁnally (c) use the robot’s dynamics model to move one step in the direction of the sampled state.
Despite its simplicity, RRTs have been wildly successful in solving motion planning problems involving high-dimensional dynamical systems.
It owes its success in large part to the Voronoi bias of the tree construction procedure (Lindemann & LaValle, 2004).
The Voronoi bias of RRT refers to property that at each iteration of the algorithm, the direction in which tree expands is proportional to the volume of the Voronoi region of an edge node.
The volume of the Voronoi region of a state can thus be seen as a measure of “how unexplored” the region around that state is; randomly selecting points to expand the tree therefore causes it to naturally grow in the direction of the unexplored frontiers of the state-space.
Since RRT assumes a given dynamics model, it cannot be directly used in the RL setting.
However, as we will see in Section 3, our graph construction procedure is heavily inspired by RRT.
Finally, although inspired by RRT, we do not learn a single-query tree (LaValle, 2006).
Instead, much like the Probabilistic Road Map (PRM) algorithm (Kavraki et al., 1996), we learn a multiple-query graph, meaning that the discovered skill graph can be used to ﬁnd solution paths between multiple start-goal pairs.
3. Deep Skill Graphs
A skill graph is a particular arrangement of acquired skills that represents a convenient discrete abstraction of the original continuous MDP.
To construct this graph, the agent continually interleaves two steps: (a) discovery of target salient regions that will increase the coverage of the skill graph and (b) creation of skill chains that connect each new salient event to the graph.
Once acquired, the agent may construct plans to reach any state that lies in the region spanned by its skill graph.
3.1. Deﬁnitions
Salient Events : A salient event β salient is a collection of states that are deemed important by an intrinsic motivation system (Barto et al., 2004).
We formalize them as binary classiﬁers that take a state as input and return a boolean decision denoting whether that state triggered that salient event.
Skill Graph : A skill graph is a weighted and directed graph
G =
( V , E, W ) .
Each vertex i ∈ V either corresponds to an option effect set or a salient event.
There is an edge e i →
j between vertices i and j if and only if the effect set of vertex i is inside the initiation set of vertex j , i.e, E i ⊆
I j .
The edge weight w i,j ∈ W is the inverse of the fraction of times you can successfully get from vertex i to vertex j using a single option execution.
3.2. Constructing the Skill Graph
As illustrated in Figure 1, skill graph construction iteratively proceeds in three steps.
In the ﬁrst step, the DSG agent discovers a target/salient event it would like to reach.
Since this target state lies outside the graph, the agent does not yet have the necessary skills to get there using planning.
In the second step, the agent identiﬁes the node in the graph closest to the target state identiﬁed in step 1.
It then constructs a plan (shown with red arrows) inside the graph to reach this identiﬁed node (shown in green).
Once there, it uses deep skill chaining to construct a series of skills that eventually connect the target state to graph.
Now, the salient event from step 1 is now just another node in the graph that can be easily reached using planning.
This process repeats when a new salient event is generated, until the skill-graph has achieved reasonable coverage of the state-space.
3.2.1. S ALIENT E VENT D ISCOVERY
Task-agnostic skill acquisition requires the agent to identify salient regions of the state-space and construct skills that target such regions.
Regions of the state-space covered by the graph represent regions where the agent has achieved mastery (Kaelbling, 1993; Veeriah et al., 2018).
This is because the agent can plan using learned skills to reliably reach arbitrary states inside the graph.
So that the agent may incrementally increase the portion of the world over which it has achieved mastery, our salient event discovery algorithm should extend our graph into largely unexplored regions of the state-space.
We propose two strategies for generating salient events:
Deep covering options (DCO) :
Given a buffer of n transitions B = ( s, s (cid:48) ) 1: n , DCO learns a one-dimensional embedding called the Fiedler vector f : s → R .
The states s 1 = arg max s ∈B f ( s ) and s 2 = arg min s ∈B f ( s ) represent the two states in B that are furthest apart in the agent’s decision space.
Jinnai et al.
(2020) note that adding an edge between s 1 , s 2 maximally increases the connectivity of the graph (makes it easier to explore the underlying MDP).
They assume that such an edge corresponds to a single option execution.
However, in general s 1 , s 2 might be very far away from each other and hence require an arbitrary number of skills to reliably go from one to the other.
Algorithmically, DCO examines the replay buffer of the agent every K episodes (where K is a hyperparameter).
It then generates two salient events: β s 1 : s → { 0 , 1 } that targets state s 1 and β s 2 : s → { 0 , 1 } that targets state s 2 .
We then call upon deep skill chaining to learn as many skills as it needs to connect β s 1 and β s 2 to the existing portion of the skill graph.
Once these events are in the graph, the agent can then plan with its existing skills to get from one to the other—reﬂecting the increased connectivity of the resulting underlying SMDP.
Random sampling : Covering options is a principled approach to generating salient events for intrinsically motivated graph expansion.
Its success in high-dimensional spaces however, is dependent on the quality of its approximation of the graph
Laplacian
(Wu et al., 2019).
A far simpler alternative would be to generate salient events in the same way the RRT algorithm does—uniformly at random.
This simple strategy can lead to the graph growing in the directions of the largest unexplored regions of the state-space.
Its disadvantage, however, is that it assumes that one can sample feasible states from the environment.
When such an oracle is available, random sampling of salient events may provide a competitive alternative to its more computationally expensive counterparts.
3.2.2. P ICKING A TARGET SALIENT EVENT
Given the current list of salient events ˜ β
= { β 1 , β 2 , ..., β n } , the agent must ﬁrst pick which one to target in the current episode.
One could imagine using heuristics based on which which skills to practice and reﬁne the most (Stout & Barto, 2010), but for simplicity, we pick a target randomly from this list.
Once the agent successfully reaches β , it picks a new target from ˜ β , and the process continues.
3.2.3. I DENTIFYING THE NEAREST NODE IN THE GRAPH
Given a target salient event β that lies outside the graph,
DSG must ﬁrst identify its nearest neighbor in the graph.
Let this vertex in the graph be v nn .
For the experiments presented in this paper, we chose the node with the lowest
Euclidean norm.
This measure is not applicable for all problems, but we leave discovering and using a more appropriate measure related to MDPs (Mahadevan & Maggioni, 2007;
Taylor et al., 2011) for future work.
3.2.4. A CTING INSIDE THE GRAPH
Given target salient event β and its nearest neighbor inside the graph v nn , the DSG agent must now ﬁgure out a way to get to that target node v nn from its current state s t .
We can ﬁnd the vertex in the graph corresponding to s t by querying the initiation set classiﬁers of all the options in the graph.
Let o t be the option whose initiation set s t is in, i.e, I o t ( s t )
= 1 , ∃ o t ∈
O .
We now use graph search (Dijkstra, 1959) to ﬁnd a plan that goes from o t to v nn .
We then execute the ﬁrst option in the plan, land in a new state s t + τ , and then re-plan to go from s t + τ to v nn .
This process continues until we reach v nn .
3.2.5. E XTENDING THE GRAPH
Once DSG has used its planning based control loop to reach v nn , which is usually a vertex in the boundary of the graph, it leaves the graph in an effort to reach the target event β .
Here, it uses deep skill chaining to learn options that eventually chain back from β to v nn .
After that, the agent can go back to using its planner to reach β in the future.
3.2.6. A DDING EDGES TO THE GRAPH
Now we will describe how skill chains targeting different salient events may be connected together in the skill graph.
For simplicity, we can break our discussion into two different cases:
Option-option edges : For plans in the abstract MDP to correspond to feasible solutions in the ground MDP, Konidaris et al.
(2018) showed that any two options o 1 and o 2 can have an edge e 1 , 2 between them if and only if there is a guarantee that successful execution of o 1 will allow the agent to execute o 2 , i.e, e 1 , 2 exists iff E o 1 ⊆
I o 2 .
To implement this rule, we store all the states in which o 1 execution successfully terminated and check if all of them lie inside I o 2 .
If at some point E o 1 expands and we get a sample from its effect set that lies outside I o 2 , we must delete edge e 1 , 2 from the skill graph.
Option-event edges : An edge can exist from an option to a salient event if the option’s effect set is a subset of the salient event—this typically only happens when the option is trained to target that salient event.
For an edge to go from a salient event β salient to option o , the option’s initiation set must contain all the points at which that salient event can be triggered, i.e, β salient ⊆
I o .
To implement this logic, we keep track of all the points at which any option has ever triggered β and ensure that I o includes all of those points.
To develop greater intuition about when it is appropriate to connect options in the graph, readers may refer to ﬁguresand 8 in the appendix.
3.2.7. L EARNING BACKWARD OPTIONS
So far, we have only described how the skills in the graph take the agent from the start state to discovered salient events.
However, to equip the agent with the ability to navigate between events, it must learn options that go backward from discovered salient events to the start state (and to other salient events).
The logic for learning these “backward options” is similar to learning “forward options”, but requires some additional care which is described in section
A.3 of the appendix.
3.3. Using the Skill Graph
Once the agent has ﬁnished constructing the skill graph, it can use it to target goals not seen during training.
Suppose that at test time, the agent is asked
to reach a goal region β s g (in continuous domains, this is typically deﬁned as a small region around a goal state s g ).
As illustrated in ﬁgure 2, the goal region
β s g , at test time can fall into one of the three possible scenarios
• β s g completely contains the effect set of an option: Since reaching that effect set would imply reaching β s g , the agent may use its planning based control loop to reach s g without any additional learning.
• β s g is inside the initiation set of some option in the graph: Suppose that the β s g does not contain the effect set of an option in the graph but is inside initiation set of option o train .
In this case, we have options that can take the agent close to the goal but do not have a controller that will drive us to s g .
As a result, we create a new option
o test having the same initiation set as option o train .
The agent then uses planning to get to I o test and then execute π o test to reach s g .
Since existing skills reliably bring the agent close to s g , it can learn π o test fairly quickly 1 .
• Goal is outside the graph: It is of course possible that s g entirely lies outside our graph.
In this case, DSG uses the same logic that it used to trigger salient events outside the graph during training, i.e, it follows the steps outlined in sections 3.2.3 – 3.2.6.
4. Experiments
We test our algorithm in a series of continuous control tasks adapted from the “Dataset for RL” benchmark (Fu et al., 2020; Todorov et al., 2012).
2 Chosen tasks either involve the point agent from Nachum et al.
(2018) or the quadruped ant robot from Brockman et al.
These tasks are challenging for non-hierarchical methods, which make littleto-no learning progress in these tasks (Duan et al., 2016).
We evaluate two versions of the proposed algorithm: one using deep covering options ( DSG-DCO ) and the other using RRT-style random sampling for salient event discovery ( DSG-RRT ).
We compare Deep Skill Graphs to goalconditioned RL (Kaelbling, 1993; Schaul et al., 2015) and deep covering options (Jinnai et al., 2020).
Goal-conditioned RL algorithms learn policies that generalize across goals by taking the goal state as an additional input during training.
Because of their relative simplicity and their widespread use in RL (Nachum et al., 2018; Levy et al., 2019; Eysenbach et al., 2019b; Nasiriany et al., 2019), we compare against them as a representative non-hierarchical method that works in the multi-task setting.
We choose DCO as our HRL baseline because (a) the DSGDCO variant of our algorithm uses DCO for salience discovery and (b) DCO outperformed other methods that learn options for exploration (Eysenbach et al., 2019a; Machado et al., 2017).
4.1. Qualitative Evaluation
Exploration Property of DSG : We compare the states visited by the DSG agent and those visited under a random walk.
Figure 4 shows that in the ant-maze environment, skill discovery can lead to temporally extended exploration as opposed to the dithering behavior of random walks, even in the absence of an extrinsic reward function.
Incremental Graph Expansion : Figures 3 and 6 provides some intuition on why DSG can effectively explore large regions of the state-space—the skill-graph begins at the start state and incrementally expands into unexplored regions of the state-space.
By planning and executing learned skills inside the graph, the agent can reliably get back to the frontier of its knowledge (as represented by the outermost vertices in the graph).
Exploration from the frontier in turn allows it to amass the experiences it needs to further expand the graph.
By interleaving planning and chaining in this way, the DSG agent incrementally achieves mastery of ever increasing proportions of the state-space.
4.2. Quantitative Evaluation
After training each algorithm for the same number of episodes, we evaluate their ability to navigate from randomly chosen start states to randomly chosen goal states.
As is common in the multi-task setting, we plot the “success rate” of each algorithm, which is the fraction of times it was able to successfully reach the goal in the given episodic budget.
All learning curves in ﬁgure 5 are averaged over the same set of 20 random start and goal states.
4.2.1. C OMPARATIVE A NALYSES
Comparison with goal-conditioned RL : Figure 5 shows that we comfortably outperform our ﬂat goal-conditioned
RL baseline.
Our results support the ﬁndings of Nasiriany et al.
that while UVFAs can in principle be used to solve any goal-reaching task, their effectiveness seems to be limited to situations where the goal state is near the agent’s start state.
By contrast, DSG learns skills that specialize in different parts of the state-space, making them more effective at solving long-horizon problems.
Comparison with deep covering options : DCO learns options that allow it to effectively explore the MDP.
However, these options are isolated, so the agent must use SMDP
Q-learning (Bradtke & Duff, 1995) to compose options and primitive actions to reach a goal given at test-time.
Learning an effective policy over options in this setting demands a lot of data, making DCO under-perform when being tested for few-shot generalization.
DSG, on the other hand, uses planning to compose options at test time and hence requires little-to-no additional interactions with the environment.
Impact of salience discovery method : Figure 5 shows that
DSG-RRT consistently outperforms the DSG-DCO variant of our algorithm.
We found that the skill-graph produced by randomly sampling salient events was denser than the one constructed by using salient events generated by DCO.
As a result, a test-time goal state tended to be inside the graph more often for DSG-RRT than for DSG-DCO.
This meant that DSG-RRT could plan and execute skills it had already learned to move close to the goal, while DSG-DCO often had to extend the graph towards the goal state—a process that can be a relatively sample inefﬁcient.
An analysis of
DSG-DCO’s poor performance on ant-maze is presented in section A.1.3 of the appendix.
5. Related Work
Skill Discovery in high-dimensional spaces : Recent work on goal-directed skill acquisition in large continuous spaces can be broadly divided into three categories: (1) optioncritic methods (Bacon et al., 2017; Harutyunyan et al., 2019;
Khetarpal & Precup, 2019; Klissarov et al., 2017; Tiwari &
Thomas, 2019; Riemer et al., 2018; Harb et al., 2018) which describe an end-to-end architecture for learning options in high-dimensional spaces, (2) feudal methods (Dayan & Hinton, 1993; Vezhnevets et al., 2017; Nachum et al., 2018;
Levy et al., 2019; Li et al., 2019) in which a higher-level manager outputs goals for lower level workers to achieve and (3) skill chaining methods (Konidaris & Barto, 2009;
Konidaris et al., 2012; Konidaris, 2016; Shoeleh & Asadpour, 2017; Metzen & Kirchner, 2013; Bagaria & Konidaris, 2020) which deconstruct the solution path of an MDP into a series of shorter horizon skills.
While these methods demonstrate substantial progress on the skill discovery question, they optimize for a task-speciﬁc objective and hence cannot be directly used for the multitask setting.
Combining model-free RL and planning : Some recent works have sought to bridge model-free RL and planning.
SoRB (Eysenbach et al., 2019b) uses graph search on the replay buffer (Lin, 1993) to ﬁnd efﬁcient paths between observations made by a model-free policy.
PRM-RL (Faust et al., 2018) replaces the local planner used in PRM methods (LaValle, 2006) with an RL policy.
Both SoRB and
PRM-RL assume access either to a well trained value function or an RL policy that can be meaningfully queried in arbitrary parts of the state-space.
By contrast, we learn skill policies that specialize in different parts of the state-space.
In LEAP (Nasiriany et al., 2019), a planner generates subgoals for a low level model-free policy to meet.
However, their planner uses a pre-trained generative model to generate high-dimensional goals in parts of the state-space the
RL agent may not have seen.
Our algorithm automatically extends the skill graph towards unexplored regions.
SPTM (Savinov et al., 2018) is a memory augmented RL agent that plans over raw landmark observations in navigation problems.
Our method bypasses the difﬁculty of planning in high-dimensional spaces by instead planning over abstract states.
Most recently, DADS (Sharma et al., 2020) achieved impressive results by using Model Predictive Control to compose learned skills to reach novel goal states at test time.
However, their algorithm does not interleave skill-discovery and high-level planning, making them unable to exploit the effectiveness of temporal abstraction to explore from states that are difﬁcult to get to.
Exploration in RL :
In the non-hierarchical setting, notions of novelty (Strehl & Littman, 2008; Bellemare et al., 2016) and prediction error (Oudeyer et al., 2008; Barto et al., 2004;
Burda et al., 2019; Pathak et al., 2017; Badia et al., 2020) have been used to drive the agent to previously unseen regions of the state-space (Ta ¨ ıga et al., 2019).
Recent work in HRL based intrinsic motivation has fallen into two broad categories: (1) empowerment (Klyubin et al., 2005) driven methods (Florensa et al., 2016; Hausman et al., 2018;
Eysenbach et al., 2019a; Gregor et al., 2016;
Heess et al., 2017;
Sharma et al., 2020) which optimize an entropy based objective and (2) spectral methods (Mahadevan & Maggioni, 2007; Machado et al., 2017; Jinnai et al.; 2020) that exploit structure in the transition dynamics of the environment.
While these methods generally focus on the exploration question alone, we show that additionally combining them with options for exploitation eventually facilitates better exploration.
6. Conclusion
We introduced a practical algorithm that compresses a large continuous MDP into a small abstract representation suitable for planning.
Like all RL agents, DSG starts by exploring the environment since it does not know enough about it to plan.
It then discovers salient events and the skills it needs to reliably trigger those salient events.
Together, these skills and target events enable high-level planning.
We showed that skill graphs grow outward from the startstate towards large unexplored regions, reﬂecting mastery over ever increasing portions of the state-space.
Finally, we tested our algorithm on a series of maze navigation tasks and showed that DSG can be used to reach novel goals at test time in a small number of trials.
We compared the few-shot generalization capability of our algorithm to that of popular ﬂat and hierarchical alternatives and showed that DSG can signiﬁcantly outperform them.
A. Appendix
A.1. Using Deep Covering Options for Salient Event Discovery
DCO approximates the Fiedler vector, which is a function that projects states into a one dimensional embedding.
In
DCO, all states that have a ﬁedler value lower than some pre-deﬁned threshold are marked as the termination condition of an option.
However, as noted by Jinnai et al., this scheme can often lead to termination sets that are either too small or those that are too big.
If we were to directly use such termination conditions as salient events, the resulting salient events themselves could be too small or too big.
As a result, we discard the termination condition discovered by
DCO and simply pick the states with the highest and low- est values under the approximate Fiedler vector.
We then construct an (cid:15) -ball around that state and treat it as a salient event.
Creating this (cid:15) -ball assumes a meaningful metric, which is readily available in the domains considered here.
However, scaling our method to domains with pixel observations would require a deeper understanding of how the size of DCO termination conditions can be better controlled (Harutyunyan et al., 2019).
A.1.1. F ILTERING SALIENT EVENTS
So as to not create options that target redundant salient events, we may choose to reject a generated salient event before we add them to the list of target events.
A generated event is rejected if it is either (a) already satisﬁed by an existing event or (b) the generated event is inside the initiation classiﬁer of an option already in the graph.
A.1.2. W HEN TO GENERATE NEW SALIENT EVENTS
We attempt to generate a new pair of salient events when any of the following three conditions are satisﬁed:
The most recent generated salient event has been added to the graph 2.
It has been K = 30 episodes since a salient event was generated and accepted 3.
It has been R = 10 episodes since the last pair of salient events were generated, and both were rejected (according to criteria described in A.1.1).
A.1.3. DSG-DCO ON A NT -M AZE
Figure 9 shows that in the Ant-Maze domain, DCO fails to discover salient events in the top left portion of the statespace.
As a result, DSG is unable to expand the graph into that region of the state space and it thus fails to reliably reach goal states there.
DSG-DCO’s failure to expand the skillgraph in the top part of the maze is principally responsible for its poor performance in Ant-Maze.
A.2. Setting edge weights in the skill graph
As described in section 3.1, the edge weight between vertices i, j is given by w i,j = 1 success rate ( i,j ) , where success rate ( i, j ) is the fraction of times option i successfully executed its policy to reach option j ’s initiation set.
The same formula works when vertex i corresponds to an option and vertex
j corresponds to a salient event.
However, when vertex i corresponds to a salient event and vertex
j corresponds to an option, then the edge weight w i,j = 0 because being inside the salient event implies being inside the initiation set of option o j .
As described in section 3.1, the edge weight between vertices i, j is given by w i,j = 1 success rate ( i,j ) , where success rate ( i, j ) is the fraction of times option i successfully executed its policy to reach option j ’s initiation set.
The same formula works when vertex i corresponds to an option and vertex
j corresponds to a salient event.
However, when vertex i corresponds to a salient event and vertex
j corresponds to an option, then the edge weight w i,j = 0 because being inside the salient event implies being inside the initiation set of option o j .
A.3. More details on learning backward options
Consider a skill chain C that starts at β 0 and targets β salient .
C is considered “complete” when it learns an option o such that β 0 ⊆
I o .
At this point, we start learning a new skill chain C back that targets β 0 and chains until β salient .
As is typical with skill chaining, C back learns as many options as it needs to reliably drive the agent from β salient to β 0 .
A.4. Test environments
We evaluated our algorithm in four tasks that exhibit a strong hierarchical structure (Nachum et al., 2018; Fu et al., 2020;
Brockman et al., 2016; Duan et al., 2016): (1) Point Maze (Medium), (2) Point Maze (Hard), (3) Ant U-Maze and (4)
Ant Reacher.
Learning curves in ﬁgure 5 are shown for environments 1 - 3 .
In ant reacher, there is no maze, and the ant is required to navigate an open area spanned by [ − 10 , 10] 2 .
Only qualitative results are presented in ﬁgurefor the Ant Reacher domain.
In each task, the agent is reset back to its start state (a small distribution around (0 , 0) ) after 1000 steps per episode.
These learning curves in ﬁgure 5 are generated after we train each algorithm for 2000 episodes in problems involving the point robot and for 4000 episodes in problems involving the more complex ant robot.
Each episode lasts 1000 time steps.
A.5. Baseline Implementation Details
Following Bagaria & Konidaris (2020), we use the Deep
Deterministic Policy Gradients (DDPG) algorithm (Lillicrap et al., 2015) to train option policies.
As a result, we compare against a goal-conditioned version of DDPG as our goal-conditioned RL baseline.
In the same way that our algorithm samples a salient event every 30 episodes, the goal-conditioned DDPG samples a goal state at random every 30 episodes.
It gets a positive terminal reward for successfully reaching this goal.
All hyper-parameters for our implementation of DCO were taken from Jinnai et al.
In the same way that DSG generates a new salient event every 30 episodes, so does
DCO.
Unlike DSG, DCO requires the number of skills to learn to be speciﬁed as an input to the algorithm.
We evaluated DCO with 3 , 4 , 5 , 10 options and present the best performing ones in ﬁgure 5.
A.6. System Architecture
Figure 10 shows the system architecture of the deep skill graph agent.
Both the salient event discovery module and the deep skill chaining agent (Skill Discovery module) may ﬁnd salient events in the MDP, all of which are stored in a single list.
When the agent is in a state inside the graph and there exists a path in the graph to the chosen target event, the planner selects the option to execute.
If the agent is outside the graph or when there is no path from the current state to the selected goal state, the policy-over-options in the DSC module picks the option to execute.
All options execute their closed loop policies until they reach their termination set or timeout at 200 steps.
A.7. The x-y prior
Option initiation sets are learned over the same state variables that factor in the reward function.
This is the same assumption as related work (Bagaria & Konidaris, 2020;
Levy et al., 2019; Nachum et al., 2018; Sharma et al., 2020;
Eysenbach et al., 2019a).
A.8. DDPG Hyperparameters
We used DDPG to train option policies and as the basis for our goal-conditioned RL baseline.
We did not try to optimize over the space of DDPG hyperparameters and used the ones reported in previous work (Lillicrap et al., 2015; Fujimoto et al., 2018).
[Table]
A.9. Compute Infrastructure
We used 1 NVIDIA GeForce 2080 Ti and 4 NVIDIA
GeForce 2070 Ti GPUs to perform all experiments reported in this paper.