Learning Options via Compression
Anonymous Author(s)
AffiliationAddress email
Abstract
Identifying statistical regularities in solutions to some tasks in multi-task reinforcement learning can accelerate learning new tasks.
Skill learning offers one way ofextracting these regularities by decomposing pre-collected experience into a sequence of skills.
A popular approach to skill learning is maximizing the likelihoodof the pre-collected experience with latent variable models.
However, there areoften many different solutions that maximize the likelihood equally well, including degenerate solutions.
To address this underspecification, we propose a newobjective that combines the maximum likelihood objective with a penalty on thedescription length of the skills.
This penalty incentivizes the skills to maximallyidentify and extract common structure from the experiences.
We demonstrate theeffectiveness of our method on a multi-task benchmark from prior work.
Further,while most prior works in the offline multi-task setting focus on low-dimensionaltasks, we demonstrate that our method can scale to challenging tasks with imageobservations.
Additionally, the acquired skills can be used to solve downstreamtasks with up to 8 × fewer samples compared to skills from maximizing likelihood.
Introduction
Reinforcement learning (RL) algorithms for solving tasks tabula rasa are sample inefficient [29, 8].
However, many domains, such as autonomous driving [10] and robotics [15] contain a wealth ofpre-collected experience for various tasks, which can be leveraged to quickly solve new tasks.
Thispre-collected experience can be used in numerous ways, including learning a dynamics model [18, 37],learning compact representations of observations [83], learning behavioral priors [74], and extractingmeaningful skills or, more specifically, options [76, 41, 2, 57] for hierarchical reinforcement learning(HRL) [6].
Our work studies this last approach.
Given the experience, skill learning methods extract commonly occurring behaviors.
For example,in an indoor navigation task, skills may correspond to navigating to different rooms.
Once learned,these skills can be used in place of low-level actions to quickly solve new tasks via planning [67, 49]or RL [53, 46, 50].
Prior methods learn skills by maximizing the likelihood of the pre-collected experience [42, 41, 2, 86].
However, this maximum likelihood objective (or the lower bounds on it) is underspecified : it oftenadmits many solutions, only some of which would help learning new tasks (see Figure 1).
For example,one degenerate solution is to learn a single skill for each entire trajectory; another degenerate solutionlearns a separate skill for each action.
Both of these decompositions can perfectly reconstruct thepre-collected experiences (and, hence, maximize likelihood), but they are of limited use for learningto solve new tasks.
Empirically, we observe the underspecification problem can occur even on simpletasks: maximizing likelihood yields unnatural decompositions that achieve the same likelihood assemantically meaningful decompositions.
We will address this underspecification problem by introducing an additional objective which biasesskill learning to acquire skills that can accelerate the learning of new tasks.
To construct such anobjective, we make two key observations:
• Skills (i.e., options) must extract reusable temporally extended structure from the precollected experiences to accelerate learning new tasks.
• Compressing data also requires identifying and extracting common structure.
Hence, we hypothesize that compression is a good objective for skill learning.
We formalizethis notion of compression with the information-theoretic principle of minimum description length(MDL) [66].
Concretely, we combine the maximum likelihood objective from prior work with a newterm to minimize the number of bits required to represent the pre-collected experience with skills.
Maximizing likelihood forces the skills to encode the experience.
Then our compression objectiveforces the skills to find common structure by our second observation.
Crucially, while methodsrelated to compression typically involve discrete optimization and hence are not differentiable, wealso provide a fully differentiable model built on top of variational temporal abstraction (VTA)
[38],which optimizes our objective (Section 4).
Overall, our main contribution is an objective for extracting skills from offline experience, whichaddresses the underspecification problem with the maximum likelihood objective.
We call theresulting approach L OVE (
L earning O ptions V ia compr E ssion).
Using the standard multi-taskbenchmark from prior work [41], we demonstrate that L OVE learns skills that enable faster RL andare more semantically meaningful compared to prior methods.
While prior works focus on tasks withlow-dimensional observations, we demonstrate that our regularizer allows L OVE to scale to taskswith image observations.
Related Works
We study the problem of using offline experience from one set of tasks to quickly learn new tasks.
While we use the experience to learn skills, prior works also consider other approaches of leveragingthe offline experience, including to learn a dynamics model [37], to learn a compact representation ofobservations [83], and to learn behavioral priors for exploring new tasks [74].
We build on a rich literature on extracting skills [76] from offline experience [60, 21, 42, 71, 41, 7,64 , 88, 87, 45, 2, 69, 68, 57, 86, 51, 89, 65, 77, 84].
These works predominantly learn skills using alatent variable model, where the latent variables partition the experience into skills, and the overallmodel is learned by maximizing (a lower bound on) the likelihood of the experiences.
This approachis structurally similar to a long line of work in hierarchical Bayesian modeling [23, 9, 20, 48, 36, 14,47, 38, 27, 16].
We also follow this approach and build off of VTA [38] for the latent variable model,but differ by introducing a new compression term to address the underspecification problem withmaximizing likelihood.
Zhang et al.
[86] show that maximizing likelihood with variational inferencecan itself be seen as a form of compression, but their proposed approach compresses the latentvariables at each time step, which does in general yield optimal compression as our compressionobjective does, and further only learns open-loop policies that do not condition on the state.
We notethat several other approaches [87, 22] also use compression, but only compress action sequences andnot the state, which also yields open-loop skills.
In contrast, L OVE provides a fully-differentiableobjective that can compress the actions conditional on the state to produce closed-loop skills.
Beyond learning skills from offline experience, prior works also consider learning skills with additional supervision [3, 61, 72, 34, 75], from online interaction without reward labels [24, 19, 17, 70, 5],and from online interaction with reward labels [78, 43, 4, 81, 28, 56].
Preliminaries
Problem setting.
We consider the problem of using an offline dataset of experience to quicklysolve new RL tasks, where each task is identified by a reward function.
The offline dataset is a set oftrajectories D
= { τ i } Ni =1 , where each trajectory is a sequence of states x ∈ X and actions a ∈ A : τ i = { x 0 , a 0 , x 1 , . . . } .
Each trajectory is collected by some unknown policy interacting with a
Markov decision process (MDP) with dynamics P ( x t +1 | x t , a t ) .
Following prior work [18, 1, 15],we do not assume access to the rewards (i.e., the task) of the offline trajectories.
Using this dataset, our aim is to quickly solve a new task.
The new task involves interacting withthe same MDP as the data collection policy in the offline dataset, except with a new reward function
R ( x , a ) .
The objective is to maximize the expected returns in as few time steps of interaction aspossible.
Importantly, this reward function is only provided after the method has extracted skills fromthe offline dataset.
Variational Temporal Abstraction.
Our method builds upon variational temporal abstraction(VTA)
[38], which is a method for decomposing non-control sequential data (i.e., data withoutactions) into subsequences.
We overview VTA below and extend it to handle actions it Section 4.1.
VTA assumes that each observation x t is associated with anabstract representation s t and a subsequence descriptor z t .
An additional, binary random variable m t indicates whethera new subsequence begins at the current observation.
VTAuses the following generative model:
Determine whether to begin a new subsequence:
m t ∼ p ( m t | s t − 1 ) .
A new subsequence alwaysbegins on the first time step, i.e., m 1 = 1 .2.
Sample the descriptor:
z t ∼ p ( z t | z <t , m
t ) .
If m t =0 , the previous descriptor is copied, i.e., z t =
z t − 1 .
Sample abstract representation: s t ∼ p ( s <t | z t , m t ) .
Sample the observation: x t ∼ p ( x t | s t ) .
We visualize this sampling procedure in Figure 2.
Formally,we can write this generative model as:
[Equation]
110 VTA maximizes the likelihood of the observed data under this latent variable model using the standardevidence lower bound (ELBO):
[Equation]
This lower bound holds for any choice of q θ .
VTA chooses to factor this distribution as:
[Equation]
As discussed above and as we observe in Section 6, the maximum likelihood objective is underspecified and may yield degenerate or unhelpful solutions.
We address this in the next section with a newobjective that compresses the latent variables z , while also maximizing the ELBO.
L OVE : Learning Options via Compression
In this section, we describe our method for learning skills from pre-collected experience.
We firstextend VTA to handle experience labeled with actions, and then introduce a compression objectivethat encourages extracting common structure from the experience.
4.1 A Model for Interaction Data
We now extend the VTA model [38] to handle sequential data labeled with actions, where descriptorsz now represent skills.
From a high level, the model partitions a trajectory into subsequences withthe boundary variables m and labels each subsequence as a skill z .
We wish to learn a state-conditional policy rather than the joint distribution of the whole trajectory.
To do this, we write a new generative model for the actions a 1: T conditional on the state x 1: T :p (
[Equation]
[Equation] [Equation]
=
T
[Equation]
This differs from the original VTA generative model in two ways: (1) we introduce a p ( a t | s t ) term 127 that indicates how actions are sampled; and (2) the distributions over z t and s t do not depend on allprevious z <t and s <t to encode the Markov property.
We then augment the variational distribution 129 such that the posterior over skills z also depends on actions:
[Equation]
Overall, this yields a model with 3 learned components:
• A state abstraction posterior q θ ( s t | z t , x t ) and decoder p θ ( a t | s t ) , which together form adistribution over actions conditioned on the skill z t and current state x t .
We refer to these jointly
as the skill policy π θ ( a t | z t , x t ) =
E s t ∼ q θ ( s t | z t , x t ) [ p θ ( a t | s t )] .
• A termination policy q θ ( m t | x 1: t ) , which determines whether the previous skill terminates or not.
• A skill posterior q θ ( z t | z t − 1 , m t , x 1: T , a 1: T ) , which outputs the current skill z t conditioned
on all states x 1: T and actions a 1: T .
This depends on m t and z t − 1 , since the boundary variables
determine whether the previous skill z t − 1 terminates: when m t = 0 , the previous skill does not
terminate
and z t =
z t − 1 .
Once this model is learned, the skill policy and termination policy represent the skills, without aneed for the skill posterior: Given a skill variable z , the skill policy encodes what actions the skilltakes, and the termination policy determines when the skill stops.
In the next section, we describe ourobjective for learning this model.
Then, in Section 5, we describe how we use the skill policy andtermination policy to learn new tasks.
4.2 Discovering Structure via Compression
As previously discussed, the maximum likelihood objective is underspecified for skill learning,because many skills can maximize likelihood, independent of whether they are useful for learningnew tasks.
In this section, we address this with a new objective that attempts to measure how usefulskills will be for learning new tasks in terms of compression.
Our objective is based on the intuitionthat effectively compressing a sequence of data requires factoring out common structure, and factoringout common structure is critical for learning useful skills.
We measure the complexity of a learned skill decomposition as the amount of information requiredto communicate the sequence of skills that encode the pre-collected experience.
The latent variablemodel introduced in the previous section encodes each trajectory as a sequence of skills z 1: T andboundaries m 1: T .
However, using the skill and termination policies, it is possible to recover eachtrajectory from only the skill variables at the boundary points: i.e., at the time steps { t ∈ [ T ] | m t =1 } .
For a prior on skills p z , an optimal code requires − log p z ( z t ) bits to send a skill z t [13, Chapter5.2].
Hence, the expected code length of communicating a trajectory x 1: T is:
[Equation]
where the expectation is under trajectories ( x 0 , a 0 , . .
, x T ) from the pre-collected experience andsampling z 1: T from the skill posterior and m 1: T from the termination policy.
The choice of priorthat minimizes the average code length is one that equals the empirical distribution of skills under thepre-collected experience [13, Chapter 5.3]:
[Equation]
( θ
[Equation]
p
[Equation]
) , where p ⋆ z ( z ) = 1 n s E x 1: T , m 1: T , z 1: T (cid:34) T (cid:88) t = δ ( z t = z ) m t (cid:35) , and n s ≜ E x 1: T , m 1: T (cid:104)(cid:80) Tt =1 m t (cid:105) .
In Appendix A.1, we show that this marginal p ⋆ z is a properdensity for both continuous and discrete z .
Substituting in this optimal choice of prior, we can showthat the code length can be expressed as the marginal entropy H p ⋆ z
[ z ] times the number of skills pertrajectory n s (see Appendix A.2 for proof):
[Equation]
Intuitively, this is equal to the average code length of a skill multiplied by the the average number ofskills per trajectory.
Note that compression is only beneficial if the model achieves high likelihood ofthe data.
We capture this by solving the following constrained optimization problem:
[Equation]
where L ELBO ( θ ) a negated evidence lower bound on the log-likelihood (detailed in Appendix B).
Remarks.
We discuss a connection between this objective and variational inference in Appendix A.3.
Additionally, while this work focuses on the RL setting, our objective generally appliesto sequential modeling problems.
4.3 Connections to Minimal Description Length
Our approach closely relates to the minimum description length (MDL) principle [66].
This principleequates learning with finding regularity in data, which can be used to compress the data.
Informally,the best model is the one that encodes the data with the lowest description length.
Given a model θthat encodes the data D into some message, one way to formalize the description length of the data
L ( D ) is with a crude two-part code [25].
This decomposes the description length of the data as thelength of the message plus the length of the model:
[Equation]
In our case, the message is the latent variables z t at the boundary points {
t ∈
[ T ] | m t = 1 } , whichcan be decoded into the data D with the skill and termination policies (representing the model).
Hence,our approach can be seen as an instance of minimizing the description length
L ( D ) .
Optimizingour objective in Equation 1 is equivalent to minimizing the message length
L ( D | θ ) .
While wedo not directly attempt to minimize the model length term L ( θ ) , many works [58, 79] indicate thatdeep learning has implicit regularization , which biases optimization toward low complexity solutionswithout an explicit regularizer.
In general, computing the true description length or appropriatenotion of complexity for neural networks is a tall order [59, 85, 35].
However, there is a rich spaceof methods to be explored here and we therefore leave the extension of our approach to directlyminimizing the model length term for future work.
4.4 A Practical Implementation
Model.
We instantiate our model by defining discrete skills z ∈
[ K ] , state abstractions s ∈ R d ,and binary boundary variables m ∈ { 0 , 1 } .
We parameterize all components of our model as neuralnetworks.
See Appendix D for architecture details.
Optimization.
We apply Gumbel-softmax [33, 52] to optimize over the discrete random variablesz and m .
We rewrite the compression objective L CL as a product of n s and H p
⋆ z
[ z t ] in Equation 1to improve the stability of optimization.
This rewriting allows computing a gradient in terms of adistribution over the skill variables p ⋆ z , rather than samples z t , which yields a more accurate finitesample gradient.
Empirically, this leads to stabler optimization and convergence to better solutions.
We approximate the optimal skill prior p ⋆ z using our skill posterior as:
[Equation]
where (cid:98) E denotes the empirical expectation over minibatches of x 1: T , a 1: T from D and sampling 201 m 1: T from the termination policy.
We solve the constrained optimization problem (Equation 2)with dual gradient descent on the Lagrangian.
In Appendix G, we summarize the overall trainingprocedure in Algorithm 1 and report details about the Lagrangian.
Enforcing a minimum skill length.
Though degenerate solutions such as skills that only take asingle action achieve a low value in our compression objective, empirically, such solutions createlocal optima that are difficult to escape.
To avoid this, we mask the boundary variables during trainingto ensure that each skill z t operates for at least T min = 3 time steps.
We find that when the skills are atleast minimally temporally extended, optimization of the compression objective appears to be stablerand achieves better values.
We remove these masks at test time, when we learn a task with our skills.
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Using the Learned Skills for HRL
We now describe how we quickly learn new tasks, given the skills learned from the pre-collectedexperience.
Overall, we simply augment the agent’s action space with the learned skills and learn anew policy that can take either low-level actions or learned skills, similar to Kipf et al.
However,our compression objective may result in unused skills, since this decreases the encoding cost of theused skills.
Therefore, we first filter down to only the skills that are used to compress the pre-collectedexperience by selecting the skills where the marginal is over some threshold α :
[Equation]
Then, on a new task with action space A , we train an agent using the augmented action space
A + = A ∪ Z .
When the agent selects a skill z ∈ Z , we follow the procedure in Algorithm 2 in the
Appendix.
We take actions following the skill policy π θ ( a t | z , x t ) (lines 2–3), until the terminationpolicy q θ ( m t | x 1: t ) outputs a termination m t = 1 (line 5).
At that point, the agent observes thenext state x t +1 and selects the next action or skill.
A Didactic Experiment: Frame Prediction
Before studying the RL setting, we first illustrate the effect of L OVE in the simpler setting of sequentialdata without actions.
In this setting, we use the original VTA model, which partitions a sequencex 1: T into contiguous subsequences with the boundary variables m and labels each subsequence witha descriptor z , as described in Section 3.
Here, VTA’s objective is to maximize the likelihood of thesequence x 1: T , and our compression objective applies to this setting completely unmodified.
Wecompare L OVE and only maximizing likelihood, i.e., VTA [38], by measuring how well the learnedsubsequences correspond to underlying patterns in the data.
To isolate the effect of the compressionobjective, we do not enforce a minimum skill length.
Dataset.
We consider two simple datasets Simple Colors and Conditional Colors , which consistof sequences of 32 × 32 monochromatic frames, with repeating underlying patterns.
Simple Colorsconsists of 3 patterns: 3 consecutive yellow frames, 3 consecutive blue frames and 3 consecutivegreen frames.
The dataset is generated by sampling these patterns with probability 0.4, 0.4, and0.2 respectively and concatenating the results.
Each pattern is sampled independent of history.
Thedataset is then divided into sequences of 6 patterns, equal to 6 × 3 = 18 frames.
In contrast to Simple Colors , Conditional Colors tests learning patterns that depend on history.
Itconsists of 4 patterns: the 3 patterns from Simple Colors with an additional pattern of 3 consecutivepurple frames.
The dataset is generated by uniformly sampling the patterns from Simple Colors .
To make the current timestep pattern dependent on the previous timestep, the yellow frames arere-colored to purple, if the previous pattern was blue or yellow.
As in Simple Colors , the dataset isdivided into sequences of 6 patterns.
In both datasets, the optimal encoding strategy is to learn 3 subsequence descriptors (i.e., skills), onefor each pattern in Simple Colors .
In Conditional Colors , the descriptor corresponding to yellowin Simple Colors either outputs yellow or purple, depending on the history.
This encoding strategyachieves an expected code length of 6 .
32 nats in Simple Colors and 6 .
59 nats in Conditional Colors .
Results.
We visualize the learned descriptors and boundaries in Figure 3.
L OVE successfullysegments the sequences into the patterns and assigns a consistent descriptor z to each pattern.
Incontrast, despite the simple structure of this problem, VTA learns subsequences that span overmultiple patterns or last for fewer timesteps than a single pattern.
Quantitatively, we measure the (1) precision, recall, and F1 scores of the boundary prediction, (2) the
ELBO of the maximum likelihood objective and (3) the average code length L CL in Table 1.
Whileboth VTA and L OVE achieve approximately the same negated ELBO (lower is better), L OVE recoversthe correct boundaries with much higher precision, recall, and F1 scores.
This illustrates that theunderspecification problem can occur even in simple sequential data.
Additionally, we find that L OVEachieves an encoding cost close to the optimal value.
One interesting, though rare, failure mode isthat L OVE sometimes even achieves a lower encoding cost than the optimal value by over-weightingthe compression term and imperfectly reconstructing the data.
In Appendix C, we conduct a ablationstudy on the weights of L CL in optimization.
Experiments
[Figure]
We aim to answer three questions: (1) Does L OVE learn semantically meaningful skills?
(2) Do skills learned with L OVEaccelerate learning for new tasks?
(3) Can L OVE scale to highdim pixel observations?
To answer these questions, we learnskills from demonstrations and use these skills to learn newtasks, first on a 2D multi-task domain, and later on a 3D variant.
Multi-task domain.
We consider the multi-task 10 × 10 gridworld introduced by Kipf et al.
[41], a representative skill learning approach (Figure 4).
This domain features a challengingcontrol problem that requires the agent to collect up to 5 objects,and we later add a perception challenge to it with the 3D variant.
We use our own custom implementation, since the code from
Kipf et al.
[41] is not publicly available.
In each task, 6 objectsare randomly sampled from a set of N obj =
10 different objects and placed at random positions in thegrid.
Additionally, impassable walls are randomly added to the grid, and the agent is also placed at arandom initial grid cell.
Each task also includes an instruction list of N pick = 3 or 5 objects that theagent must pick up in order.
Within each task, the agent’s actions are to move up, down, left, right, or to pick up an item at theagent’s grid cell.
The state consists of two parts: (1) a 10 × 10 × ( N obj +2) grid observation, indicatingif the agent, a wall, or any of the N obj different object types is present at each of the 10 × 10 gridcells; (2) the instruction list encoded as a list of ordered one-hot integers.
Following Kipf et al.
[41],we consider two reward variants: In the sparse reward variant, the agent only receives +1 rewardafter picking up all N pick objects in the correct order.
In the dense reward variant, the agent receives+1 reward after picking up each specified object in the correct order.
Our dense reward variant isslightly harder than the variant in Kipf et al.
[41], which gives the agent +1 reward for picking upobjects in any order.
The agent receives 0 reward in all other time steps.
The episode ends when theagent has picked up all the objects in the correct order, or after 50 timesteps.
Pre-collected experience.
We follow the setting in Kipf et al.
We set the pre-collectedexperience to be 2000 demonstrations generated via breadth-first search on randomly generated taskswith only N pick = 3 and test if the agent can generalize to N pick = 5 when learning a new task.
Thesedemonstrations are not labeled with rewards and also do not contain the instruction list observations.
Points of comparison.
To study our compression objective, we compare with two representativesof learning skills via the maximum likelihood objective:
• VTA [38], modified to handle interaction data as in Section 4.1.
• Discovery of deep options (DDO) [21].
Since the original implementation is unavailable, weimplement DDO’s maximum likelihood objective and graphical model on top of VTA’s variational
inference optimization procedure, instead of expectation-gradients used in the original paper.
For
fairness
, we also
compare with variants of these that implement the minimum skill lengthconstraint from Section 4.4.
CompILE [41] is another approach that learns skill by maximizinglikelihood and was introduced in the same paper as this grid world.
Because the implementationis unavailable, we do not compare with it.
However, we note that CompILE requires additionalsupervision that L OVE , VTA, and DDO do not: namely, it requires knowing how many skills eachdemonstration is composed of.
This supervision can be challenging to obtain without already knowingwhat the skills should be.
[Figure]
t u r n s
N pick = 3 (Dense)
[Figure]
Sparse
[Figure]
Since latent variable models are prone to local optima [44], it is common to learn such models withmultiple restarts [55].
We therefore run each method with 3 random initializations and pick the bestmodel according to the compression objective L CL for L OVE and according to the ELBO of themaximum likelihood objective L ELBO for the others.
Notably, this does not require any additionaldemonstrations or experience.
We begin by analyzing the skills learned from the pre-collected experience before analyzing the utilityof these skills for learning downstream tasks in the next sections.
7.1 Analyzing the Learned Skills
Precision
Recall
F
DDO [21] 0.
0.32 DDO + min.
skill length 0.
0.35 VTA [38] 0.
0.32 VTA + min.
skill length 0.
0.36 L OVE (ours) 0.
[Table]
We analyze the learned skills by comparing
[Table]
them to a natural decomposition that partitions
[Table]
the demonstration in N pick sequences of movingto and picking up an object.
Specifically, wemeasure the precision and recall of each methodin outputting the correct boundaries of these
N pick sequences.
We find that only L OVE recovers the correctboundaries with both high precision and recall(Table 2).
Visually examining the skills learnedwith L OVE shows that each skill moves to an object and picks it up.
We visualize L OVE ’s skills in
Appendix H.
In contrast, maximizing likelihood with either DDO or VTA learns degenerate skillsthat output only a single action, leading to high recall, but low precision.
Adding the minimumskill length constraint to these approaches prevents such degenerate solutions, but still does notlead to the correct boundaries.
Skills learned by VTA with the minimum skill length constraintexhibit no apparent structure, but skills learned by DDO with the minimum skill length constraintappear to move toward objects, though they terminate at seemingly random places.
These resultsfurther illustrate the underspecification problem.
All approaches successfully learn to reconstruct thedemonstrations and achieve high likelihood, but only L OVE and DDO with the minimum skill lengthconstraint learn skills that appear to be semantically meaningful.
7.2 Learning New Tasks
Next, we evaluate the utility of the skills for learning new tasks.
To evaluate the skills, we sample anew task in each of 4 settings: with sparse or dense rewards and with N pick = 3 or N pick = 5 .
Thenwe learn the new tasks following the procedure in Section 5: we augment the action space with theskills and train an agent over the augmented action space.
To understand the impact of skills, we alsocompare with a low-level policy that learns these tasks using only the original action space, and thesame low-level policy that incorporates the demonstrations by pre-training with behavioral cloning.
We parametrize the policy for all approaches with dueling double deep Q-networks [54, 82, 80] withϵ -greedy exploration.
We report the returns of evaluation episodes with ϵ = 0 , which are run every10 episodes.
We report returns averaged over 5 seeds.
See Appendix F.3 for full details.
Results.
Overall, L OVE learns new tasks across all 4 settings comparably to or faster than bothskill methods based on maximizing likelihood and incorporating demonstrations via behavior cloning(Figure 5).
More specifically, when N pick = 3 , DDO with the min.
skill length constraint performscomparably to L OVE , despite not segmenting the demonstrations into sequences of moving to andpicking up objects.
We observe that while a single DDO skill does not move to and pick up an object, the skills consistently move toward objects, which likely helps with exploration.
Imitating thedemonstrations with behavior cloning also accelerates learning when N pick = 3 with dense rewards,though not as much as L OVE or DDO, and yields little benefit in all other settings.
Skill learning with
VTA yields no benefit over no skill learning at all.
Recall that N pick = 3 in all of the demonstrations.
L OVE learns new tasks in the generalization settingof N pick = 5 much faster than the other methods.
With dense rewards, DDO (min.
skill length) alsoeventually solves the task, but requires over 8 × more timesteps.
The other approaches learn to pickup some objects, but never achieve optimal returns of 5.
With sparse rewards, L OVE is the onlyapproach to achieve high returns, while all other approaches achieve 0 returns.
This likely occursbecause this setting creates a challenging exploration problem, so exploring with low-level actions orskills learned with VTA or DDO rarely achieves reward.
By contrast, exploring with skills learnedwith L OVE achieves rewards much more frequently, which enables L OVE to ultimately solve the task.
7.3 Scaling to High-dimensional Observations
Prior works in the offline multi-task setting have learned skillsfrom low-dim states and then transferred them to high-dimpixel observations [65], or have learned hierarchical modelsfrom pixel observations [38, 21].
However, to the best of ourknowledge, prior works do not directly learn skills from highdim pixel observations, only from low-dim states.
Hence, inthis section, we test if L OVE can scale up to high-dim pixelobservations by considering a challenging 3D image-basedvariant of the multi-task domain above, illustrated in Figure 6.
The goal is still to pick up N pick objects in the correct order,where objects are blocks of different colors.
Each task includes 3 objects randomly sampled froma set of N obj = 6 different colors.
The observations are 400 × 60 × 3 egocentric panoramic RGBarrays.
The actions are to move forward /
[Figure]
We consider only the harder sparse reward andgeneralization setting, where the agent must pickup N pick = 3 objects in the correct order, afterreceiving 2000 pre-collected demonstrations of theapproximate shortest path of picking up N pick = 2objects in the correct order.
We use the samehyperparameters as in multi-task domain and onlychange the observation encoder and action decoder(full details in Appendix D.4).
Figure 7 shows the results.
Again, L OVE learns skills that each navigateto and pick up an object, which quickly achieves optimal returns, indicating its ability to scale tohigh-dim observations.
In contrast, the other approaches perform much worse on pixel observationsthan the previous low-dim observations.
DDO also makes progress on the task, but requires far moresamples, as it again learns skills that only operate for a few timesteps, though they move toward theobjects.
All other approaches fail to learn at all, including the variants of VTA and DDO without themin.
skill length, which are not plotted.
Conclusion
We started by highlighting the underspecification problem: maximizing likelihood does not necessarily yield skills that are useful for learning new tasks.
To overcome this problem, we drew aconnection between skill learning and compression and proposed a new objective for skill learningvia compression and a differentiable model for optimizing our objective.
Empirically, we found thatthe underspecification problem occurs even on simple tasks and learning skills with our objectiveallows learning new tasks much faster than learning skills by maximizing likelihood.
Still, important future work remains.
L OVE applies when there are useful structures that can beextracted from multiple trajectories.
This is often present in multi-task demonstrations, which solverelated tasks in similar ways.
However, an open future challenge for adapting to general offline datais to ensure that the learned skills do not overfit to potentially noisy or unhelpful behaviors oftenpresent in offline data.
In addition, We showed a connection between skill learning and compressionby minimizing the description length of a crude two-part code.
However, we only proposed a wayto optimize the message length term and not the model length.
Completing this connection byaccounting for model length may therefore be an exciting direction for future work.
A
Derivations
A.1 p ⋆ z is a valid density
Proposition A.1.
p ⋆ z ( ξ ) is a proper probability density function.
Proof.
Recall that
[Equation]
A function is a probability density function if it is non-negative, continuous, and integrates to 1.
By construction, p ⋆ z ( ξ ) is non-negative since m t ≥ 0 and δ ( z t = ξ ) ≥ 0 .
Further, δ ( z t = ξ )is continuous in ξ so the numerator E x 1: T , m 1: T , z 1: T
[Equation]
Tt
[Equation]
1 δ ( z t = ξ ) m t (cid:105) , an expectation containingδ ( z t = ξ ) , is also continuous in ξ .
Finally,(cid:90)
[Equation]
Fubini’s theorem)
[Equation]
=
[Equation]
Therefore, p ⋆ z ( ξ ) satisfies all 3 properties of a probability density function.
Remarks.
This result generalizes to the case for discrete z by replacing the delta with an indicatorfunction and integration with summation.
A.2 Two expressions of L CL
Proposition A.2.
The expected coding length of a trajectory is equal to the expected number skillsmultiplied by the marginal entropy:
[Equation]
Proof.
−
[Equation]
=
[Equation]
(cid:88)
[Equation]
(Fubini’s theorem) =
[Equation]
(cid:90)
[Equation] [Equation]
(Linearity of expectation)
[Equation]
=
E x 1: T , m 1: T , z 1: T (cid:34) T (cid:88) t
= m t (cid:35) (cid:124) (cid:123)(cid:122) (cid:125) n s (cid:90) ξ − p ⋆ z ( ξ ) log p ⋆ z ( ξ ) dξ (cid:124) (cid:123)(cid:122) (cid:125)
H p⋆ z
[ z ] = n
[Equation]
Remarks.
This result generalizes to the case for discrete z by replacing the delta with an indicatorfunction and integration with summation.
Further, this rewriting not only offers an insightfulinterpretation of the objective but also has different finite-sample gradient compared to the originalform.
We discuss its implication further in Section 4.4.
A.
But you just picked the wrong prior
Is the compression objective a prior?
Yes and no.
A bridge that connects variational inferenceand source coding is the bits-back coding argument [30, 31].
Under this scheme, the cost ofcommunicating a message is equal to the KL divergence between the approximate posterior andthe chosen prior distribution.
This is because the posterior often does not match the prior and thesender must use more bits to send the message compared to using the prior exactly.
In modern deeplatent variable models such as VAE [40], the prior is in general chosen to be an isotropic Gaussiandistribution or a sequence of isotropic Gaussian distributions that decomposes over the time steps.
The bit-back coding argument shows that variational inference as a coding scheme is asymptoticallyoptimal (in this limit of sending large numbers of messages), but it does not immediately guaranteegood representation learning .
As a thought experiments, imagine we choose the dimension of thelatent variable in a VAE to be equal to the dimension of the data itself.
It is clear that the model is notincentivized to perform good representation learning, but the bit-backing coding scheme still applies.
Instead, good representation can only emerge if the prior is chosen properly.
Contrary to this prevailing design choice, our latent variables z 1: T and m 1: T are not independentfrom each other and the timesteps are not independent from each other.
In fact, the interactionbetween the latent variables is crucial for compressing the trajectory optimally in our framework.
Therefore, our objective takes into account the global structure of the latent variables.
Note that
L CL also has the natural interpretation of measuring the average number of bits that is required tosend a trajectory under the coding scheme of this work.
As both quantities measure the cost of thecommunication, the philosophical parallel between KL divergence and L CL is likely not a coincidence.
In practice, we observe that L CL also has comparable regularization effect on the latent code, and insome cases it is possible to learn a good model without any KL divergence if L CL is present.
Fromthis perspective, our compression objective is indeed a “prior" insofar as it specifies the desiredproperties of the posterior.
Nonetheless, it is an open question whether L CL has a precise counterpart in Bayesian inference,that is, it is not immediately obvious if it can be written as the KL divergence between the posteriorq ( z 1: T , m 1: T | x 1: T ) and some prior distribution p ( z 1: T , m 1: T ) .
One complication, for example, 679 is the fact that the log density of m 1: T does not participate in the computation of L CL .
It may bepossible to construct some form of hierarchical priors that enables a fully Bayesian interpretation of
L CL , but it is also well-known that the MDL framework can accommodate codes that are not Bayesian(e.g.
, the Shtarkov normalized maximum likelihood code [73]).
We do not offer a definitive answerto this question in this work, but the connection between Bayesian inference and L CL is certainly aninteresting direction for future works.
B Evidence Lower BoundB.1 VTA
We reproduce the data generating process of Kim et al.
[38] here:p
[Equation]
VTA [38] shows that the ELBO can be written as:
[Equation]
Like most ELBO, this ELBO can be deconstructed into a reconstruction term:
[Equation]
, and the remaining KL term L KL ( θ , ϕ ) .
B. L OVE
We reproduce the data generating process of L OVE here.
Instead of using ϕ for the parameters ofpriors and decoder, we will merge all parameters into θ :p ( a 1: T , z 1: T , s 1: T , m 1: T | x 1: T ) =
T
[Equation]
t =
[Equation]
p
[Equation]
m
[Equation]
) p ( z
[Equation]
| x t , z <t , m t −
[Equation]
For L OVE , the ELBO is re-written as:
[Equation]
( s t | s t − 1 , m t − 1 )) .
Since z t is a categorical distribution, we chose the uniform prior p ( z t ) similar to Oord et al.
Similar to VTA, the loss is also decomposed into a reconstruction term and a KL term.
C Sensitivity of λ
Here we study how sensitive L OVE is to the choice of λ , the coefficient of L CL .
We tested 5 values ofλ ∈ [0 . 01 , 1] on Simple and Cond.
Colors and applied dual gradient descent to automatically tune λ .λ
=0 .λ
=0 .λ =0 .λ
=
.λ
=1 .λ (dual GD)
VTA
Simple (F1) .
67 ±
[Table] [Table]
80 ±
[Table]
[Table] [Table]
Cond.
[Table] [Table]
[Table]
We see that while λ can change the quality of solutions found, the performance is stable within a± 3 × regions around λ = 0 .
Further, using dual GD consistently outperforms handpicked λ withan ELBO threshold C slightly (e.g., 5% ) higher than the L ELBO achieved without compression, whichcan be obtained with minimal prior knowledge about the task.
Hence λ likely can be tuned withoutstrong prior knowledge.
This contrasts with the other methods such as Kipf et al.
[41] which requireknowing the number of skills in each demonstration to perform well while achieving the similar orbetter performance.
D Architecture
D.1 Frame Prediction
Our base architecture is similar to Kim et al.
and we highlight the difference blue.
Theparameters of the posteriors are implemented with neural networks.
First, the observation x 1: T areencoded into a lower dimension embedding ˆ x 1: T with a observation encoder .
The embedding is then passed through a boundary posterior decoder that outputs the parameters of m 1: T .
Theembedding is further passed through a GRU [11], f s-rnn-fwd which runs on ˆ x 1: T and give the cell stateh 1: T .
For z 1: T , we have two GRU’s with cell, f z-rnn-bwd and f z-rnn-bwd , which run on ˆ x 1: T in different 715 temporal direction (i.e., f z-rnn-bwd sees the future) to generate cell states c fwd 1: T and c bwd 1: T .
For eachtime step t , a candidate z ′ t is sampled from a n z -dimensional categorical distribution 2 whose logprobability is a linear projection of c fwd t ∥ c bwd t .
The sampled z ′ t are embedded linearly into a vectorof size 128.
If m 1 = 1 , the z t = z ′ t ; otherwise, z t =
z t − 1 3 .
Then we concatenate h t − 1 ∥
z t andlinearly project the vector into the mean and standard deviation of a isotropic Gaussian in R 8 .
A ˆ s t issampled from this Gaussian.
Th model also keeps two “belief" vectors that keep track of the history of the computation.
Thesevectors are modulated by two RNN, f h-rnn and f c-rnn .
The abstraction belief c t = m t · f c-rnn ( z t , c t − 1 )+(1 − m t ) · c t − 1 .
The observation belief h t = m t c t − 1 + (1 − m t ) · f h-rnn (ˆ s t ∥ c t ∥
z t , h t − 1 ) .
Finally 724 the state abstraction is computed as the linear projection of s t = proj ( h t ∥
ˆ s t ) .
The projection layer is256 × 128 .
Finally, s t is passed through a observation decoder and decode into the reconstructionof x t .
Refer to Kim et al.
[38] for more details and an illustration of the computation graph.
For each convolutional layer or transposed convolution layer, the tuple’s values correspond to inputchannel, output channel, kernel size, stride, padding.
With that notation in mind, the specifichyperparameters for each components are:
• observation encoder : 4-layer convolutional neural network with {(3, 128, 4, 2, 1), (128,128, 4, 2, 1), (128, 128, 4, 2, 1), (128, 128, 4, 1, 0)} with ELU activation [12] and batchnormalization [32].
The output is flattened linearly projected to vector of size 128.• boundary posterior decoder is a 5-layer 1-D causal temporal convolution [62] with{(128, 128, 5, 1, 2) × 5, (128, 2, 5, 1, 2)} with ELU activation and batch normalization.
Theoutput is the logits for the m 1: T .•
f z-rnn-bwd , f z-rnn-fwd , f s-rnn-fwd , f c-rnn , f h-rnn : GRU with cell size 128• observation decoder : 4-layer transposed convolutional layers {(128, 128, 4, 1, 0),(128, 128, 4, 2, 1), (128, 128, 4, 2, 1), (128, 3, 4, 2, 1)} with ELU activation and batchnormalization on all but the last layer.
The last layer has no normalization and has tanhactivation.
Removing Boundary Regularization.
An important difference from Kim et al.
[38] is that we donot enforce maximum number of subsequence N max and the maximum length of subsequence l maxthrough the boundary prior (This effectively amounts to setting both to a value larger than the sequencelength).
These two hyperparameters’ effect are similar to that of picking the number of segments in
CompILE [41] and provide strong training signal.
This assumption is in general problematic since wecannot expect to know a priori what the good values are for these hyperparameters.
Kipf et al.
[41]demonstrates that the performance can suffer if this kind of supervision is wrong.
On the other hand,we do not assume anything about the the duration of the subsequence or how many subsequencesthere are in each demonstration.
We do, however, enforce a minimum length on the skill.
While it is largely an optimization choice, webelieve this prior is significantly weaker since useful options are almost always temporally extended.
Indeed, with only the minimum length constraint, VTA is unable to learn good skills conducive fordownstream tasks.
D.2 Interaction Data
For trajectories, we make some larger changes to the architecture to encode properties we want in amodel for learning options, e.g., Markov property.
Though similar in spirit, a large portion of thearchitecture is different from the base VTA model, so the difference will not be highlighted.
For the posterior, we have an observation encoder and an action encoder that embeds boththe state observation x 1: T and actions a 1: T to get embedding a 1: T and x 1: T each of size 128.
The 760 concatenation a t ∥ x t linear projected down to a vector of size 128.
The embedding is then passedthrough a boundary posterior decoder that outputs the parameters of m 1: T .
The embeddingis further passed through a GRU [11], f s-rnn-fwd which runs on ˆ x 1: T and give the cell state h 1: T .
For z 1: T , we have two GRU’s with cell, f z-rnn-bwd and f z-rnn-bwd , which run on ˆ x 1: T in different temporal 764 direction (i.e., f z-rnn-bwd sees the future) to generate cell states c fwd 1: T and c bwd 1: T .
For sampling z is done differently for interaction data for better gradient and representation learning.
Instead of doing straight-through estimator for the categorical random variable, we opt to usevector-quantization [63] with straight-through estimators.
First, c fwd t ∥ c bwd t is linearly projected intodimension 128.
Then a ReLU is applied and another 128 by 128 linear layer is applied.
The VQcodebook is of size n z × 128 .
Instead of taking argmax of the codebook, we approximate thedistribution to be proportional to the softmax over the distance over a temperature t V Q .
We cansample from this distribution and apply the straight-through gradient on the sampled code candidatez ′
: T .
If m 1 = 1 , the z t = z ′ t ; otherwise, z t =
z t − 1 .
We construct directly s t =
z t ∥ x t .
and linearly project the vector into the mean and standard deviationof a isotropic Gaussian R 8 .
s t is sampled from this Gaussian and then decoded with the actiondecoder into the reconstructed action in A .
For each convolutional layer or transposed convolution layer, the tuple’s values correspond to inputchannel, output channel, kernel size, stride, padding.
With that notation in mind, the specifichyperparameters for each components are:
• observation encoder :
This takes as input the 10 × 10 × N obj + 2 and is implementedas a two 2D convolutional layers with ReLU activations, followed by a linear layer witha ReLU activation and a linear layer with no activation.
The convolutional layers have
hyperparameters (12 , 64 , 3 , 1 , 0) and (64 , 64 , 3 , 1 , 0) respectively.
The first linear has inputdimension 6 × 6 × 64 and output dimension 128 .
The second linear layer has input and
output dimensions 128 .
• action encoder :
The actions are embedded with an embedding matrix with embeddingdimension 128.
• boundary posterior decoder is a 5-layer 1-D causal temporal convolution [62] with{(128, 128, 5, 1, 2) × 5, (128, 2, 5, 1, 2)} with ELU activation and batch normalization.
Theoutput is the logits for the m 1: T .
• f z-rnn-bwd , f z-rnn-fwd , f s-rnn-fwd : GRU with cell size
• action decoder : The decoder is implemented as three linear layers with ReLU activations,followed by a linear layer with no activation.
The (input, output) dimensions of these layersare as follows: (128 , 128) , (128 , 128) , (128 , 128) , (128 , 5) , where the last layer outputslogits over the actions.
D.3 3D Navigation Environment
The architecture we use here is almost identical to that of multi-task environment D.2, but wechange the output of the action decoder to be 4 and the observation encoder to be a 3-layerconvolutional neural networks with (3 , 32 , 5 , 2 , 0) , (32 , 32 , 5 , 2 , 0) , (32 , 32 , 4 , 2 , 0) , followed twolinear layers (7520 , 128) , (128 , 128) with a ReLU activation in between.
The same encoder is used for learning new tasks with HRL, but with an object embedding concatenatedto the output (details in Appendix D.4).
D.4 Hierarchical Reinforcement Learning
For all approaches, we parametrize the policy as a double dueling deep Q-network [54, 82, 80].
Theparametrization of the Q-function consists of a state embedding followed by two linear layer heads forthe state-value V θ ( s ) function and the advantage
A θ ( s, a ) function.
Then the Q-function is computedas Q θ ( s, a ) =
V θ ( s ) + A ( s, a ) − 1 |A| (cid:80)
a ′ ∈A
A ( s, a ) .
The state embedding is computed as embedding the grid state observation g as e ( g ) , embedding theinstruction state observation i as e ( i ) , and applying a final linear layer with output dimension 128 toa ReLU on their concatenation: Linear(ReLU([e(g); e(i)])) .
The grid embedding is computedusing the same architecture as the observation encoder in the previous section.
The instructionembedding is computed with a 16-dimensional embedding matrix.
E Hardware
All our experiments are conducted on a GeForce RTX 2080 Ti or a GeForce RTX A6000.
F HyperparametersF.1 Frame Prediction
For these set of experiments, we use the same architecture and hyperparameters as Kim et al.
The hyperparameters used for both Simple Colors and Conditional Colors are:
• β : 1 .
• λ (fixed) : 0 .
1• mini-batch size: 512• training iteration: 30000• n z :
F.2 Interaction Data
For training the interaction data model, we use the following hyperparameters:
• β :
– We found that the compression objective readily provides strong and sufficient regularization for the latent code.
Adding additional KL divergence often results in the modelnot being able to reconstruct the actions properly.
• λ :
– We use a adaptive scheduling that approximate dual gradient descent.
λ is initializedto 0 .
After every gradient step, λ is increased by 2 .
× 10 − 5 if L ELBO ≤ 0 .
(Sinceβ = 0 , this is effectively L rec ); otherwise, λ is decreased by 2 .
× 10 − 5 .
After eachupdate, the value of λ is clipped to [0 , 0 . 05] .
We find this setting provides the moststable training.
• mini-batch size: 64• training iteration: 20000• n z : 10• t VQ : 0.
F.3 Hierarchical Reinforcement Learning
We use the same hyperparameters for training the policy for all approaches.
Specifically, thesehyperparameter values are as follows:
• ϵ -greedy schedule: We linearly decay ϵ from 1 to 0 .
01 over 500 K timesteps for densereward settings and over 5 M timesteps for sparse reward settings.
• Discount factor γ : We use γ = 0 .
99 in all of the DQN updates.
• Maximum replay buffer size: 1M
• Minimum replay buffer size before updating: 500• Learning rate: 0.0001 with the Adam optimizer [39]• Batch size:
[Table]
• Update frequency: every 4 timesteps
[Table]
• Target syncing frequency: every 50K updates
[Table]
• Gradient ℓ 2 -norm clipping:
[Table]
• Marginal threshold α : 0.
[Table]
Since the demonstrations do not contain the instruction list observations, we set those to 0 in thedemonstrations for the behavior cloning approach.
G Algorithm
The Lagrangian for the unconstrained optimization problem is:
[Equation]
In standard KKT condition, the dual variable is introduced on the ELBO, but the two problems areequivalent by inverting the dual variable.
Since the only constraint on λ is that it is non-negative, thistransformation does not change the optimal solution.
Further note that we find that in many cases,choosing a fixed constant value for λ is sufficient for solving the problem.
1: Initialize model parameters θ 2: while not converged do 3:
Sample a trajectory ( x 0 , a 0 , . . .
, x T ) from the pre-collected experience D 4: for t = 1 , 2 , . . .
, T do 5:
Sample boundary conditioned on entire trajectory m t ∼ q θ ( m t | x 1: t ) 6:
Sample skill from skill posterior z t ∼ q θ
( z t | m t , x 1: t , a 1: t ) 7:
Sample abstraction from abstraction posterior s t ∼ q θ ( s t | z t , x t ) 8:
Compute probability of the correct action from the decoder p θ ( a t | s t ) 9: end for 10:
Compute lower bound on maximum likelihood objective L ELBO according to Equation11:
Compute compression objective L CL according to Equation12:
Update θ ← θ − η ∇ ( L ELBO + λ L CL ) and λ 13: end while
Algorithm z 1: while skill has not terminated do 2:
Compute state abstraction µ s t =
E s ∼ q θ ( s t | z , x t )
[ s ] 3:
Take action a t = arg max a p θ ( a | µ s t ) 4:
Observe next state x t + 5:
Terminate if E m t +1 ∼ q θ ( m t +1 | x 1: t +1 )
[ m t +1 ] > 0 .
5 6: end while
[Figure] [Figure]
-7- -357 V e r t
i c a l c h a n g e
Skill
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
-7- -3V
e r t i c a l c h a n g e
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
In Figure 8, we visualize the behavior of following each of L OVE ’s 10 skills on an example task.
Each skill moves to and picks up an object, and the total set of skills covers 3 of the 4 object types inthe task.
These skills pick up both nearby objects, such as the tree in skill 2, as well as more distantobjects, such as the diamond in skill 1.
This contrasts DDO and VTA, which do not learn skills thatmove to and pick up objects.
To further understand what each of L OVE ’s skills does, we analyze whether there is a correlationbetween each skill and either the type of object it picks up, or the location of the object it picks up.
We find that there appears to be little correlation between each skill and the type of object it picks up.
Plotting the frequency of picking up each object type by skill shows that each skill picks up eachobject type with roughly the same frequency.
Instead, there appears to be a correlation between eachskill and the location of the object it picks up, illustrated in Figure 9.
For example, skill 1 appears tospecialize in moving to and picking up objects that are above the agent, while skill 7 tends to pickup objects that are up and to the right of the agent.
It is also interesting to note that some skills aremuch more biased to move in certain direction (e.g., skill 1, 6, 7) while some appear to be moregeneral (e.g., 4, 5, 8, 9) and move in any direction.
Note in Figure 8, it may seem L OVE ’s skillsseem short / redundant.
This is due to the fact that L OVE ’s skills each pick up an object, so theynaturally appear short when the agent is close to the objects, as in Figure 8.
However, when the agent is far away from the objects, the skills still pick up the objects and are much longer.
From
Figure 9, we can see from the heatmap that even the skills that appear to do same thing in Figure 8have very distinct behaviors depending on the state they are in.
Also, recall L OVE imposes a sparsedistribution over skills (Section 5).
After filtering, skill 3 , 7 and 10 in Figure 8 are dropped sincethey have low marginal probability and are not used to describe a significant part of the trajectories.
Figure 8 includes redundant skills L OVE does not use; the used skills in the sparse distribution havelittle redundancy.
I Option Critic Results
[Figure]
Option critic [4] is a classical online HRL algorithm.
We show the results of running option critic 4on two of the four RL tasks we considered in Figure 10.
The best return achieved over the trainingis shown in Table 4.
We set the number of option to 8 following Bacon et al.
[4] and leave otherhyperparameters untouched.
N pick = 3 with dense reward is the easily setting and N pick = 5 withsparse reward is the hardest setting.
We see that in N pick = 3 , option critic is able to reach reward of 2(maximum possible is 3 ) at around 1M environment steps but the performance deterioates afterwards.
In N pick = 5 , the algorithm fails to make any meaningful progress.
These observations may suggestonline HRL algorithms like option critic may be insufficient for solving these tasks.
[Figure]
[Equation]
[Table]
J Comparison to Zhang et al. [86]
[Table]
Zhang et al.
[86] learn open-loop skills, which do not condition on the state and therefore cannotadapt to different states.
Further, the MDL used in Zhang et al.
[86] is equivalent to the variationalinference used by VTA (on a different graphical model), which can be seen as greedily compressing each skill independently, rather than compressing a whole trajectory as L OVE does.
Table 5 reportssegmentation results on the Color domain and grid world, akin to Tables 2 and 3.
Zhang et al.
[86]performs the same as VTA on the Color domains – as they both use variational inference and there isno state – which achieves lower precision / recall than L OVE .
On the grid world navigation, Zhanget al.
[86] fails to recover the boundaries, unlike L OVE , because skills that navigate to objects requireobserving the state.
This problem is shared by all open-loop approaches.
K Number of Initial Skills Ablation
[Table]
The number of skills to extract from demonstrations is often not known a priori, so choosing anappropriate value of the number of initial skills before filtering K is important.
Intuitively, wehypothesize that K can be set conservatively:
L OVE requires at least a minimum number of skillsin order to fit the behaviors in the demonstrations, but may be able to gracefully prune out skillsif K is set too high via the filtering described in Section 5.
We test this intuition by varying
K ∈ { 2 , 5 , 10 , 15 , 20 , 30 , 50 } and measuring the segmentation performance on the grid world multitask domain, and find that it appears to hold true in Table 6.
For smaller values of K , such as K = 2and K = 5 , L OVE ’s F1 scores significantly degrade.
However, performance remains high for allvalues where K is sufficiently large.
Hence, we conservatively sizing K to be a large number.
L Comparison to Different Regularizers
[Table]
Several prior methods encourage skills to act for more timesteps or prevent skills from switching toofrequently, which can similarly help avoid degenerate solutions like L OVE .
We note that such priormethods do not necessarily yield skills that help learn new tasks, while the maximum of L OVE ’sobjective achieves the information-theoretic limit of compressing the trajectories (under a fixedfunction class), which is not done in other works.
We empirically compare L OVE with several suchmethods on the segmentation of the multi-task grid world domain, including:
Entropy : This approach regularizes the entropy of skill marginal akin to
Shankar et al
which they
refer to
this as parsimony
Specifically,
this approach adds a
regularization term
H p ⋆ z [ z ] to
the objective
L ELBO
Num switches : This approach penalizes switching between skills, similar to Harb et al.
[26],a variant of the Option-Critic framework [4].
Specifically, this approach adds a penalty n s to
the objective L ELBO equal to the number of switches (i
e., the number of times m t = 1 ) in a
demonstration.
VTA( N max , l max ) :
VTA includes a prior on its boundary variables, which effectively sets the
maximum number
of skills per episode
to be N max and sets the maximum number of actionsa skill can take to l max .
We experiment with several settings of N max and l max , includingthose that use prior knowledge about the domain not used by L OVE .
The results are summarized in Table 7.
We find that regularizing the number of switches performsfairly well, but L OVE achieves higher F1 than other approaches that regularize the skills.