Learning Options via Compression
Abstract
Identifying statistical regularities in solutions to some tasks in multi-task reinforcement learning can accelerate the learning of new tasks.
Skill learning offers one way of identifying these regularities by decomposing pre-collected experiences into a sequence of skills.
A popular approach to skill learning is maximizing the likelihood of the pre-collected experience with latent variable models, where the latent variables represent the skills.
However, there are often many solutions that maximize the likelihood equally well, including degenerate solutions.
To address this underspecification, we propose a new objective that combines the maximum likelihood objective with a penalty on the description length of the skills.
This penalty incentivizes the skills to maximally extract common structures from the experiences.
Empirically, our objective learns skills that solve downstream tasks in fewer samples compared to skills learned from only maximizing likelihood. Further, while most prior works in the offline multi-task setting focus on tasks with low-dimensional observations, our objective can scale to challenging tasks with high-dimensional image observations.
Introduction
While learning tasks from scratch with reinforcement learning (RL) is often sample inefficient [30, 8], leveraging datasets of pre-collected experience from various tasks can accelerate the learning of new tasks.
This experience can be used in numerous ways, including to learn a dynamics model [18, 38], to learn compact representations of observations [87], to learn behavioral priors [77] or to extract meaningful skills or options [79, 42, 2, 59] for hierarchical reinforcement learning (HRL) [6].
Our work studies this last approach.
The central idea is to extract commonly occurring behaviors from the pre-collected experience as skills, which can then be used in place of primitive low-level actions to accelerate learning new tasks via planning [70, 51] or RL [55, 47, 52]. For example, in a navigation domain, learned skills may correspond to navigating to different rooms or objects.
Prior methods learn skills by maximizing the likelihood of the pre-collected experience [43, 42, 2, 90].
However, this maximum likelihood objective (or the lower bounds on it) is underspecified : it often admits many solutions, only some of which would help learning new tasks (see Figure 1).
For example, one degenerate solution is to learn a single skill for each entire trajectory; another degenerate solution is to learn skills that operate for a single timestep, equivalent to the original action space.
Both of these decompositions can perfectly reconstruct the pre-collected experiences (and, hence, maximize likelihood), but they are of limited use for learning to solve new tasks.
Overall, the maximum likelihood objective cannot distinguish between such decompositions and potentially more useful [Figure] decompositions, and we find that this underspecification problem can empirically occur even on simple tasks in our experiments.
To address this underspecification problem, we propose to introduce an additional objective which biases skill learning to acquire skills that can accelerate the learning of new tasks.
To construct such an objective, we make two key observations:
• Skills must extract reusable temporally-extended structure from the pre-collected experiences to accelerate learning of new tasks.
• Compressing data also requires identifying and extracting common structure.
Hence, we hypothesize that compression is a good objective for skill learning.
We formalize this notion of compression with the principle of minimum description length (MDL) [69].
Concretely, we combine the maximum likelihood objective (used in prior works) with a new term to minimize the number of bits required to represent the pre-collected experience with skills, which incentivizes the skills to find common structure.
Additionally, while prior compression-based methods typically involve discrete optimization and hence are not differentiable, we also provide a method to minimize our compression objective via gradient descent (Section 4), which enables optimizing our objective with neural networks.
Overall, our main contribution is an objective for extracting skills from offline experience, which addresses the underspecification problem with the maximum likelihood objective.
We call the resulting approach L OVE ( L earning O ptions V ia compr E ssion).
Using multi-task benchmarks from prior work [42], we find that L OVE can learn skills that enable faster RL and are more semantically meaningful compared to skills learned with prior methods.
We also find that L OVE can scale to tasks with high-dimensional image observations, whereas most prior works focus on tasks with low-dimensional observations.
Related Works
We study the problem of using offline experience from one set of tasks to quickly learn new tasks.
While we use the experience to learn skills, prior works also consider other approaches of leveraging the offline experience, including to learn a dynamics model [38], to learn a compact representation of observations [87], and to learn behavioral priors for exploring new tasks [77].
We build on a rich literature on extracting skills [79] from offline experience [62, 21, 43, 74, 42, 7, 66, 92, 91, 46, 2, 72, 71, 59, 90, 53, 93, 68, 80, 88].
These works predominantly learn skills using a latent variable model, where the latent variables partition the experience into skills, and the overall model is learned by maximizing (a lower bound on) the likelihood of the experiences.
This approach is structurally similar to a long line of work in hierarchical Bayesian modeling [24, 9, 20, 49, 37, 13, 48, 39, 28, 15].
We also follow this approach and build off of VTA [39] for the latent variable model, but differ by introducing a new compression term to address the underspecification problem with maximizing likelihood.
Several prior approaches [90, 91, 23] also use compression, but yield open-loop skills, whereas we aim to learn closed-loop skills that can react to the state.
Additionally,
Zhang et al.
[90] show that maximizing likelihood with variational inference can itself be seen as a form of compression, but this work compresses the latent variables at each time step, which does not in general yield optimal compression as our objective does.
Beyond learning skills from offline experience, prior works also consider learning skills with additional supervision [3, 63, 75, 35, 78], from online interaction without reward labels [25, 19, 17, 73, 5], and from online interaction with reward labels [81, 44, 4, 84, 29, 58]. Additionally, a large body of work on meta-reinforcement learning [16, 85, 67, 94, 50] also leverages prior experience to quickly learn new tasks, though not necessarily through skill learning.
Preliminaries
Problem setting.
We consider the problem of using an offline dataset of experience to quickly solve new RL tasks, where each task is identified by a reward function.
The offline dataset is a set of trajectories D
= { τ i } Ni =1 , where each trajectory is a sequence of states x ∈ X and actions a ∈ A : τ i = { ( x 1 , a 1 ) , ( x 2 , a 2 ) , . . . } .
Each trajectory is collected by some unknown policy interacting with a Markov decision process (MDP) with dynamics P ( x t +1 | x t , a t ) .
Following prior work [18, 1, 14], we do not assume access to the rewards (i.e., the task) of the offline trajectories.
Using this dataset, our aim is to quickly solve a new task.
The new task involves interacting with the same MDP as the data collection policy in the offline dataset, except with a new reward function
R ( x , a ) .
The objective is to learn a policy that maximizes the expected returns in as few numbers of environment interactions as possible.
Variational Temporal Abstraction.
Our method builds upon the graphical model from variational temporal abstraction (VTA) [39], which is a method for decomposing non-control sequential data (i.e., data without actions) into subsequences.
We overview VTA below and extend it to handle actions it Section 4.1.
VTA assumes that each observation x t is associated with an abstract representation s t and a subsequence descriptor z t .
An additional, binary random variable m t indicates whether a new subsequence begins at the current observation.
VTA uses the following generative model:
Determine whether to begin a new subsequence: m t ∼ p ( m t | s t − 1 ) .
A new subsequence always begins on the first time step, i e., m 1 = 1 . 2.
Sample the descriptor:
z t ∼ p ( z t | z <t , m t ) .
If m t = 0 , the previous descriptor is copied, i.e., z t =
z t − 1 . 3.
Sample the next state abstract representation: s t ∼ p ( s t | s <t , z t , m t ) .
Sample the observation: x t ∼ p ( x t | s t ) .
We visualize this sampling procedure in Figure 2.
Formally, we can write this generative model as:
[Equation]
VTA maximizes the likelihood of the observed data under this latent variable model using the standard evidence lower bound (ELBO):
[Equation]
This lower bound holds for any choice of q θ .
VTA chooses to factor this distribution as:
[Equation]
As discussed above and as we observe in Section 6, the maximum likelihood objective is underspecified and may yield degenerate or unhelpful solutions, which we address in the next section.
Specifically, we use the graphical model from VTA, but introduce a new objective that compresses the latent variables z , while also maximizing the ELBO.
L OVE : Learning Options via Compression
In this section, we describe our method for learning skills from pre-collected experience.
We first extend the VTA graphical model to handle experience labeled with actions, and then introduce a compression objective that encourages extracting common structure from the experience.
4.1 A Graphical Model for Interaction Data
We now extend the VTA graphical model [39] to handle sequential data labeled with actions, where descriptors z now represent skills.
From a high level, the model partitions a trajectory into subsequences with the boundary variables m and labels each subsequence as a skill z .
We wish to learn a state-conditional policy rather than the joint distribution of the whole trajectory.
To do this, we write a new generative model for the actions a 1: T conditional on the state x 1: T : p ( z 1: T , s 1: T , m 1: T , a 1: T | x 1: T ) = T (cid:89) t = p (
[Equation]
p ( m t | s t
[Equation]
p ( s t | x t
[Equation]
z
[Equation]
p ( z t | x t , z t − 1 , m t − 1 ) . This differs from the original VTA generative model in two ways: (1) we introduce a p ( a t | s t ) term that indicates how actions are sampled; and (2) the distributions over z t and s t do not depend on all previous z 1: t and s 1: t to encode the Markov property.
We then augment the variational distribution such that the posterior over skills z also depends on actions:
[Equation]
[Table]
Overall, this yields a model with 3 learned components:
• A state abstraction posterior q θ ( s t | z t , x t ) and decoder p θ ( a t | s t ) , which together form a distribution over actions conditioned on the skill z t and current state x t .
We refer to these jointly as the skill policy π θ ( a t | z t , x t ) =
E s t ∼ q θ ( s t | z t , x t ) [ p θ ( a t | s t )] .
• A termination policy q θ ( m t | x 1: t ) , which determines whether the previous skill terminates.
• A skill posterior q θ ( z t | z t − 1 , m t , x 1: T , a 1: T ) , which outputs the current skill z t conditioned on all states x 1: T and actions a 1: T .
This depends on m t and z t − 1 , since the boundary variables determine whether the previous skill z t − 1 terminates: when m t = 0 , the previous skill does not terminate and z t =
z t − 1 .
Once this model is learned, the skill policy and termination policy represent the skills, without a need for the skill posterior: Given a skill variable z , the skill policy encodes what actions the skill takes, and the termination policy determines when the skill stops.
In the next section, we describe our objective for learning this model.
Then, in Section 5, we describe how we use the skill policy and termination policy to learn new tasks.
4.2 Discovering Structure via Compression
As previously discussed, the maximum likelihood objective is underspecified for skill learning, because many skills can maximize likelihood, independent of whether they are useful for learning new tasks.
In this section, we address this with a new objective that attempts to measure how useful skills will be for learning new tasks in terms of compression.
Our objective is based on the intuition that effectively compressing a sequence of data requires factoring out common structure, and factoring out common structure is critical for learning useful skills.
We measure the complexity of a skill decomposition as the amount of information required to communicate the sequence of skills that encode the pre-collected experience.
The latent variable model introduced in the previous section encodes each trajectory as a sequence of skills z 1: T and boundaries m 1: T .
However, using the skill and termination policies, it is possible to recover each trajectory from only the skill variables at the boundary points: i.e., at the time steps { t ∈ [ T ] | m t = 1 } .
For a prior on skills p z , an optimal code requires − log p z ( z t ) bits to send a skill z t [12, Chapter 5.2].
Hence, the expected code length of communicating a trajectory τ 1: T = { ( x 1 , a 1 ) , . . , ( x T , a T ) } is:
[Equation]
where the expectation is under trajectories τ 1: T from the pre-collected experience and sampling z 1: T from the skill posterior and m 1: T from the termination policy.
The choice of prior that minimizes the average code length is one that equals the empirical distribution of skills under the pre-collected experience [12, Chapter 5.3]:
[Equation]
where p ⋆ z
( z ) =
1 n s E τ 1: T , m 1: T , z 1: T (cid:34) T (cid:88) t = δ ( z t = z ) m t (cid:35) , and n s ≜ E τ 1: T , m 1: T (cid:104)(cid:80) Tt =1 m t (cid:105) .
In Appendix A.1, we show that this marginal p ⋆ z is a proper density for both continuous and discrete z .
Substituting in this optimal choice of prior, we can show that the code length can be expressed as the marginal entropy H p ⋆ z
[ z ] times the number of skills per trajectory n s (see Appendix A.2 for proof):
[Equation]
Intuitively, this is equal to the average code length of a skill multiplied by the the average number of skills per trajectory.
Note that compression is only beneficial if the model also achieves high likelihood of the data.
We capture this by solving the following constrained optimization problem:
[Equation]
where L ELBO ( θ ) a negated evidence lower bound on the log-likelihood (detailed in Appendix B).
Remarks.
We discuss a connection between this objective and variational inference in Appendix A.3.
Additionally, while this work focuses on the RL setting, our objective generally applies to sequential modeling problems.
We believe that it could be useful for many applications beyond option learning.
4.3 Connections to Minimal Description Length
Our approach closely relates to the minimum description length (MDL) principle [69].
This principle equates learning with finding regularity in data, which can be used to compress the data.
Informally, the best model is the one that encodes the data with the lowest description length.
Given a model θ that encodes the data D into some message, one way to formalize the description length of the data
L ( D ) is with a crude two-part code [26].
This decomposes the description length of the data as the length of the message plus the length of the model:
[Equation]
In our case, the message is the latent variables z t at the boundary points {
t ∈
[ T ] | m t = 1 } , which can be decoded into the data D with the skill and termination policies (representing the model).
Hence, our approach can be seen as an instance of minimizing the description length
L ( D ) .
Optimizing our objective in Equation 1 is equivalent to minimizing the message length
L ( D | θ ) .
While we do not directly attempt to minimize the model length term L ( θ ) , many works [60, 82] indicate that deep learning has implicit regularization , which biases optimization toward low complexity solutions without an explicit regularizer.
In general, computing the true description length or appropriate notion of complexity for neural networks is a tall order [61, 89, 36]; however, there is a rich space of methods to be explored here and we therefore leave the extension of our approach to directly minimizing the model length term for future work.
4.4 A Practical Implementation
Model.
We instantiate our model by defining discrete skills z ∈
[ K ] , state abstractions s ∈ R d , and binary boundary variables m ∈ { 0 , 1 } .
We parameterize all components of our model as neural networks.
See Appendix D for architecture details.
Optimization.
We apply Gumbel-softmax [34, 54] to optimize over the discrete random variables z and m .
We rewrite the compression objective L CL as a product of n s and H p
⋆ z
[ z t ] in Equationto improve the stability of optimization.
This rewriting allows computing a gradient in terms of a distribution over the skill variables p ⋆ z , rather than samples z t , which yields a more accurate finite sample gradient.
Empirically, this leads to stabler optimization and convergence to better solutions.
We approximate the optimal skill prior p ⋆ z using our skill posterior as:
[Equation]
where (cid:98) E denotes the empirical expectation over minibatches of x 1: T , a 1: T from D and sampling m 1: T from the termination policy.
We solve the constrained optimization problem (Equation 2) with dual gradient descent on the Lagrangian.
In Appendix G, we summarize the overall training procedure in Algorithm 1 and report details about the Lagrangian.
Enforcing a minimum skill length.
Though degenerate solutions such as skills that only take a single action score poorly in our compression objective, empirically, such solutions create local optima that are difficult to escape.
To avoid this, we mask the boundary variables during training to ensure that each skill z t operates for at least T min = 3 time steps.
We find that when the skills are at least minimally temporally extended, optimization of the compression objective appears to be stabler and achieves better values.
We remove these masks at test time, when we learn a task with our skills.
Using the Learned Skills for Hierarchical RL
We now describe how we quickly learn new tasks, given the skills learned from the pre-collected experience.
Overall, we simply augment the agent’s action space with the learned skills and learn a new policy that can take either low-level actions or learned skills, similar to Kipf et al.
However, our compression objective may result in unused skills, since this decreases the encoding cost of the used skills.
Therefore, we first filter down to only the skills that are used to compress the pre-collected experience by selecting the skills where the marginal is over some threshold α :
[Equation]
Then, on a new task with action space A , we train an agent using the augmented action space
A + = A ∪ Z .
When the agent selects a skill z ∈ Z , we follow the procedure in Algorithm 2 in the
Appendix.
We take actions following the skill policy π θ ( a t | z , x t ) (lines 2–3), until the termination policy q θ ( m t | x 1: t ) outputs a termination m t = 1 (line 5).
At that point, the agent observes the next state x t +1 and selects the next action or skill.
A Didactic Experiment: Frame Prediction
Before studying the RL setting, we first illustrate the effect of L OVE in the simpler setting of sequential data without actions.
In this setting, we use the original VTA model, which partitions a sequence x 1: T into contiguous subsequences with the boundary variables m and labels each subsequence with a descriptor z , as described in Section 3.
Here, VTA’s objective is to maximize the likelihood of the sequence x 1: T , and our compression objective applies to this setting completely unmodified.
We compare L OVE and only maximizing likelihood, i.e., VTA [39], by measuring how well the learned subsequences correspond to underlying patterns in the data.
To isolate the effect of the compression objective, we do not enforce a minimum skill length.
Dataset.
We consider two simple datasets Simple Colors and Conditional Colors , which consist of sequences of 32 × 32 monochromatic frames, with repeating underlying patterns.
Simple Colors consists of 3 patterns: 3 consecutive yellow frames, 3 consecutive blue frames and 3 consecutive green frames.
The dataset is generated by sampling these patterns with probability 0.4, 0.4, and 0.2 respectively and concatenating the results.
Each pattern is sampled independent of history.
The dataset is then divided into sequences of 6 patterns, equal to 6 × 3 = 18 frames.
In contrast to Simple Colors , Conditional Colors tests learning patterns that depend on history.
It consists of 4 patterns: the 3 patterns from Simple Colors with an additional pattern of 3 consecutive [Figure] [Table] purple frames.
The dataset is generated by uniformly sampling the patterns from Simple Colors .
To make the current timestep pattern dependent on the previous timestep, the yellow frames are re-colored to purple, if the previous pattern was blue or yellow.
As in Simple Colors , the dataset is divided into sequences of 6 patterns
In both datasets the optimal encoding strategy is to learn 3 subsequence descriptors (i e., skills), one for each pattern in Simple Colors .
In Conditional Colors , the descriptor corresponding to yellow in Simple Colors either outputs yellow or purple, depending on the history.
This encoding strategy achieves an expected code length of 6 .
32 nats in Simple Colors and 6 .
59 nats in Conditional Colors .
Results.
We visualize the learned descriptors and boundaries in Figure 3.
L OVE successfully segments the sequences into the patterns and assigns a consistent descriptor z to each pattern.
In contrast, despite the simple structure of this problem, VTA learns subsequences that span over multiple patterns or last for fewer timesteps than a single pattern.
Quantitatively, we measure the (1) precision, recall, and F1 scores of the boundary prediction, (2) the
ELBO of the maximum likelihood objective and (3) the average code length L CL in Table 1.
While both VTA and L OVE achieve approximately the same negated ELBO (lower is better), L OVE recovers the correct boundaries with much higher precision, recall, and F1 scores.
This illustrates that the underspecification problem can occur even in simple sequential data.
Additionally, we find that L OVE achieves an encoding cost close to the optimal value.
One interesting, though rare, failure mode is that L OVE sometimes even achieves a lower encoding cost than the optimal value by over-weighting the compression term and imperfectly reconstructing the data.
In Appendix C, we conduct a ablation study on the weights of L CL in optimization.
Experiments
[Figure]
We aim to answer three questions: (1) Does L OVE learn semantically meaningful skills?
(2) Do skills learned with L OVE accelerate learning for new tasks?
(3) Can L OVE scale to high-dim pixel observations?
To answer these questions, we learn skills from demonstrations and use these skills to learn new tasks, first on a 2D multi-task domain, and later on a 3D variant.
Multi-task domain.
We consider the multi-task 10 ×grid world introduced by Kipf et al.
[42], a representative skill learning approach (Figure 4).
This domain features a challenging control problem that requires the agent to collect up to 5 objects, and we later add a perception challenge to it with the 3D variant.
We use our own custom implementation, since the code from Kipf et al.
[42] is not publicly available.
In each task, 6 objects are randomly sampled from a set of N obj =
10 different objects and placed at random positions in the grid.
Additionally, impassable walls are randomly added to the grid, and the agent is also placed at a random initial grid cell.
Each task also includes an instruction list of N pick = 3 or 5 objects that the agent must pick up in order.
Within each task, the agent’s actions are to move up, down, left, right, or to pick up an item at the agent’s grid cell.
The state consists of two parts: (1) a 10 × 10 × ( N obj + 2) grid observation, indicating if the agent, a wall, or any of the N obj different object types is present at each of the 10 ×grid cells; (2) the instruction corresponding to the next object to pick up, encoded as a one-hot integer.
Following Kipf et al.
[42], we consider two reward variants: In the sparse reward variant, the agent only receives +1 reward after picking up all N pick objects in the correct order.
In the dense reward variant, the agent receives +1 reward after picking up each specified object in the correct order.
Our dense reward variant is slightly harder than the variant in Kipf et al.
[42], which gives the agent + reward for picking up objects in any order.
The agent receives 0 reward in all other time steps.
The episode ends when the agent has picked up all the objects in the correct order, or after 50 timesteps.
Pre-collected experience.
We follow the setting in Kipf et al.
We set the pre-collected experience to be 2000 demonstrations generated via breadth-first search on randomly generated tasks with only N pick = 3 and test if the agent can generalize to N pick = 5 when learning a new task.
These demonstrations are not labeled with rewards and also do not contain the instruction list observations.
Points of comparison.
To study our compression objective, we compare with two representatives of learning skills via the maximum likelihood objective:
• VTA [39], modified to handle interaction data as in Section 4.1.
• Discovery of deep options (DDO) [21].
We implement DDO’s maximum likelihood objective and graphical model on top of VTA’s variational inference optimization procedure, instead of expectation-gradients used in the original paper.
For fairness, we also compare with variants of these that implement the minimum skill length constraint from Section 4.4.
CompILE [42] is another approach that learns skill by maximizing likelihood and was introduced in the same paper as this grid world.
Because the implementation is unavailable, we do not compare with it.
However, we note that CompILE requires additional supervision that L OVE , VTA, and DDO do not: namely, it requires knowing how many skills each demonstration is composed of.
This supervision can be challenging to obtain without already knowing what the skills should be.
Since latent variable models are prone to local optima [45], it is common to learn such models with multiple restarts [57].
We therefore run each method with 3 random initializations and pick the best model according to the compression objective L CL for L OVE and according to the ELBO of the maximum likelihood objective L ELBO for the others.
Notably, this does not require any additional demonstrations or experience.
We begin by analyzing the skills learned from the pre-collected experience before analyzing the utility of these skills for learning downstream tasks in the next sections.
7.1 Analyzing the Learned Skills
[Table]
We analyze the learned skills by comparing them to a natural decomposition that partitions the demonstration in N pick sequences of moving to and picking up an object.
Specifically, we measure the precision and recall of each method in outputting the correct boundaries of these N pick sequences.
We find that only L OVE recovers the correct boundaries with both high precision and recall (Table 2).
Visually examining the skills learned with L OVE shows that each skill moves to an object and picks it up.
We visualize L OVE ’s skills in Appendix H.
In contrast, maximizing likelihood with either DDO or VTA learns degenerate skills that output only a single action, leading to high recall, but low precision.
Adding the minimum skill length constraint to these approaches prevents such degenerate solutions, but still does not lead to the correct boundaries.
Skills learned by VTA with the minimum skill length constraint exhibit no apparent structure, but skills learned by DDO with the minimum skill length constraint appear to move toward objects, though they terminate at seemingly random places.
These results further illustrate the underspecification problem.
All approaches successfully learn to reconstruct the [Figure] a [Figure]
t u r n s
N pick
[Figure]
(Dense)
[Figure]
Sparse
[Figure]
demonstrations and achieve high likelihood, but only L OVE and DDO with the minimum skill length constraint learn skills that appear to be semantically meaningful.
7.2 Learning New Tasks
Next, we evaluate the utility of the skills for learning new tasks.
To evaluate the skills, we sample a new task in each of 4 settings: with sparse or dense rewards and with N pick = 3 or N pick = 5 .
Then we learn the new tasks following the procedure in Section 5: we augment the action space with the skills and train an agent over the augmented action space.
To understand the impact of skills, we also compare with a low-level policy that learns these tasks using only the original action space, and the same low-level policy that incorporates the demonstrations by pre-training with behavioral cloning.
We parametrize the policy for all approaches with dueling double deep Q-networks [56, 86, 83] with ϵ -greedy exploration.
We report the returns of evaluation episodes with ϵ = 0 , which are run every 10 episodes.
We report returns averaged over 5 seeds.
See Appendix F.4 for full details.
Results.
Overall, L OVE learns new tasks across all 4 settings comparably to or faster than both skill methods based on maximizing likelihood and incorporating demonstrations via behavior cloning (Figure 5).
More specifically, when N pick = 3 , DDO with the min.
skill length constraint performs comparably to L OVE , despite not segmenting the demonstrations into sequences of moving to and picking up objects.
We observe that while a single DDO skill does not move to and pick up an object, the skills consistently move toward objects, which likely helps with exploration.
Imitating the demonstrations with behavior cloning also accelerates learning when N pick = 3 with dense rewards, though not as much as L OVE or DDO, and yields insignificant benefit in all other settings.
Skill learning with VTA yields no benefit over no skill learning at all.
Recall that N pick = 3 in all of the demonstrations.
L OVE learns new tasks in the generalization setting of N pick = 5 much faster than the other methods.
With dense rewards, DDO (min.
skill length) also eventually solves the task, but requires over 8 × more timesteps.
The other approaches learn to pick up some objects, but never achieve optimal returns of 5.
With sparse rewards, L OVE is the only approach that achieves high returns, while all other approaches achieve 0 returns.
This likely occurs because this setting creates a challenging exploration problem, so exploring with low-level actions or skills learned with VTA or DDO rarely achieves reward.
By contrast, exploring with skills learned with L OVE achieves rewards much more frequently, which enables L OVE to ultimately solve the task.
7.3 Scaling to High-dimensional Observations
Prior works in the offline multi-task setting have learned skills from low-dimensional states and then transferred them to high-dim pixel observations [68], or have learned hierarchical models from pixel observations [39, 21].
However, to the best of our knowledge, prior works in this setting have not directly learned skills from high-dim pixel observations, only from low-dimensional states.
Hence, in this section, we test if L OVE can scale up to high-dim pixel observations by considering a challenging 3D image-based variant of the multi-task domain above, illustrated in Figure 6.
The goal is still to pick up N pick objects in the correct order, where objects are blocks of different colors.
Each task includes 3 objects randomly sampled from a set of N obj = 6 different colors.
The observations are 400 × 60 × 3 egocentric panoramic RGB arrays.
The actions are to move forward / backward, turn left / right, and pick up.
[Figure]
We consider only the harder sparse reward and generalization setting, where the agent must pick up N pick = 4 objects in the correct order, after receiving 2,000 pre-collected demonstrations of the approximate shortest path of picking up N pick = 2 objects in the correct order.
We use the same hyperparameters as in multi-task domain and only change the observation encoder and action decoder (full details in Appendix D.3).
Figure 7 shows the results.
Again, L OVE learns skills that each navigate to and pick up an object, which quickly achieves optimal returns, indicating its ability to scale to high-dim observations.
In contrast, the other approaches perform much worse on pixel observations than the previous low-dim observations.
DDO also makes progress on the task, but requires far more samples, as it again learns skills that only operate for a few timesteps, though they move toward the objects.
All other approaches fail to learn at all, including the variants of VTA and DDO without the min.
skill length, which are not plotted.
Conclusion
We started by highlighting the underspecification problem: maximizing likelihood does not necessarily yield skills that are useful for learning new tasks.
To overcome this problem, we drew a connection between skill learning and compression and proposed a new objective for skill learning via compression and a differentiable model for optimizing our objective.
Empirically, we found that the underspecification problem occurs even on simple tasks and learning skills with our objective allows learning new tasks much faster than learning skills by maximizing likelihood.
Still, important future work remains.
L OVE applies when there are useful and consistent structures that can be extracted from multiple trajectories.
This is often present in multi-task demonstrations, which solve related tasks in similar ways.
However, an open challenge for adapting to general offline data like D4RL [22], is to ensure that the learned skills do not overfit to potentially noisy or unhelpful behaviors often present in offline data.
In addition, we showed a connection between skill learning and compression by minimizing the description length of a crude two-part code.
However, we only proposed a way to optimize the message length term and not the model length.
Completing this connection by accounting for model length may therefore be an promising direction for future work. Reproducibility.
Our code is publicly available at https://github.com/yidingjiang/love .
Acknowledgements. We would like to thank Karol Hausman, Abhishek Gupta, Archit Sharma, Sergey Levine, Annie Xie, Zhe Dong, Samuel Sokota for valuable discussions during the course of this work, and Victor Akinwande, Christina Baek, Swaminathan Gurumurthy for comments on the draft. We would also like to thank the anonymous reviewers for their valuable feedback.
Last, but certainly not least, YJ and EZL thank the beautiful beach of Kona for initiating this project.
YJ is supported by funding from the Bosch Center for Artificial Intelligence.
EZL is supported by a
National Science Foundation Graduate Research Fellowship under Grant No. DGE-1656518.
CF is a Fellow in the CIFAR Learning in Machines and Brains Program.
This work was also supported in part by the ONR grant N00014-21-1-2685, Google, and Intel.
Icons in this work were made by
ThoseIcons, FreePik, VectorsMarket, from FlatIcon.
A Derivations
A. p ⋆ z is a valid density
Proposition A.1.
p ⋆ z ( ξ ) is a proper probability density function.
Proof.
Recall that
[Equation]
A function is a probability density function if it is non-negative, continuous [Equation] and integrates to 1.
By construction, p ⋆ z ( ξ ) is non-negative since m t ≥ 0 and δ ( z t = ξ ) ≥ 0 .
Further, δ ( z t = ξ ) is continuous in ξ so the numerator E τ 1: T , m 1: T , z 1: T (cid:104)(cid:80) Tt =1 δ ( z t = ξ )
m t (cid:105) , an expectation containing δ ( z t = ξ ) , is also continuous in ξ .
Finally, (cid:90)
[Equation]
(Fubini’s theorem)
[Equation]
Therefore, p ⋆ z ( ξ ) satisfies all 3 properties of a probability density function.
Remarks.
This result generalizes to the case for discrete z by replacing the delta with an indicator function and integration with summation.
A.2 Two expressions of L CL
Proposition A.2.
The expected coding length of a trajectory is equal to the expected number skills multiplied by the marginal entropy:
[Equation]
Proof.
−
[Equation]
=
[Equation]
E [Equation] (cid:34)(cid:90)
[Equation]
m [Equation] ) dξ (cid:35) (Fubini’s theorem) = − (cid:90)
[Equation]
E
[Equation]
[Equation]
[Equation]
(cid:34)
[Equation]
(cid:88) t
[Equation]
m
[Equation]
[Equation]
ξ ) (cid:35)
[Equation]
p [Equation] [Equation] ) dξ (Linearity of expectation) =
− E τ 1: T , m 1: T (cid:34) T
[Equation]
=
E τ 1: T , m 1: T , z 1: T (cid:34) T (cid:88) t
= m t (cid:35) (cid:124) (cid:123)(cid:122) (cid:125) n s (cid:90) ξ − p ⋆ z ( ξ ) log p ⋆ z ( ξ ) dξ (cid:124) (cid:123)(cid:122) (cid:125)
H p⋆ z
[ z ] = n s H p ⋆ z [ z ]
Remarks.
This result generalizes to the case for discrete z by replacing the delta with an indicator function and integration with summation.
Further, this rewriting not only offers an insightful interpretation of the objective but also has different finite-sample gradient compared to the original form.
We discuss its implication further in Section 4.4.
[-A.
 But you just picked the wrong prior-]A.3 Connection between L OVE and Variational Inference
Is the compression objective a prior in the framework of variational inference (VI)?
The answer to this question is surprisingly nuanced.
A bridge that connects VI and source coding is the bits-back coding argument [31, 32].
Under this scheme, the additional cost of communicating a message is equal to the KL divergence between the approximate posterior and the chosen prior distribution.
This is because the posterior often does not match the prior and the sender must use more bits to send the message compared to using the prior exactly.
In modern deep latent variable models such as VAE [41], the prior is in general chosen to be an isotropic Gaussian distribution or a sequence of isotropic Gaussian distributions that decomposes over the time steps.
The bit-back coding argument shows that VI as a coding scheme is asymptotically optimal (in the limit of sending large numbers of messages), but it does not immediately guarantee good representation learning .
As a thought experiment, imagine we choose the dimension of the latent variable in a VAE to be equal to the dimension of the data itself.
It is clear that the model is not incentivized to perform good representation learning, but the bit-backing coding scheme still guarantees asymptotic optimality.
Instead, good representation can only emerge if the prior is chosen properly.
Contrary to this prevailing design choice, our latent variables z 1: T and m 1: T are not independent from each other and the timesteps are not independent from each other.
In fact, the interaction between the latent variables is crucial for compressing the trajectory optimally in our framework.
Therefore, our objective takes into account the global structure of the latent variables.
Note that
L CL also has the natural interpretation of measuring the average number of bits that is required to send a trajectory under the coding scheme of this work.
As both quantities measure the cost of the communication, the philosophical parallel between KL divergence and L CL is likely not a coincidence.
In practice, we observe that L CL also has comparable regularization effect on the latent code, and in some cases it is possible to learn a good model without any KL divergence if L CL is present.
From this perspective, our compression objective is indeed a “prior" insofar as it specifies the desired properties of the posterior.
Another interpretation of L OVE is that it is optimizing the prior directly .
Nonetheless, it is an open question whether L CL has a precise counterpart in Bayesian inference, that is, it is not immediately obvious if it can be written as the KL divergence between the posterior q ( z 1: T , m 1: T | x 1: T ) and some prior distribution p ( z 1: T , m 1: T ) .
One complication, for example, is the fact that the log density of m 1: T does not participate in the computation of L CL .
It may be possible to construct some form of hierarchical priors that enables a fully Bayesian interpretation of
L CL , but it is also well-known that the MDL framework can accommodate codes that are not Bayesian (e.g., the Shtarkov normalized maximum likelihood code [76]).
We do not offer a definitive answer to this question in this work, but the connection between Bayesian inference and L CL is certainly an interesting direction for future works.
B Evidence Lower Bound B.1 VTA
We reproduce the data generating process of Kim et al.
[39] here: [Equation] [Equation] =
[Equation]
[Equation]
[Equation]
− 1 ) .
VTA [39] shows that the ELBO can be written as:
[Equation]
Like most ELBO, this ELBO can be deconstructed into a reconstruction term:
[Equation]
and the remaining KL term L KL ( θ , ϕ ) .
B. L OVE
We reproduce the data generating process of L OVE here.
Instead of using ϕ for the parameters of priors and decoder, we will merge all parameters into θ : p ( a 1: T , z 1: T , s 1: T , m
[Equation]
x
[Equation]
[Equation]
For L OVE , the ELBO is re-written as:
[Equation]
− 1 )) .
Since z t is a categorical distribution, we chose the uniform prior p ( z t ) similar to Oord et al.
Similar to VTA, the loss is also decomposed into a reconstruction term and a KL term.
C Sensitivity to Compression Objective Weighting
Here we study how sensitive L OVE is to the choice of λ , the coefficient of L CL .
We tested 5 values of λ ∈ [0 . 01 , 1] on Simple and Cond.
Colors and applied dual gradient descent to automatically tune λ .
[-=
-]
Stopping point is selected based on the shortest code length achieved.
[Table]
We see that while λ can change the quality of solutions found
[Table]
the performance is stable within a ± 3 × regions around λ = 0 .
Further, using dual GD consistently outperforms handpicked λ with an ELBO threshold C slightly (e.g., 5% ) higher than the L ELBO achieved without compression, which can be obtained with minimal prior knowledge about the task.
Hence λ likely can be tuned without strong prior knowledge.
This contrasts with the other methods such as Kipf et al.
[42] which require knowing the number of skills in each demonstration to perform well while achieving the similar or better performance.
D Architecture D.1 Frame Prediction
Our base architecture is similar to Kim et al.
and we highlight the difference blue.
The parameters of the posteriors are implemented with neural networks.
First, the observation x 1: T are encoded into a lower dimension embedding ˆ x 1: T with a observation encoder .
The embedding is then passed through a boundary posterior decoder that outputs the parameters of m 1: T .
The embedding is further passed through a GRU [10], f s-rnn-fwd which runs on ˆ x 1: T and give the cell state h 1: T .
For z 1: T , we have two GRU’s with cell, f z-rnn-bwd and f z-rnn-bwd , which run on ˆ x 1: T in different temporal direction (i.e., f z-rnn-bwd sees the future) to generate cell states c fwd 1: T and c bwd 1: T .
For each time step t , a candidate z ′ t is sampled from a n z -dimensional categorical distribution 3 whose log probability is a linear projection of c fwd t ∥ c bwd t .
The sampled z ′ t are embedded linearly into a vector of size 128.
If m 1 = 1 , the z t = z ′ t ; otherwise, z t =
z t − 1 4 .
Then we concatenate h t − 1 ∥ z t and linearly project the vector into the mean and standard deviation of a isotropic Gaussian in R 8 .
A ˆ s t is sampled from this Gaussian.
Th model also keeps two “belief” vectors that keep track of the history of the computation.
These vectors are modulated by two RNN, f h-rnn and f c-rnn .
The abstraction belief c t = m t · f c-rnn ( z t , c t − 1 )+ (1 − m t ) · c t − 1 .
The observation belief h t = m t c t − 1 + (1 − m t ) · f h-rnn (ˆ s t ∥ c t ∥
z t , h t − 1 ) .
Finally the state abstraction is computed as the linear projection of s t = proj ( h t ∥
ˆ s t ) .
The projection layer is 256 × 128 .
Finally, s t is passed through a observation decoder and decode into the reconstruction of x t .
Refer to Kim et al.
[39] for more details and an illustration of the computation graph.
For each convolutional layer or transposed convolution layer, the tuple’s values correspond to input channel, output channel, kernel size, stride, padding.
With that notation in mind, the specific hyperparameters for each components are:
• observation encoder : 4-layer convolutional neural network with {(3, 128, 4, 2, 1), (128, 128, 4, 2, 1), (128, 128, 4, 2, 1), (128, 128, 4, 1, 0)} with ELU activation [11] and batch normalization [33].
The output is flattened linearly projected to vector of size 128. • boundary posterior decoder is a 5-layer 1-D causal temporal convolution [64] with {(128, 128, 5, 1, 2) × 5, (128, 2, 5, 1, 2)} with ELU activation and batch normalization.
The output is the logits for the m 1: T .
• f z-rnn-bwd , f z-rnn-fwd , f s-rnn-fwd , f c-rnn , f h-rnn : GRU with cell size• observation decoder : 4-layer transposed convolutional layers {(128, 128, 4, 1, 0), (128, 128, 4, 2, 1), (128, 128, 4, 2, 1), (128, 3, 4, 2, 1)} with ELU activation and batch normalization on all but the last layer.
The last layer has no normalization and has tanh activation.
Removing Boundary Regularization.
An important difference from Kim et al.
[39] is that we do not enforce maximum number of subsequence N max and the maximum length of subsequence l max through the boundary prior (This effectively amounts to setting both to a value larger than the sequence length).
These two hyperparameters’ effect are similar to that of picking the number of segments in
CompILE [42] and provide strong training signal.
This assumption is in general problematic since we cannot expect to know a priori what the good values are for these hyperparameters.
Kipf et al.
[42] demonstrates that the performance can suffer if this kind of supervision is wrong.
On the other hand, we do not assume anything about the the duration of the subsequence or how many subsequences there are in each demonstration.
We do, however, enforce a minimum length on the skill.
While it is largely an optimization choice, we believe this prior is significantly weaker since useful options are almost always temporally extended.
Indeed, with only the minimum length constraint, VTA is unable to learn good skills conducive for downstream tasks.
D.2 Multi-task Grid World Environment
For trajectories, we make some larger changes to the architecture to encode properties we want in a model for learning options, e.g., Markov property.
Though similar in spirit, a large portion of the architecture is different from the base VTA model, so the difference will not be highlighted.
For the posterior, we have an observation encoder and an action encoder that embeds both the state observation x 1: T and actions a 1: T to get embedding a 1: T and x 1: T each of size 128.
The concatenation a t ∥ x t linear projected down to a vector of size 128.
The embedding is then passed through a boundary posterior decoder that outputs the parameters of m 1: T .
The embedding is further passed through a GRU [10], f s-rnn-fwd which runs on ˆ x 1: T and give the cell state h 1: T .
For z 1: T , we have two GRU’s with cell, f z-rnn-bwd and f z-rnn-bwd , which run on ˆ x 1: T in different temporal direction (i.e., f z-rnn-bwd sees the future) to generate cell states c fwd 1: T and c bwd 1: T .
For sampling z is done differently for interaction data for better gradient and representation learning.
Instead of doing straight-through estimator for the categorical random variable, we opt to use vector-quantization [65] with straight-through estimators.
First, c fwd t ∥ c bwd t is linearly projected into dimension 128.
Then a ReLU is applied and another 128 by 128 linear layer is applied.
The VQ codebook is of size n z × 128 .
Instead of taking argmax of the codebook, we approximate the distribution to be proportional to the softmax over the distance over a temperature t V Q .
We can sample from this distribution and apply the straight-through gradient on the sampled code candidate z ′ [-
 :-]1: T .
If m 1 = 1 , the z t = z ′ t ; otherwise, z t =
z t − 1 .
We construct directly s t =
z t ∥ x t .
and linearly project the vector into the mean and standard deviation of a isotropic Gaussian R 8 .
s t is sampled from this Gaussian and then decoded with the action decoder into the reconstructed action in A .
For each convolutional layer or transposed convolution layer, the tuple’s values correspond to input channel, output channel, kernel size, stride, padding.
With that notation in mind, the specific hyperparameters for each components are:
• observation encoder :
This takes as input the 10 × 10 × N obj + 2 and is implemented as a two 2D convolutional layers with ReLU activations, followed by a linear layer with a ReLU activation and a linear layer with no activation.
The convolutional layers have hyperparameters (12 , 64 , 3 , 1 , 0) and (64 , 64 , 3 , 1 , 0) respectively.
The first linear has input dimension 6 × 6 × 64 and output dimension 128 .
The second linear layer has input and output dimensions 128 . • action encoder :
The actions are embedded with an embedding matrix with embedding dimension 128.
• boundary posterior decoder is a 5-layer 1-D causal temporal convolution [64] with {(128, 128, 5, 1, 2) × 5, (128, 2, 5, 1, 2)} with ELU activation and batch normalization.
The output is the logits for the m 1: T .
• f z-rnn-bwd , f z-rnn-fwd , f s-rnn-fwd : GRU with cell size• action decoder : The decoder is implemented as three linear layers with ReLU activations, followed by a linear layer with no activation.
The (input, output) dimensions of these layers are as follows: (128 , 128) , (128 , 128) , (128 , 128) , (128 , 5) , where the last layer outputs logits over the actions.
D.3 3D Navigation Environment
The architecture used in the 3D navigation environment is identical to that of the multi-task environment detailed above, with the exception that the observation encoder is changed to handle visual inputs. Specifically, the observation encoder is a a 3-layer convolutional neural network parameters (3 , 32 , 5 , 2 , 0) , (32 , 32 , 5 , 2 , 0) , (32 , 32 , 4 , 2 , 0) , followed two linear layers (7520 , 128) , (128 , 128) with a ReLU activation in between.
D.4 Hierarchical Reinforcement Learning
For all approaches, we parametrize the policy as a double dueling deep Q-network [56, 86, 83].
The parametrization of the Q-function consists of a state embedding followed by two linear layer heads for the state-value V θ ( s ) function and the advantage
A θ ( s, a ) function.
Then the Q-function is computed as Q θ ( s, a ) =
V θ ( s ) + A θ ( s, a ) − 1 |A| (cid:80) a ′ ∈A
A θ ( s, a ) .
The value function V θ ( s ) and advantage function A θ ( s, a ) are computed as linear layers with output dimension 1 and |A| respectively on top of the embedding e ( s ) of the state s . Recall that the state s consists of two portions: the observation o of the grid or 3D environment, and the instruction i corresponding to the next object to pick up, where the instruction is not present in the demonstrations.
The embedding e ( s ) of the state s is computed by embedding the observation e ( o ) with the observation encoder defined in the previous sections, and embedding the instruction e ( i ) with a 16-dimensional embedding matrix.
Then, the embedding e ( s ) is a final linear layer with output dimension 128 applied to a ReLU of the concatenation of e ( o ) and e ( i ) .
E Hardware
All our experiments are conducted on a GeForce RTX 2080 Ti or a GeForce RTX A6000.
F Hyperparameters F.1 Frame Prediction
For these set of experiments, we use the same architecture and hyperparameters as Kim et al.
The hyperparameters used for both Simple Colors and Conditional Colors are:
• KL divergence weight β : 1 .• MDL objective weight λ (fixed) : 0 .•
Mini-batch size:• Training iteration:• Number of skills n z[-: 
-]
= F.2 Multi-task Grid World Segmentation
We use the following hyperparameters for skill learning from demonstrations on the multi-task grid world environment experiments.
• KL divergence weight β
= 0 : – We found that the compression objective readily provides strong and sufficient regularization for the latent code.
Adding additional KL divergence often results in the model not being able to reconstruct the actions properly.
• MDL objective weight λ : – We use a adaptive scheduling that approximate dual gradient descent.
λ is initialized to 0 .
After every gradient step, λ is increased by 2 .
× 10 − 5 if L ELBO ≤ 0 .
(Since β = 0 , this is effectively L rec ); otherwise, λ is decreased by 2 .
× 10 − 5 .
After each update, the value of λ is clipped to [0 , 0 . 05] .
We find this setting provides the most stable training.
• Mini-batch size:• Training iteration:• Number of skills n z =• t VQ [-: 0.
-]= 0 .•
Learning rate: 0 . 0005 with Adam Optimizer
F.3 3D Visual Navigation Segmentation
We use the same hyperparameter as Multi-task grid world experiments except that the maximum
L ELBO is capped at 0 .
001 , i.e. , L ELBO ≤ 0 .
F.4 Hierarchical Reinforcement Learning
We use the same hyperparameters for training the policy for all approaches.
Specifically, these hyperparameter values are as follows:
• ϵ -greedy schedule: We linearly decay ϵ from 1 to 0 .
01 over 500 K timesteps for dense reward settings and over 5 M timesteps for sparse reward settings in the multi-task grid world environment.
In the 3D visual navigation environment, we decay over 250 K timesteps.
• Discount factor γ : We use γ = 0 .
99 in all of the DQN updates.
• Maximum replay buffer size: 50K • Minimum replay buffer size before updating:• Learning rate: 0.0001 with the Adam optimizer [40] • Batch size:• Update frequency: every 4 timesteps • Target syncing frequency: every 50K updates for multi-task grid world environment, every 30K updates for 3D navigation • Gradient ℓ 2 -norm
clipping:• Marginal threshold α = 0 .
Since the demonstrations do not contain the instruction list observations, we set those to 0 in the demonstrations for the behavior cloning approach.
Though we use α = 0 .
001 in all of the experiments for consistency, we also found that slightly higher values of α yielded greater sample efficiency for
L OVE .
G Algorithm
The Lagrangian for the unconstrained optimization problem is:
[Equation]
In standard KKT condition, the dual variable is introduced on the ELBO, but the two problems are equivalent by inverting the dual variable.
Since the only constraint on λ is that it is non-negative, this transformation does not change the optimal solution.
Further note that we find that in many cases, choosing a fixed constant value for λ is sufficient for solving the problem.
Sample a trajectory ( x a
. . . , x T ) from the pre-collected experience D
t
Sample boundary conditioned on entire trajectory m t ∼ q θ ( m t | x 1: t ) 6:
Sample skill from skill posterior z t ∼ q θ
( z t | m t , x 1: t , a 1: t ) 7:
Sample abstraction from abstraction posterior s t ∼ q θ ( s t | z t , x t ) 8:
Compute probability of the correct action from the decoder p θ ( a t | s t ) 9: end for 10:
Compute lower bound on maximum likelihood objective L ELBO according to Equation11:
Compute compression objective L CL according to Equation12:
Update θ ← θ − η ∇ ( L ELBO + λ L CL ) and λ 13: end while
1: while skill has not terminated do 2:
Compute state abstraction µ s t =
E s ∼ q θ ( s t | z , x t )
[ s ] 3:
Take action a t = arg max a p θ ( a | µ s t ) 4:
Observe next state x t + 5:
Terminate if E m t +1 ∼ q θ ( m t +1 | x 1: t +1 )
[ m t +1 ] > 0 .
5 6: end while
[Figure] [Figure]
57 V e r t i c a l c h a n g e
[Figure]
V
[Figure]
r
[Figure]
i c
[Figure]
l
[Figure]
In Figure 8, we visualize the behavior of following each of L OVE ’s 10 skills on an example task.
Each skill moves to and picks up an object, and the total set of skills covers 3 of the 4 object types in the task.
These skills pick up both nearby objects, such as the tree in skill 2, as well as more distant objects, such as the diamond in skill 1.
This contrasts DDO and VTA, which do not learn skills that move to and pick up objects.
To further understand what each of L OVE ’s skills does, we analyze whether there is a correlation between each skill and either the type of object it picks up, or the location of the object it picks up.
We find that there appears to be little correlation between each skill and the type of object it picks up.
Plotting the frequency of picking up each object type by skill shows that each skill picks up each object type with roughly the same frequency.
Instead, there appears to be a correlation between each skill and the location of the object it picks up, illustrated in Figure 9.
For example, skill 1 appears to specialize in moving to and picking up objects that are above the agent, while skill 7 tends to pick up objects that are up and to the right of the agent.
It is also interesting to note that some skills are much more biased to move in certain direction (e.g., skill 1, 6, 7) while some appear to be more general (e.g., 4, 5, 8, 9) and move in any direction.
Note in Figure 8, it may seem L OVE ’s skills seem short / redundant.
This is due to the fact that L OVE ’s skills each pick up an object, so they naturally appear short when the agent is close to the objects, as in Figure 8.
However, when the agent is far away from the objects, the skills still pick up the objects and are much longer.
From
Figure 9, we can see from the heatmap that even the skills that appear to do same thing in Figurehave very distinct behaviors depending on the state they are in.
Also, recall L OVE imposes a sparse distribution over skills (Section 5).
After filtering, skill 3 , 7 and 10 in Figure 8 are dropped since they have low marginal probability and are not used to describe a significant part of the trajectories.
Figure 8 includes redundant skills L OVE does not use; the used skills in the sparse distribution have little redundancy.
I Option Critic Results
[Figure]
Option critic [4] is a classical online HRL algorithm.
We show the results of running option criticon two of the four RL tasks we considered in Figure 10.
The best return achieved over the training is shown in Table 4.
We set the number of option to 8 following Bacon et al.
[4] and leave other hyperparameters untouched.
N pick = 3 with dense reward is the easily setting and N pick = 5 with sparse reward is the hardest setting.
We see that in N pick = 3 , option critic is able to reach reward of(maximum possible is 3 ) at around 1M environment steps but the performance deterioates afterwards.
In N pick = 5 , the algorithm fails to make any meaningful progress.
These observations may suggest online HRL algorithms like option critic may be insufficient for solving these tasks.
[Equation]
[Table]
J Comparison to Zhang et al. [90]
[Table]
Colors
[Table]
[Table]
Zhang et al.
[90] learn open-loop skills, which do not condition on the state and therefore cannot adapt to different states.
Further, the MDL used in Zhang et al.
[90] is equivalent to the variational inference used by VTA (on a different graphical model), which can be seen as greedily compressing each skill independently, rather than compressing a whole trajectory as L OVE does.
Table 5 reports segmentation results on the Color domain and grid world, akin to Tables 2 and 3.
Zhang et al.
[90] performs the same as VTA on the Color domains – as they both use variational inference and there is no state – which achieves lower precision / recall than L OVE .
On the grid world navigation, Zhang et al.
[90] fails to recover the boundaries, unlike L OVE , because skills that navigate to objects require observing the state.
This problem is shared by all open-loop approaches.
K Number of Initial Skills Ablation
[Table]
The number of skills to extract from demonstrations is often not known a priori, so choosing an appropriate value of the number of initial skills before filtering K is important.
Intuitively, we hypothesize that K can be set conservatively:
L OVE requires at least a minimum number of skills in order to fit the behaviors in the demonstrations, but may be able to gracefully prune out skills if K is set too high via the filtering described in Section 5.
We test this intuition by varying
K ∈ { 2 , 5 , 10 , 15 , 20 , 30 , 50 } and measuring the segmentation performance on the grid world multitask domain, and find that it appears to hold true in Table 6.
For smaller values of K , such as K =and K = 5 , L OVE ’s F1 scores significantly degrade.
However, performance remains high for all values where K is sufficiently large.
Hence, we conservatively sizing K to be a large number.
L Comparison to Different Regularizers
[Table]
Several prior methods encourage skills to act for more timesteps or prevent skills from switching too frequently, which can similarly help avoid degenerate solutions like L OVE .
We note that such prior methods do not necessarily yield skills that help learn new tasks, while the maximum of L OVE ’s objective achieves the information-theoretic limit of compressing the trajectories (under a fixed function class), which is not done in other works.
We empirically compare L OVE with several such methods on the segmentation of the multi-task grid world domain, including:
Entropy : This approach regularizes the entropy of skill marginal akin to Shankar et al.
[72], which they refer to this as parsimony .
Specifically, this approach adds a regularization term H p
⋆ z [ z ] to the objective L ELBO .
Num switches : This approach penalizes switching between skills, similar to Harb et al.
[27], a variant of the Option-Critic framework [4].
Specifically, this approach adds a penalty n s to the objective L ELBO equal to the number of switches (i.e., the number of times m t = 1 ) in a demonstration.
VTA( N max , l max ) :
VTA includes a prior on its boundary variables, which effectively sets the maximum number of skills per episode to be N max and sets the maximum number of actions a skill can take to l max .
We experiment with several settings of N max and l max , including those that use prior knowledge about the domain not used by L OVE .
The results are summarized in Table 7.
We find that regularizing the number of switches performs fairly well, but L OVE achieves higher F1 than other approaches that regularize the skills.