D YNAMICS -A WARE U NSUPERVISED D ISCOVERY OF S KILLS
A BSTRACT
Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment.
A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks.
However, learning an accurate model for complex dynamical systems is difﬁcult, and even then, the model might not generalize well outside the distribution of states on which it was trained.
In this work, we combine model-based learning with model-free learning of primitives that make modelbased planning easy.
To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict?
We propose an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics.
Our method can leverage continuous skill spaces, theoretically, allowing us to learn inﬁnitely many behaviors even for high-dimensional state-spaces.
We demonstrate that zero-shot planning in the learned latent space signiﬁcantly outperforms standard MBRL and model-free goal-conditioned RL, can handle sparsereward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery.
We have open-sourced our implementation at: https://github.com/google-research/dads
I NTRODUCTION
Deep reinforcement learning (RL) enables autonomous learning of diverse and complex tasks with rich sensory inputs, temporally extended goals, and challenging dynamics, such as discrete gameplaying domains (Mnih et al., 2013; Silver et al., 2016), and continuous control domains including locomotion (Schulman et al., 2015; Heess et al., 2017) and manipulation (Rajeswaran et al., 2017;
Kalashnikov et al., 2018; Gu et al., 2017).
Most of the deep RL approaches learn a Q-function or a policy that are directly optimized for the training task, which limits their generalization to new scenarios.
In contrast, MBRL methods (Li & Todorov, 2004; Deisenroth & Rasmussen, 2011;
Watter et al., 2015) can acquire dynamics models that may be utilized to perform unseen tasks at test time.
While this capability has been demonstrated in some of the recent works (Levine et al., 2016;
Nagabandi et al., 2018; Chua et al., 2018b; Kurutach et al., 2018; Ha & Schmidhuber, 2018), learning an accurate global model that works for all state-action pairs can be exceedingly challenging, especially for high-dimensional system with complex and discontinuous dynamics.
The problem is further exacerbated as the learned global model has limited generalization outside of the state distribution it was trained on and exploring the whole state space is generally infeasible.
Can we retain the ﬂexibility of model-based RL, while using model-free RL to acquire proﬁcient low-level behaviors under complex dynamics?
While learning a global dynamics model that captures all the different behaviors for the entire statespace can be extremely challenging, learning a model for a speciﬁc behavior that acts only in a small part of the state-space can be much easier.
For example, consider learning a model for dynamics of all gaits of a quadruped versus a model which only works for a speciﬁc gait.
If we can learn many such behaviors and their corresponding dynamics, we can leverage model-predictive control to plan in the behavior space , as opposed to planning in the action space.
The question then becomes: how do we acquire such behaviors, considering that behaviors could be random and unpredictable?
To this end, we propose Dynamics-Aware Discovery of Skills (DADS), an unsupervised RL framework for learning low-level skills using model-free RL with the explicit aim of making model-based control easy.
Skills obtained using DADS are directly optimized for predictability , providing a better representation on top of which predictive models can be learned.
Crucially, the skills do not require any supervision to learn, and are acquired entirely through autonomous exploration.
This means that the repertoire of skills and their predictive model are learned before the agent has been tasked with any goal or reward function.
When a task is provided at test-time, the agent utilizes the previously learned skills and model to immediately perform the task without any further training.
The key contribution of our work is an unsupervised reinforcement learning algorithm, DADS, grounded in mutual-information-based exploration.
We demonstrate that our objective can embed learned primitives in continuous spaces, which allows us to learn a large, diverse set of skills.
Crucially, our algorithm also learns to model the dynamics of the skills, which enables the use of model-based planning algorithms for downstream tasks.
We adapt the conventional model predictive control algorithms to plan in the space of primitives, and demonstrate that we can compose the learned primitives to solve downstream tasks without any additional training.
P RELIMINARIES
Mutual information can been used as an objective to encourage exploration in reinforcement learning (Houthooft et al., 2016; Mohamed & Rezende, 2015).
According to its deﬁnition, I ( X ; Y ) =
H ( X ) − H ( X | Y ) , maximizing mutual information I with respect to Y amounts to maximizing the entropy H of X while minimizing the conditional entropy H ( X | Y ) .
In the context of RL, X is usually a function of the state and Y a function of actions.
Maximizing this objective encourages the state entropy to be high, making the underlying policy to be exploratory.
Recently, multiple works (Eysenbach et al., 2018; Gregor et al., 2016; Achiam et al., 2018) apply this idea to learn diverse skills which maximally cover the state space.
To leverage planning-based control, MBRL estimates the true dynamics of the environment by learning a model ˆ p ( s (cid:48) | s, a ) .
This allows it to predict a trajectory of states ˆ τ H
=
( s t , ˆ s t +1 , . . .
ˆ s t
+ H ) resulting from a sequence of actions without any additional interaction with the environment.
While model-based RL methods have been demonstrated to be sample efﬁcient compared to their modelfree counterparts, learning an effective model for the whole state-space is challenging.
An openproblem in model-based RL is to incorporate temporal abstraction in model-based control, to enable high-level planning and move-away from planning at the granular level of actions.
These seemingly unrelated ideas can be combined into a single optimization scheme, where we ﬁrst discover skills (and their models) without any extrinsic reward and then compose these skills to optimize for the task deﬁned at test time using model-based planning.
At train time, we assume a Markov Decision Process (MDP) M 1 ≡ ( S , A , p ) .
The state space S and action space A are assumed to be continuous, and the A bounded.
We assume the transition dynamics p to be stochastic, such that p : S ×
A × S (cid:55)→ [0 , ∞ ) .
We learn a skill-conditioned policy π ( a | s, z ) , where the skills z belongs to the space Z , detailed in Section 3.
We assume that the skills are sampled from a prior p ( z ) over Z .
We simultaneously learn a skill-conditioned transition function q ( s (cid:48) | s, z ) , coined as skill-dynamics , which predicts the transition to the next state s (cid:48) from the current state s for the skill z under the given dynamics p .
At test time, we assume an MDP M 2 ≡ ( S , A , p, r ) , where S , A , p match those deﬁned in M 1 , and the reward function r : S × A (cid:55)→ ( −∞ , ∞ ) .
We plan in Z using q ( s (cid:48) | s, z ) to compose the learned skills z for optimizing r in M 2 , which we detail in Section 4.
D YNAMICS -A WARE D ISCOVERY OF S KILLS (DADS)
Initialize π, q φ ; while not converged do
Sample a skill z ∼ p ( z ) every episode;Collectnew M on-policy samples;
Update q φ using K 1 steps of gradient descent on M transitions;
Compute r z ( s, a, s (cid:48) ) for M transitions; Update π using any RL algorithm; end
We use the information theoretic paradigm of mutual information to obtain our unsupervised skill discovery algorithm.
In particular, we propose to maximize the mutual information between the next state s (cid:48) and current skill z conditioned on the current state s .
[Equation]
Mutual information in Equation 1 quantiﬁes how much can be known about s (cid:48) given z and s , or
symmetrically, z given the transition from s → s (cid:48) .
From Equation 2, maximizing this objective corresponds to maximizing the diversity of transitions produced in the environment, that is denoted by the entropy H ( s (cid:48) | s ) , while making z informative about the next state s (cid:48) by minimizing the entropy H ( s (cid:48) | s, z ) .
Intuitively, skills z can be interpreted as abstracted action sequences which are identiﬁable by the transitions generated in the environment (and not just by the current state).
Thus, optimizing this mutual information can be understood as encoding a diverse set of skills in the latent space Z , while making the transitions for a given z ∈ Z predictable.
We use the entropydecomposition in Equation 2 to connect this objective with model-based control.
We want to optimize the our skill-conditioned controller π ( a | s, z ) such that the latent space
z ∼ p ( z ) is maximally informative about the transitions s → s (cid:48) .
Using the deﬁnition of conditional mutual information, we can rewrite Equation 2 as:
[Equation]
We assume the following generative model: p ( z, s, s (cid:48) )
= p ( z ) p ( s | z ) p ( s (cid:48) | s, z ) , where p ( z ) is user speciﬁed prior over Z , p ( s | z ) denotes the stationary state-distribution induced by π ( a | s, z ) for a skill z and p ( s (cid:48) | s, z ) denotes the transition distribution under skill z .
Note, p ( s (cid:48) | s, z )
=
(cid:82) p ( s (cid:48) | s, a ) π ( a | s, z ) da is intractable to compute because the underlying dynamics are unknown.
However, we can variationally lower bound the objective as follows:
[Equation]
where we have used the non-negativity of KL-divergence, that is D KL ≥ 0 .
Note, skill-dynamics q φ represents the variational approximation for the transition function p ( s (cid:48) | s, z ) , which enables model-based control as described in Section 4.
Equation 4 suggests an alternating optimization between q φ and π , summarized in Algorithm 1.
In every iteration: ( Tighten variational lower bound )
We minimize D KL ( p ( s (cid:48) | s, z )
||
q φ ( s (cid:48) | s, z )) with respect to the parameters φ on z, s ∼ p to tighten the lower bound.
For general function approximators like neural networks, we can write the gradient for φ as follows:
[Equation]
which corresponds to maximizing the likelihood of the samples from p under q φ .
( Maximize approximate lower bound ) After ﬁtting q φ , we can optimize π to maximize E z,s,s (cid:48)
[log q φ ( s (cid:48) | s, z ) − log p ( s (cid:48) | s )] .
Note, this is a reinforcement-learning style optimization with a reward function log q φ ( s (cid:48) | s, z ) − log p ( s (cid:48) | s ) .
However, log p ( s (cid:48) | s ) is intractable to compute, so we approximate the reward function for π :
[Equation]
The approximation is motivated as follows: p ( s (cid:48) | s )
= (cid:82) p ( s (cid:48) | s, z ) p ( z | s )
dz ≈ (cid:82) q φ ( s (cid:48) | s, z ) p ( z ) dz
≈ 1 L (cid:80) Li =
1 q φ ( s (cid:48) | s, z i ) for z
i ∼ p
( z ) , where L denotes the number of samples from the prior.
We are using the marginal of variational approximation q φ over the prior p ( z ) to approximate the marginal distribution of transitions.
We discuss this approximation in Appendix C. Note, the ﬁnal reward function r z encourages the policy π to produce transitions that are (a) predictable under q φ ( predictability ) and (b) different from the transitions produced under z i ∼ p ( z ) ( diversity ).
To generate samples from p ( z, s, s (cid:48) ) , we use the rollouts from the current policy π for multiple samples z ∼ p ( z ) in an episodic setting for a ﬁxed horizon T .
We also introduce entropy regularization for π ( a | s, z ) , which encourages the policy to discover action-sequences with similar state-transitions and to be clustered under the same skill z , making the policy robust besides encouraging exploration (Haarnoja et al., 2018a).
The use of entropy regularization can be justiﬁed from an information bottleneck perspective as discussed for Information Maximization algorithm in (Mohamed & Rezende, 2015).
This is even more extensively discussed from the graphical model perspective in Appendix B, which connects unsupervised skill discovery and information bottleneck literature, while also revealing the temporal nature of skills z .
Details corresponding to implementation and hyperparameters are discussed in Appendix A.
P LANNING USING S KILL D YNAMICS
Given the learned skills π ( a | s, z ) and their respective skill-transition dynamics q φ ( s (cid:48) | s, z ) , we can perform model-based planning in the latent space Z to optimize for a reward r that is given to the agent at test time.
Note, that this essentially allows us to perform zero-shot planning given the unsupervised pre-training procedure described in Section 3.
In order to perform planning, we employ the model-predictive-control (MPC) paradigm Garcia et al.
(1989), which in a standard setting generates a set of action plans P k
= ( a k, 1 , . . .
a k,H ) ∼ P for a planning horizon H .
The MPC plans can be generated due to the fact that the planner is able to simulate the trajectory ˆ τ k
=
( s k, 1 , a k, 1 . . .
s k,H +1 ) assuming access to the transition dynamics ˆ p ( s (cid:48) | s, a ) .
In addition, each plan computes the reward r (ˆ τ k ) for its trajectory according to the reward function r that is provided for the test-time task.
Following the MPC principle, the planner selects the best plan according to the reward function r and executes its ﬁrst action a 1 .
The planning algorithm repeats this procedure for the next state iteratively until it achieves its goal.
We use a similar strategy to design an MPC planner to exploit previously-learned skill-transition dynamics q φ ( s (cid:48) | s, z ) .
Note that unlike conventional model-based RL, we generate a plan P k
=
( z k, 1 , . . .
z k,H P ) in the latent space Z as opposed to the action space
A that would be used by a standard planner.
Since the primitives are temporally meaningful, it is beneﬁcial to hold a primitive [Figure] for a horizon H Z > 1 , unlike actions which are usually held for a single step.
Thus, effectively, the planning horizon for our latent space planner is H
= H P
×
H Z , enabling longer-horizon planning using fewer primitives.
Similar to the standard MPC setting, the latent space planner simulates the trajectory ˆ τ k
=
( s k, 1 , z k, 1 , a k, 1 , s k, 2 , z k, 2 , a k, 2 , . . .
s k,H +1 ) and computes the reward r (ˆ τ k ) .
After a small number of trajectory samples, the planner selects the ﬁrst latent action z 1 of the best plan, executes it for H Z steps in the environment, and the repeats the process until goal completion.
The latent planner P maintains a distribution of latent plans, each of length H P .
Each element in the sequence represents the distribution of the primitive to be executed at that time step.
For continuous spaces, each element of the sequence can be modelled using a normal distribution,
N ( µ 1 , Σ) , . . .
N ( µ H P , Σ) .
We reﬁne the planning distributions for R steps, using K samples of latent plans P k , and compute the r k for the simulated trajectory ˆ τ k .
The update for the parameters follows that in Model Predictive Path Integral (MPPI) controller Williams et al.
[Equation]
While we keep the covariance matrix of the distributions ﬁxed, it is possible to update that as well as shown in Williams et al.
We show an overview of the planning algorithm in Figure 3, and provide more implementation details in Appendix A.
R ELATED W ORK
Central to our method is the concept of skill discovery via mutual information maximization.
This principle, proposed in prior work that utilized purely model-free unsupervised RL methods (Daniel et al., 2012; Florensa et al., 2017; Eysenbach et al., 2018; Gregor et al., 2016; Warde-Farley et al., 2018; Thomas et al., 2018), aims to learn diverse skills via a discriminability objective: a good set of skills is one where it is easy to distinguish the skills from each other, which means they perform distinct tasks and cover the space of possible behaviors.
Building on this prior work, we distinguish our skills based on how they modify the original uncontrolled dynamics of the system.
This simultaneously encourages the skills to be both diverse and predictable .
We also demonstrate that constraining the skills to be predictable makes them more amenable for hierarchical composition and thus, more useful on downstream tasks.
Another line of work that is conceptually close to our method copes with intrinsic motivation (Oudeyer & Kaplan, 2009; Oudeyer et al., 2007; Schmidhuber, 2010) which is used to drive the agent’s exploration.
Examples of such works include empowerment Klyubin et al.
; Mohamed & Rezende (2015), count-based exploration Bellemare et al.
Oh et al. (2015); Tang et al.
Fu et al.
(2017), information gain about agent’s et al.
and forward-inverse dynamics models Pathak et al.
While our method uses an informationtheoretic objective that is similar to these approaches, it is used to learn a variety of skills that can be directly used for model-based planning, which is in contrast to learning a better exploration policy for a single skill.
The skills discovered using our approach can also provide extended actions and temporal abstraction, which enable more efﬁcient exploration for the agent to solve various tasks, reminiscent of hierarchical RL (HRL) approaches.
This ranges from the classic option-critic architecture (Sutton et al., 1999; Stolle & Precup, 2002; Perkins et al., 1999) to some of the more recent work (Bacon et al., 2017; Vezhnevets et al., 2017; Nachum et al., 2018; Hausman et al., 2018).
However, in contrast to end-to-end HRL approaches (Heess et al., 2016; Peng et al., 2017), we can leverage a stable, two-phase learning setup.
The primitives learned through our method provide action and temporal abstraction, while planning with skill-dynamics enables hierarchical composition of these primitives, bypassing many problems of end-to-end HRL.
In the second phase of our approach, we use the learned skill-transition dynamics models to perform model-based planning - an idea that has been explored numerous times in the literature.
Model-based reinforcement learning has been traditionally approached with methods that are well-suited for lowdata regimes such as Gaussian Processes (Rasmussen, 2003) showing signiﬁcant data-efﬁciency gains over model-free approaches (Deisenroth et al., 2013; Kamthe & Deisenroth, 2017; Kocijan et al., 2004;
Ko et al., 2007).
More recently, due to the challenges of applying these methods to highdimensional state spaces, MBRL approaches employs Bayesian deep neural networks (Nagabandi et al., 2018;
Chua et al., 2018b; Gal et al., 2016; Fu et al., 2016; Lenz et al., 2015) to learn dynamics models.
In our approach, we take advantage of the deep dynamics models that are conditioned on the skill being executed, simplifying the modelling problem.
In addition, the skills themselves are being learned with the objective of being predictable, further assists with the learning of the dynamics model.
There also have been multiple approaches addressing the planning component of MBRL including linear controllers for local models (Levine et al., 2016; Kumar et al., 2016;
Chebotar et al., 2017), uncertainty-aware (Chua et al., 2018b; Gal et al., 2016) or deterministic planners (Nagabandi et al., 2018) and stochastic optimization methods (Williams et al., 2016).
The main contribution of our work lies in discovering model-based skill primitives that can be further combined by a standard model-based planner, therefore we take advantage of an existing planning approach - Model Predictive Path Integral (Williams et al., 2016) that can leverage our pre-trained setting.
E XPERIMENTS
Through our experiments, we aim to demonstrate that: (a) DADS as a general purpose skill discovery algorithm can scale to high-dimensional problems; (b) discovered skills are amenable to hierarchical composition and; (c) not only is planning in the learned latent space feasible, but it is competitive to strong baselines.
In Section 6.1, we provide visualizations and qualitative analysis of the skills learned using DADS.
We demonstrate in Section 6.2 and Section 6.4 that optimizing the primitives for predictability renders skills more amenable to temporal composition that can be used for Hierarchical RL.We benchmark against state-of-the-art model-based RL baseline in Section 6.3, and against goal-conditioned RL in Section 6.5.
6. Q UALITATIVE A NALYSIS
In this section, we provide a qualitative discussion of the unsupervised skills learned using DADS.
We use the MuJoCo environments (Todorov et al., 2012) from the OpenAI gym as our test-bed (Brockman et al., 2016).
We ﬁnd that our proposed algorithm can learn diverse skills without any reward, even in problems with high-dimensional state and actuation, as illustrated in Figure 4.
DADS can discover primitives for Half-Cheetah to run forward and backward with multiple different gaits, for Ant to navigate the environment using diverse locomotion primitives and for Humanoid to walk using stable locomotion primitives with diverse gaits and direction.
The videos of the discovered primitives are available at: https://sites.google.com/view/dads-skill
Qualitatively, we ﬁnd the skills discovered by DADS to be predictable and stable, in line with implicit constraints of the proposed objective.
While the Half-Cheetah will learn to run in both backward and forward directions, DADS will disincentivize skills which make Half-Cheetah ﬂip owing to the reduced predictability on landing.
Similarly, skills discovered for Ant rarely ﬂip over, and tend to provide stable navigation primitives in the environment.
This also incentivizes the Humanoid, which is characteristically prone to collapsing and extremely unstable by design, to discover gaits which are stable for sustainable locomotion.
One of the signiﬁcant advantages of the proposed objective is that it is compatible with continuous skill spaces, which has not been shown in prior work on skill discovery (Eysenbach et al., 2018).
Not only does this allow us to embed a large and diverse set of skills into a compact latent space, but also the smoothness of the learned space allows us to interpolate between behaviors generated in the environment.
We demonstrate this on the Ant environment (Figure 5), where we learn two-dimensional continuous skill space with a uniform prior over ( − 1 , 1) in each dimension, and compare it to a discrete skill space with a uniform prior over 20 skills.
Similar to Eysenbach et al.
(2018), we restrict the observation space of the skill-dynamics q to the cartesian coordinates ( x, y ) .
We hereby call this the x-y prior , and discuss its role in Section 6.2.
[Figure]
In Figure 5, we project the trajectories of the learned Ant skills from both discrete and continuous spaces onto the Cartesian plane.
From the traces of the skills, it is clear that the continuous latent space can generate more diverse trajectories.
We demonstrate in Section 6.3, that continuous primitives are more amenable to hierarchical composition and generally perform better on downstream tasks.
More importantly, we observe that the learned skill space is semantically meaningful.
The heatmap in Figure 5 shows the orientation of the trajectory (with respect to the x -axis) as a function of the skill z ∈ Z , which varies smoothly as z is varied, with explicit interpolations shown in
Appendix D.
6. S KILL V ARIANCE A NALYSIS
In an unsupervised skill learning setup, it is important to optimize the primitives to be diverse.
However, we argue that diversity is not sufﬁcient for the learned primitives to be useful for downstream tasks.
Primitives must exhibit low-variance behavior, which enables long-horizon composition of the learned skills in a hierarchical setup.
We analyze the variance of the x - y trajectories in the environment, where we also benchmark the variance of the primitives learned by DIAYN (Eysenbach et al., 2018).
For DIAYN, we use the x - y prior for the skill-discriminator, which biases the discovered skills to diversify in the x - y space.
This step was necessary for that baseline to obtain a [Figure] competitive set of navigation skills.
Figure 6 (Top-Left) demonstrates that DADS, which optimizes the primitives for predictability and diversity, yields signiﬁcantly lower-variance primitives when compared to DIAYN, which only optimizes for diversity.
This is starkly demonstrated in the plots of X-Y traces of skills learned in different setups.
Skills learned by DADS show signiﬁcant control over the trajectories generated in the environment, while skills from DIAYN exhibit high variance in the environment, which limits their utility for hierarchical control.
This is further demonstrated quantitatively in Section 6.4.
While optimizing for predictability already signiﬁcantly reduces the variance of the trajectories generated by a primitive, we ﬁnd that using the x - y prior with DADS brings down the skill variance even further.
For quantitative benchmarks in the next sections, we assume that the Ant skills are learned using an x - y prior on the observation space, for both DADS and DIAYN.
6. M ODEL -B ASED R EINFORCEMENT L EARNING
The key utility of learning a parametric model q φ ( s (cid:48) | s, z ) is to take advantage of planning algorithms for downstream tasks, which can be extremely sample-efﬁcient.
In our setup, we can solve testtime tasks in zero-shot, that is without any learning on the downstream task .
We compare with the state-of-the-art model-based RL method (Chua et al., 2018a), which learns a dynamics model parameterized as p ( s (cid:48) | s, a ) , on the task of the Ant navigating to a speciﬁed goal with a dense reward.
Given a goal g , reward at any position u is given by r ( u ) = −(cid:107) g − u (cid:107) 2 .
We benchmark our method against the following variants:
• Random-MBRL
( rMBRL ): We train the model p ( s (cid:48) | s, a ) on randomly collected trajectories, and test the zero-shot generalization of the model on a distribution of goals.
• Weak-oracle MBRL ( WO-MBRL ):
We train the model p ( s (cid:48) | s, a ) on trajectories generated by the planner to navigate to a goal, randomly sampled in every episode.
The distribution of goals during training matches the distribution at test time.
• Strong-oracle MBRL ( SO-MBRL ):
We train the model p ( s (cid:48) | s, a ) on a trajectories generated by the planner to navigate to a speciﬁc goal, which is ﬁxed for both training and test time.
Amongst the variants, only the rMBRL matches our assumptions of having an unsupervised taskagnostic training.
Both WO-MBRL and SO-MBRL beneﬁt from goal-directed exploration during training, a signiﬁcant advantage over DADS, which only uses mutual-information-based exploration.
We use ∆ =
(cid:80)
Ht =
− r ( u ) H (cid:107) g
(cid:107) 2 as the metric, which represents the distance to the goal g averaged over the episode (with the same ﬁxed horizon H for all models and experiments), normalized by the initial distance to the goal g .
Therefore, lower ∆ indicates better performance and 0 < ∆ ≤ 1 (assuming the agent goes closer to the goal).
The test set of goals is ﬁxed for all the methods, sampled from [ − 15 , 15] 2 .
Figure 7 demonstrates that the zero-shot planning signiﬁcantly outperforms all model-based RL baselines, despite the advantage of the baselines being trained on the test goal(s).
For the experiment depicted in Figure 7 (Right), DADS has an unsupervised pre-training phase, unlike SO-MBRL which is training directly for the task.
A comparison with Random-MBRL shows the signiﬁcance of mutual-information-based exploration, especially with the right parameterization and priors.
This experiment also demonstrates the advantage of learning a continuous space of primitives, which outperforms planning on discrete primitives.
6. H IERARCHICAL C ONTROL WITH U NSUPERVISED P RIMITIVES
We benchmark hierarchical control for primitives learned without supervision, against our proposed scheme using an MPPI based planner on top of DADS-learned skills.
We persist with the task of
Ant-navigation as described in 6.3.
We benchmark against Hierarchical DIAYN (Eysenbach et al., 2018), which learns the skills using the DIAYN objective, freezes the low-level policy and learns a meta-controller that outputs the skill to be executed for the next H Z steps.
We provide the x - y prior to the DIAYN’s disciminator while learning the skills for the Ant agent.
The performance of the meta-controller is constrained by the low-level policy, however, this hierarchical scheme is agnostic to the algorithm used to learn the low-level policy.
To contrast the quality of primitives learned by the DADS and DIAYN, we also benchmark against Hierarchical DADS, which learns a meta-controller the same way as Hierarchical DIAYN, but learns the skills using DADS.
From Figure 8 (Left)
We ﬁnd that the meta-controller is unable to compose the skills learned by
DIAYN, while the same meta-controller can learn to compose skills by DADS to navigate the Ant to different goals.
This result seems to conﬁrm our intuition described in Section 6.2, that the high variance of the DIAYN skills limits their temporal compositionality.
Interestingly, learning a RL meta-controller reaches similar performance to the MPPI controller, taking an additional 200 ,samples per goal.
6. G OAL - CONDITIONED RL
To demonstrate the beneﬁts of our approach over model-free RL, we benchmark against goalconditioned RL on two versions of Ant-navigation: (a) with a dense reward r ( u ) and (b) with a sparse reward r ( u )
= 1 if (cid:107) u
− g (cid:107) 2 ≤ (cid:15) , else 0.
We train the goal-conditioned RL agent using soft actor-critic, where the state variable of the agent is augmented with u − g , the position delta to the goal.
The agent gets a randomly sampled goal from [ − 10 , 10] 2 at the beginning of the episode.
In Figure 8 (Right), we measure the average performance of the all the methods as a function of the initial distance of the goal, ranging from 5 to 30 metres.
For dense reward navigation, we observe that while model-based planning on DADS-learned skills degrades smoothly as the initial distance to goal to increases, goal-conditioned RL experiences a sudden deterioration outside the goal distribution it was trained on.
Even within the goal distribution observed during training of goal-conditioned RL model, skill-space planning performs competitively to it.
With sparse reward navigation, goal-conditioned RL is unable to navigate, while MPPI demonstrates comparable performance to the dense reward up to about 20 metres.
This highlights the utility of learning task-agnostic skills, which makes them more general while showing that latent space planning can be leveraged for tasks requiring long-horizon planning.
C ONCLUSION
We have proposed a novel unsupervised skill learning algorithm that is amenable to model-based planning for hierarchical control on downstream tasks.
We show that our skill learning method can scale to high-dimensional state-spaces, while discovering a diverse set of low-variance skills.
In addition, we demonstrated that, without any training on the speciﬁed task, we can compose the learned skills to outperform competitive model-based baselines that were trained with the knowledge of the test tasks.
We plan to extend the algorithm to work with off-policy data, potentially using relabelling tricks (Andrychowicz et al., 2017; Nachum et al., 2018) and explore more nuanced planning algorithms.
We plan to apply the hereby-introduced method to different domains, such as manipulation and enable skill/model discovery directly from images.  A CKNOWLEDGEMENTS
We would like to thank Evan Liu, Ben Eysenbach, Anusha Nagabandi for their help in reproducing the baselines for this work.
We are thankful to Ben Eysenbach for their comments and discussion on the initial drafts.
We would also like to acknowledge Oﬁr Nachum, Alex Alemi, Daniel Freeman, Yiding Jiang, Allan Zhou and other colleagues at Google Brain for their helpful feedback and discussions at various stages of this work.
We are also thankful to Michael Ahn and others in Adept team for their support, especially with the infrastructure setup and scaling up the experiments.
A I MPLEMENTATION D ETAILS
All of our models are written in the open source Tensorﬂow-Agents (Sergio Guadarrama, Anoop
Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina,
Chris Harris, Vincent Vanhoucke, Eugene Brevdo, 2018), based on Tensorﬂow (Abadi et al., 2015).
A. S KILL S PACES
When using discrete spaces we parameterize Z as one-hot vectors.
These one-hot vectors are randomly sampled from the uniform prior p ( z )
= 1 D , where D is the number of skills.
We experiment with D ≤ 128 .
For discrete skills learnt for MuJoCo Ant in Section 6.3, we use D = 20 .
For continuous spaces, we sample z ∼ Uniform ( − 1 , 1) D .
We experiment with D = 2 for Ant learnt with x-y prior, D = 3 for Ant learnt without x-y prior (that is full observation space), to D = 5 for
Humanoid on full observation spaces.
The skills are sampled once in the beginning of the episode and ﬁxed for the rest of the episode.
However, it is possible to resample the skill from the prior within the episode, which allows for every skill to experience a different distribution than the initialization distribution.
This also encourages discovery of skills which can be composed temporally.
However, this increases the hardness of problem, especially if the skills are re-sampled from the prior frequently.
A. A GENT
We use SAC as the optimizer for our agent π ( a | s, z ) , in particular, EC-SAC (Haarnoja et al., 2018b).
The s input to the policy generally excludes global co-ordinates ( x, y ) of the centre-ofmass, available for a lot of enviroments in OpenAI gym, which helps produce skills agnostic to the location of the agent.
We restrict to two hidden layers for our policy and critic networks.
However, to improve the expressivity of skills, it is beneﬁcial to increase the capacity of the networks.
The hidden layer sizes can vary from (128 , 128) for Half-Cheetah to (512 , 512) for Ant and (1024 , 1024) for Humanoid.
The critic Q ( s, a, z ) is similarly parameterized.
The target function for critic Q is updated every iteration using a soft updates with co-efﬁcient of 0 .
We use Adam (Kingma &
Ba, 2014) optimizer with a ﬁxed learning rate of 3 e − 4 , and a ﬁxed initial entropy co-efﬁcient
β
= 0 .
While the policy is parameterized as a normal distribution N ( µ ( s, z ) , Σ( s, z )) where Σ is a diagonal covariance matrix, it undergoes through tanh transformation, to transform the output to the range ( − 1 , 1) and constrain to the action bounds.
A. S KILL -D YNAMICS
Skill-dynamics, denoted by q ( s (cid:48) | s, z ) , is parameterized by a deep neural network.
A common trick in model-based RL is to predict the ∆ s = s (cid:48) − s , rather than the full state s (cid:48) .
Hence, the prediction network is q (∆ s | s, z ) .
Note, both parameterizations can represent the same set of functions.
However, the latter will be easy to learn as ∆ s will be centred around 0 .
We exclude the global coordinates from from the state input to q .
However, we can (and we still do) predict ∆ x , ∆ y , because reward functions for goal-based navigation generally rely on the position prediction from the model.
This represents another beneﬁt of predicting state-deltas, as we can still predict changes in position without explicitly knowing the global position.
The output distribution is modelled as a Mixture-of-Experts (Jacobs et al., 1991).
We ﬁx the number of experts to be 4.
We model each expert as a Gaussian distribution.
The input ( s, z ) goes through two hidden layers (the same capacity as that of policy and critic networks, for example (512 , 512) for Ant).
The output of the two hidden layers is used as an input to the mixture-of-experts, which is linearly transformed to output the parameters of the Gaussian distribution, and a discrete distribution over the experts using a softmax distribution.
In practice, we ﬁx the covariance matrix of the
Gaussian experts to be an identity matrix, so we only need to output the means for the experts.
We use batch-normalization for both input and the hidden layers.
We normalize the output targets using their batch-average and batch-standard deviation, similar to batch-normalization.
A. O THER H YPERPARAMETERS
The episode horizon is generally kept shorter for stable agents like Ant ( 200 ), while longer for unstable agents like Humanoid ( 1000 ).
For Ant, longer episodes do not add value, but Humanoid can beneﬁt from longer episodes as it helps it ﬁlter skills which are unstable.
The optimization scheme is on-policy, and we collect 2000 steps for Ant and 4000 steps for Humanoid in one iteration.
The intuition is to experience trajectories generated by multiple skills (approximately 10 ) in a batch.
Re-sampling skills can enable experiencing larger number of skills.
Once a batch of episodes is collected, the skill-dynamics is updated using Adam optimizer with a ﬁxed learning rate of 3 e − 4 .
The batch size is 128 , and we carry out 32 steps of gradient descent.
To compute the intrinsic reward, we need to resample the prior for computing the denominator.
For continuous spaces, we set L = 500 .
For discrete spaces, we can marginalize over all skills.
After the intrinsic reward is computed, the policy and critic networks are updated for 128 steps with a batch size of 128 .
The intuition is to ensure that every sample in the batch is seen for policy and critic updates about 3 −times in expectation.
A. P LANNING AND E VALUATION S ETUPS
For evaluation, we ﬁx the episode horizon to 200 for all models in all evaluation setups.
Depending upon the size of the latent space and planning horizon, the number of samples from the planning distribution P is varied between 10 − 200 .
For H P = 1 , H Z = 10 and a 2 D latent space, we use 50 samples from the planning distribution P .
The co-efﬁcient γ for MPPI is ﬁxed to 10 .
We use a setting of H P = 1 and H Z = 10 for dense-reward navigation, in which case we set the number of reﬁne steps R = 10 .
However, for sparse reward navigation it is important to have a longer horizon planning, in which case we set H P = 4 , H Z = 25 with a higher number of samples from the planning distribution ( 200 from P ).
Also, when using longer planning horizons, we found that smoothing the sampled plans help.
Thus, if the sampled plan is z 1 , z 2 , z 3 , z 4 . .
, we smooth the plan to make z 2 =
βz
+ (1 − β ) z
2 and so on, with β = 0 .
For hierarchical controllers being learnt on top of low-level unsupervised primitives, we use PPO (Schulman et al., 2017) for discrete action skills, while we use SAC for continuous skills.
We keep the number of steps after which the meta-action is decided as 10 (that is H Z = 10 ).
The hidden layer sizes of the meta-controller are (128 , 128) .
We use a learning rate of 1 e − 4 for PPO and 3 e −for SAC.
For our model-based RL baseline PETS, we use an ensemble size of 3 , with a ﬁxed planning horizon of 20 .
For the model, we use a neural network with two hidden layers of size 400 .
In our experiments, we found that MPPI outperforms CEM, so we report the results using the MPPI as our controller.
B G RAPHICAL MODELS , I NFORMATION B OTTLENECK AND U NSUPERVISED S KILL L EARNING
We now present a novel perspective on unsupervised skill learning, motivated from the literature on information bottleneck.
This section takes inspiration from (Alemi & Fischer, 2018), which helps us provide a rigorous justiﬁcation for our objective proposed earlier.
To obtain our unsupervised RL objective, we setup a graphical model P as shown in Figure 9, which represents the distribution of trajectories generated by a given policy π .
The joint distribution is given by:
[Equation]
z a
[Figure]
a [Figure] a [Figure]
s [Figure] s [Figure]
s
[Figure]
We setup another graphical model N , which represents the desired model of the world.
In particular, we are interested in approximating p ( s (cid:48) | s, z ) , which represents the transition function for a particular primitive.
This abstraction helps us get away from knowing the exact actions, enabling model-based planning in behavior space (as discussed in the main paper).
The joint distribution for N shown in
Figure 10 is given by:
[Equation]
The goal of our approach is to optimize the distribution π ( a | s, z ) in the graphical model P to minimize the distance between the two distributions, when transforming to the representation of the graphical model Z .
In particular, we are interested in minimizing the KL divergence between p and η , that is D KL
( p || η ) .
Note, if N had the same structure as P , the information lost in projection would be 0 for any valid P .
Interestingly, we can exploit the following result from in Friedman et al.
(2001) to setup the objective for π , without explicitly knowing η :
[Equation]
where I P and I N represents the multi-information for distribution P on the respective graphical models.
Note, min η ∈ N D KL ( p || η ) , which is the reverse information projection (Csisz´ar & Matus, 2003).
The multi-information (Slonim et al., 2005) for a graphical model G with nodes g i is deﬁned as:
[Equation]
where Pa ( g i ) denotes the nodes upon which g i has direct conditional dependence in G .
Using this deﬁnition, we can compute the multi-information terms:
[Equation]
(cid:88) t
[Equation]
[Equation]
(cid:88) t =
I ( s t ; { s t − 1 , a t − 1 } ) and I N = T (cid:88) t = I ( s t ; { s t − 1 , z } ) . Following the Optimal Frontier argument in (Alemi & Fischer, 2018), we introduce Lagrange multipliers β t ≥ 0 , δ t ≥ 0 for the information terms in I P to setup an objective R ( π ) to be maximized with respect to π :
[Equation]
As the underlying dynamics are ﬁxed and unknown, we simplify the optimization by setting δ t = 0 which intuitively corresponds to us neglecting the unchangeable information of the underlying dynamics.
This gives us
[Equation]
Here, we have used the chain rule of mutual information: I ( s t +1 ; { s t , z } )
=
I ( s t +1 ; s t )
+
I ( s t +1 ; z | s t ) ≥ I ( s t +1 ; z | s t ) , resulting from the non-negativity of mutual information.
This yield us an information bottleneck style objective where we maximize the mutual information motivated in Section 3, while minimizing I ( a t ; { s t , z } ) .
We can show that the minimization of the latter mutual information corresponds to entropy regularization of π ( a t | s t , z ) , as follows:
[Equation]
for some arbitrary distribution log p ( a t ) (for example uniform).
Again, we have used the nonnegativity of D KL to get the inequality.
We use Equation 19 in Equation 16 to get:
[Equation]
where we have ignored p ( a t ) as it is a constant with respect to optimization for π .
This motivates the use of entropy regularization.
We can follow the arguments in Section 3 to obtain an approximate lower bound for I ( s t +1 ; z | s t ) .
The above discussion shows how DADS can be motivated from a graphical modelling perspective, while justifying the use of entropy regularization from an information bottleneck perspective.
This objective also explicates the temporally extended nature of z , and how it corresponds to a sequence of actions producing a predictable sequence of transitions in the environment. [Figure]
[Equation]
z s
We can carry out the exercise for the reward function in Eysenbach et al.
(2018) (DIAYN) to provide a graphical model interpretation of the objective used in the paper.
To conform with objective in the paper, we assume to be sampling to be state-action pairs from skill-conditioned stationary distributions in the world P , rather than trajectories.
The objective to be maximized is given by:
[Equation] [Equation]
where we have used the variational inequalities to replace p ( z | s ) with q φ ( z | s ) and π ( a ) with a uniform prior over bounded actions p ( a ) (which is ignored as a constant).
C A PPROXIMATING THE R EWARD F UNCTION
We revisit Equation 4 and the resulting approximate reward function constructed in Equation 6.
The maximization objective for policy was:
[Equation]
[Equation] The computational problem arises from the intractability of p ( s (cid:48)
[Equation]
s
[Equation]
|
[Equation]
|
[Equation]
where both p ( s (cid:48) | s, z ) and p ( z | s ) ∝ p ( s | z ) p ( z ) are intractable.
Unfortunately, any variational approximation results in an improper lower bound for the objective.
To see that:
[Equation]
where the inequality goes the wrong way for any variational approximation q ( s (cid:48) | s ) .
Our approximation can be seen as a special instantiation of q ( s (cid:48) | s )
= (cid:82) q φ ( s (cid:48) | s, z ) p ( z ) dz .
This approximation is simple to compute as generating samples from the prior p ( z ) is inexpensive and effectively requires only a forward pass through q φ .
Reusing q φ to approximate p ( s (cid:48) | s ) makes intuitive sense because we want q φ to reasonably approximate p ( s (cid:48) | s, z ) (which is why we collect large batches of data and take multiple steps of gradient descent for ﬁtting q φ ).
While sampling from the prior p ( z ) is crude, sampling p ( z | s ) can be computationally prohibitive.
For a certain class of problems, especially locomotion, sampling from p ( z ) is a reasonable approximation as well.
We want our primitives/skills to be usable from any state, which is especially the case with locomotion.
Empirically, we have found our current approximation provides satisfactory results.
We also discuss some other potential solutions (and their limitations):
(a) One could potentially use another network q β ( z | s ) to approximate p ( z | s ) by minimizing E s,z ∼ p (cid:2) D KL ( p ( z | s ) ||
q β ( z | s )) (cid:3) .
Note, the resulting approximation would still be an improper lower bound for R ( π | q φ ) .
However, sampling from this q β might result in a better approximation than sampling from the prior p ( z ) for some problems.
(b) We can bypass the computational intractability of p ( s (cid:48) | s ) by exploiting the variational lower bounds from Agakov (2004).
We use the following inequality, used in Hausman et al.
[Equation]
) where q is a variational approximation to the posterior p ( z | x ) .
[Equation]
where we have used the inequality for H ( s (cid:48) | s ) to introduce the variational posterior for skill inference q α ( z | s (cid:48) , s ) besides the conventional variational lower bound to introduce q ( s (cid:48) | s, z ) .
Further decomposing the leftover entropy:
[Equation]
Reusing the variational lower bound for marginal entropy from Agakov (2004), we get:
[Equation]
Since, the choice of posterior is upon us, we can choose q ( a | s (cid:48) , s, z )
= 1 /c to induce a uniform distribution for the bounded action space.
For H ( s (cid:48) | s, a, z ) , notice that the underlying dynamics p ( s (cid:48) | s, a ) are independent of z , but the actions do depend upon z .
Therefore, this corresponds to entropy-regularized RL when the dynamics of the system are deterministic.
Even for stochastic dynamics, the analogy might be a good approximation , assuming the underlying dynamics are not very entropic.
The ﬁnal objective (making this low-entropy dynamics assumption) can be written as:
[Equation]
While this does bypass the intractability of p ( s (cid:48) | s ) , it runs into the intractable p ( z | s ) , despite deploying signiﬁcant mathematical machinery and additional assumptions.
Any variational approximation for p ( z | s ) would again result in an improper lower bound for I ( s (cid:48) ; z | s ) .
(c) One way to a make our approximation q ( s (cid:48) | s ) to more closely resemble p ( s (cid:48) | s ) is to change our generative model p ( z, s, s (cid:48) ) .
In particular, if we resample z ∼ p ( z ) for every timestep of the rollout from π , we can indeed write p ( z | s ) =
p ( z ) .
Note, p ( s (cid:48) | s ) is still intractable to compute, but marginalizing q φ ( s (cid:48) | s, z ) over p ( z ) becomes a better approximation of p ( s (cid:48) | s ) .
However, this severely dampens the interpretation of our latent space Z as temporally extended actions (or skills).
It becomes better to interpret the latent space Z as dimensional reduction of action space.
Empirically, we found that this signiﬁcantly throttles the learning, not yielding useful or interpretable skills.
D I NTERPOLATION IN C ONTINUOUS L ATENT S PACE
E M ODEL P REDICTION
From Figure 14, we observe that skill-dynamics can provide robust state-predictions over long planning horizons.
When learning skill-dynamics with x − y prior, we observe that the error in prediction rises slower with horizon as compared to the norm of the actual position.
This provides strong evidence of cooperation between the primitives and skill-dynamics learned using DADS with x − y prior.
As the error-growth for skill-dynamics learned on full-observation space is sub-exponential, similar argument can be made for DADS without x − y prior as well (albeit to a weaker extent).