D IVERSITY IS A LL Y OU N EED : L EARNING S KILLS WITHOUT A R EWARD F UNCTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Intelligent creatures can explore their environments and learn useful skills without supervision.
In this paper, we propose “Diversity is All You Need”(DIAYN), a method for learning useful skills without a reward function.
Our proposed method learns skills by maximizing an information theoretic objective using a maximum entropy policy.
On a variety of simulated robotic tasks, we show that this simple objective results in the unsupervised emergence of diverse skills, such as walking and jumping.
In a number of reinforcement learning benchmark environments, our method is able to learn a skill that solves the benchmark task despite never receiving the true task reward.
We show how pretrained skills can provide a good parameter initialization for downstream tasks, and can be composed hierarchically to solve complex, sparse reward tasks.
Our results suggest that unsupervised discovery of skills can serve as an effective pretraining mechanism for overcoming challenges of exploration and data efﬁciency in reinforcement learning
I NTRODUCTION
Deep reinforcement learning (RL) has been demonstrated to effectively learn a wide range of rewarddriven skills, including playing games (Mnih et al., 2013; Silver et al., 2016), controlling robots (Gu et al., 2017; Schulman et al., 2015b), and navigating complex environments (Zhu et al., 2017;
Mirowski et al., 2016).
However, intelligent creatures can explore their environments and learn useful skills even without supervision, so that when they are later faced with speciﬁc goals, they can use those skills to satisfy the new goals quickly and efﬁciently.
Learning skills without reward has several practical applications.
Environments with sparse rewards effectively have no reward until the agent randomly reaches a goal state.
Learning useful skills without supervision may help address challenges in exploration in these environments.
For long horizon tasks, skills discovered without reward can serve as primitives for hierarchical RL, effectively shortening the episode length.
In many practical settings, interacting with the environment is essentially free, but evaluating the reward requires human feedback (Christiano et al., 2017).
Unsupervised learning of skills may reduce the amount of supervision necessary to learn a task.
While we can take the human out of the loop by designing a reward function, it is challenging to design a reward function that elicits the desired behaviors from the agent (Hadﬁeld-Menell et al., 2017).
Finally, when given an unfamiliar environment, it is challenging to determine what tasks an agent should be able to learn.
Unsupervised skill discovery partially answers this question.
Autonomous acquisition of useful skills without any reward signal is an exceedingly challenging problem.
A skill is a latent-conditioned policy that alters the state of the environment in a consistent way.
We consider the setting where the reward function is unknown, so we want to learn a set of skills by maximizing the utility of this set.
Making progress on this problem requires specifying a learning objective that ensures that each skill individually is distinct and that the skills collectively explore large parts of the state space.
In this paper, we show how a simple objective based on mutual information can enable RL agents to autonomously discover such skills.
These skills are useful for a number of applications, including hierarchical reinforcement learning and imitation learning.
We propose a method for learning diverse skills with deep RL in the absence of any rewards.
We hypothesize that in order to acquire skills that are useful, we must train the skills so that they maximize coverage over the set of possible behaviors.
While one skill might perform a useless behavior like random dithering, other skills should perform behaviors that are distinguishable from random dithering, and therefore more useful.
A key idea in our work is to use discriminability between skills as an objective.
Further, skills that are distinguishable are not necessarily maximally diverse – a slight difference in states makes two skills distinguishable, but not necessarily diverse in a semantically meaningful way.
To combat this problem, we want to learn skills that not only are distinguishable, but also are as diverse as possible.
By learning distinguishable skills that are as random as possible, we can “push” the skills away from each other, making each skill robust to perturbations and effectively exploring the environment.
By maximizing this objective, we can learn skills that run forward, do backﬂips, skip backwards, and perform face ﬂops (see Figure 3).
Our paper makes ﬁve contributions.
First, we propose a method for learning useful skills without any rewards.
We formalize our discriminability goal as maximizing an information theoretic objective with a maximum entropy policy.
Second, we show that this simple exploration objective results in the unsupervised emergence of diverse skills, such as running and jumping, on several simulated robotic tasks.
In a number of RL benchmark environments, our method is able to solve the benchmark task despite never receiving the true task reward.
In these environments, some of the learned skills correspond to solving the task, and each skill that solves the task does so in a distinct manner.
Third, we propose a simple method for using learned skills for hierarchical RL and ﬁnd this methods solves challenging tasks.
Four, we demonstrate how skills discovered can be quickly adapted to solve a new task.
Finally, we show how skills discovered can be used for imitation learning.
R ELATED W ORK
Previous work on hierarchical RL has learned skills to maximize a single, known, reward function by jointly learning a set of skills and a meta-controller (e.g., (Bacon et al., 2017; Heess et al., 2016;
Dayan & Hinton, 1993; Frans et al., 2017; Krishnan et al., 2017; Florensa et al., 2017)).
One problem with joint training (also noted by Shazeer et al.
(2017)) is that the meta-policy does not select “bad” options, so these options do not receive any reward signal to improve.
Our work prevents this degeneracy by using a random meta-policy during unsupervised skill-learning, such that neither the skills nor the meta-policy are aiming to solve any single task.
A second importance difference is that our approach learns skills with no reward .
Eschewing a reward function not only avoids the difﬁcult problem of reward design, but also allows our method to learn task-agnostic.
Related work has also examined connections between RL and information theory (Ziebart et al., 2008; Schulman et al., 2017; Nachum et al., 2017; Haarnoja et al., 2017) and developed maximum entropy algorithms with these ideas Haarnoja et al.
Recent work has also applied tools from information theory to skill discovery.
Mohamed & Rezende (2015) and Jung et al.
(2011) use the mutual information between states and actions as a notion of empowerment for an intrinsically motivated agent.
Our method maximizes the mutual information between states and skills , which can be interpreted as maximizing the empowerment of a hierarchical agent whose action space is the set of skills.
Hausman et al.
(2018), Florensa et al.
(2017), and Gregor et al.
(2016) showed that a discriminability objective is equivalent to maximizing the mutual information between the latent skill z and some aspect of the corresponding trajectory.
Hausman et al.
(2018) considered the setting with many tasks and reward functions and Florensa et al.
(2017) considered the setting with a single task reward.
Three important distinctions allow us to apply our method to tasks signiﬁcantly more complex than the gridworlds in Gregor et al.
First, we use maximum entropy policies to force our skills to be diverse.
Our theoretical analysis shows that including entropy maximization in the RL objective results in the mixture of skills being maximum entropy in aggregate.
Second, we ﬁx the prior distribution over skills, rather than learning it.
Doing so prevents our method from collapsing to sampling only a handful of skills.
Third, while the discriminator in Gregor et al.
(2016) only looks at the ﬁnal state, our discriminator looks at every state, which provides additional reward signal.
These three crucial differences help explain how our method learns useful skills in complex environments.
Prior work in neuroevolution and evolutionary algorithms has studied how complex behaviors can be learned by directly maximizing diversity (Lehman & Stanley, 2011a;b;
Woolley & Stanley, 2011;
Stanley & Miikkulainen, 2002; Such et al., 2017; Pugh et al., 2016; Mouret & Doncieux, 2009).
While this prior work uses diversity maximization to obtain better solutions, we aim to acquire complex skills with minimal supervision to improve efﬁciency (i.e., reduce the number of objective function queries) and as a stepping stone for imitation learning and hierarchical RL.
We focus on
[Figure]
while not converged do
Sample skill z ∼ p ( z ) and initial state
[Table]
[Figure]
[Table]
[Figure]
( s ) for
[Table]
←
[Table]
to steps _ per
[Table]
[Figure]
do
[Figure]
[Table]
[Figure]
[Table]
[Figure]
deriving a general, information-theoretic objective that does not require manual design of distance metrics and can be applied to any RL task without additional engineering.
Previous work has studied intrinsic motivation in humans and learned agents.
Ryan & Deci (2000);
Bellemare et al.
Fu et al.
(2017); Schmidhuber (2010); Oudeyer et al.
; Pathak et al.
(2017); Baranes & Oudeyer (2013).
While these previous works use an intrinsic motivation objective to learn a single policy, we propose an objective for learning many , diverse policies.
Concurrent work Achiam et al.
(2017) draws ties between learning discriminable skills and variational autoencoders.
We show that our method scales to more complex tasks, likely because of algorithmic design choices, such as our use of an off-policy RL algorithm and conditioning the discriminator on individual states.
D IVERSITY IS A LL Y OU N EED
We consider an unsupervised RL paradigm in this work, where the agent is allowed an unsupervised “exploration” stage followed by a supervised stage.
In our work, the aim of the unsupervised stage is to learn skills that eventually will make it easier to maximize the task reward in the supervised stage.
Conveniently, because skills are learned without a priori knowledge of the task, the learned skills can be used for many different tasks.
3. H OW IT W ORKS
Our method for unsupervised skill discovery, DIAYN (“Diversity is All You Need”), builds off of three ideas.
First, for skills to be useful, we want the skill to dictate the states that the agent visits.
Different skills should visit different states, and hence be distinguishable.
Second, we want to use states, not actions, to distinguish skills, because actions that do not affect the environment are not visible to an outside observer.
For example, an outside observer cannot tell how much force a robotic arm applies when grasping a cup if the cup does not move.
Finally, we encourage exploration and incentivize the skills to be as diverse as possible by learning skills that act as randomly as possible.
Skills with high entropy that remain discriminable must explore a part of the state space far away from other skills, lest the randomness in its actions lead it to states where it cannot be distinguished.
We construct our objective using notation from information theory: S and A are random variables for states and actions, respectively; Z ∼ p ( z ) is a latent variable, on which we condition our policy; we refer to a the policy conditioned on a ﬁxed Z as a “skill”; I ( · ; · ) and H [ · ] refer to mutual information and Shannon entropy, both computed with base e .
In our objective, we maximize the mutual information between skills and states, I ( S ; Z ) , to encode the idea that the skill should control which states the agent visits.
Conveniently, this mutual information dictates that we can infer the skill from the states visited.
To ensure that states, not actions, are used to distinguish skills, we minimize the mutual information between skills and actions given the state, I ( A ; Z | S ) .
Viewing all skills together with p ( z ) as a mixture of policies, we maximize the entropy H
[ A | S ] of this mixture policy.
In summary, we maximize the following objective with respect to our policy parameters, θ :
[Equation]
We rearranged our objective in Equation 2 to give intuition on how we optimize it.
The ﬁrst term encourages our prior distribution over p ( z ) to have high entropy.
We ﬁx p ( z ) to be uniform in our approach, guaranteeing that it has maximum entropy.
The second term suggests that it should be easy to infer the skill z from the current state.
The third term suggests that each skill should act as randomly as possible, which we achieve by using a maximum entropy policy to represent each skill.
As we cannot integrate over all states and skills to compute p ( z | s ) exactly, we approximate this posterior with a learned discriminator q φ ( z | s ) .
Jensen’s Inequality tells us that replacing p ( z | s ) with q φ ( z | s ) gives us a variational lower bound G ( θ, φ ) on our objective F ( θ ) (see (Agakov, 2004) for a detailed derivation):
[Equation]
3. I MPLEMENTATION
We implement DIAYN with soft actor critic (SAC) (Haarnoja et al., 2018), learning a policy π θ ( a | s, z ) that is conditioned on the latent variable z .
Soft actor critic maximizes the policy’s entropy over actions, which takes care of the entropy term in our objective G .
Following Haarnoja et al.
(2018), we scale the entropy regularizer H [ a | s, z ] by α .
We found empirically that an α = 0 .provided a good trade-off between exploration and discriminability.
We maximize the expectation in G by replacing the task reward with the following pseudo-reward:
[Equation]
We use a categorical distribution for p ( z ) .
During unsupervised learning, we sample a skill z ∼ p ( z ) at the start of each episode, and act according to that skill throughout the episode.
The agent is rewarded for visiting states that are easy to discriminate, while the discriminator is updated to better infer the skill z from states visited.
Entropy regularization occurs as part of the SAC update.
3. S TABILITY
Unlike prior adversarial unsupervised RL methods (e.g., Sukhbaatar et al.
, DIAYN forms a cooperative game, which avoids many of the instabilities of adversarial saddle-point formulations.
On gridworlds, we can compute analytically that the unique optimum to the DIAYN optimization problem is to evenly partition the states between skills, with each skill assuming a uniform stationary distribution over its partition (proof in Appendix B).
In the continuous and approximate setting, convergence guarantees would be desirable, but this is a very tall order: even standard RL methods with function approximation (e.g., DQN) lack convergence guarantees, yet such techniques are still useful.
Empirically, we ﬁnd DIAYN to be robust to random seed; varying the random seed does not noticeably affect the skills learned, and has little effect on downstream tasks (see Fig.s 4, 6, and 13).
E XPERIMENTS
In this section, we evaluate DIAYN and compare to prior work.
First, we analyze the skills themselves, providing intuition for the types of skills learned, the training dynamics, and how we avoid problematic behavior in previous work.
In the second half, we show how the skills can be used for downstream tasks, via policy initialization, hierarchy, imitation, outperforming competitive baselines on most tasks.
We encourage readers to view videos 3 and code 4 for our experiments.
4. A NALYSIS OF L EARNED S KILLS
Question 1.
What skills does DIAYN learn?
[Figure]
We study the skills learned by DIAYN on tasks of increasing complexity, ranging from point navigation (2 dimensions) to ant locomotion (111 dimensions).
We ﬁrst applied DIAYN to a simple 2D navigation environment.
The agent starts in the center of the box, and can take actions to directly move its ( x, y ) position.
Figure 2a illustrates how the 6 skills learned for this task move away from each other to remain distinguishable.
Next, we applied DIAYN to two classic control tasks, inverted pendulum and mountain car.
Not only does our approach learn skills that solve the task without rewards, it learns multiple distinct skills for solving the task.
(See Appendix D for further analysis.)
Finally, we applied DIAYN to three continuous control tasks (Brockman et al., 2016): half cheetah, hopper, and ant.
As shown in Figure 3, we learn a diverse set of primitive behaviors for all tasks.
For half cheetah, we learn skills for running forwards and backwards at various speeds, as well as skills for doing ﬂips and falling over; ant learns skills for jumping and walking in many types of curved trajectories (though none walk in a straight line); hopper learns skills for balancing, hopping forward and backwards, and diving.
See Appendix D.4 for a comparison with VIME.
Question 2.
How does the distribution of skills change during training?
While DIAYN learns skills without a reward function, as an outside observer, can we evaluate the skills throughout training to understand the training dynamics.
Figure 2 shows how the skills for inverted pendulum and mountain car become increasingly diverse throughout training (Fig.
13 repeats this experiment for 5 random seeds, and shows that results are robust to initialization).
Recall that our skills are learned with no reward, so it is natural that some skills correspond to small task reward while others correspond to large task reward.
Question 3.
Does discriminating on single states restrict DIAYN to learn skills that visit disjoint sets of states?
Our discriminator operates at the level of states, not trajectories.
While DIAYN favors skills that do not overlap, our method is not limited to learning skills that visit entirely disjoint sets of states.
Figure 2b shows a simple experiment illustrating this.
The agent starts in a hallway (green star), and can move more freely once exiting the end of the hallway into a large room.
Because RL agents are incentivized to maximize their cumulative reward, they may take actions that initially give no reward to reach states that eventually give high reward.
In this environment, DIAYN learns skills that exit the hallway to make them mutually distinguishable.
Question 4.
How does DIAYN differ from Variational Intrinsic Control (VIC) (Gregor et al., 2016)?
The key difference from the most similar prior work on unsupervised skill discovery, VIC, is our decision to not learn the prior p ( z ) .
We found that VIC suffers from the “Matthew Effect” Merton (1968): VIC’s learned prior p ( z ) will sample the more diverse skills more frequently, and hence only those skills will receive training signal to improve.
To study this, we evaluated DIAYN and VIC on the half-cheetah environment, and plotting the effective number of skills (measured as exp( H [ Z ]) ) throughout training (details and more ﬁgures in Appendix E.2).
The ﬁgure to the right shows how VIC quickly converges to a setting where it only samples a handful of skills.
In contrast, DIAYN ﬁxes the distribution over skills, which allows us to discover more diverse skills.
4. H ARNESSING L EARNED S KILLS
The perhaps surprising ﬁnding that we can discover diverse skills without a reward function creates a building block for many problems in RL.
For example, to ﬁnd a policy that achieves a high reward on a task, it is often sufﬁcient to simply choose the skill with largest reward.
Three less obvious applications are adapting skills to maximize a reward, hierarchical RL, and imitation learning.
4.2. A CCELERATING L EARNING WITH P OLICY I NITIALIZATION
After DIAYN learns task-agnostic skills without supervision, we can quickly adapt the skills to solve a desired task.
Akin to the use of pre-trained models in computer vision, we propose that DIAYN can serve as unsupervised pre-training for more sample-efﬁcient ﬁnetuning of task-speciﬁc policies.
Question 5.
Can we use learned skills to directly maximize the task reward?
We take the skill with highest reward for each benchmark task and further ﬁnetune this skill using the task-speciﬁc reward function.
We compare to a “random initialization” baseline that is initialized from scratch.
Our approach differs from this baseline only in how weights are initialized.
We initialize both the policy and value networks with weights learned during unsupervised pretraining.
Although the critic networks learned during pretraining corresponds to the pseudo-reward from the discriminator (Eq. 3) and not the true task reward, we found empirically that the pseudo-reward was close to the true task reward for the best skill, and initializing the critic in addition to the actor further sped up learning.
Figure 5 shows both methods applied to half cheetah, hopper, and ant.
We assume that the unsupervised pretraining is free (e.g., only the reward function is expensive to compute) or can be amortized across many tasks, so we omit pretraining steps from this plot.
On all tasks, unsupervised pretraining enables the agent to learn the benchmark task more quickly.
4.2. U SING S KILLS FOR H IERARCHICAL RL
In theory, hierarchical RL should decompose a complex task into motion primitives, which may be reused for multiple tasks.
In practice, algorithms for hierarchical RL can encounter many problems: (1) each motion primitive reduces to a single action (Bacon et al., 2017), (2) the hierarchical policy only samples a single motion primitive (Gregor et al., 2016), or (3) all motion primitives attempt to do the entire task.
In contrast, DIAYN discovers diverse, task-agnostic skills, which hold the promise of acting as a building block for hierarchical RL.
Question 6.
Are skills discovered by DIAYN useful for hierarchical RL?
We propose a simple extension to DIAYN for hierarchical RL, and ﬁnd that simple algorithm outperforms competitive baselines on two challenging tasks.
To use the discovered skills for hierarchical
RL, we learn a meta-controller whose actions are to choose which skill to execute for the next k steps (100 for ant navigation, 10 for cheetah hurdle).
The meta-controller has the same observation space as the skills.
As an initial test, we applied the hierarchical RL algorithm to a simple 2D point navigation task (details in Appendix C.2).
Figure 6 illustrates how the reward on this task increases with the number of skills; error bars show the standard deviation across 5 random seeds.
To ensure that our goals were not cherry picked, we sampledgoals evenly from the state space, and evaluated each random seed on all goals.
We also compared to Variational Information Maximizing Exploration (VIME) (Houthooft et al., 2016).
Note that even the best random seed from VIME signiﬁcantly under-performs DIAYN.
This is not surprising: whereas DIAYN learns a set of skills that effectively partition the state space, VIME attempts to learn a single policy that visits many states.
Next, we applied the hierarchical algorithm to two challenging simulated robotics environment.
On the cheetah hurdle task, the agent is rewarded for bounding up and over hurdles, while in the ant navigation task, the agent must walk to a set of 5 waypoints in a speciﬁc order, receiving only a sparse reward upon reaching each waypoint.
The sparse reward and obstacles in these environments make them exceedingly difﬁcult for non-hierarchical RL algorithms.
Indeed, state of the art RL algorithms that do not use hierarchies perform poorly on these tasks.
Figure 8 shows how DIAYN outperforms state of the art on-policy RL (TRPO (Schulman et al., 2015a)), off-policy RL (SAC (Haarnoja et al., 2018)), and exploration bonuses (VIME).
This experiment suggests that unsupervised skill learning provides an effective mechanism for combating challenges of exploration and sparse rewards in RL.
Question 7.
How can DIAYN leverage prior knowledge about what skills will be useful?
If the number of possible skills grows exponentially with the dimension of the task observation, one might imagine that DIAYN would fail to learn skills necessary to solve some tasks.
While we found that DIAYN does scale to tasks with more than 100 dimensions (ant has 111), we can also use a simple modiﬁcation to bias DIAYN towards discovering particular types of skills.
We can condition
[Figure]
the discriminator on only a subset of the observation space, or any other function of the observations.
In this case, the discriminator maximizes E [log q φ ( z | f ( s ))] .
For example, in the ant navigation task, f ( s ) could compute the agent’s center of mass, and DIAYN would learn skills that correspond to changing the center of mass.
The “DIAYN+prior” result in Figure 8 (right) shows how incorporating this prior knowledge can aid DIAYN in discovering useful skills and boost performance on the hierarchical task.
(No other experiments or ﬁgures in this paper used this prior.)
The key takeaway is that while DIAYN is primarily an unsupervised RL algorithm, there is a simple mechanism for incorporating supervision when it is available.
Unsurprisingly, we perform better on hierarchical tasks when incorporating more supervision.
4.2. I MITATING AN E XPERT
Question 8.
Can we use learned skills to imitate an expert?
Aside from maximizing reward with ﬁnetuning and hierarchical RL, we can also use learned skills to follow expert demonstrations.
One use-case is where a human manually controls the agent to complete a task that we would like to automate.
Simply replaying the human’s actions fails in stochastic environments, cases where closed-loop control is necessary.
A second use-case involves an existing agent with a hard coded, manually designed policy.
Imitation learning replaces the existing policy with a similar yet differentiable policy, which might be easier to update in response to new constraints or objectives.
We consider the setting where we are given an expert trajectory consisting of states, without actions, deﬁned as τ ∗ =
{ ( s i ) } 1 ≤ i ≤ N .
Our goal is to obtain a feedback controller that will reach the same states.
Given the expert trajectory, we use our learned discriminator to estimate which skill was most likely to have generated the trajectory.
This optimization problem, which we solve for categorical z by enumeration, is equivalent to an M-projection (Bishop, 2016):
[Equation]
We qualitatively evaluate this approach to imitation learning on half cheetah.
Figure 9 (left) shows four imitation tasks, three of which our method successfully imitates.
We quantitatively evaluate this imitation method on classic control tasks in Appendix G.
C ONCLUSION
In this paper, we present DIAYN, a method for learning skills without reward functions.
We show that DIAYN learns diverse skills for complex tasks, often solving benchmark tasks with one of the learned skills without actually receiving any task reward.
We further proposed methods for using the learned skills (1) to quickly adapt to a new task, (2) to solve complex tasks via hierarchical RL, and (3) to imitate an expert.
As a rule of thumb, DIAYN may make learning a task easier by replacing the task’s complex action space with a set of useful skills.
DIAYN could be combined with methods for augmenting the observation space and reward function.
Using the common language of information theory, a joint objective can likely be derived.
DIAYN may also more efﬁciently learn from human preferences by having humans select among learned skills.
Finally, the skills produced by DIAYN might be used by game designers to allow players to control complex robots and by artists to animate characters.
B O PTIMUM FOR G RIDWORLDS
For simple environments, we can compute an analytic solution to the DIAYN objective.
For example, consider a N × N gridworld, where actions are to move up/down/left/right.
Any action can be taken in any state, but the agent will stay in place if it attempts to move out of the gridworld.
We use ( x, y ) to refer to states, where x, y ∈ { 1 , 2 , · · · , N } .
For simplicity, we assume that, for every skill, the distribution of states visited exactly equals that skill’s stationary distribution over states.
To clarify, we will use π z to refer to the policy for skill z .
We use ρ π z to indicate skill z ’s stationary distribution over states, and ˆ ρ π z as the empirical distribution over states within a single episode.
Our assumption is equivalent to saying
[Equation]
One way to ensure this is to assume inﬁnite-length episodes.
We want to show that a set of skills that evenly partitions the state space is the optimum of the DIAYN objective for this task.
While we will show this only for the 2-skill case, the 4 skill case is analogous.
SKILLSKILL
[Figure]
The optimum policies for a set of two skills are those which evenly partition the state space.
We will show that a top/bottom partition is one such (global) optima.
The left/right case is analogous.
Lemma B.1.
A pair of skills with state distributions given below (and shown in Figure 10) are an optimum for the DIAYN objective with no entropy regularization ( α = 0 ).
[Equation]
and
[Equation]
Before proving Lemma B.1, we note that there exist policies that achieve these stationary distributions.
Figure 10b shows one such policy, were each arrow indicates a transition with probability 14 .
Note that when the agent is in the bottom row of yellow states, it does not transition to the green states, and instead stays in place with probability 14 .
Note that the distribution in Equation 4 satisﬁes the detailed balance equations (Murphy, 2012).
Proof.
Recall that the DIAYN objective with no entropy regularization is:
[Equation]
Because the skills partition the states, we can always infer the skill from the state, so
H
[ Z | S ] = 0 .
By construction, the prior distribution over H [ Z ] is uniform, so H [ Z ] = log(2) is maximized.
Thus, a set of two skills that partition the state space maximizes the un-regularized DIAYN objective.
Next, we consider the regularized objective.
In this case, we will show that while an even partition is not perfectly optimal, it is “close” to optimal, and its “distance” from optimal goes to zero as the gridworld grows in size.
This analysis will give us additional insight into the skills preferred by the
DIAYN objective.
Lemma B.2.
A pair of skills with state distributions given given in Equation 4 achieve an DIAYN objective within a factor of O (1 /N ) of the optimum, where N is the gridworld size.
Proof.
Recall that the DIAYN objective with no entropy regularization is:
[Equation]
We have already computed the second two terms in the previous proof: H
[ Z | S ] = 0 and
H [ Z ] = log(2) .
For computing the ﬁrst term, it is helpful to deﬁne the set of “border states” for a particular skill as those that do not neighbor another skill.
For skill 1 deﬁned in Figure 10 (colored yellow), the border states are: { ( x, y ) | y = 4 } .
Now, computing the ﬁrst term is straightforward:
[Equation]
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) non-border states log(
[Equation]
(cid:124)(cid:123)(cid:122)(cid:125) border states
[Equation]
[Equation]
) (cid:19)
[Equation]
Thus, the overall objective is within log(4)2 N of optimum. >
[Figure]
>
[Figure]
Note that the term for maximum entropy over actions ( H [ A | S, Z ] ) comes into conﬂict with the term for discriminability ( −H [ Z | S ] ) at states along the border between two skills.
Everything else being equal, this conﬂict encourages DIAYN to produce skills that have small borders, as shown in
Figure 11.
For example, in a gridworld with dimensions N < M , a pair of skills that split along the ﬁrst dimension (producing partitions of size ( N, M/ 2) ) would achieve a larger (better) objective than skills that split along the second dimension.
This same intuition that DIAYN seeks to minimize the border length between skills results in DIAYN preferring partitions that correspond to bottleneck states (see Figure 11b).
C E XPERIMENTAL D ETAILS
In our experiments, we use the same hyperparameters as those in Haarnoja et al.
(2018), with one notable exception.
For the Q function, value function, and policy, we use neural networks withhidden units instead of 128 units.
We found that increasing the model capacity was necessary to learn many diverse skills.
When comparing the “skill initialization” to the “random initialization” in
Section 4.2, we use the same model architecture for both methods.
To pass skill z to the Q function, value function, and policy, we simply concatenate z to the current state s t .
As in Haarnoja et al.
(2018), epochs are 1000 episodes long.
For all environments, episodes are at most 1000 steps long, but may be shorter.
For example, the standard benchmark hopper environment terminates the episode once it falls over.
Figures 2 and 5 show up to 1000 epochs, which corresponds to at most 1 million steps.
We found that learning was most stable when we scaled the maximum entropy objective ( H [ A | S, Z ] in Eq. 1) by α = 0 .
We use this scaling for all experiments.
C. E NVIRONMENTS
Most of our experiments used the following, standard RL environments (Brockman et al., 2016):
HalfCheetah-v1, Ant-v1, Hopper-v1, MountainCarContinuous-v0, and InvertedPendulum-v1.
The simple 2D navigation task used in Figures 2a and 6 was constructed as follows.
The agent starts in the center of the unit box.
Observations s ∈ [0 , 1] 2 are the agent’s position.
Actions a ∈ [ − 0 .
1]directly change the agent’s position.
If the agent takes an action to leave the box, it is projected to the closest point inside the box.
The cheetah hurdle environment is a modiﬁcation of HalfCheetah-v1, where we added boxes with shape H = 0 .
25 m, W = 0 .
1 m, D = 1 . 0 m , where the width dimension is along the same axis as the cheetah’s forward movement.
We placed the boxes ever 3 meters, start at x = − 1 m .
The ant navigation environment is a modiﬁcation of Ant-v1.
To improve stability, we follow Pong et al. (2018) and lower the gear ratio of all joints to 30.
The goals are the corners of a square, centered at the origin, with side length of 4 meters: [(2 , 2) , (2 , − 2) , ( − 2 , − 2) , ( − 2 , 2) , (2 , 2)] .
The ant starts at the origin, and receives a reward of +1 when its center of mass is within 0.5 meters of the correct next goal.
Each reward can only be received once, so the maximum possible reward is +5.
C. H IERARCHICAL RL E XPERIMENT
For the 2D navigation experiment shown in Figure 6, we ﬁrst learned a set of skills on the point environment.
Next, we introduced a reward function r g ( s ) =
−(cid:107)
s − g (cid:107) 22 penalizing the distance from the agent’s state to some goal, and applied the hierarchical algorithm above.
In this task, the
DIAYN skills provided sufﬁcient coverage of the state space that the hierarchical policy only needed to take a single action (i.e., choose a single skill) to complete the task.
D M ORE A NALYSIS OF DIAYN S KILLS D. T RAINING O BJECTIVES
To provide further intuition into our approach, Figure 12 plots the two terms in our objective throughout training.
Our skills become increasingly diverse throughout training without converging to deterministic policies.
Figure 13: We repeated the experiment from Figure 2 with 5 random seeds to illustrate the robustness of our method to random seed.
To illustrate the stability of DIAYN to random seed, we repeated the experiment in Figure 2 forrandom seeds.
Figure 13 illustrates that the random seed has little effect on the training dynamics.
D. E FFECT OF E NTROPY R EGULARIZATION
Question 9.
Does entropy regularization lead to more diverse skills?
α =
.α
=
α =
To answer this question, we apply our method to a 2D point mass.
The agent controls the orientation and forward velocity of the point, with is conﬁned within a 2D box.
We vary the entropy regularization α , with larger values of α corresponding to policies with more stochastic actions.
With small α , we learn skills that move large distances in different directions but fail to explore large parts of the state space.
Increasing α makes the skills visit a more diverse set of states, which may help with exploration in complex state spaces.
It is difﬁcult to discriminate skills when α is further increased.
D. D ISTRIBUTION OVER T ASK R EWARD
[Figure]
In Figure 15, we take the skills learned without any rewards, and evaluate each of them on the standard benchmark reward function.
We compare to random (untrained) skills.
The wide distribution over rewards is evidence that the skills learned are diverse.
For hopper, some skills hop or stand for the entire episode, receiving a reward of at least 1000.
Other skills aggressively hop forwards or dive backwards, and receive rewards between 100 and 1000.
Other skills fall over immediately and receive rewards of less than 100.
The benchmark half cheetah reward includes a control penalty for taking actions.
Unlike random skills, learned skills rarely have task reward near zero, indicating that all take actions to become distinguishable.
Skills that run in place, ﬂop on their nose, or do backﬂips receive reward of -100.
Skills that receive substantially smaller reward correspond to running quickly backwards, while skills that receive substantially larger reward correspond to running forward.
Similarly, the benchmark ant task reward includes both a control penalty and a survival bonus, so random skills that do nothing receive a task reward near 1000.
While no single learned skill learns to run directly forward and obtain a task reward greater than 1000, our learned skills run in different patterns to become discriminable, resulting in a lower task reward.
D. E XPLORATION
Question 10.
Does DIAYN explore effectively in complex environments?
We apply DIAYN to three standard RL benchmark environments: half-cheetah, hopper, and ant.
In all environments, we learn diverse locomotion primitives, as shown in Figure 3.
Despite never receiving any reward, the half cheetah and hopper learn skills that move forward and achieve large task reward on the corresponding RL benchmarks, which all require them to move forward at a fast pace.
Half cheetah and hopper also learn skills that move backwards, corresponding to receiving a task reward much smaller than what a random policy would receive.
Unlike hopper and half cheetah, the ant is free to move in the XY plane.
While it learns skills that move in different directions, most skills move in arcs rather than straight lines, meaning that we rarely learn a single skill that achieves large task reward on the typical task of running forward.
In the appendix, we visualize the objective throughout training.
In Figure 16, we evaluate all skills on three reward functions: running (maximize X coordinate), jumping (maximize Z coordinate) and moving (maximize L2 distance from origin).
For each skill,
DIAYN learns some skills that achieve high reward.
We compare to single policy trained with a pure exploration objective (VIME (Houthooft et al., 2016)).
Whereas previous work (e.g., Pathak et al.
; Bellemare et al.
; Houthooft et al.
(2016)) ﬁnds a single policy that explores well,
DIAYN optimizes a collection of policies, which enables more diverse exploration.
E L EARNING p ( z )
We used our method as a starting point when comparing to VIC (Gregor et al., 2016) in Section 4.2.
While p ( z ) is ﬁxed in our method, we implement VIC by learning p ( z ) .
In this section, we describe how we learned p ( z ) , and show the effect of learning p ( z ) rather than leaving it ﬁxed.
E. H OW TO L EARN p ( z )
We choose p ( z ) to optimize the following objective, where p z ( s ) is the distribution over states induced by skill s :
[Equation]
For clarity, we deﬁne p tz ( s ) as the distribution over states induced by skill z at epoch t , and deﬁne (cid:96) t ( z ) as an approximation of E [log p ( z | s )] using the policy and discriminator from epoch t :
[Equation]
Noting that p ( z ) is constrained to sum to 1, we can optimize this objective using the method of
Lagrange multipliers.
The corresponding Lagrangian is
[Equation]
whose derivative is
[Equation]
Setting the derivative equal to zero
[Equation]
we get
[Equation]
and ﬁnally arrive at
[Equation]
E. E FFECT OF L EARNING p ( z )
In this section, we brieﬂy discuss the effect of learning p ( z ) rather than leaving it ﬁxed.
To study the effect of learning p ( z ) , we compared the entropy of p ( z ) throughout training.
When p ( z ) is ﬁxed, the entropy is a constant ( log(50) ≈ 3 . 9 ).
To convert nats to a more interpretable quantity, we compute the effective number of skills by exponentiation the entropy:
[Equation]
Figure 17 shows the effective number of skills for half cheetah, inverted pendulum, and mountain car.
Note how the effective number of skills drops by a factor of 10x when we learn p ( z ) .
This observation supports our claim that learning p ( z ) results in learning fewer diverse skills.
Figure 18 is a repeat of the experiment in Figure 17, where we varying the dimension of z .
Note that the dimension of z equals the maximum number of skills that the agent could learn.
We observe that the effective number of skills plummets throughout training, even when using a high-dimensional vector for z .
F V ISUALIZING L EARNED S KILLS F. C LASSIC C ONTROL T ASKS
In this section, we visualize the skills learned for inverted pendulum and mountain car without a reward.
Not only does our approach learn skills that solve the task without rewards, it learns multiple distinct skills for solving the task.
Figure 19 shows the X position of the agent across time, within one episode.
For inverted pendulum (Fig. 19a), we plot only skills that solve the task.
Horizontal lines with different X coordinates correspond to skills balancing the pendulum at different positions along the track.
The periodic lines correspond to skills that oscillate back and forth while balancing the pendulum.
Note that skills that oscillate have different X positions, amplitudes, and periods.
For [Figure] mountain car (Fig. 19b), skills that climb the mountain employ a variety of strategies for to do so.
Most start moving backwards to gather enough speed to summit the mountain, while others start forwards, then go backwards, and then turn around to summit the mountain.
Additionally, note that skills differ in when the turn around and in their velocity (slope of the green lines).
F. S IMULATED R OBOT T ASKS
Figures 20, 21, and 22 show more skills learned without reward .
G I MITATION L EARNING
Given the expert trajectory, we use our learned discriminator to estimate which skill was most likely to have generated the trajectory:
[Equation]
As motivation for this optimization problem, note that each skill induces a distribution over states, p z (cid:44) p ( s | z ) .
We use p ∗ to denote the distribution over states for the expert policy.
With a ﬁxed prior distribution p ( z ) and a perfect discriminator q φ ( z | s )
=
p ( z | s ) , we have p ( s | z ) ∝ q φ ( z | s ) as a function of z .
Thus, Equation G is an M-projection of the expert distribution over states onto the family of distributions over states, P = { p z } :
[Equation]
For clarity, we omit a constant that depends only on p ∗ .
Note that the use of an M-projection, rather than an I-projection, helps guarantee that the retrieved skill will visit all states that the expert visits (Bishop, 2016).
In our experiments, we solve Equation 5 by simply iterating over skills.
G. I MITATION L EARNING E XPERIMENTS
The “expert” trajectories are actually generated synthetically in these experiments, by running a different random seed of our algorithm.
A different seed is used to ensure that the trajectories are not actually produced by any of the currently available skills.
Of course, in practice, the expert trajectories might be provided by any other means, including a human.
For each expert trajectory, we retrieve the closest DIAYN skill ˆ
z using Equation 4.2.3.
Evaluating q φ (ˆ z | τ ∗ ) gives us an estimate of the probability that the imitation will match the expert (e.g., for a safety critical setting).
This quantity is useful for predicting how accurately our method will imitate an expert before executing the imitation policy.
In a safety critical setting, a user may avoid attempting tasks where this score is low.
We compare our method to three baselines.
The “low entropy” baseline is a variant on our method with lower entropy regularization.
The “learned p ( z ) ” baseline learns the distribution over skills.
Note that Variational Intrinsic Control (Gregor et al., 2016) is a combination of the “low entropy” baseline and the “learned p ( z ) ” baseline.
Finally, the “few skills” baseline learns only 5 skills, whereas all other methods learn 50.
Figure 23 shows the results aggregated across 600 imitation tasks.
The
X-axis shows the discriminator score, our estimate for how well the imitation policy will match the expert.
The Y-axis shows the true distance between the trajectories, as measured by L2 distance in state space.
For all methods, the distance between the expert and the imitation decreases as the discriminator’s score increases, indicating that the discriminator’s score is a good predictor of task performance.
Our method consistently achieves the lowest trajectory distance among all methods.
The “low entropy” baseline is slightly worse, motivating our decision to learn maximum entropy skills.
When imitating tasks using the “few skills” baseline, the imitation trajectories are even further from the expert trajectory.
This is expected – by learning more skills, we obtain a better “coverage” over the space of skills.
A “learn p ( z ) ” baseline that learns the distribution over skills also performs poorly.
Recalling that Gregor et al.
(2016) is a combination of the “low entropy” baseline and the “learn p ( z ) ” baseline, this plot provides evidence that using maximum entropy policies and ﬁxing the distribution for p ( z ) are two factors that enabled our method to scale to more complex tasks.