L EARNING TRANSITIONAL SKILLS WITH INTRINSIC MOTIVATION
Anonymous authors
Paper under double-blind review
A BSTRACT
By maximizing an information theoretic objective, a few recent methods empower the agent to explore the environment and learn skills without extrinsic reward.
However, when considering to use multiple consecutive skills to complete a speciﬁc task, the transition from one to another cannot guarantee the success of the process due to the evident gap between skills.
In this paper, we propose to learn transitional skills (LTS) in addition to creating diverse primitive skills.
By introducing an extra latent variable for transitional skills and a compensation term for intrinsic reward, our LTS method discovers both primitive and transitional skills by maximizing the mutual information with compensation.
By considering various robotic tasks, our results demonstrate the effectiveness of LTS on learning both diverse primitive skills and transitional skills, and show its superiority in smooth transition of skills over the state-of-the-art baseline DIAYN.
In addition, we further show the signiﬁcance of transitional skills for learning downstream tasks.
I NTRODUCTION
Deep reinforcement learning (DRL) has shown its great effectiveness in learning various rewarddriven skills in wide domains, such as performing robotic manipulation tasks (Levine et al.
playing video games (Mnih et al.
, playing adversarial board games (Silver et al.
and implementing robot navigation in complex environments (Wang et al.
Nevertheless, for the majority of real applications, there is no reward in a long term until the agent reaches a goal state (Wu & Chen (2007)), especially in unseen environments.
In such cases, DRL has difﬁculty in carrying out the tasks.
By observing the human intelligence that can explore their surroundings and learn valuable skills without reward, a couple of prior works have been recently proposed to generate skills without supervision by incorporating intrinsic motivation into DRL methods (Barto (2013),Ryan & Deci (2000)).
Diverse skills can be autonomously acquired without extrinsic reward by maximizing an information theoretic objective using a maximum entropy policy (DIAYN (Eysenbach et al.
VIC (Gregor et al.
; DAS (Sharma et al.
Discovered skills may help the exploration in complex environments, and can also serve as primitive skills for hierarchical DRL.
Particularly, a high-level meta-policy could be adopted in the hierarchical framework to choose corresponding low-level primitive skills to complete tasks in order.
Although discovered skills are both distinguishable and diverse, it is still exceedingly difﬁcult to integrate such skills for a complex task that requires smooth transitions between skills (Lee et al.
Take the basketball as an example: learning the passing, catching and shooting skills in an isolated way cannot guarantee to score in the court due to the possible failure in the process of transitions between skills.
To address this problem, we propose to further learn transitional skills (LTS), where discovered primitive skills, same as prior works (Eysenbach et al.
(2018)), are distinguishable and as diverse as possible.
More concretely, our LTS method learns both primitive and transitional skills by optimizing an information theoretic objective, where extra transitional skills are generated to ﬁll in the gap between diverse primitive skills.
For such purpose, aside from using the latent variable on which we condition primitive skills, an extra latent variable is introduced on which we condition our transitional policy.
Furthermore, a compensation term has to be considered in the objective function because the distinct between primitive skills and transitional skills will lead to the decline of multual information between the latent variable corresponding to primitive skills and states generated by transitional skills.
This compensation considers the divergence of the latent variable corresponding to primitive skills and that for transitional skills.
Different from learning primitive skills, our learning process considers two arbitrary primitive skills and multiple transitional skills between them, where both primitive and transitional skillls are unknown and to be learned.
By maximizing the multual information with compensation, both primitive skills and transitional skills are discovered, which can be used to effectively learn downstream tasks.
On four simulated robotic tasks, experimental results show that our LTS can discover both primitive skills and transitional skills, successfully perform the transition between primitive skills that are distinguishable, and achieve a better peformance in comparison to the state-of-the-art baseline
DIAYN.
The main contributions of our work can be summarized as follows.
Our proposed LTS can learn both primitive and transitional skills without extrinsic reward, where the primitive skills are distinguishable and diverse, and the transitional skills can accomplish smooth transitions between primitive skills.
And extensive experiments are conducted, which demonstrates the effectiveness of our LTS method in solving downstream tasks, performing the transition between primitive skills as well as the weighted way to compose skills.
P RELIMINARIES
RL :
In the standard RL setup, an agent interacts with an environment over discrete time.
At time step t , the agent observes the current state s t and selects an action a t according to a policy π ( a t
| s t ) .
Then, the agent receives a reward r t and comes to the next state s t +1 .
The objective of learning is to maximize the discounted return R = (cid:80) ∞ t =
γ t r t of the policy π , where γ ∈ [0 , 1] is a discount factor.
Learn Skills with RL : Using the notation from information theory: we introduce two random variables S and A for states and actions, respectively.
To discover diverse skills, a latent variable ω n ∼ p ( ω ) is deﬁned on which we condition the policy π ( a t | s t , ω n ) .
Such policy is deﬁned as a skill (Eysenbach et al.
Prior works verify that maximizing the mutual information between the states S and the skills ω n will successfully generate lots of distinguishable, diverse and useful skills.
By primarily maximizing the mutual information between the ﬁnal state S f and the skills ω n given the initial state S 0 ,
[Equation]
the variational intrinsic control (VIC) (Gregor et al. (2016)) shows the success of acquiring distinguishable skills from the ﬁnal state.
Furthermore, in order to enhance the diversity of skills as much as possible, DIAYN primarily maximizes the mutual information between the state S at all time stamps and the skills ω n (Eysenbach et al.
[Equation]
which indicates that different skills generate different trajectories that traverse different states, and such diverse skills can be identiﬁed distinguishably.
Both VIC and DIAYN successfully discover primitive skills by maximizing the mutual information betweens state and skills.
However, prior works cannot accomplish a variety of robot tasks that require smooth transition between skills.
To address such problem, we propose the LTS scheme in this paper to learn both primitive and transitional skills by using an information theoretic objective function.
[Figure]
M ETHODOLOGY
In this section, we elaborate our proposed LTS method to discover both primitive and transitional skills without extrinsic reward.
Deﬁne by S and A states and actions, respectively.
Deﬁne by Ω n ∼ p ( ω ) , 1 ≤ n ≤ N , N latent variables on which we condition the primitive skill π ( a t | s t , ω ) .
Given the starting and ending primitive skill that are conditioned on ω i and ω j , respectively, we deﬁne z
i,j,k , 1 ≤ k ≤ K − 1 , as K latent variables on which we condition the transitional skill π ( a t | s t , z i,j,k ) .
Figure 1 shows the block diagram of primitive and transitional skills as well as the correponding states.
Our objective is to discover primitive skills π ( a t | s t , ω i ) , π ( a t
| s t , ω j ) , and transitional skills π ( a t | s t , z i,j,k ) .
Speciﬁcally, we discover skills by maximizing the following mutual information
[Equation]
) where S z represents one of states S ti,j,k at time stamp t , 1 ≤ k ≤ K − 1 , in Figure 1.
It is worth being noted that when the starting primitive skill is same with the ending primitive skill, no transitional skill needs to be discovered.
In such case, maximizing L ( θ ) in (3) is equivalent to maximizing
I ( S z ; Ω i ) , which is identical to learning only primitive skills Gregor et al. (2016);
Eysenbach et al.
The challege is how to solve such problem associated with two primitive skills and acquire the intrinsic reward.
Our solution is to use a compensation for individual mutual information so that two components become separable.
It is observed that, when the transitional skill π ( a t | s t , z i,j,k ) transits from the starting primitive skill π ( a t | s t , ω i ) to the ending primitive skill π ( a t | s t , ω j ) , the mutual information
I ( S z ; Ω i ) decreases progressively while the mutual information I ( S z ; Ω j ) increases progressively.
In order to compensate the mutual information, we deﬁne f
dω i ,z i,j,k as the divergence of the latent variables ω
i and z i,j,k , which indicates the deviation of z i,j,k from ω i .
Similarly, we deﬁne f
dz i,j,k ,ω j as the divergence of the latent variables
z i,j,k and ω j , which indicates the deviation of ω j from z i,j,k .
The divergence deﬁned here can be measured in a couple of ways 3 .
Therefore, the objective function to be maximized becomes
[Equation]
The sufﬁcient condition of maximizing the objecive function in (4) is to simultaneously maximize
L 1 ( θ ) and L 2 ( θ ) .
It is further observed that, when the transitional skill π ( a t | s t , z i,j,k ) transits from
For details, π ( a t | s t , ω i ) to π ( a t | s t , ω j ) , the divergence f
dω i ,z i,j,k increases progressively while the divergence f dz i,j,k ,ω
j decreases progressively.
Thus, the ﬁrst component L 1 ( θ ) that has a decreasing mutual information and an increasing divergence, is symmetric to the second component L 2 ( θ ) that has an increasing mutual information and a decreasing divergence.
The sole difference lies in the transition from π ( a t | s t , ω i ) to π ( a t | s t , ω j ) or from π ( a t | s t , ω j ) to π ( a t | s t , ω i ) .
So maximizing L 1 ( θ ) is equivalent to maximizing L 2 ( θ ) .
Consequently, maximizing the overall objective function L ( θ ) is equivalent to maximizing L 1 ( θ ) or
L 2 ( θ ) .
In such case, we have
[Equation]
Because it is difﬁcult to ﬁnd p ( ω i | s z ) , it is common to approximate p ( ω i | s z ) with a learned discriminator q φ ( ω i | s z ) .
According to Jensen’s Inequality, we know that replacing p ( ω i | s z ) with q φ ( ω i | s z ) gives us a variational lower bound G ( θ, φ ) of L ( θ ) .
So we have
[Equation]
I MPLEMENTATION 4. H INDSIGHT AND O NE - HOT E NCODING
In this section, we discuss about the implementation of learning transitional skills, where there are a couple of issues with the usage of the latent variable corresponding to transitional skills.
Problem 1 :
Along with the growth of the number of primitive skills N and transitional skills K − 1 , we have a high training complexity.
In our approach, we calculate the conditional probability q φ ( ω i | s z ) with a low efﬁciency because q φ ( ω i | s z ) can just keep the consistency of z i,j,k with ω i but ignore the relation with ω j .
To enhance the efﬁciency, we utilize the hindsight experience reply mechanism to allow samplebased learning from the sparse reward.
Problem 2 : The categorical distribution of the latent variable ω
i , 1 ≤ i ≤ N , suffers from a dilemma when the discriminator q φ ( ω i | s z ) judges the states.
Our solution is to use one-hot encoding for the latent variable ω i , 1 ≤ i ≤ N , corresponding to primitive skills.
In such case, the latent variable z i,j,k corresponding to transitional skills has a different expression and further analysis is given in Appendix B.
In the implementation, we change q φ ( ω i | s z ) in (6) to the following conditional probability distribution:
[Equation]
Correspondingly, the divergence of f di,j,k is organized as the following f di,k ,
[Equation]
(8) where k (cid:54) = i .
Consequently, the objective is to maximize the following G ( θ, φ ) ,
[Equation] [Equation]
where z i,k ∼ p
z
( z | ω i )
= { z i,k | z i,k =
z i,j,k , 0 ≤ j ≤ N − 1 , z i,j,k ∈ p ( z i,j | ω i , ω j ) } , α is a hyperparameter, and we ignore the term p ( ω i ) by taking a uniform distribution for ω i .
1: while NOT converged do 2:
Sample ω i ∼ p ( ω ) 3:
Sample a skill z ∼ p
z ( z | ω i ) and an initial state s 0 ∼ p 0 ( s ) 4: for t ← 1 to steps per episode do 5:
Sample an action a t ∼ π θ ( a t | s t , z ) ; 6:
Interact with the environment: s t +1 ∼ p
( s t +1 | s t , a t ) ; 7:
Compute D t = 1 N (cid:107) q φ (˜Ω | s t +1 )
+ αf di,k (cid:107) 2 with the discriminator ( φ ) ; 8:
Set the reward for current skill: r t =
D t .
By using SAC, update the policy ( θ ) to maximize the discounted return R = (cid:80) ∞ t =
γ t r t ; 10:
Update the discriminator ( φ ) to maximize D t with SGD.
11: end for 12: end while
4. A LGORITHM
We summarize our LTS method in Algorithm 1.
At each roll-out, given the latent variable of primitive skill ω
i , we sample a transitional skill z from a ﬁxed distribution p z ( z | ω i ) .
After the agent interacts with the environment at time step t , the discriminator ﬁnds the discriminability by
[Equation]
(10) where s zt +1 denotes the states at time step t. As mentioned above, we encode primitive skills using one-hot encoding.
We constrain (cid:80)
j z i,j,k = 1 , 0 ≤ z i,j,k ≤ 1 , so we simplify the divergence as f di,k = Array
( z i,j,k ) , where Array ( · ) means convert one-hot encoding into an array.
(See example in Appendix B.1 for more details (16, 17)).
In addition, we adopt soft actor-critic (SAC) algorithm to train our policy, adding the regularization E i,j [ H
[ A | S, Z i,j ] to maximize the policy’s entropy over actions given states and skills.
R ELATED WORK
Real-world tasks often require diverse behaviors.
Wang et al.
(2017) notes that building versatile embodied agents capable of performing a wide and diverse set of behaviors is one of the long-standing challenges of AI.
And learning continuous control of diverse behaviors in locomotion (Merel et al.
(2017); Heess et al.
; Peng et al.
and robotic manipulation (Ghosh et al.
Gu et al.
(2017)) is an active research area.
In this scenario, although some complex tasks can be solved through extensive reward engineering, undesired behaviors often emerge because of the sparse nature of reward (Riedmiller et al.
Moreover, training complex skills from scratch is not computationally practical.
These issues can be addressed by use of intrinsic motivation (Barto (2013);
Chentanez et al.
; Singh et al.
(2010)), which is a reward-free learning method.
Historically, the intrinsic motivation comes from the tendency of organisms to play and explore their environment without any reward (Ryan & Deci (2000), White (1959)).
Another line of work that is conceptually close to our method copes with information theories that are used to drive the agent’s exploration.
The information gain is a reward based on the reduction of uncertainty on environment’s dynamics (Little & Sommer (2013); Oudeyer & Kaplan (2007)), which can also be assimilated to learning progress (Frank et al.
; Oudeyer & Kaplan (2007)).
This can push agents into unknown areas on the one hand, and prevent them from being attracted to random areas on the other.
Recent work has also applied information theory for skill discovery.
VIC (Gregor et al.
(2016)) is an optional discovery technique by optimizing a variational lower bound on the mutual information between the context and the ﬁnal state in a trajectory, conditioned on the initial state.
Furthermore,
DIAYN (Eysenbach et al.
(2018)) maximizes the mutual information between states and skill to achieve diversity and shows the interest as a pre-training for hierarchical reinforcement learning or as an initialization for learning a task.
While discriminative embedding reward networks (DISCERN) (Warde-Farley et al.
(2018) aim to simultaneously learn a goal-conditioned policy and a goal achievement reward function by maximizing the mutual information between the goal state and the achieved state.
Let us notice that the skill space here is discrete, with just one or multiple policies.
However, we considered the relationship between different skills during the training process and ﬁnally formed a continuous skill space, likely because of inducing a novel latent variable for transitional skills.
In addition
[Figure]
it is important to point out that our skills are transitional with an intrinsically driven approach, which is very different from numerous previous works.
While Sharma et al.
(2019) discovers predictable behaviors to let the single skill more predictable, it need an external model-predictivecontrol (MPC) paradigm (Garcia et al.
to connect skills.
Peng et al.
(2019) learns reusable motor primitives that can be composed to produce a continuous spectrum of skills.
To bridge the gap between skills, Lee et al.
(2018) propose a transition policy to get a new smooth skill.
In comparison, our method captures intrinsic transition, which is independent from external tasks, and could eliminate the extra ﬁne-tuning process.
E XPERIMENTS
In our experiments
[Figure]
we aim to demonstrate the effectiveness of our approach for learning primitive and transitional skills.
We evaluate LTS and compare it to prior works.
6. D IVERSITY AND T RANSITIONAL S KILLS
[Figure]
shows the changes of the feature values of 4 diverse skills.
(b) shows the corresponding statistical characteristics of 4 skills in (a), where a single box-plot denotes a skill.
(c) shows 3 diverse skills of the half-cheetah.
In this section, we provide visualizations and quantitative analysis for our LTS method.
The tasks of CartPole-v0, MountainCar-v0, Pendulum-v0 and HalfCheetah-v3 are based on OpenAI gym 4 .
To evaluate different skills in different environment, we extract features from the observation of the agent, e.g. in the MountainCar environment, where we use the variance of the car’s altitude as the feature.
For more details on the features in other environments, please refer to Appendix C.
Figure 2 illustrates the discovered primitive skills in the MountainCar and Halfcheetah environment.
As shown in Figure 2(a), all four skills move in an periodic manner.
Corresponding to allskills in Figure 2(a), the statistical values of features are shown in Figure 2(b) using Box-plot.
It is observed that, these four skills have different movement patterns so that these skills are easy to be distinguished, so as the skills shown in Figure 2(c).
Moreover, we consider a different number of primitive skills and different environments as in Appendix E, from which it is observed that all primitive skills have an evident difference in feature statistics and are easy to be distinguished.
Transition.
Furthermore, we use the transitional skill z 0 , 1 ,k to show the performance of transition from one primitive skill ω
= [1 , 0 , 0 , 0] ) to another ω 1 = [0 , 1 , 0 , 0] ), where the number of transitional skills is set as 9 .
5000 Step 1.
F e a t u r e v a l u e (a) Trajectory of transition.
Transition from 0 to 1. 1.
S t a t i s t
i c a l c h a r a c t e r
i s t
i c s ( [Figure] [Figure]
[Figure]
In Figure 3(a) and 3(b), it is observed that the primitive skill ω 0 smoothly changes to ω 1 viatransitional skills.
More speciﬁcally, there exists a slight increment on the amplitude of features in the ﬁrst three skills, which is followed by consecutive declines until the ending primitive skill ωis discovered.
In Figure 3(c), we can see the clear transition process between two primitive skills.
More experiments on skill transition are given in Appendix F.
This demonstrates the effectiveness of our LTS method on discovering transitional skills and accomplishing the successful transition between two primitive skills.
6. C OMPARISON WITH DIAYN
In this subsection, we compare our LTS method with the state-of-the-art DIAYN (Eysenbach et al.
in terms of learning diverse primitive skills and transitional skills.
Experimental results show that LTS achieves an approximate performance on diversity of primitive skills, and a much better performance on skill transition.
Diversity: Different skills have different means or variances from their own trajectories.
In this subsection, we use (1) variance of means and (2) mean of variances of the features as metrics to evaluate the diversity of learned primitive skills using LTS and DIAYN
Figure 4(a) shows the variance of means while Figure 4(b) shows the mean of variances, where the black perpendicular line represents the range of values from a single experiment and we collect all trajectories of various primitive skills.
In Figure 4(a), the height of this line depicts the range of variance of means by considering all trajectories of primitive skills.
As shown, LTS obtains a lower variance of means in compare to DIAYN because the learned transitional skills from LTS affect the variance of learned primitive skills.
The lower variance of means does not degrade the diversity of learned primitive skills, which can be observed from Figure 4(b).
Although there is a large difference between experiments with random seeds, LTS has generally a similar mean of variance as DIAYN, indicating that our method performs similar with the baseline in terms of learning diverse primitive skills.
[Figure]
[Figure]
Transition: We also conduct experiments to evaluate the transitional skills of LTS and DIAYN, where we use identical encoding scheme for both and the number of transitional skills is set as 9.
Figure 5 shows the all statistical characteristics of 11 skills that including two primitive skills, where the value denotes the mean of features like the middle bar in Figure 3(b).
It is observed that LTS experiences a smooth transition state from one primitive skill to another while the transitional states in DIAYN suffer from one sharp-rising phase and two steady phases indicating a rigid changes of transition states.
The steady phase indicates that changing z will not change states very much, and the sharp change indicates that a little ﬂuctuate of z will cause a huge change of states, which is not reasonable in practice.
Because of the huge gap, controlling the skills learned by DIAYN changing from ω 0 to ω 1 will not push the agent standing in the state space of ω 2 .
On the contrary, we control the transitional skills in
LTS changing from ω 0 to ω 1 as in Figure 2(b), and the agent reaches the target states.
More studies including real statistical characteristics of different transitional skills are conducted and the results are reported in Appendix G.
6. G ENERALIZATION
[Figure]
Our LTS method suffers from the high training complexity to learn transitional skills.
A reasonable approach to tackle this problem is to use a ﬁxed number of transitional skills.
In this experiment, we set this number as 3 , indicating that the nonzero element of z i,j,k is from the set { 0 . 25 , 0 .
75 } in the training phase.
Figure 6 shows the transitional skills, where the training phase considers 3 transitional skills and adopting 3 and 50 transitional skills to evaluate the generalization of our LTS method.
It is observed that the transitional skills in blue (traning and testing both with 3 transitional skills) suffers from severe declines, possibly leading to the failure of the transition process in practice.
Fortunately, the transitional skills in red (training with 3 transitional skills while testing with 50) go through a steady and smooth process from one
[Table]
[Figure]
6. E QUIP THE AGENT WITH MORE TRANSITIONAL SKILLS
Figure 7 shows much more skills and corresponding transitions between them, where each small
It is observed that our LTS method is able to accomplish a smooth transition between two arbitrary skills, whatever primitive skills or transitional skills.
However, DIAYN suffers from a rigid transition between two skills, resulting in a possible failure in practical transition.
Furthermore, these results provide a deep insight that LTS has the ability to learn a larger continuous skill space.
The agent equipped with such numerous skills is expected to become much more powerful.
6. T RANSITION WITH H IERARCHICAL F RAMEWORK
In this subsection, we conduct experiments to answer the question: how do transitional skills yield beneﬁt to the downstream tasks?
To evaluate the effect of the transition policies, we take as benchmark hierarchical control for primitives learned by DIAYN.
Moreover, we conduct hierarchical framework to weight (or choose) the action modeled by different primitive skills.
[Figure]
For primitive skills modeled by DIAYN, we consider the following approaches:1) DIAYN-random :
Transition policy with random weights to affect actions; 2) DIAYN-weighted : Transition policy with smoothly changed weights (artiﬁcially designed) to affect actions; 3) DIAYN-master : Transition policy with optimal weights learned by meta-policy.
For transitional skills modeled by LTS, we consider the following approaches: 1) LTS-random : Randomly choose transitional skills between two primitive skills; 2) LTS-smooth :
Choosing continously changed transitional skills z i,j,k ( 1 ≤ k ≤ K − 1 ); 3) LTS-master : Transfer policy with optimal weights learned by meta-policy.
Table 1 shows the performance.
It is observed that we choose skills randomly or weighted actions modeled by primitive skills, the transition process is more likely to fail.
Even with an extensively forecast through the hierarchical framework, DIAYN-master is possible to fail.
We believe this is because the primitive skills have a large gap and therefore, one skill are unable to associate with another.
Even with meta-policy, it is also hard to transit between primitive skills.
In contrast, LTSsmooth has a better performance than DIAYN-master without any meta-policy because of abundant transitional skills.
Moreover, LTS-master could select most appropriate transitional skills to ﬁll in the gap between two arbitrary skills.
6. V ISUALIZING T RANSITION T RAJECTORIES
Figure 8 shows the two-dimensional t-SNE embedding of the agent’s states generated by different skills.
Different colors represent different skills (including primitive skills and transitional skills).
The red lines represent the agents’ transition trajectories.
In Figure 8(a) and 8(b), we can see a clear boundary between two kinds of skills, which is consistent with the boundary of Figure 7.
Starting from the initiatial state, the agent tries to transit to another state.
But both of agents with
DIAYN-smooth-policy and DIAYN-weighted-policy fail for transiton.
The state space of different skills in Figure 8(c) seems to be more diverse, while it is a pseudomorphism.
Because the space is a two-dimensional mapping, the difference between Figure 8(a) and 8(b) only discloses that different skills do exist a conclusive gap between two kinds of skills, but this uncovers that the learned transitional skills are also distinguishable.
We can ﬁnd in Figure 8(c) also exits many intersections which are the crucial transition states between two different skills.
In the t-SNE space, distant dots with same color are also connective, so there exits many long line.
And the intersections existing two endpoints must go through a transition between two skills.
Particularly, the trajectory of the agent in Figure 8(c) shows the transition between different 11 skills.
C ONCLUSION
In this paper, we introduce a novel LTS method to learn transitional skills without extrinsic reward by using two kinds of latent varibles to depict different skills.
As a result, LTS can discover both primitive skills and transitional skills.
Also, LTS achieves a great success in the smooth transition from one primitive skill to another and exhibits its potential in learning a large continuous skill space.
Extensive experiments demonstrate the effectiveness of our LTS in the discovery of diverse skills and the smooth transition between skills even for complex down-stream tasks
Appendices A M ETHODOLOGY A. DERIVATION OF THE V ARIATION B OUND ON M UTUAL I NFORMATION
Here we derive the variational bound:
[Equation]
B I MPLEMENTATION B. H INDSIGHT AND O NE - HOT E NCODING
In our approach, we model the conditional probability given by the discriminator q φ ( ω i | s t ) w.r.t.
the divergence (or distance) between transitional skill z i,j,k and speciﬁc primitive skill ω i .
Although (4) constrains two mutual information terms, but ( ?? ) is adopted to train our model, which constrains only one term.
These efﬁciency is relatively low.
For example, if we want to simultaneously constrain the probability q φ ( ω i 1 | s t ) and q φ ( ω i 2 | s t ) in terms of the same transition state s t generated by the same skill z
( z = z i 1 ,j 1 ,k 1 =
z i 2 ,j 2 ,k 2 ) but two different primitive skills ( ω i 1 and ω i 2 ) in the training stage, we must wait for the next time step to sample the same z and a different ω in the experience.
The efﬁciency is relatively low because that q φ ( ω i | s t ) just constrain the consistency of z i,j,k and ω i which ignores the consistency to other primitive skills.
So we utilize the hindsight experience reply mechanism to allow sample-efﬁcient learning form sparse rewards.
We calculate a distribution of the conditional probability given by the discriminator instead of a single value.
So we change q φ ( ω i | s t ) to the conditional probability distribution:
[Equation]
By doing this, we could simultaneously constrain the similarity probability distribution given by the discriminator with related to all primitive skills.
+
Moreover, we change the criterion of f di,j,k into f di,k :
[Equation]
On the other hand, for categorical encoded primitive skills, when ω i − ω j (cid:54) = ± 1 ( i (cid:54) = j ), the discriminator q φ will face a dilemma: the intersection of transition states and primitive states are not empty, i.e. S Ti,j ∩ S P (cid:54) = (cid:11) , leading to a conﬂict between diversity and transition.
Here we give an example: considering transferring two skills from ω i to ω j ( ω i (cid:54) =
ω j ), the former categorical encoding will cause extra consumption: if ω i − ω j (cid:54) =
± 1 , primitive states with related to primitive skills { ω | ω ∈ [ ω i , ω j ] or [ ω j , ω i ] } will occur in the transition states.
For three primitive skills ω i 1 , ω i 2 and ω i 3 , given the optimal discriminator q ∗ φ , there should be q ∗ φ ( ω i 3 | s (cid:48) t ) =
( s (cid:48)
t ∼ π ( ω i 1 ) and ω i 1 (cid:54) = ω i 3 ) and q ∗ φ ( ω i 3 | s (cid:48)(cid:48) t )
=
( s (cid:48)(cid:48)
t ∼ π ( ω i 2 ) and ω i 2 (cid:54)
= ω
i 3 ).
While if ω i 1 < ω i 3 < ω i 2 , q ∗ φ ( ω 3 | s (cid:48) t )
= log p ( ω i 1 ) − f d ( ω i 1 ) + c and q ∗ φ ( ω 3 | s (cid:48)(cid:48) t )
= log p ( ω i 2 ) − f d ( ω i 2 ) + c 5 , which is in contrast to the conditional probability of 1 ( ω i 3 is also a primitive skill).
So, we encode ω ∼ p ( ω ) with one-hot way:
[Equation]
And we denote primitive skills and transitional skills as a set Z
+ i,j :
[Equation]

[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
− kK ... kK ......
...
  ( K +1)
× N
[Equation] [Equation]
where the value of i -th and j -th column keeps decreasing and increasing respectively.
Other column always keep 0 , which could constrain the incoherence between transition states and other primitive skills.
Without causing any misunderstanding, following z i,j,k all comes from Z + i,j .
For transition, we assure that the change of z i,j,k only happens on the corresponding dimension, which overcomes the conﬂict caused by categorical encoding.
As show in Fig.9, all transition skills in Z 3 , 1 and primitive skills ω 3 are orthogonal.
The transition only reﬂects on the plane deﬁned by the corresponding primitive skills.
Although these will induce a problem of the generation issue, but this could also learning continuous transitional skills as shown in Figure 6 and 7.
In fact, there is more than one transitional path, which can be a directed line or any directed curve.
As in Figure 7(b), we can ﬁnd more than one transitional paths.
An example Z + i,j shows below:
[Equation]
where N = 4 , K − 1 = 3 .
A transition path could be ω
→ z 3 , 1 , 1 → z 3 , 1 , 2 → z 3 , 1 , 3 → ω 1 .
And an alternative way could be [0 , 0 , 0 , 1 , 0]
→ [0 , 0 . 2 , 0 , 0 . 8 , 0] → [0 . 1 , 0 . 3 , 0 . 2 , 0 . 3 , 0 . 1] → [0 , 0 . 8 , 0 , 0 . 2 , 0]
→ [0 , 1 , 0 , 1 , 0] .
For the similairity f di,k , examples are shown below (generated by
Z +3 , 1 in Formula(16) ):
[Equation]
where f si,j = 1 − f di,j .
C EXPERIMENTAL ENVIRONMENT
The experiments were carried out over four opened reinforcement learning environments (CartPole 6 ,
MountainCar 7 , Pendulum 8 and HalfCheetah 9 .
C. CARTPOLE
In this environment, a pole is attached by an un-actuated joint to a cart, which moves along a frictionless track.
The system is controlled by applying a force of +1 or -1 to the cart.
The pendulum starts upright, and the goal is to prevent it from falling over by increaseing and reducing the cart’s velocity.
The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center.
C. MOUNTAINCAR
A car is on a one-dimensional track, positioned between two ”mountains”.
The goal is to drive up the mountain on the right; however, the car’s engine is not strong enough to scale the mountain in a single pass.
Therefore, the only way to succeed is to drive back and forth to build up momentum.
C. PENDULUM
The inverted pendulum swingup problem is a classic problem in the control literature.
The problem of the pendulum starts in a random position, and the goal is to swing it up so it stays upright.
C. HALF C HEETAH
In the HalfCheetah environment, there is a two-legged robot, restricted to a vertical plane, meaning it can only run forward or backward.
The agent has 17 dimensions of state space and 6 dimensions of action space.
D HYPERPARAMETERS
For all RL algorithm in our experiments, we use the SAC (Haarnoja et al.
as implementation framework.
The hyperparameters are summed up in the Table
and we use ADAM (Kingma & Ba (2014)) optimizer.
E VISUALIZING PRIMITIVE SKILLS
In order to better visualize the distinction between skills, we did various experiments and ﬁnally determined some optimal observations as feature vector for each skill (see Table 3).
Please note that the feature values adopted are aimed at visualizing the skills, and what we put into the discriminator is the all state of the agent instead the single feature value.
The following experiments show that it makes sense to calculate the statistical characteristics of skills’ characteristics to represent a skill.
Three experiments’ performance was shown in Figure 10, Figure 11 and Figure 12.
F VISUALIZING TRANSITION PROCESS
For Cartpole, MountainCar, Pendulum and HalfCheetah, we get 4 primitive skills, and control variant z only takes 3 ﬁxed values (0.25 0.5 0.75) during training.
However, at the test phase of transition, z is added every 0.1.
This ways reduce the training complexity as analyzed in Section 6.3.
So we can obtain 9 transition skills between any two primitive skills( z
=
(See Figure 13, Figure 14, and Figure 15).
https://gym.openai.com/envs/CartPole-v0/https://gym.openai.com/envs/MountainCar-v0/https://gym.openai.com/envs/Pendulum-v0/https://gym.openai.com/envs/HalfCheetah-v2/
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
t e r i s t
i c s (h)
Primitive skills 0.
S t a t i s t
i c a l c h a r a c t e r
i s t
i c s (i)
3 Primitive skills 0.
S t a t i s t
i c a l c h a r a c t e r
i s t
i c s (j)    400 Step 0.10 0.
F e a t u r e v a l u e 0(k)
G TRANSITIONAL SKILLS COMPARISON
Studies including motion trail and statistical characteristics on MountainCar was reported in Figure 16.
The value of the y-axis denotes the mean of feature values (which is also the statistical characteristic).
Here we do not test the transition processes between skills and we just run different transitional skills independently.
There should be many isolated dots instead a transition line.
Skills with two distant features will means a larger probability of failure of transition.
For DIAYN’s primitive skills, there exists a large gap which notes an harder transition between the two skills, comparing the smooth line of LTS’s skills.
Please note that there exits no faster transition instead a large gap.
[Table]
[Figure]
c
[Figure]
e r i s t i c
[Figure]
(h)
[Figure]
S t
[Figure]
t
[Figure]
s t
i c a l c h a r a c t e r
i s t
i c
[Figure]
(i)
[Figure] [Figure]
r e
[Figure]
[Figure] [Figure]
i
[Figure]
(c
[Figure]
3 Primitive skills 0.6 0.
S t a t i s t
i c a l c h a r a c t e r
i s t
i c s (d)
[Table] [Table]
[Table] [Table]
[Table]
S t a t i s t
i c a l c h a r a c t e r
i s t
i c s (e)
[Table]
skills
[Table] [Table]
[Table] [Table]
[Table]
S t a t
[Table]
c h
[Table]
c t
[Table]
i s
[Table]
c
[Table]
(f)
[Table] [Table]
[Table] [Table]
[Table]
S t a t i s t
i c a l c h a r a c t e r
i s t
i c s (g)
[Table]
[Table]
S t a t i s t
i c a l c h a r a c t e r
i s t
i c s (h)
[Table]
skills
[Table] [Table]
[Table] [Table]
S t a t i s t
i c a l c h a r a c t e r
i s t
i c s (i)
[Table]
skills 0.
[Table] [Table]
[Table] [Table]
[Table]
S t a t i s t
i c a l c h a r a c t e r
i s t
i c s (j)
[Figure]
e a t u r e
[Figure]
l
[Figure] [Figure]
[Figure] [Figure]
a
[Figure]
s
[Figure]
c a l c
[Figure]
a
[Figure]
a c
[Figure]
r
[Figure] [Figure]
(g) are from different 4 trials respectively.
Each transitional skill holds 500 steps and then transfers the ﬁnal state to the next skill.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
n
[Table]
f f e a t u r e v a l u e s -DIAYNLTS (a)
[Table]
e
[Table]
a l
[Table]
e s
[Table]
(b)
[Table] [Table]
[Table]
M e a n o f f e a t u r e v a l u e s
[Table]
(c)
[Table] [Table]
[Table] [Table]
[Table]
M e a n o f f e a t u r e v a l u e s -DIAYNLTS (d)
[Table]
[Figure]
[Table]
M e a n o f f e a t u r e v a l u e s -
[Table]
(e)
[Table]
(f)
[Table]
e
[Table]
(g)
[Table] [Figure]