Unsupervised Skill Discovery via Recurrent Skill Training
Abstract
Being able to discover diverse useful skills without external reward functions is beneficial in reinforcement learning research.
Previous unsupervised skill discovery approaches mainly train different skills in parallel.
Although impressive results have been provided, we found that parallel training procedure can sometimes block exploration when the state visited by different skills overlap, which leads to poor state coverage and restricts the diversity of learned skills.
In this paper, we take a deeper look into this phenomenon and propose a novel framework to address this issue, which we call Recurrent Skill Training (ReST).
Instead of training all the skills in parallel, ReST trains different skills one after another recurrently, along with a state coverage based intrinsic reward.
We conduct experiments on a number of challenging 2D navigation environments and robotic locomotion environments.
Evaluation results show that our proposed approach outperforms previous parallel training approaches in terms of state coverage and skill diversity.
Videos of the discovered skills are available at https://sites.google.com/ view/neurips22-rest .
Introduction
[Figure]
Recent advances in deep reinforcement learning have shown its promising performance in domains ranging from game playing [2, 3], robotics [4, 5] and recommender systems [6].
These applications of reinforcement learning rely on task-specific reward functions for the agents to successfully accomplish the tasks.
However, intelligent creatures can automatically explore the environments and learn diverse useful skills in the absence of external supervision.
Such ability is beneficial in a variety of situations.
For tasks where rewards are non-trivial to design or where the reward signal is sparse, unsupervised skill discovery approaches can provide intrinsic re- wards to help accomplish tasks.
Moreover, in hierarchical control problems, unsupervised skill discovery can serve as low-level policies for downstream tasks [7, 1].
Although existing works have shown great potential in discovering diverse useful skills in an unsupervised manner [1, 7, 8], one of the key problems of such methods, as observed in our preliminary experiments and some recent works [9, 10], is that they might suffer from poor state coverage, which may lead to failures in learning desirable useful skills.
For instance, in robotic locomotion environments, previous approaches tend to learn ‘posing’ skills instead of dynamic, far-reaching skills [1].
A straightforward explanation for this phenomenon would be the lack of exploration [9].
However, we argue that this is not the only reason for the poor state coverage.
Counterintuitively, we observed that the skills after convergence may avoid visiting certain states even if they are explored during training .
For instance, as shown in our preliminary 2D navigation experiments, the discovered skills fail to cover the states passing through the bottleneck to the right (Figure 1b), even if they have been explored during training (Figure 1a).
We call this phenomenon Exploration degradation .
In this paper, we take a deeper look into the above phenomenon and show that it is mainly caused by the parallel training paradigm, which is a common choice for most existing works [1, 7, 8, 10].
When multiple skills trained in parallel have visited the same state, such state will be prevented from being visited again.
Detailed analysis is provided in Section 3.1.
Based on the analysis, we propose Re current S kill T raining (ReST), an unsupervised skill discovery algorithm that addresses the exploration degradation issue.
Instead of training all the skills in parallel, ReST trains the skills one after another in a recurrent fashion, along with an intrinsic reward that discourages covering frequently visited states of other skills.
A preliminary result of using ReST is shown in Figure 1c, where the exploration degradation problem is eliminated and the converged skills are able to visit the space in the right room.
Evaluation results on complex 2D navigation and robot locomotion tasks show that our approach can achieve better state coverage and skill divergence compared to baselines.
Our contributions are summarized as follows:
• We discover a new phenomenon that reduces state coverage called exploration degradation, which indicates that some certain states are discouraged from being visited by the learned skills, even if they have been explored during training.
• We show that the main reason causing exploration degradation is that multiple skills visiting the same states can reduce the MI reward in the parallel training paradigm.
We then propose Re current S kill T raining (ReST), a recurrent training paradigm along with a state coverage based intrinsic reward, which prevents multiple skills from visiting the same states and alleviates the exploration degradation issue.
• We conduct experiments on various 2D navigation tasks and robot locomotion tasks.
Evaluation results show that our method achieves better state coverage and divergence compared to baseline methods.
Moreover, ReST learns diverse meaningful robot locomotion skills that have not been shown in previous works.
Preliminaries 2.1 Markov Decision Process
Markov decision process (MDP) can be used to find a reward maximizing policy.
It is represented by a tuple ( S , A , P , R, γ, µ ) , where S , A are the state and action spaces.
P : S ×
A × S → [0 , 1] is the transition dynamic that maps the state and action into a probability distribution over the next state.
R : S ×
A → R is the reward function.
γ is the discount factor for the reward function while µ : S → [0 , 1] is the initial state distribution.
The expected discounted cumulative reward can be formulated as J R ( π ) =
E τ ∼ π
[ (cid:80) ∞ t =
 γ t R ( s t , a t )] .
Thus the overall optimization problem can be written as π ∗ = arg max π J R ( π ) .
2.2 Unsupervised Skill Discovery
Generally speaking, unsupervised skill discovery aims to find a family of skills conditioned on latent z , which results in a latent-conditioned policy π ( a | s, z ) that maximizes the mutual information between state and latent:
[Equation]
where s ∈ S , a ∈ A , z ∈ Z are the state, action and latent respectively.
Let ( S, Z ) ∼ p
( s, z ) as the random variables of the state distribution and the latent distribution.
As suggested by [11], a variational lower bound for Equation (1) can be derived as:
[Equation]
where q ϕ ( z | s ) is a learned discriminator approximating p ( z | s ) .
Such lower bound also exists for
Equation (2):
[Equation]
where q ϕ ( s | z ) is a learned function approximator of p ( s | z ) .
Recurrent Skill Training
[Figure]
In this section, we first introduce the exploration degradation phenomenon of previous skill discovery approaches with parallel training paradigm.
Then we present our proposed recurrent skill training method that addresses this issue.
[Figure]
Initialize: random initialized skill π , RND networks ˆ f and f for i = 1 to N do
Set skill π i = π
Set RND network ˆ f i
=
ˆ f , f i = f
Collect on-policy samples with π i
Update RND network
ˆ f i by minimizing loss in Equation (5) end for repeatfor i = 1 to N ( N is the number of skills) do for j = 1 to M ( M is the number of training epochs for each skill)
do
Collect on-policy samples with π i ;
Calculate reward using Equation (7);
Update skill policy π i using any RL algorithms;
Update RND network
ˆ f i by minimizing loss in Equation (5) using the latest on-policy samples; end for end for until convergence 3.1 Exploration Degradation
As we observed in our preliminary experiment shown in Figure 1, previous unsupervised skill discovery approaches might suffer from the exploration degradation problem, such that some explored states (e.g, states near the bottleneck and in the right room in Figure 1a) are prevented from being visited by the learned skills.
We now provide a simplified analysis to show that this phenomenon is mainly caused by the parallel training paradigm commonly used in previous works.
Let us focus on discrete latent case with N different skills.
Consider at state s , skill k visited state s with p ( z k | s ) probability, which is modeled by the discriminator q ϕ ( z k | s ) .
Assume state s 0 is only visited by skill k
whereas s 1 is visited by multiple skills, which means p ( z k | s 0 )
= 1 and p ( z k
| s 1 ) < 1 .
As long as the discriminator q ϕ can model the difference and output q ϕ ( z k | s 0 ) >
q ϕ ( z k | s 1 ) , which is not a strong assumption, the intrinsic reward used by MI approaches would encourage the visitation of state s 0 by skill k and discourage visiting s 1 .
This means MI based approaches end up with skills that prefer exploiting states only visited by themselves in the previous training epochs, causing exploration degradation.
For clarity, we further explain this issue with a toy example.
Consider a 2 × 2 grid world as shown in Figure 2.
There are four states 0 , 1 , 2 , 3 and at each state, there are three actions to take: go to the two adjacent states or stay where it is.
For simplicity, we use the number of the next state to denote the action, for instance, if the agent is in state 0 and chooses to go to state 1 , then the action will be denoted as 1 .
We use the mutual information described in Equation (1) and assume that the discriminator is perfect: q ϕ ( z | s ) =
p π ( z | s ) .
Consider a case when the number of skills is N =and during the first collection of samples, skill 1 visited { 0 , 2 , 3 } while skill 2 visited { 0 , 1 , 3 } , as shown in Figure 2a.
Since we have a perfect discriminator, for state 3 the discriminator would output q ϕ ( z 1 | 3)
= q ϕ ( z 2 | 3) = 0 . 5 , which results in 0 reward for both skills.
Therefore, the optimal converged policy for skill 1 would generate trajectory { 0 , 2 , 2 } while skill 2 would generate trajectory { 0 , 1 , 1 } , as shown in Figure 2b.
State 3 is not covered, which is undesirable.
This example indicates that even in such an extremely simple case, the exploration degradation phenomenon still exists.
3.2 Recurrent Skill Training
Instead of training all skills in parallel, we propose a recurrent training paradigm, along with a state coverage intrinsic reward.
We now introduce the details of our proposed method.
Recurrent Training Paradigm.
As analyzed in Section 3.1, the main reason causing the exploration degradation issue is that the same states are visited by multiple skills in the parallel training paradigm.
A natural way to alleviate this issue is to train the skills one after another recurrently.
In this way, we can encourage the latter trained skills to avoid entering the same states covered by the previous skills.
Figure 3 illustrates the recurrent training paradigm.
Starting with N randomly initialized skills ( N = 3 in this case), the recurrent training paradigm trains one skill at each epoch whereas the parallel training paradigm trains all the skills together.
Furthermore, in order to improve convergence, the recurrent training paradigm updates each skill for M epochs ( M = 2 in this case) before switching to another skill.
In this work, we use N independent neural networks to parametrize the N skills, which can be considered as discrete latent conditioned policies.
State Coverage Intrinsic Reward.
When implementing the above recurrent training paradigm, the latter trained skill needs to avoid visiting the states frequently visited by other skills.
In order to accomplish this objective, we need to identify how frequently each state is visited by each skill, or equally, how novel a given state is to a skill.
In this paper, we adopt random network distillation (RND) [12], a simple yet scalable novelty detection approach, to estimate the novelty of a state to a specific skill.
Random network distillation (RND) [12] mainly involves two networks for state novelty detection: a randomly initialized target neural network whose parameters are kept fixed throughout the whole training process and a predictor network trying to fit the target neural network.
The target network f : S → R k , where k is the dimension of the output vector, maps a state to a vector while the predictor network ˆ f : S → R k is trained to minimize the expected mean square error between the output vector and the target output vector:
|| ˆ f ( s ) − f ( s ) || 2 , which distills the target network into the predictor.
Therefore, the predictor should have low prediction error on the data it is trained on while have high prediction error on other states.
We call such target network and predictor a pair of RND networks .
We assign each skill with a pair of RND networks.
Each skill’s predictor network is trained on the the on-policy data it visited during the last rollout, which means for each skill with latent z i ∈ Z , where i ∈ { 1 , 2 , ..., N } is the index of the skill, with state distribution s ∼ p ( s | z i ) , we can train the predictor network using gradient decent by minimizing the mean square error loss L i
[Equation]
As mentioned above, the learned predictor network’s prediction error
||
ˆ f i ( s ) − f i ( s ) || 2 can indicate visitation frequency of a certain state s .
Intuitively, a higher prediction error indicates higher uncertainty of ˆ f i on this state, which further implies its higher novelty.
Since we need to avoid visiting states visited by other skills when training a certain skill, a straightforward solution is to define a reward function for skill i with state s t and action a t as:
[Equation]
such that states frequently visited by any other skills are less desired.
However, this minimum operator would make the reward landscape rugged, which might lead to poor convergence property.
To stabilize the training process, we introduce a soft version of the minimum operator in (6):
[Equation]
where α is a task-specific temperature parameter.
Practical Algorithm.
We summarize our algorithm in Figure 3 and Algorithm 1.
Firstly, we randomly initialize a policy network π i and a pair of RND networks f i and ˆ f i for each skill i ∈ { 1 , 2 , ..., N } .
Before training skills, ReST first collects on-policy samples for each initialized skill and train their
RND networks.
Then we train each skill’s policy network and the corresponding RND networks for
M times during each training epoch.
The skill networks are trained recurrently until convergence.
For maximizing the intrinsic reward, the ReST algorithm can be combined with an arbitrary reinforcement learning algorithm.
In this paper we choose Proximal Policy Optimization (PPO) [13] with generalized advantage estimation (GAE) [14].
Grid World Example.
We further illustrate the effectiveness of the proposed algorithm using the 2 × 2 grid world example with the same setting as in Section 3.1.
As suggested by our previous analysis, the parallel training paradigm will end up with two skills that neither of them visits state 3 .
As mentioned in Section 3.1, the initial two skills visits { 0 , 2 , 3 } and { 0 , 1 , 3 } respectively.
Here we assume that the RND networks f i and ˆ f i perfectly obtain the state visitation frequency, which means:
[Equation]
if state s is visited by skill i r otherwise ( [Equation] ) where r > 0 is the prediction error.
ReST starts by training the corresponding RND networks of each skill and then recurrently train different skills to maximize the intrinsic reward.
For skill 1 , visiting state 2 would gain reward r while visiting state 1 and 3 would get zero rewards.
Therefore, the optimal policy for skill 1 would be visiting { 0 , 2 , 2 } in sequence.
After updating the RND networks of skill 1 correspondingly, for skill 2, visiting state 1 and 3 would gain reward r while visitingwould get zero rewards.
Therefore, the initial skill 2 is already the optimal policy and the training converges.
This way, the optimal trajectory for skill 1 is { 0 , 2 , 2 } while for skill 2 it is { 0 , 1 , 3 } , as shown in Figure 2c.
Therefore, the state space of the 2 × 2 grid world is fully covered by the two skills.
Experiments
[Figure]
corresponding navigation environments.
We trained 10 skills for each algorithm where 20 trajectories are rendered for each discovered skill.
The right two columns of the figure show the quantitative results of different algorithms.
SC stands for state coverage while MI stands for mutual information.
SC and MI are calculated based on converged models of each algorithm with three different random seeds.
We calculate the SC and MI metrics three times for each converged skill to draw the histogram.
Detailed analysis of the evaluation metrics can be found in the Appendix We conduct experiments on several challenging 2D navigation environments and several robotic locomotion tasks. We compare our proposed approach with two of the most popular unsupervised [Figure]
[Figure] Generally speaking, baseline methods discover ‘posing’ skills instead of dynamic locomotion skills whereas ReST discovers dynamic far-reaching skills [Figure] skill discovery approaches, DIAYN [1] and DADS [7].
Both of these approaches use the parallel training paradigm to optimize different skills.
Since our proposed approach parameterizes each skill with an independent neural network, we also compare our proposed approach with DIAYN and DADS using independent neural networks, which we call DIAYN-i and DADS-i respectively, for a fair comparison.
We present our empirical results both qualitatively and quantitatively.
The quantitative results introduce a state coverage metric and a mutual information metric.
Our proposed approach significantly outperforms previous skill discovery approaches in terms of state coverage while staying comparable with previous approaches in terms of mutual information.
Qualitative results include visualizations of the converged navigation experiments, the state visitation of robotic locomotion tasks, and rendered videos of novel skills discovered using our proposed approach, which can be found in our project website https://sites.google.com/view/neurips22-rest .
Details of our implementation can be found in the Appendix.
4.1 2D Navigation Tasks
Environments.
We conducted experiments on several challenging 2D navigation environments.
The agent is a point mass navigating in a 2D plane with boundaries [0 , 1] × [0 , 1] .
The observation space of the environment has 4 dimensions, including the x ∈ [0 , 1] and y ∈ [0 , 1] position on the plane and the corresponding velocity v x and v y belonging to the velocity space { ( v x , v y ) | v 2 x + v 2 y ≤ 0 . 01 } .
The action space has 2 dimensions, including the acceleration a x ∈ [0 , 0 . 1] and a y ∈ [0 , 0 . 1] of the point mass agent.
There are walls inside the plane and the agent cannot go through the walls.
We designed diverse placements of the walls with increasing difficulties for exploring the environments, which could be used to test the effectiveness of our proposed approach.
Evaluation Metrics.
We introduce two metrics to compare our proposed approach with our comparison baselines.
The first one is the state coverage metric.
State coverage matters since not covering enough state space might result in failures in learning desirable useful skills.
We evaluate the state coverage on the X - Y plane by first decomposing the environment into cells and then testing whether each cell is successfully visited.
We roll out 1 trajectory for each skill and use the visited states to calculate the state coverage metric.
The percentage of cells visited by at least one of the skills is the state coverage.
Moreover, merely covering the state space is not enough.
The skills need to be informative about which states they are going to visit in the environment so that the skills are meaningful.
We use mutual information between skill latent z and the corresponding covered states s to quantify how informative the skill is.
Similarly, we decompose the state space into multiple cells and roll out 20 trajectories for each skill to record the probability distribution of states p ( s ) for all the states visited by the skills.
Then we record the probability distribution of states visited by each skill p ( s | z ) and calculate the entropy H ( S ) and H ( S | Z i ) .
Since the skill distribution p ( z ) is a uniform distribution for all skills, we can calculate the mutual information as
I ( S ; Z ) = H ( S ) − 1 N
[Table]
Qualitative Results.
We train N = 10 skills for each of the algorithms and rollout 20 trajectories to obtain the qualitative results.
We conduct experiments on 4 challenging 2D navigation mazes, which we call the DoorMaze , CenterMaze , 4RoomMaze and 9RoomMaze , which are shown in Figure 4.
The left 5 columns show the qualitative results.
Each rolled-out trajectory is rendered as a curved line and different colors represent different skills.
The results indicate that our proposed ReST algorithm can reach more diverse states in the environments whereas baseline approaches like DIAYN or
DADS can only reach a small portion of the environments.
Usually, the baseline approaches cannot pass through the ‘bottlenecks’ in the environment.
The results support our insight that the parallel training paradigm is one of the causes of previous skill discovery approaches’ state coverage issues.
Moreover, when using independent neural networks to parameterize different skills, we observe that the performance is not so different from the latent-conditioned parameterization version.
Quantitative Results.
Beyond qualitative results, we quantify the state coverage and the informativeness of different algorithms using the aforementioned two metrics.
As shown in the right two columns of Figure 4, our proposed ReST algorithm significantly outperforms baseline approaches with a parallel training paradigm in terms of state coverage and is comparable with baseline approaches in terms of mutual information I ( S ; Z ) .
The above results indicate that our proposed approach can cover diverse sets of states without too much sacrifice on informativeness.
Quantitative results also evidenced that the difference in performance is not because of the different parameterization since
DIAYN and DADS using independent neural networks are not so different from the original ones.
4.2 Robotic Locomotion Tasks
We present qualitative results of robotic locomotion skills discovered using our proposed ReST algorithm in MuJoCo [15].
Generally speaking, our proposed ReST algorithm can discover dynamic, far-reaching robotic locomotion skills whereas DIAYN and DADS tend to discover ‘posing’ skills.
Figure 5 shows visualizations of parts of skills discovered using ReST.
There are several novel skills discovered, such as Hopper backflip, that have not been presented in previous works to the best of our knowledge.
More rendered results can be found in the Appendix and our project website https://sites.google.com/view/neurips22-rest .
Moreover, we provide semi-quantitative results of the proposed approach.
As shown in Figure 6, we draw the agent’s x position over timestep, using the skills discovered by ReST and the comparison baselines.
The results are evaluated on HalfCheetah , Hopper and Walker2d tasks.
We use the
OpenAI Gym [16] settings of the three tasks, where HalfCheetah is trained with fixed episode length whereas Hopper and Walker2d terminate when the agents fall during training.
The resulting timestep- x curve indicates that our proposed approach learns skills that are more far-reaching, diverse, and dynamic than baseline methods.
This also evidenced that our recurrent training paradigm outperforms the parallel training-based baselines by alleviating their state coverage issues.
Related Work
Unsupervised Skill Discovery.
Previous unsupervised skill discovery approaches mainly focus on maximizing the mutual information I ( S ; Z ) to obtain meaningful skills.
As discussed in Equation (1) and Equation (2), the mutual information has two forms.
Several works follow the Equation (1).
VIC [17] uses the mutual information between the final states and the skill latent as the intrinsic reward and optimizes it via reinforcement learning.
DIAYN [1] fixes the prior p ( z ) and uses the mutual information between the skill and its visited states as its intrinsic reward to learn meaningful skills, which has superior performance compared to VIC.
VALOR [8] further improves DIAYN by replacing the state-based objective with a trajectory-based objective.
DADS [7], on the other hand, uses the Equation (2) form of mutual information and learns a transition model q ϕ ( s [-
-]′ | s, z ) and uses model predictive control to solve downstream tasks.
EDL [9] provides insight into why previous approaches suffer from lack of exploration and proposes an algorithm using a fixed state prior p ( s ) that to alleviate the issues.
IBOL [10] tries to relieve the difficulty of reaching diverse states by introducing a low-level controller.
Besides learning a set of skills, SMM [18] formulates skill discovery as a state marginal matching problem and optimizes the KL divergence between the expected state distribution and the current policy’s state distribution.
MUSIC [19] improves previous unsupervised skill discovery algorithms by adding the mutual information between the surrounding state and the agent state.
DDL [20] learns one skill that maximizes the dynamical distance functions of the previous skill.
Plan2Explore [21] makes use of self-supervised reward signals to optimize one policy to efficiently gather state-covering dataset.
In contrast, ReST optimizes multiple skills and maximize online state coverage.
LEXA [22] learns a goal-conditioned policy using self-supervised reward.
ReST, on the other hand, does not require the agent to achieve specific goals, which is a more general formulation.
Besides the above approaches, there are also other unsupervised skill discovery methods [23, 24, 25, 26].
Intrinsic Reward.
Another stream of works related to this work is intrinsic reward.
In our proposed approach, we use intrinsic reward as an objective to help learn a set of meaningful skills.
Intrinsic reward can also help with cases where rewards are sparse by augmenting them to the original reward function.
Count-based exploration [27, 28, 29, 30] uses pseudo count to identify the frequently visited states and the less frequently visited ones and adds the count-based bonus as intrinsic rewards to accelerate exploration.
Prediction error exploration methods [31, 32, 12, 33, 34] make use of prediction errors as intrinsic rewards based on an insight that states with high prediction error should have higher novelty.
Intrinsic reward can also benefit novelty seeking tasks [35, 36], where RSPO [35] made use of an iterative paradigm when training different strategies, which is similar to recurrent training paradigm in this work.
Other works augment an information-theoretic intrinsic reward with extrinsic rewards that encourage information gain about the environments [37, 38, 39].
Discussion
In this paper, we proposed a novel, effective yet simple algorithm called Recurrent Skill Training (ReST).
We began by introducing a new phenomenon called exploration degradation which reduces state coverage of the learned skills.
We found the key reason for this phenomenon is the parallel training paradigm commonly used in previous skill discovery approaches, such that the same states visited by multiple skills are discouraged from being visited again.
Instead of training all skills in parallel at each epoch, ReST trains different skills one after another recurrently.
This recurrent training paradigm is supported by an effective prediction error-based intrinsic reward inspired by novelty detection methods.
We then conducted experiments on 2D maze navigation to continuous robotic control tasks.
Both qualitative and quantitative results show that ReST is able to discover more diverse skills with better state coverage compared to baseline algorithms.
Moreover, we demonstrated several novel and dynamic robot locomotion skills that have not been presented in previous works.
There are also some limitations of the proposed algorithm.
First of all, compared with previous approaches, our proposed approach has worse sample complexity during skill discovery since only one skill is trained at each epoch, as shown in Figure 3.
Moreover, the computational complexity is higher than approaches like [1] or [10] since ReST needs to compute intrinsic reward based on all other skills’ prediction errors.
Finally, due to the recurrent training paradigm, ReST is currently not scalable to continuous latent, which is in general a better choice as a low-level controller for hierarchical control in downstream tasks.
Future works include addressing the above limitations and apply ReST to downstream/hierarchical tasks Acknowledgement
This work is supported by the Ministry of Science and Technology of the People’s Republic of China, the 2030 Innovation Megaprojects “Program on New Generation Artificial Intelligence” (Grant No.
2021AAA0150000).
We thank Wei Fu and the anonymous reviewers for many insightful suggestions which improved the manuscript
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 6 for details.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] The code would be released immediately after acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A]
We did not use existing assets.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See http://sites.google.com/view/neurips22-rest (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
We present the rendered robotic locomotion tasks in this section. Videos of different locomotion skills can be found on our project website https://sites.google.com/view/neurips22-rest . A.1 HalfCheetah
We used HalfCheetah-v3 environment in OpenAI Gym [16]. [Figure] A.2 Hopper We used Hopper-v3 environment in OpenAI Gym [16].
[Figure]
A.3 Hopper Without Termination
We set the terminate_when_unhealthy parameter to False in the Hopper-v3 environment in
OpenAI Gym [16].
[Figure]
A.4 Walker2d
We used Walker2d-v3 environment in OpenAI Gym [16]
[Figure]
B Implementation Details
In this section, we introduce our implementation details of our proposed ReST algorithm and our comparison baselines.
B.1 ReST
Our proposed ReST algorithm can incorporate with any reinforcement learning algorithm to maximize the intrinsic reward.
We choose Proximal Policy Optimization (PPO) [13] as our algorithm to maximize intrinsic reward.
Detailed PPO related hyper-parameters can be found in Table 1.
We set the temperature coefficient α = 10 for 2D navigation environments and α = 1 for MuJoCo environments.
We train each skill for M = 20 epochs in all the environments.
We used k = 5 output channels for the RND networks.
We update the RND networks 10 times per epoch and the learning rate is same as the learning rate of the critic.
For the RND networks to better record the visitation of a certain skill, it needs to slightly overfit and we set the hidden dimensions for the RND networks as [500 , 500] .
B.2 DIAYN
We use our own reproduction of DIAYN in the experiments talked about in this paper.
For fair comparison, we also used PPO to optimize the intrinsic reward of DIAYN.
We basically followed the hyper-parameter setting from the original DIAYN paper and added entropy term α = 0 .
1 to the original DIAYN intrinsic reward.
We update the discriminator 10 times per epoch during training and the learning rate is the same as the learning rate of the critic.
For the independent neural network parameterization version of DIAYN, we used same initialization for all the skills for fair comparison.
We used exactly the same network architecture as our proposed algorithm.
Detailed PPO hyper-parameters can be found in Table 1.
B.3 DADS
We use our own reproduction of DADS in the experiments mentioned above.
For a fair comparison, we used PPO to optimize the intrinsic reward of DADS.
We basically followed the hyper-parameter setting, network architectures, and implementation details from the original DADS paper, like predicting ∆ s in the skill-dynamics and using a Mixture-of-Experts [40] with 4 Gaussian experts to model the output distribution.
We don’t let the dynamics to predict ∆ x, ∆ y
if the global coordinates are excluded from the state s , as there is no goal-based navigation problems in our experiments.
The input ( s, z ) first goes through a two hidden layers MLP, whose capacity is the same as that of the policy and critic networks, then the output will be linear tranformed to be the means of the Gaussian experts and a discrete distribution over them.
We fix the covariance matrix of each expert to be an
Identity Matrix.
The skill dynamics is updated 32 times per epoch during training and the learning rate is the same as that of the critic.
For the independent neural network parameterization version of DADS, we used same initialization for all the skills for fair comparison.
We used exactly the same network architecture as our proposed algorithm.
Detailed PPO hyper-parameters can be found in Table
B.4 Other Implementation Details
We implement all the algorithm using PyTorch 1.10.1 and Python 3.9.7.
All the networks are trained using NVIDIA RTX 3090 GPU and the CPU used for MuJoCo simulation is AMD EPYC 7H12.
C Details of Evaluation Metrics C.1 State Coverage
We particlize the X - Y plane of the 2D navigation environments into 20 × 20 particles.
We rollouttrajectories for each skill each algorithm.
Therefore, 200 trajectories are generated for each algorithm.
We count the particles visited by any or the 200 trajectories and record the number as n v , then the state coverage rate is n v 400 .
C.2 Mutual Information
We particlize the 4 dimensional state space, which is x, y, v x , v y into 20 × 20 × 20 × 20 particles.
We rollout 20 trajectories for each skill each algorithm.
We first count the visitation frequency of particles p ( s ) where s is a 4 dimensional particle and calculate H ( S )
= − (cid:80) s p ( s ) log p ( s ) .
We then calculate the visitation frequency of particles for each skill p ( s | z ) and calculate H ( S | Z )
= − (cid:80) s p ( s | z ) log p ( s | z .
Therefore, we derive the mutual information between skills and states:
I
[Equation]
[Table]
D Environment Details D.1 2D Navigation Environments
We use our own implementation for the 2D navigation environments in Section 4.1.
The environment is a GPU based environment implemented using PyTorch 1.10.1 and Python 3.9.7.
The state space is the concatnation of ( x, y, v x , v y ) and the action space is the acceleration ( a x , a y ) .
We use a fixed episode length of 1000 in all 2D navigation environments.
The X - Y plane of these environments is [0 . 1] × [0 , 1] whereas the velocity space is { ( v x , v y ) | v 2 x + v 2 y ≤ 0 . 01 } .
D.2 MuJoCo Environments
We use OpenAI Gym [16] as our evaluation environment in Section 4.2.
The maximum episode length of the environments are set to 1000 .
We follow the setting of OpenAI gym where HalfCheetah-v does not terminate and run for 1000 time steps each episode whereas Hopper-v3 and Walker2d-v terminates when unhealthy or reaches the maximum time step.
We expose all the state information to the discriminator or dynamic model or RND modules and exclude current position from the observation of the agents.
We also added Ant-v3 to our rendering of the learned skills.
The Ant-v3 environments are adopted from OpenAI Gym but does not terminate when unhealthy.
Moreover, some of the skills demonstrated in the rendering using Hopper-v3 also do not terminate when unhealthy.
These are done by setting the terminate_when_unhealthy=
False .
This modification makes some flipping skills and other interesting ones emerged during training.
E Why ReST does not have the lack of exploration issue mentioned by EDL [9]
Our proposed algorithm ReST not only addresses the parallel training issue, but can also solve the issue mentioned by EDL [9].
EDL argues that the reason for the poor state coverage of previous information theoretic unsupervised skill discovery approaches is that the mutual information reward inherently do not encourage exploration.
For a visited state, the intrinsic reward for the skill that visited the state would be log N while for an unexplored state, the reward for arbitrary skill would be 0 .
In this section, we demonstrate that our proposed algorithm doew not have this undesirable property.
We make the same assumption as in Equation 8, which means we have perfect RND modules.
For a state s already visited by skill z
i , the intrinsic reward would be:
[Equation]
whereas for an unexplored state s ′ , the reward function stays the same.
We note that the above reward function does not contain any term related to skill z i , which indicates that the reward function for z
i at a state has nothing to do with whether state s is an explored state by z i .
F More Comparison Baselines
We compare our proposed algorithm to several more comparison baselines in the 2D navigation environments.
The added comparison baselines are discussed in the following subsections.
F.1 DIAYN/DADS with state coverage-based intrinsic reward
We trained agents using the parallel training paradigm as DIAYN or DADS did with state coveragebased intrinsic reward as shown in Equation (7).
We added this comparison baseline to show the effect of the recurrent training paradigm and the state coverage-based intrinsic reward respectively.
The result shows that without the recurrent training paradigm, the discovered skills perform worse than those with the recurrent training paradigm while still performs better than those using mutual information as intrinsic reward.
As shown in Fig 11, we denote this baseline as ReST-p.
F.2 ReST with MI as
intrinsic reward
We also trained agents using the reverse formulation of the mutual information reward in Equation (1) as suggested by [9] together with the recurrent training paradigm.
We added this comparison baseline to demonstrate the effect of the recurrent training paradigm and the state coverage-based intrinsic reward respectively.
As shown in Fig 11, we denote this baseline as MIrecurrent.
F.3 LEXA
We trained agents using LEXA [22].
The empirical results show that LEXA does not perform well in the 2D navigation environments.
The LEXA framework does not work well in the 2D navigation settings mainly because of it’s goal-conditioned setting.
The LEXA framework takes in an image as a goal and the intrinsic reward for the achiever π g is encouraging the agent to go to and stay at the goal.
This is helpful for many real-world robotic tasks, for instance, pick and place manipulation.
However, despite the effectiveness, this goal based setting introduces some limitations in other domains.
For instance, the goal based intrinsic reward is one of the reasons why the DeepMind Control experiments demonstrated by LEXA contain mainly ‘posing’ skills, since the reward encourages the agent to stay at the state provided by the goal image.
This setting is more problematic in our setting since the observation space of the 2D navigation environment is a vector composed of the position in the x - y plane and velocities v x , v y .
If the achiever is encouraged to stay at the goal, suppose it reached the goal position and velocity at a certain timestep, then in the next time step, if v x and v y are not 0, the agent would run away from the goal state immediately, which makes the agent get low reward.
This kind of scenario is common in robotics.
For instance, if we use the LEXA framework in the legged
LEXA
MIrecurrent
[Figure]
locomotion tasks, the observation space would contain joint positions and joint velocities, which might bring trouble to the training process.
We believe this is one of the reasons why LEXA does not work in our low dimensional state settings.
Our proposed approach ReST, however, does not have such issues since it directly learns a set of skills and no explicit goal is given for the agent to reach and stay during the training process.
G Quantitative Results for MuJoCo
We quantitatively evaluated the MuJoCo robotic locomotion tasks by calculating the variance of the ending x position of each skill.
The results are shown in Table 2.
H Broader Impact
Being able to acquire useful skills in the absence of reward functions is beneficial for a wide range of applications.
To this end, we introduce our Recurrent Skill Training algorithm to effectively explore the state space of the environment without external supervision.
This enables a fully autonomous acquisition of skills, which might bring both positive and negative social impacts.
On the positive side, for instance, the fully autonomous procedure of acquiring diverse and state covering skills makes
[Table]
it possible for robots to explore in dangerous, unknown and unstructured environments to cover all valid states, preventing human from being exposed to dangers.
However, on the negative side, when deployed incorrectly in the real world, the novelty-seeking nature of the proposed approach might autonomously discover skills novel yet dangerous, which might be harmful for both the agent and the environment.
This indicates that we should be careful on the usage of such algorithms in the real world.