Unsupervised Skill Discovery via Recurrent Skill Training
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Being able to discover diverse useful skills without external reward functions isbeneﬁcial in reinforcement learning research.
Previous unsupervised skill discoveryapproaches mainly train different skills in parallel.
Although impressive resultshave been provided, we found that parallel training procedure can sometimes blockexploration when the state visited by different skills overlap, which leads to poorstate coverage and restricts the diversity of learned skills.
In this paper, we takea deeper look into this phenomenon and propose a novel framework to addressthis issue, which we call Recurrent Skill Training (ReST).
Instead of training allthe skills in parallel, ReST trains different skills one after another recurrently,along with a state coverage based intrinsic reward.
We conduct experiments ona number of challenging 2D navigation environments and robotic locomotionenvironments.
Evaluation results show that our proposed approach outperformsprevious parallel training approaches in terms of state coverage and skill diversity.
Videos of the discovered skills are available at https://sites.google.com/view/neurips22-rest .
Introduction
[Figure]
Recent advances in deep reinforcement learning have shown its promising performance indomains ranging from game playing [2, 3],robotics [4, 5] and recommender systems [6].
These applications of reinforcement learningrely on task-speciﬁc reward functions for theagents to successfully accomplish the tasks.
However, intelligent creatures can automaticallyexplore the environments and learn diverse useful skills in the absence of external supervision.
Such ability is beneﬁcial in a variety of situations.
For tasks where rewards are non-trivialto design or where the reward signal is sparse,unsupervised skill discovery approaches can provide intrinsic rewards to help accomplish tasks.
Moreover, in hierarchical control problems, unsupervised skill discovery can serve as low-levelpolicies for downstream tasks [7, 1].
Although existing works have shown great potential in discovering diverse useful skills in an unsupervised manner [1, 7, 8], one of the key problems of such methods, as observed in our preliminary experiments and some recent works [9, 10], is that they might suffer from poor state coverage,which may lead to failures in learning desirable useful skills.
For instance, in robotic locomotionenvironments, previous approaches tend to learn ‘posing’ skills instead of dynamic, far-reachingskills [1].
A straightforward explanation for this phenomenon would be the lack of exploration [9].
However, we argue that this is not the only reason for the poor state coverage.
Counterintuitively, weobserved that the skills after convergence may avoid visiting certain states even if they are exploredduring training .
For instance, as shown in our preliminary 2D navigation experiments, the discoveredskills fail to cover the states passing through the bottleneck to the right (Figure 1b), even if they havebeen explored during training (Figure 1a).
We call this phenomenon Exploration degradation .
In this paper, we take a deeper look into the above phenomenon and show that it is mainly causedby the parallel training paradigm, which is a common choice for most existing works [1, 7, 8, 10].
When multiple skills trained in parallel have visited the same state, such state will be preventedfrom being visited again.
Detailed analysis is provided in Section 3.1.
Based on the analysis, wepropose Re current S kill T raining (ReST), an unsupervised skill discovery algorithm that addressesthe exploration degradation issue.
Instead of training all the skills in parallel, ReST trains the skillsone after another in a recurrent fashion, along with an intrinsic reward that discourages coveringfrequently visited states of other skills.
A preliminary result of using ReST is shown in Figure 1c,where the exploration degradation problem is eliminated and the converged skills are able to visit thespace in the right room.
Evaluation results on complex 2D navigation and robot locomotion tasksshow that our approach can achieve better state coverage and skill divergence compared to baselines.
Our contributions are summarized as follows:
• We discover a new phenomenon that reduces state coverage called exploration degradation,which indicates that some certain states are discouraged from being visited by the learned
skills, even if they have been explored during training.• We show that the main reason causing exploration degradation is that multiple skills visitingthe same states can reduce the MI reward in the parallel training paradigm.
We then proposeRe current S kill T raining (ReST), a recurrent training paradigm along with a state coveragebased intrinsic reward, which prevents multiple skills from visiting the same states andalleviates the exploration degradation issue.•
We conduct experiments on various 2D navigation tasks and robot locomotion tasks.
Evaluation results show that our method achieves better state coverage and divergence comparedto baseline methods.
Moreover, ReST learns diverse meaningful robot locomotion skillsthat have not been shown in previous works.
Preliminaries
2.1 Markov Decision Process
Markov decision process (MDP) can be used to ﬁnd a reward maximizing policy.
It is representedby a tuple
( S , A , P , R,  , µ ) , where S , A are the state and action spaces.
P : S ⇥ A ⇥ S !
[0 , 1]is the transition dynamic that maps the state and action into a probability distribution over the nextstate.
R : S ⇥ A !
R is the reward function.
is the discount factor for the reward function whileµ : S ! [0 , 1] is the initial state distribution.
The expected discounted cumulative reward can beformulated as J R ( ⇡ )
{+= E ⌧ ⇠ ⇡ [ P 1 t =
+}
t R ( s t , a t )] .
Thus the overall optimization problem can bewritten as ⇡ ⇤ = arg max ⇡ J R ( ⇡ ) .
2.2 Unsupervised Skill Discovery
Generally speaking, unsupervised skill discovery aims to ﬁnd a family of skills conditioned on latentz , which results in a latent-conditioned policy ⇡ ( a | s, z ) that maximizes the mutual informationbetween state and latent:
[Equation]
where s 2 S , a 2 A , z 2 Z are the state, action and latent respectively.
Let ( S, Z )
⇠ p ( s, z ) as 84 the random variables of the state distribution and the latent distribution.
As suggested by [11], avariational lower bound for Equation (1) can be derived as:
[Equation]
where q  ( z | s ) is a learned discriminator approximating p ( z | s ) .
Such lower bound also exists for
Equation (2):
[Equation]
where q  ( s | z ) is a learned function approximator of p ( s | z ) .
Recurrent Skill Training
[Figure]
In this section, we ﬁrst introduce the exploration degradation phenomenon of previous skill discoveryapproaches with parallel training paradigm.
Then we present our proposed recurrent skill trainingmethod that addresses this issue.
3.1 Exploration Degradation
As we observed in our preliminary experiment shown in Figure 1, previous unsupervised skilldiscovery approaches might suffer from the exploration degradation problem, such that some exploredstates (e.g, states near the bottleneck and in the right room in Figure 1a) are prevented from beingvisited by the learned skills.
We now provide a simpliﬁed analysis to show that this phenomenon ismainly caused by the parallel training paradigm commonly used in previous works.
[Figure]
[Figure]
Algorithm Initialize: random initialized skill ⇡ , RND networks ˆ f and f for i = 1 to N do
Set skill ⇡ i = ⇡
Set RND network ˆ f i
=
ˆ f , f i = f
Collect on-policy samples with ⇡ i
Update RND network
ˆ f i by minimizing loss in Equation (5) end for repeatfor i = 1 to N ( N is the number of skills) do for j = 1 to M ( M is the number of training epochs for each skill) do
Collect on-policy samples with ⇡ i ;
Calculate reward using Equation (7);
Update skill policy ⇡ i using any RL algorithms;
Update RND network
ˆ f i by minimizing loss in Equation (5) using the latest on-policy samples; end for end for until convergence
Let us focus on discrete latent case with N different skills.
Consider at state s , skill k visited state swith p ( z k | s ) probability, which is modeled by the discriminator q  ( z k | s ) .
Assume state s 0 is onlyvisited by skill k
whereas s 1 is visited by multiple skills, which means p ( z k | s 0 )
= 1 and p ( z k
| s 1 ) < 1 .
As long as the discriminator q  can model the difference and output q  ( z k | s 0 ) >
q  ( z k | s 1 ) , which isnot a strong assumption, the intrinsic reward used by MI approaches would encourage the visitationof state s 0 by skill k and discourage visiting s 1 .
This means MI based approaches end up withskills that prefer exploiting states only visited by themselves in the previous training epochs, causingexploration degradation.
For clarity, we further explain this issue with a toy example.
Consider a 2 ⇥ 2 grid world as shownin Figure 2.
There are four states 0 , 1 , 2 , 3 and at each state, there are three actions to take: go tothe two adjacent states or stay where it is.
For simplicity, we use the number of the next state todenote the action, for instance, if the agent is in state 0 and chooses to go to state 1 , then the actionwill be denoted as 1 .
We use the mutual information described in Equation (1) and assume that thediscriminator is perfect: q  ( z | s ) =
p ⇡ ( z | s ) .
Consider a case when the number of skills is N = 2and during the ﬁrst collection of samples, skill 1 visited { 0 , 2 , 3 } while skill 2 visited { 0 , 1 , 3 } , asshown in Figure 2a.
Since we have a perfect discriminator, for state 3 the discriminator wouldoutput q  ( z 1 | 3) =
q  ( z 2 | 3) = 0 . 5 , which results in 0 reward for both skills.
Therefore, the optimalconverged policy for skill 1 would generate trajectory { 0 , 2 , 2 } while skill 2 would generate trajectory{ 0 , 1 , 1 } , as shown in Figure 2b.
State 3 is not covered, which is undesirable.
This example indicatesthat even in such an extremely simple case, the exploration degradation phenomenon still exists.
3.2 Recurrent Skill Training
Instead of training all skills in parallel, we propose a recurrent training paradigm, along with a statecoverage intrinsic reward.
We now introduce the details of our proposed method.
Recurrent Training Paradigm.
As analyzed in Section 3.1, the main reason causing the explorationdegradation issue is that the same states are visited by multiple skills in the parallel training paradigm.
A natural way to alleviate this issue is to train the skills one after another recurrently.
In this way,we can encourage the latter trained skills to avoid entering the same states covered by the previousskills.
Figure 3 illustrates the recurrent training paradigm.
Starting with N randomly initialized skills( N = 3 in this case), the recurrent training paradigm trains one skill at each epoch whereas the paralleltraining paradigm trains all the skills together.
Furthermore, in order to improve convergence, therecurrent training paradigm updates each skill for M epochs ( M = 2 in this case) before switchingto another skill.
In this work, we use N independent neural networks to parametrize the N skills,which can be considered as discrete latent conditioned policies.
State Coverage Intrinsic Reward.
When implementing the above recurrent training paradigm,the latter trained skill needs to avoid visiting the states frequently visited by other skills.
In orderto accomplish this objective, we need to identify how frequently each state is visited by each skill,or equally, how novel a given state is to a skill.
In this paper, we adopt random network distillation(RND)
[12], a simple yet scalable novelty detection approach, to estimate the novelty of a state to aspeciﬁc skill.
Random network distillation (RND) [12] mainly involves two networks for state novelty detection: arandomly initialized target neural network whose parameters are kept ﬁxed throughout the wholetraining process and a predictor network trying to ﬁt the target neural network. The target networkf : S !
R k , where k is the dimension of the output vector, maps a state to a vector while thepredictor network ˆ f : S ! R k is trained to minimize the expected mean square error between theoutput vector and the target output vector: || ˆ f ( s )  f ( s ) || 2 , which distills the target network into thepredictor. Therefore, the predictor should have low prediction error on the data it is trained on whilehave high prediction error on other states.
We call such target network and predictor a pair of RNDnetworks . We assign each skill with a pair of RND networks. Each skill’s predictor network is trained on thethe on-policy data it visited during the last rollout, which means for each skill with latent z i 2 Z ,where i 2 { 1 , 2 , ..., N } is the index of the skill, with state distribution s ⇠ p ( s | z i ) , we can train thepredictor network using gradient decent by minimizing the mean square error loss L i
[Equation]
As mentioned above, the learned predictor network’s prediction error
||
ˆ f i ( s )  f i ( s ) || 2 can indicate 152 visitation frequency of a certain state s .
Intuitively, a higher prediction error indicates higheruncertainty of ˆ f i on this state, which further implies its higher novelty.
Since we need to avoidvisiting states visited by other skills when training a certain skill, a straightforward solution is todeﬁne a reward function for skill i with state s t and action a t as:
[Equation]
such that states frequently visited by any other skills are less desired.
However, this minimumoperator would make the reward landscape rugged, which might lead to poor convergence property.
To stabilize the training process, we introduce a soft version of the minimum operator in (6):
[Equation]
where ↵ is a task-speciﬁc temperature parameter.
Practical Algorithm.
We summarize our algorithm in Figure 3 and Algorithm 1.
Firstly, we randomlyinitialize a policy network ⇡ i and a pair of RND networks f i and ˆ f i for each skill i 2 { 1 , 2 , ..., N } .
Before training skills, ReST ﬁrst collects on-policy samples for each initialized skill and train their RND networks.
Then we train each skill’s policy network and the corresponding RND networks for M times during each training epoch.
The skill networks are trained recurrently until convergence.
Formaximizing the intrinsic reward, the ReST algorithm can be combined with an arbitrary reinforcementlearning algorithm.
In this paper we choose Proximal Policy Optimization (PPO) [13] with generalizedadvantage estimation (GAE) [14].
Grid World Example.
We further illustrate the effectiveness of the proposed algorithm using the2 ⇥ 2 grid world example with the same setting as in Section 3.1.
As suggested by our previousanalysis, the parallel training paradigm will end up with two skills that neither of them visits state 3 .
As mentioned in Section 3.1, the initial two skills visits { 0 , 2 , 3 } and { 0 , 1 , 3 } respectively.
Here weassume that the RND networks f i and ˆ f i perfectly obtain the state visitation frequency, which means:
[Equation]
if state s is visited by skill i r otherwise (8) where r > 0 is the prediction error.
ReST starts by training the corresponding RND networks of eachskill and then recurrently train different skills to maximize the intrinsic reward.
For skill 1 , visitingstate 2 would gain reward r while visiting state 1 and 3 would get zero rewards.
Therefore, theoptimal policy for skill 1 would be visiting { 0 , 2 , 2 } in sequence.
After updating the RND networksof skill 1 correspondingly, for skill 2, visiting state 1 and 3 would gain reward r while visiting 2would get zero rewards.
Therefore, the initial skill 2 is already the optimal policy and the trainingconverges.
This way, the optimal trajectory for skill 1 is { 0 , 2 , 2 } while for skill 2 it is { 0 , 1 , 3 } , asshown in Figure 2c.
Therefore, the state space of the 2 ⇥ 2 grid world is fully covered by the twoskills.
Experiments
[Figure]
can be found in the Appendix We conduct experiments on several challenging 2D navigation environments and several roboticlocomotion tasks.
We compare our proposed approach with two of the most popular unsupervisedskill discovery approaches, DIAYN [1] and DADS [7].
Both of these approaches use the paralleltraining paradigm to optimize different skills.
Since our proposed approach parameterizes each skillwith an independent neural network, we also compare our proposed approach with DIAYN and DADSusing independent neural networks, which we call DIAYN-i and DADS-i respectively, for a faircomparison.
We present our empirical results both qualitatively and quantitatively.
The quantitativeresults introduce a state coverage metric and a mutual information metric.
Our proposed approach
[Figure]
[Figure]
Generally speaking, baseline methods discover ‘posing’ skills instead of dynamic locomotion skills whereas ReST discovers dynamic, far-reaching locomotion skills.
signiﬁcantly outperforms previous skill discovery approaches in terms of state coverage while stayingcomparable with previous approaches in terms of mutual information.
Qualitative results includevisualizations of the converged navigation experiments, the state visitation of robotic locomotiontasks, and rendered videos of novel skills discovered using our proposed approach, which can befound in our project website https://sites.google.com/view/neurips22-rest .
Details ofour implementation can be found in the Appendix.
4.1 2D Navigation Tasks
Environments.
We conducted experiments on several challenging 2D navigation environments.
Theagent is a point mass navigating in a 2D plane with boundaries [0 , 1] ⇥ [0 , 1] .
The observation spaceof the environment has 4 dimensions, including the x 2 [0 , 1] and y 2 [0 , 1] position on the planeand the corresponding velocity v x and v y belonging to the velocity space { ( v x , v y ) | v 2 x + v 2 y  0 . 01 } .
The action space has 2 dimensions, including the acceleration a x 2 [0 , 0 . 1] and a y 2 [0 , 0 . 1] of the point mass agent.
There are walls inside the plane and the agent cannot go through the walls.
Wedesigned diverse placements of the walls with increasing difﬁculties for exploring the environments,which could be used to test the effectiveness of our proposed approach.
Evaluation Metrics.
We introduce two metrics to compare our proposed approach with our comparison baselines.
The ﬁrst one is the state coverage metric.
State coverage matters since notcovering enough state space might result in failures in learning desirable useful skills.
We evaluatethe state coverage on the X - Y plane by ﬁrst decomposing the environment into cells and then testingwhether each cell is successfully visited.
We roll out 1 trajectory for each skill and use the visitedstates to calculate the state coverage metric.
The percentage of cells visited by at least one of theskills is the state coverage.
Moreover, merely covering the state space is not enough.
The skillsneed to be informative about which states they are going to visit in the environment so that theskills are meaningful.
We use mutual information between skill latent z and the correspondingcovered states s to quantify how informative the skill is.
Similarly, we decompose the state spaceinto multiple cells and roll out 20 trajectories for each skill to record the probability distributionof states p ( s ) for all the states visited by the skills.
Then we record the probability distribution ofstates visited by each skill p ( s | z ) and calculate the entropy H ( S ) and H ( S | Z i ) .
Since the skilldistribution p ( z ) is a uniform distribution for all skills, we can calculate the mutual information as I [Equation] S ;
[Equation]
Qualitative Results.
We train N = 10 skills for each of the algorithms and rollout 20 trajectories toobtain the qualitative results.
We conduct experiments on 4 challenging 2D navigation mazes, whichwe call the DoorMaze , CenterMaze , 4RoomMaze and 9RoomMaze , which are shown in Figure 4.
Theleft 5 columns show the qualitative results.
Each rolled-out trajectory is rendered as a curved lineand different colors represent different skills.
The results indicate that our proposed ReST algorithmcan reach more diverse states in the environments whereas baseline approaches like DIAYN or
DADS can only reach a small portion of the environments.
Usually, the baseline approaches cannotpass through the ‘bottlenecks’ in the environment.
The results support our insight that the paralleltraining paradigm is one of the causes of previous skill discovery approaches’ state coverage issues.
Moreover, when using independent neural networks to parameterize different skills, we observe thatthe performance is not so different from the latent-conditioned parameterization version.
Quantitative Results.
Beyond qualitative results, we quantify the state coverage and the informativeness of different algorithms using the aforementioned two metrics.
As shown in the right two columnsof Figure 4, our proposed ReST algorithm signiﬁcantly outperforms baseline approaches with aparallel training paradigm in terms of state coverage and is comparable with baseline approachesin terms of mutual information I ( S ; Z ) .
The above results indicate that our proposed approach cancover diverse sets of states without too much sacriﬁce on informativeness.
Quantitative results alsoevidenced that the difference in performance is not because of the different parameterization since
DIAYN and DADS using independent neural networks are not so different from the original ones.
4.2 Robotic Locomotion Tasks
We present qualitative results of robotic locomotion skills discovered using our proposed ReSTalgorithm in MuJoCo [15].
Generally speaking, our proposed ReST algorithm can discover dynamic,far-reaching robotic locomotion skills whereas DIAYN and DADS tend to discover ‘posing’ skills.
Figure 5 shows visualizations of parts of skills discovered using ReST.
There are several novel skillsdiscovered, such as Hopper backﬂip, that have not been presented in previous works to the bestof our knowledge.
More rendered results can be found in the Appendix and our project websitehttps://sites.google.com/view/neurips22-rest .
Moreover, we provide semi-quantitative results of the proposed approach.
As shown in Figure 6, wedraw the agent’s x position over timestep, using the skills discovered by ReST and the comparisonbaselines.
The results are evaluated on HalfCheetah , Hopper and Walker2d tasks.
We use the
OpenAI Gym [16] settings of the three tasks, where HalfCheetah is trained with ﬁxed episodelength whereas Hopper and Walker2d terminate when the agents fall during training.
The resultingtimestep- x curve indicates that our proposed approach learns skills that are more far-reaching, diverse,and dynamic than baseline methods.
This also evidenced that our recurrent training paradigmoutperforms the parallel training-based baselines by alleviating their state coverage issues.
Related Work
Unsupervised Skill Discovery.
Previous unsupervised skill discovery approaches mainly focus onmaximizing the mutual information I ( S ; Z ) to obtain meaningful skills.
As discussed in Equation(1) and Equation (2), the mutual information has two forms.
Several works follow the Equation (1).
VIC [17] uses the mutual information between the ﬁnal states and the skill latent as the intrinsicreward and optimizes it via reinforcement learning.
DIAYN [1] ﬁxes the prior p ( z ) and uses themutual information between the skill and its visited states as its intrinsic reward to learn meaningfulskills, which has superior performance compared to VIC.
VALOR [8] further improves DIAYNby replacing the state-based objective with a trajectory-based objective.
DADS [7], on the otherhand, uses the Equation (2) form of mutual information and learns a transition model q  ( s {+
+} | s, z )and uses model predictive control to solve downstream tasks.
EDL [9] provides insight into whyprevious approaches suffer from lack of exploration and proposes an algorithm using a ﬁxed stateprior p ( s ) that to alleviate the issues.
IBOL [10] tries to relieve the difﬁculty of reaching diversestates by introducing a low-level controller.
Besides learning a set of skills, SMM [18] formulatesskill discovery as a state marginal matching problem and optimizes the KL divergence between theexpected state distribution and the current policy’s state distribution.
MUSIC [19] improves previousunsupervised skill discovery algorithms by adding the mutual information between the surroundingstate and the agent state.
DDL [20] learns one skill that maximizes the dynamical distance functionsof the previous skill.
Plan2Explore [21] makes use of self-supervised reward signals to optimize onepolicy to efﬁciently gather state-covering dataset.
In contrast, ReST optimizes multiple skills andmaximize online state coverage.
LEXA [22] learns a goal-conditioned policy using self-supervisedreward.
ReST, on the other hand, does not require the agent to achieve speciﬁc goals, which is a moregeneral formulation.
Besides the above approaches, there are also other unsupervised skill discoverymethods [23, 24, 25, 26].
Intrinsic Reward.
Another stream of works related to this work is intrinsic reward.
In our proposedapproach, we use intrinsic reward as an objective to help learn a set of meaningful skills.
Intrinsicreward can also help with cases where rewards are sparse by augmenting them to the original rewardfunction.
Count-based exploration [27, 28, 29, 30] uses pseudo count to identify the frequentlyvisited states and the less frequently visited ones and adds the count-based bonus as intrinsic rewardsto accelerate exploration.
Prediction error exploration methods [31, 32, 12, 33, 34] make use ofprediction errors as intrinsic rewards based on an insight that states with high prediction error shouldhave higher novelty.
Other works augment an information-theoretic intrinsic reward with extrinsicrewards that encourage information gain about the environments [35, 36, 37].
Discussion
In this paper, we proposed a novel, effective yet simple algorithm called Recurrent Skill Training(ReST).
We began by introducing a new phenomenon called exploration degradation which reducesstate coverage of the learned skills.
We found the key reason for this phenomenon is the paralleltraining paradigm commonly used in previous skill discovery approaches, such that the same statesvisited by multiple skills are discouraged from being visited again.
Instead of training all skillsin parallel at each epoch, ReST trains different skills one after another recurrently.
This recurrenttraining paradigm is supported by an effective prediction error-based intrinsic reward inspired bynovelty detection methods.
We then conducted experiments on 2D maze navigation to continuousrobotic control tasks.
Both qualitative and quantitative results show that ReST is able to discover morediverse skills with better state coverage compared to baseline algorithms.
Moreover, we demonstratedseveral novel and dynamic robot locomotion skills that have not been presented in previous works.
There are also some limitations of the proposed algorithm.
First of all, compared with previousapproaches, our proposed approach has worse sample complexity during skill discovery since onlyone skill is trained at each epoch, as shown in Figure 3.
Moreover, the computational complexityis higher than approaches like [1] or [10] since ReST needs to compute intrinsic reward based onall other skills’ prediction errors.
Finally, due to the recurrent training paradigm, ReST is currentlynot scalable to continuous latent, which is in general a better choice as a low-level controller forhierarchical control in downstream tasks.
Future works include addressing the above limitations andapply ReST to downstream/hierarchical tasks.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6 for details.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] The code would
be released immediately after acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the appendix.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
We did not use existing assets.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See http://sites.google.com/view/neurips22-rest
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]