LECO: Learnable Episodic Count for Task-Specific Intrinsic Reward
Anonymous Author(s)
AffiliationAddress email
Abstract
Episodic count has been widely used to design a simple yet effective intrinsicmotivation for reinforcement learning with a sparse reward.
However, the use ofepisodic count in a high-dimensional state space as well as over a long episode timerequires a thorough state compression and fast hashing, which hinders rigorousexploitation of it in such hard and complex exploration environments.
Moreover,the interference from task-irrelevant observations in the episodic count may causeits intrinsic motivation to overlook task-related important changes of states, and thenovelty in an episodic manner can lead to repeatedly revisit the familiar states acrossepisodes.
In order to resolve these issues, in this paper, we propose a learnablehash-based episodic count, which we name LECO, that efficiently performs asa task-specific intrinsic reward in hard exploration problems.
In particular, theproposed intrinsic reward consists of the episodic novelty and the task-specificmodulation where the former employs a vector quantized variational autoencoderto automatically obtain the discrete state codes for fast counting while the latterregulates the episodic novelty by learning a modulator to optimize the task-specificextrinsic reward.
The proposed LECO specifically enables the automatic transitionfrom exploration to exploitation during reinforcement learning.
We experimentallyshow that in contrast to the previous exploration methods LECO successfully solveshard exploration problems and also scales to large state spaces through the mostdifficult tasks in MiniGrid and DMLab environments.
Introduction
In reinforcement learning (RL), tackling the problem of sparse rewards still remains a great challenge.
Many works have been proposed to develop dense synthetic reward, which is generally calledintrinsic reward, to provide the motivation for exploration in the absence of extrinsic rewards fromthe environment [5, 24, 1, 26, 13, 36, 31, 10, 38, 27, 35].
Among them, count-based exploration[29, 4, 23, 22, 31, 21, 5] that encourages the RL agent to visit novel states, is one of the most popularclasses for intrinsic motivation since it is simple and effective in many tasks, and more importantly itis theoretically justified [29, 19].
In general, the state novelty can be measured within a single episodeor through an entire task across episodes.
In recent RL problems, environments are often changedover episodes or even procedurally generated within the same task [7, 3, 20, 8, 16].
Therefore,per-episode novelty by episodic count has been widely adapted in modern hard exploration problems[1, 26, 36].
However, in high-dimensional and/or continuous state spaces, state counts need to be approximatedin an abstract space since most states are likely to be unique and novel in the original space, leadingto insufficient exploration [4, 23, 26, 1].
In addition, computation of episodic count at every timestep based on the episodic memory could be burdensome if an episode lasts for a long time and therefore requires fast hashing [31].
Previous works have discretized the state space to obtain the hashcode [31, 14, 9] for a fast pseudo-count, but they have not extensively applied it to recent difficultexploration problems such as procedurally generated and partially observable environments.
Moreover, existing count-based exploration methods have mostly employed task-agnostic statenovelty, and thereby task-irrelevant count may disturb its intrinsic reward bonus to faithfully reflecttask-related information [18, 2, 17, 38].
This problem could be severe in exploration by per-episodecount.
For example, task-wise familiar states across episodes can have high novelty in each episode,which confines overall exploration into smaller areas, while some states that are highly related toextrinsic rewards need to be repeatedly revisited even in a single episode for obtaining extrinsicrewards more densely.
In this work, we propose a hash-based episodic count that is learned to provide task-specific intrinsic motivation in solving hard exploration problems.
The proposed L earnable E pisodic CO unt(LECO) produces task-specific intrinsic rewards that are efficiently and practically applicable tohigh-dimensional states, long-horizon episodes, procedurally generated environments, and highlysparse extrinsic rewards.
In specific, the proposed intrinsic reward is composed of the episodiccount-based novelty and the task-specific modulation.
Our episodic count makes use of a vectorquantized variational autoencoder (VQ-VAE) [32] to automatically obtain the discrete state codesfor a fast counting.
In addition, we exploit a learnable modulator to control the episodic novelty inrelation to task-specific extrinsic rewards.
The proposed LECO also adaptively shifts the RL phasefrom exploration to exploitation.
Experimental results on the most difficult exploration tasks inprocedurally generated environments of MiniGrid [7] and DMLab [3] show that LECO efficientlysolves such tasks and outperforms previous state-of-the-art exploration methods.
Our main contributions can be summarized as follows:
• A novel learnable episodic count is proposed to efficiently provide task-specific intrinsicmotivation to the RL agent under complex environments with sparse rewards.•
A learnable hash code based on VQ-VAE as well as a learnable reward modulator arecombined to maximize extrinsic rewards during RL.• On various hard exploration benchmark tasks, the proposed LECO demonstrates its feasibility in solving diverse tasks and consistent performance improvements over previousexploration methods.• Several ablation experiments show the benefits from the use of learnable hash codes as wellas the task-specific reward modulation.
Related Work
Count-based exploration.
Count-based exploration was first proposed in tabular tasks [29], whichpromotes the agent to reduce its uncertainty by visiting unfamiliar states corresponding to low counts.
However, it is basically ineffective in high-dimensional state spaces since every state would bedifferent.
Therefore, several recent works have utilized count-based exploration for high-dimensionalenvironments.
For instance, pseudo counts are derived from density models using context treeswitching [4] or PixelCNN [23].
More recently, RND [5] has approximated the state visitation countby the difference between a random fixed target network and a trained predictor network.
RNDhas also been used in other methods for measuring the state novelty [1, 36, 35].
On the other hand,a number of algorithms have transformed the high-dimensional state to the discretized hash codefor fast visitation counting by a hash table [31, 10].
Similarly, we obtain the episodic count by thediscretized hash code.
However, we exploit learnable hash codes by VQ-VAE that is trained toapproximate the state density.
In addition, different from [14] that directly produces a global code by
VQ-VAE, we concatenate local codes as in the original VQ-VAE for generating a single state code.
Task-specific intrinsic motivation.
While count-based exploration methods have typically utilizedthe state novelty in a task-agnostic manner, curiosity-based methods have tried to consider task-relatedinformation for intrinsic motivation by learning the environment dynamics [24, 27, 18, 26].
Theyhave developed intrinsic rewards based on the prediction error upon the agent’s actions.
Amongthem, recently, a number of approaches have exploited the task-dependent state novelty using theinformation bottleneck in order to ignore task-irrelevant distractions [18, 2, 17].
Meanwhile, there have been a number of attempts to directly learn the intrinsic reward function to ultimately optimizethe task-specific extrinsic rewards [38, 37, 34].
Their algorithms are generally performed under themeta-gradient framework.
LECO also adopts the idea of learnable rewards and the gradient-basedbi-level optimization [38] to construct a task-specific modulator for intrinsic rewards.
However,unlike those works that do not work well with sparse extrinsic rewards, LECO takes a full advantageof intrinsic motivation by the task-agnostic state novelty based on the episodic count before obtainingthe extrinsic rewards.
Exploration for procedurally generated environments.
One of the important challenging RLproblems is robust exploration in a procedurally generated environment where the environment(partially) changes between episodes, and hence the agent needs to explore a very large state spaceand have strong generalization ability.
Many recent RL benchmarks deal with procedurally generatedenvironments [7, 3, 20, 8, 16], and accordingly state-of-the-art exploration algorithms have tried toimprove sample efficiency in such environments.
RIDE [26] has used a change between successivestates on a latent space as an intrinsic reward while AGAC [13] has encouraged action diversityby an adversarial policy.
NovelD [36] has exploited regulated difference of the novelty betweenconsecutive states, and MADE [35] has devised exploration by maximizing the deviation frompreviously explored regions.
AMIGo [6] has proposed a teacher that generates count-based intrinsicgoals.
In contrast to these handcrafted or task-agnostic intrinsic rewards, LECO learns to leveragetask-level information to guide the count-based exploration especially at the episode level due tothe procedurally generated environment.
Moreover, LECO also shows improved performances on
DMLab tasks with higher-dimensional partial observations.
Learnable Episodic Count via Task-Specific Modulation
3.1 Notation
Our problem setting follows a typical RL problem described using Markov Decision Process definedby a set of states S , a set of actions A , and a transition function T : S ×
A → S .
An RL agentinteracting with the environment samples its actions from a policy, which is an action distributionfunction π → P ( A ) , and receives a reward, r : S × A → R , upon state transition.
The objective ofthe agent is to learn a policy that maximizes the expected sum of rewards G t =
E π (cid:2)(cid:80) ∞
k = t γ k r t + k +1 (cid:3) ,where γ ∈ [0 , 1] is a discount factor.
Among various sub-problems that RL faces, balancing the trade-off between exploration and exploitation still remains a fundamental problem [30].
Especially, efficient exploration is important whendealing with sparse rewards.
One approach to solving such problem is to design an algorithm thatdirectly generates exploring behavior.
More specifically, a widely adapted method is to design anintrinsic motivation through a form of a reward such that the reward which the agent receives at timet is given by r t
= r et + αr it , where r e is the extrinsic reward from the environment, r i is the intrinsicreward designed for exploration, and α is a hyperparameter for scaling [26].
3.2 LECO Intrinsic Reward
The use of state visitation count for intrinsic rewards based on the state novelty has been a coreconcept in several previous studies [26, 1, 36, 13, 28, 15] due to its simplicity, task-agnostic property,and theoretical basis.
However, such state-novelty measured from visitation counts monotonicallydecreases as more experiences accumulate, equally for both important and unimportant states.
Thus,an agent would be discouraged to visit even the task-important states after a certain period of time, andthe exploration towards those states vanishes [1].
Furthermore, an intrinsic bonus based on the statevisitation could lead the agent to only pursue any state novelty, especially in the partially-observableand sparse reward environments.
For example, imagine a task in which the goal is to find a coloredkey hidden in a box and unlock a door that has the same color as the key.
If the box changes its colorto a random color every time when it is opened, the agent would repeatedly open and close the boxjust to obtain the intrinsic bonus.
In this task, the agent should be encouraged to find the door afterobtaining the key and discouraged to repeat task-irrelevant actions.
We directly build upon this ideaby decomposing the intrinsic reward into the state-novelty and task-specific modulation, to balanceexploration and exploitation, respectively.
We design our intrinsic motivation to encourage the discovery of state novelty that is adaptivelycontrolled to maximize the extrinsic return, hence the name L earnable E pisodic CO unt for taskspecific motivation.
Specifically, we define the general form of intrinsic reward as a sum of twoparts,
[Equation]
where r ep t ( s t +1 )
= 1 / (cid:112) N ep ( s t +1 ) is the intrinsic reward from the episodic state-novelty, N ep ( s ) isthe episodic visitation count of state s , and r ta is the task-specific modulation with λ as its weightingfactor.
The first term measures the episodic novelty by state visitation count within the current episodethat is initialized to zero at the beginning of each episode.
The second term learns to modulate theepisodic novelty in the first term according to whether the state is beneficial or harmful in achievingthe task-specific objective.
Note that the previous action a t − 1 is included as the input to rewardcalculation to provide extra transition information.
A schematic diagram of the proposed intrinsicreward can be found in Figure 1.
LECO comprises of two submodules: 1) a learnable hash that encodes input observations to discretehash codes for episodic counts and 2) a modulator that maximizes the task-specific extrinsic rewardsby adaptively scaling the episodic novelty.
Here, the modulator additively scales the episodic novelty.
In comparison to multiplicative scaling, the additive modulation allows the episodic novelty to beretained solely when the modulator does not give rise to enough signal due to the scarcity of extrinsicrewards.
3.3 Episodic Count via Vector Quantized Hashing
Previous researches have extended classic state visitation counts [29] to high-dimensional state spaces[4, 31].
A generalization of the classic count-based approach by locality-sensitive hashing (LSH)on the continuous embedding space was proposed [31] where the latent space is obtained by anautoencoder (AE).
The use of a hash table in their approach enables counting the occurrences ina constant time, taking the advantage of scalability, especially in modern distributed RL systems[12, 11, 25].
However, such LSH does not take into account the spatial information due to its dataindependency [33].
As an alternative to AE, we propose to utilize vector quantized variationalautoencoder (VQ-VAE) [32] that directly optimizes the discrete compressed representation of a state,producing a hash code without LSH.
In LECO, a state s t is mapped to a hash code c t by vector quantized hashing based on VQ-VAEcomprised of an encoder and a decoder with θ enc and θ dec as their parameters, respectively.
In specific,
VQ-VAE formulates the hash function as quantization operation z q ( s t ; θ e ) to a pre-defined codebookθ e ∈ R K × D where K is the size of the discrete latent space, and D is the dimensionality of eachlatent embedding vector e i .
The state feature z e ( s t ) is mapped to the nearest embedding z q ( s t ) =
e k [-i ] w × h i =
 where-],where k = argmin
j ∥
z e ( s t ) − e j ∥ 2 .
The spatial feature z e = f enc ( s t ; θ enc ) is obtained from the CNNencoder of VQ-VAE. The hash code is then defined as the sequence of indices c t = [ k i ] w × h i =0 , wherew and h are width and height of the spatial feature, respectively.
Following the conventional VQ-VAE training, the hash function is updated according to the followinggradient
[Equation]
where sg stands for stopgradient operator, T is an unroll length of the agent, and θ vq
= { θ enc , θ e , θ dec } .
3.4 Task-Specific Intrinsic Modulation
In a sparse reward environment, when an extrinsic reward signal has little or no presence, having atask-agnostic intrinsic motivation can be significantly beneficial in guiding the agent to efficientlyexplore the environment until the discovery of more extrinsic signals.
However, a task-generalintrinsic reward such as the inverse of the episodic count may become a distracting signal in thelater exploitation since it is not reduced with the progression of learning.
We thereby propose toinclude task-specific modulation that aims to ultimately maximize the extrinsic return G t in theintrinsic reward formulation of LECO.
The task-specific modulation can be seen as a dynamic factorfor controlling the exploration and exploitation, based on the extrinsic rewards earned from theenvironment.
In order to specify the task-relevant motivation, several methods have been proposed such as reachingsubgoals generated by the agent itself [6, 10]. Although those methods are capable of solving hardexploration tasks like grid maze or Montezuma’s Revenge from the Atari games, generating goalsin high-dimensional environments that do not provide meta-data (e.g. agent position) still remainsinfeasible. On the other hand, LIRPG [38] has proposed a learning framework of parametric intrinsic rewardssimilar to that of the meta-gradient of policy gradient based on the extrinsic return. We leverage thisframework in optimizing our modulator to maximize the extrinsic reward.
In this framework, thepolicy parameters θ is updated using policy gradient ∇ θ J e + i , where J e + i =
{+E θ [ (cid:80) ∞ t =
+}
γ t ( r et + αr it )]and the intrinsic reward is constructed from a parametric model r i t =
r ep t + r ta t ( a t − 1 , s t , a t ; θ ta ) withθ ta as the modulator parameters.
Then, the modulator is updated using the gradient ∇ θ ta J e where
J e = E θ ′
{+[ (cid:80) ∞ t =
 γ t r et ] and θ ′ = θ ++} η ∇ θ J e + i is the updated policy parameters, affected by ourintrinsic reward, with the step size η .
Namely, the intrinsic module parameters θ ta is updated bycomputing the gradient by the chain rule such that ∇ θ ta J e = ∇ θ ′
J e ∇ θ ta θ ′ .
This is the well-knownbi-level optimization popularly used in meta-learning.
Model for task-specific modulation.
After executing an action a t a task-specific modulationr ta ∈ ( − 1 , 1) is modeled as
[Equation]
where θ ta is the parameters of r ta , 1 a is the one-hot vector for action a , f : R D
→ R |A| is a MLPnetwork, and z et = z e ( s t ; θ enc ) is shared with VQ-VAE based encoding to make the modulatorefficient in training.
Experiments
The key concept of LECO is the learnable and modulated episodic count.
In other words, comparedto a naive episodic count that assigns the state novelty proportional to its visitation count withinan episode, LECO aims to produce low intrinsic rewards to task-irrelevant states regardless of itsepisodic count.
We demonstrate such behavior by comparing the mean intrinsic rewards obtained by
LECO to those from exploration with just the episodic count-based intrinsic reward in Figure 2.
Theresults show that while the episodic novelty remains high, modulation in LECO decreases the intrinsicreward such that the explorative behavior has vanished towards the end of training and the agentfocuses on the task-specific exploitation.
Apart from the novelty based on the agent’s location, in anenvironments where agent can pickup and drop objects, such actions can create task-irrelevant novelstates, further complicating the search space.
The task-specific modulation of LECO can mitigatethose task-irrelevant behaviors as shown in Figure 3.
Environments.
As LECO operates on multiple components, we design the experiments to thoroughly cover various aspects, especially the adaptiveness and the scalability.
Since the real worldas an environment is nonstationary and requires high-dimensional perception, it is crucial for anagent to efficiently explore the constantly changing environment, even at a large scale.
Thus, wefirst demonstrate the explorative performance of LECO in procedurally generated environments of
MiniGrid [7].
Then, we further extend the experiments on its performance to large scale environmentsof DMLab tasks.
Baselines.
In order to present the contribution of each component of our proposed method, wecompare LECO to its variants that incrementally add each component to a baseline without theintrinsic motivation which we denote as no-Int for the rest of the section.
Similarly, a variantwith only the vector quantization episodic counts is denoted as VQ-only .
The no-Int baseline is an IMPALA-based agent [12], with minor architectural modifications according to the environments.
Details on the base model and the architectures can be found in the supplementary material.
Otherbaselines include DSC (Down Sampled Cell)
[10], AE-LSH [31], NovelD [36], and AGAC [13], where [Figure] DSC uses a non-learnable hash for state compression and AE-LSH uses a learnable hash.
NovelDand AGAC are state-of-the-art exploration methods for procedurally generated environments.
Here,for a fair assessment of LECO we implement the episodic count state of NovelD and AGAC usingthe VQ hashing as same as LECO for all of MiniGrid and DMLab tasks.
Experimental Setup.
In MiniGrid, LECO was trained on A100x2 GPUs with a batch size of 768for 18 hours.
In DMLab, we used V100x8 GPUs with a batch size of 576 for 8 hours.
We used theunroll length T = 96 for all tasks and the same LSTM-based policy networks for LECO and allbaselines.
The hyperparameters, model architectures, and detailed training settings are provided inthe supplementary material.
4.1 MiniGrid
MiniGrid [7] is a gridworld RL environment specifically developed for testing explorationalgorithms under sparse rewards, partial observations, and procedurally generated environments.
A typical world in MiniGrid consists of MxN tiles, where each tile is assigned to be objects suchas floor, wall, lava, door, key, ball, box, and goal.
The agent deployed navigates through theworld, collecting objects and solving sub-tasks such as opening a door using a key, to reach agoal placed somewhere in the world.
Of the many types of environments included, we test theperformance of LECO in three types of MultiRoom, KeyCorridor, and ObstructedMaze environments.
More specifically, we use total of six configurations of those environments namely MiniGridMultiRoom-N6, MiniGrid-KeyCorridorS4R3, MiniGrid-KeyCorridorS6R3,
MiniGrid-ObstructedMaze-1Q, MiniGrid-ObstructedMaze-2Q , and MiniGirdObstructedMaze-Full , roughly ordered in increasing level of difficulty.
The choices weremade to reflect a wide range of difficulty including the most difficult task, while evaluating on similartasks used by the previous baselines.
We emphasize that for all tasks the baselines and LECO only use the original partial input observations,in which a 7x7x3 tensor given as the partially-observable grid cell is the input observation for boththe policy network and intrinsic module as in [6].
This setting is more challenging than the othersettings such as the partial observation for the policy and full observation for the intrinsic module asin [26, 36, 28], and the partial observation for the policy and meta information such as the agent’sposition for the intrinsic module as in [13].
We first compare LECO to its variations as shown in Figure 4.
The variations include no-Int, DSC,
AE-LSH, and VQ-only.
The results show that LECO solved all six tasks while VQ-only solvedthree, MultiRoom-N6, ObstructedMaze-1Q, ObstructedMaze-2Q , and showed slow
[Figure]
learning trend in ObstructedMaze-Full .
On the other hand, compared to DSC and AE-LSH,a similar method of using just the episodic count as intrinsic reward but basing the quantization onheuristic discretization, VQ-only outperforms DSC as DSC only solved one out of the six tasks.
While AE-LSH solved same number of tasks as VQ-only, the scores of VQ-only are higher in alltasks.
As shown by the results of no-Int, a policy without any intrinsic reward is unable to solve anyof the six tasks.
Combining all comparisons indicates that 1) episodic count enables training in sparsereward exploration tasks with procedurally generated environments, 2) vector quantization-basedcounting is more effective than a heuristic cell discretization, and 3) the task-specific modulation of LECO further improves the performances of the episodic count-based approach.
Moreover, we compare the performance of LECO with other baselines of AE-LSH, AGAC, and
NovelD on the same six MiniGrid tasks.
As shown in Figure 5, only LECO solves all six tasksincluding the ObstructedMaze-Full.
Here, it is noted that the original implementations of AGAC and NovelD used the agent’s position and the full observations, respectively, for the episodic counts intheir methods while we used the original partial observations in our implementations of AGAC and
NovelD. To the best of our knowledge, LECO is the first to achieve average score of over 0 . 75 on the ObstructedMaze-Full with completely partial observation setting within 300M steps.
Noisy-TV on MiniGrid. Provided by [26] is a variation of MultiRoom task that brings the concept of Noisy-TV problem into the MiniGrid environment setting, named MiniGridMiniGridMultiRoomNoisyTV-N7-S4 . In this version of the task, the color of the ball changes to a randomcolor when the agent performs a specific action, creating a visual novelty irrelevant to the objectiveof the task, which is to reach the goal [26].
In Figure 6, we present the learning curves of LECO andbaselines on this Noisy-TV environment. The result shows that all methods that use the episodicintrinsic reward, including LECO, are able to successfully solve the noisy environments.
4.2 DMLab
DMLab [3] is a learning environment consisting of various tasks, each involving different visuals andobjectives in a simulated 3D world.
We consider DMLab for its partial-observability that naturallyarises from first-person perspective and its complexity from image-based observation with finedetails and randomly generated levels to demonstrate the scalability of LECO.
We compare LECOto no-Int, episodic count methods based on AE-LSH, DSC and VQ, and NovelD on two individualexploration-heavy tasks.
We select two specific tasks from DMLab, lasertag_three_opponents_small andlasertag_three_opponents_large , in order to show LECO’s capability to solve explo-
[Figure]
ration tasks at a large scale.
These tasks are known to be difficult to solve without advancedexploration strategy and are characterized as sparse reward problems.
The small and large standfor the size of the map, which determines the reward sparsity.
In Figure 7, we present the learningcurves of LECO and baselines on these tasks.
Each task will have its own independent agent, but thearchitectures and hyperparameters remain consistent across these tasks.
As shown in Figure 7, LECOoutperforms all of the baselines in the selected DMLab tasks.
Conclusion
[Figure] This paper proposes a learnable episodic count to produce task-specific intrinsic rewards for hardexploration problems including high-dimensional state spaces and procedurally generated environments.
In specific, the hash code based on VQ-VAE is used for computing the episodic state countand the corresponding state novelty, and this episodic novelty is regulated by the task-specific modulator that is trained to maximize the extrinsic rewards.
Experimental results demonstrate that theproposed task-specific exploration based on the episodic count significantly outperforms the previousstate-of-the-art exploration methods on the most difficult tasks of MiniGrid and DMLab.
However,the proposed intrinsic reward also has a number of limitations in solving tasks that require intensiveleveraging of the inter-episode novelty and in dealing with somewhat difficult bi-level optimization.
For future works, we will extend our work to combine the inter-episode novelty and devise a moreefficient training of task-specific intrinsic reward to further improve the scalability.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We have discussed the limitations
and corresponding future works in the conclusion section.
(c) Did you discuss any potential negative societal impacts of your work?
[No] This paper
will not create any negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We will releasethe code and data upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] More details including hyperparameters are included in appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We include the standard deviation of the results withrespect to multiple runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We used only the benchmark simulators.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] There is no personal data in our experiments.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]