Exploration via Elliptical Episodic Bonuses
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In recent years, a number of reinforcement learning (RL) methods have been proposed to explore complex environments which differ across episodes.
In this work,we show that the effectiveness of these methods critically relies on a count-basedepisodic term in their exploration bonus.
As a result, despite their success inrelatively simple, noise-free settings, these methods fall short in more realisticscenarios where the state space is vast and prone to noise.
To address this limitation, we introduce E xploration via E lliptical E pisodic B onuses ( E3B ), a newmethod which extends count-based episodic bonuses to continuous state spaces andencourages an agent to explore states that are diverse under a learned embeddingwithin each episode.
The embedding is learned using an inverse dynamics modelin order to capture controllable aspects of the environment.
Our method sets anew state-of-the-art across 22 challenging tasks from the MiniHack suite, withoutrequiring task-speciﬁc inductive biases.
E3B also matches current state-of-the-artmethods on hard exploration tasks in VizDoom, demonstrating that it can scale tohigh-dimensional pixel-based observations.
Introduction
Exploration in environments with sparse rewards is a fundamental challenge in reinforcement learning(RL).
In the tabular setting, provably optimal algorithms have existed since the early 2000s [35,10].
More recently, exploration has been studied in the context of deep RL and a number ofempirically successful methods have been proposed, such as pseudocounts [9], intrinsic curiositymodules (ICM)[51] and random network distillation (RND) [13].
These methods rely on intrinsicallygenerated exploration bonuses that reward the agent for visiting states that are novel according tosome measure.
Different measures of novelty have been proposed, such as the likelihood of a stateunder a learned density model, the error of a forward dynamics model, or the loss on a randomprediction task.
These approaches have proven effective on hard exploration problems, as exempliﬁedby the Atari games Montezuma’s Revenge and PitFall [22].
The approaches above are, however, designed for singleton RL tasks, where the agent is spawnedin the same environment in every episode.
Recently, several studies have drawn attention to thefact that RL agents exhibit poor generalization across environments, and that even minor changesto the environment can lead to substantial degradation in performance [72, 34, 71, 18, 37].
Thishas motivated the creation of benchmarks in the Contextual Markov Decision Process (CMDP)framework, where different episodes correspond to different environments that nevertheless sharecertain characteristics.
Examples of CMDPs include procedurally generated (PCG) environments[15, 56, 40, 33, 17, 7, 29, 52] or embodied AI tasks where the agent must generalize its behavior tounseen physical spaces at test time [57, 59, 28, 69].
A number of methods have been proposed whichhave shown promising performance in PCG environments with sparse rewards, such as RIDE [55],
AGAC [25] and NovelD [73].
These methods propose different intrinsic reward functions, such as the change representation in a latent space, the divergence between the predictions of a policy and an adversary, or the difference between random network prediction errors at two consecutive states.
Although not presented as a central algorithmic feature, these methods also include a count-basedbonus which is computed at the episode level.
In this work, we take a closer look at exploration in CMDPs, where each episode corresponds to adifferent environment context.
We ﬁrst show that, surprisingly, the count-based episodic bonus that isoften included as a heuristic is in fact essential for good performance, and current methods fail if it isomitted.
Furthermore, due to this dependence on a count-based term, existing methods fail on morecomplex tasks with irrelevant features or dynamic entities, where each observation is rarely seenmore than once.
We ﬁnd that performance can be improved by counting certain features extractedfrom the observations, rather than the observations themselves.
However, different features are usefulfor different tasks, making it difﬁcult to design a feature extractor that performs well across all tasks.
To address this fundamental limitation, we propose a new method, E3B, which uses an ellipticalepisodic bonus that can be seen as a natural generalization of a count-based episodic bonus tocontinuous state spaces, and that is paired with a self-supervised feature learning method using aninverse dynamics model.
Our algorithm is simple to implement, scalable to large or inﬁnite statespaces, and achieves state-of-the-art performance across 22 challenging tasks from the MiniHacksuite [56], without the need for task-speciﬁc prior knowledge.
It also matches existing methods onhard exploration tasks from the VizDoom environment [36], demonstrating that it scales well to highdimensional pixel-based observations.
Our code is available at web-link .
Background2.1 Contextual MDPs
We consider a Contextual Markov Decision Process 1 (CMDP [30]) given by M =( S , A , C , P, r, µ C , µ S ) where S is the state space, A is the action space, C is the context space, 61 P is the transition function, r is the reward function, µ C is the distribution over contexts and µ S isthe conditional initial state distribution.
At each episode, we sample a context c ⇠ µ C , an initial states 0 ⇠ µ ( ·| c ) , and subsequent states in the episode are sampled according to s t +1 ⇠ P ( ·| s t , a t , c ) .
Let 64 d c ⇡ denote the distribution over states induced by following policy ⇡ in context c .
The goal is to learn apolicy ⇡ which maximizes the expected return over all contexts, i.e. R = E c ⇠ µ C ,s ⇠ d c ⇡ ,a ⇠ ⇡ ( s ) [ r ( s, a )] .
66 Examples of CMDPs include procedurally-generated environments, such as ProcGen [17], MiniGrid[15], NetHack [40], or MiniHack [56], where each context c corresponds to the random seed usedto generate the environment; in this case, the number of contexts |C| is effectively inﬁnite.
Other 69 examples include embodied AI environments [57, 67, 28, 59, 69], where the agent is placed indifferent simulated houses and must navigate to a location or ﬁnd an object.
In this setting, eachcontext c 2 C represents a house identiﬁer and the number of houses |C| is typically between 20 and 72 500 .
For an in-depth review of the literature on CMDPs and generalization in RL, see [38].
2.2 Exploration Bonuses
If the environment rewards are sparse, learning a policy using simple ✏ -greedy exploration may requireintractably many samples.
We therefore consider methods that augment the external reward functionr with an intrinsic reward bonus b .
A number of intrinsic bonuses that encourage exploration insingleton (non-contextual) MDPs have been proposed, including pseudocounts [9], intrinsic curiositymodules (ICM) [51] and random network distillation (RND) error [13].
At a high level, these methodsdeﬁne an intrinsic reward bonus that is high if the current state is different from the previous statesvisited by the agent, and low if it is similar (according to some measure).
More recently, several methods have been proposed for and evaluated on procedurally-generated
MDPs.
All use different exploration bonuses, which are summarized in Table 1.
RIDE deﬁnes abonus based on the distance between the embeddings of two consecutive observations, NovelD usesa bonus based on the difference between two consecutive RND bonuses, and AGAC uses the KLdivergence between the predictions of the agent’s policy and those of an adversarial policy trained tomimic it (see the original works for details). In addition, all three methods have a term in their bonus
Method Exploration bonus
[Table]
[Table]
[Table]
/ p N
[Table]
( s
[Table] [Equation] [Table] (marked in blue) which depends on N e ( s t +1 ) – the number of times s t +1 has been encounteredduring the current episode .
Although presented as a heuristic, below we will show that without thiscount-based episodic term, all three methods fail to learn.
This in turn limits their effectiveness inmore complex, dynamic, and noisy environments.
Importance and Limitations of Count-Based Episodic Bonuses
We now discuss in more detail the importance and limitations of the count-based episodic bonusesused in RIDE, AGAC and NovelD, which depend on N e ( s t ) .
Figure 1 shows results for the threemethods with and without their respective count-based episodic terms, on one of the MiniGridenvironments used in prior work.
When the count-based terms are removed, all three methods fail tolearn.
Similar trends apply for other MiniGrid environments (see Appendix D.1).
This shows that theepisodic bonus is in fact essential for good performance.
deviation.
However, the count-based episodic bonus suffers from a fundamental limitation, which is similar to that faced by count-based approaches ingeneral: if each state is unique, then N e ( s t ) willalways be 1 and the episodic bonus is no longermeaningful.
This is the case for many real-worldapplications.
For example, a household robot’sstate as recorded by its camera might includemoving trees outside the window, clocks showing the time or images on a television screenwhich are not relevant for its tasks, but nevertheless make each state unique.
Previous works [55, 25, 73] have used the MiniGrid test suite [15] for evaluation, where observations are less noisy and do not typicallycontain irrelevant information.
Thus, methods relying on episodic counts have been effective in thesescenarios.
However, in more complex environments such as MiniHack [56] or with high-dimensionalpixel-based observations, episodic count-based approaches can cease to be viable.
[Figure] [Figure]
A possible alternative could be to design a function to extract relevant features from each stateand feed them to the count-based episodic bonus.
Speciﬁcally, instead of deﬁning a bonus using
N e ( s t ) , we could deﬁne the bonus using N e (  ( s t )) , where  is a hand-designed feature extractor.
For example, in the paper introducing the MiniHack suite [56], the RIDE implementation uses
( s t ) =
( x t , y t ) , where ( x t , y t ) is the spatial location of the agent at time t .
However, this approachrelies heavily on task-speciﬁc knowledge.
Figure 2 shows results on two tasks from the MiniHack suite for three NovelD variants.
The ﬁrst, N OVEL D, denotes the standard formulation which uses the bonus I [ N e ( s t ) = 1] .
The second, N OVEL D- POSITION , uses the positional feature encoding described above, i.e. the episodic bonus isdeﬁned as I [ N e (  ( s t ))
= 1] with  ( s t ) =
( x t , y t ) .
The third, N OVEL D- MESSAGE , uses a featureencoding where  ( s t ) extracts the message portion of the state s t similarly to [46] (both encodingsare explained in more detail in Section 5).
In contrast to the MiniGrid environments, here standard N OVEL D fails completely due to the presence of a time counter feature in the MiniHack observations,which makes each observation in the episode unique.
Using the positional encoding enables N OVEL DPOSITION to solve the LavaCross-S19-N13 task, but this method fails on the Levitate-Potiontask.
On the other hand, using the message encoding enables N OVEL D- MESSAGE to succeed onthe Levitate-Potion task, but it fails on the LavaCrossing-S19-N13 one.
This illustrates thatdifferent feature extractors are effective on different tasks, and that designing one which is broadlyeffective is challenging.
Therefore, robust new methods which do not require task-speciﬁc engineeringare needed.
Elliptical Episodic Bonuses
In this section we describe Exploration via Elliptical Episodic Bonuses , ( E3B ), our algorithm forexploration in contextual MDPs.
It is designed to address the shortcomings of count-based episodicbonuses described above, with two aims in mind.
First, we would like an episodic bonus that canbe used with continuous state representations, unlike the count-based bonus which requires discretestates.
Second, we would like a representation learning method that only captures information aboutthe environment that is relevant for the task at hand.
The ﬁrst requirement is met by using an elliptical bonus, which provides a continuous analog to the count-based bonus, while the second requirementis met by using a representation learned with an inverse dynamics model [51, 55].
A summary of the method is shown in Figure 3.
We deﬁne an intrinsic reward based on the positionof the current state’s embedding with respect to an ellipse ﬁt on the embeddings of previous statesencountered within the same episode.
This bonus is then combined with the environment reward andused to update the agent’s policy.
The elliptical bonus and the embedding method are described inthe next two sections, and the full algorithm is detailed in Appendix B.
4.1 Elliptical Episodic Bonus
Given a feature encoding  , at each time step t in the episode the elliptical bonus b is deﬁned asfollows:
[Equation]
Here  I is a regularization term to ensure that the matrix C t  1 is non-singular, where  is a scalarcoefﬁcient and I is the identity matrix.
Intuition.
One perspective is that the elliptical bonus is a natural generalization of a count-basedepisodic bonus [2].
To see this, observe that if the problem is tabular and  is a one-hot encodingof the state, then C t  1 will be a diagonal matrix whose entries contain the counts correspondingto each state encountered in the episode.
Its inverse C  1 t  1 will also be a diagonal matrix whoseentries are inverse state visitation counts, and the bilinear form  ( s t ) > C  1 t  1  ( s t ) reads off the entrycorresponding to the current state s t , yielding a bonus of 1 /N e ( s t ) .
For a more general geometric interpretation, if ( s 0 ) , ...,  ( s t  1 ) are roughly centered at zero, then C t  1can be viewed as their unnormalized covariance matrix.
Now consider the eigendecomposition C t  1 =
U >
⇤ U ,where ⇤ is the diagonal matrix whose entries are the eigenvalues  1 , ...,  n (these are real since C t  1 is symmetric).
Letting z
= U  ( s t ) =
( z 1 , ..., z n ) be the set of coordinates of  ( s t ) in the eigenspace of C t  1 , we can rewritethe elliptical bonus as:
[Equation]
The bonus increases the more  ( s t ) is aligned with theeigenvectors corresponding to smaller eigenvalues of C t  1 172 (directions of low data density), and decreases the more itis aligned with eigenvectors corresponding to larger eigenvalues (directions of high data density).
An illustrationfor n = 2 is shown in Figure 4.
In our experiments, n istypically between 256 and 1024 .
The elliptical bonus is related to the Mahalanobis distance [43] which uses a similar bilinear form.
However, the Mahalanobis distance would normalize the matrix C t  1 in equation 4.1 by the numberof observations t  1 in the episode, whereas the elliptical bonus does not.
The elliptical bonus thustends to decrease with the number of observations, similarly to the count-based bonus.
Efﬁcient Computation.
The matrix C t  1 needs to be inverted at each step t in the episode, anoperation which is cubic in the dimension of  and may thus be expensive.
To address this, we usethe Sherman-Morrison matrix identity [60] to perform fast rank- 1 updates in quadratic time:
[Equation]
This results in an approximately 3 ⇥ speedup over naive matrix inversion (details in Appendix D.2).
4.2 Learned Feature Encoder
[Figure]
Any feature learning method could in principle be used to learn  .
Here we use the inverse dynamicsmodel approach proposed in [51], which trains a model g along with  to map each pair of consecutiveembeddings  ( s t ) ,  ( s t +1 ) to a distribution over actions a t linking them.
In our setup,  is separatefrom the policy network.
The g model is trained jointly with  using the following per-sample loss:
[Equation]
The motivation is that the mapping  will discard information about the environment which is notuseful for predicting the agent’s actions.
Previous work [51] has shown that this can make learningmore robust to random noise or other parts of the state which are not controllable by the agent.
In ourexperiments, we compare this to other approaches such as using the policy network [5] or randomnetworks [11, 2] to produce state embeddings.
Experiments
[Figure]
5.1 MiniHack Suite
[Figure]
In order to probe the capabilities of existing methods andevaluate E3B, we seek CMDP environments which exhibit challenges associated with realistic scenarios, suchas sparse rewards, noisy or irrelevant features, and largestate spaces.
We opted for the procedurally generatedtasks from the MiniHack suite [56], which is itself basedon the NetHack Learning Environment [40].
NetHack isa notoriously challenging roguelike video game wherethe agent must navigate through procedurally generateddungeons to recover a magical amulet.
The MiniHacktasks contain numerous challenges such as ﬁnding andusing magical objects, navigating through levels whileavoiding lava, and ﬁghting monsters.
Furthermore, rewards are sparse and as detailed below, the state representation contains a large amount of information, onlysome of which is relevant for a given task.
For all our experiments we use the Torchbeast [39] implementation of IMPALA [23] as our base RL algorithm.
For certain skill-based tasks, we restricted the actionspace to only the actions which are useful for solvingthe task at hand, since we found that none of the methodswere able to make progress with the full action space(see Appendix C.1.4 for more discussion).
Full experimental details can be found in Appendix C.
5.1.1 Modalities for Episodic Bonus
The MiniHack environments provide an observation at each step that includes three different modalities: i) a symbolic image, which indicates the location of the agent, monsters or other entities, objectsand different types of terrain (such as walls, lava, water); ii) a statistics vector which indicates theagent’s current ( x, y ) location, hit points, time step t , and other features such as strength, dexterity andconstitution, and iii) a textual message which gives different types of feedback about the environment("you see a key of master thievery", "the wall is solid rock").
These are illustrated in Figure 5.
We now draw attention to some important subtleties regarding how existing methods implementcount-based episodic bonuses—as we will see, these can have a large impact on performance.
Theworks of [55, 25, 73], which use the MiniGrid suite [15] for evaluation, use a hash table whose keysare the full observations (in this case symbolic images) encountered by the agent.
The occurrences ofthese observations are then counted to compute the episodic bonus.
The baselines run in the MiniHacktasks [56], on the other hand, use a hash table whose keys are the ( x, y ) positions of the agent, whichare extracted from the observation and are then similarly counted to compute the episodic bonus. [Figure]
Which of these is preferable?
On one hand, using the full observation does not discard any potentiallyuseful information, but if there are irrelevant features in the observations, similar states might becounted as separate, which can make the count-based episodic bonus meaningless.
On the other hand,using the ( x, y ) positions might be useful for navigation-based tasks, and can discard distractinginformation, but might discard useful information for tasks which are not navigation based.
In order to understand the effect of these different input modalities on count-based episodic bonuses,we consider three variants of the NovelD algorithm, in addition to the original formulation.
Thesemethods all have a count-based episodic bonus of the form I [ N e (  ( s t ))
= 1] , with different choicesof  detailed below.
We decided to focus our study on variants of NovelD, since it was previouslyshown to outperform competing methods on MiniGrid [73]. Furthermore, in preliminary experimentswe were not able to get analogous variants of RIDE to yield good results (see Appendix D.3).
N OVEL D: in the original version of the algorithm,  is the identity.
Here the input to the count-basedepisodic bonus is the full state, namely the concatenation of the symbol image, the message and thestats vector.
This makes no assumptions about which part of the state is most useful.
N OVEL D- POSITION : in this variant,  extracts the ( x, y ) position of the agent from the statisticsvector, which reﬂects a strong inductive bias that the task is navigation-based.
N OVEL D- IMAGE : in this variant,  extracts the symbol image only.
This reﬂects an inductive biasthat the message and the stats vector are not useful since they are discarded.
N OVEL D-
MESSAGE : in this variant,  extracts the message only.
This reﬂects an inductive bias thatthe messages are important but the stats and symbolic images are not.
For E3B, we feed the full state to the algorithm and do not make any assumptions about which part isuseful for the task at hand.
Despite the lack of task-speciﬁc inductive biases, our method is still able to extract the relevant features for each task, thus outperforming the other exploration approaches (ormatching their performance when prior knowledge of the task is used).
5.1.2 Results on MiniHack
Aggregate results for IMPALA, RND, RIDE, ICM, NovelD (with the three variants described above)and
E3B over 22 sparse reward tasks from the MiniHack suite are shown in Figure 6.
Out of 22environments, 11 are based on the MiniGrid suite, but use the MiniHack interface and observationspace (environment details are included in Appendix C.1.3).
We used the performance metrics andbootstrapping protocol from [4] to compute conﬁdence intervals, which are more informative thansimple point estimates.
We see that over all tasks (top row), E3B outperforms all other methods by astatistically signiﬁcant margin across all three performance metrics.
Standard N OVEL D performs poorly due to the fact that the MiniHack observations contain a timecounter in the statistics vector (see Figure 5), which makes each observation in the episode uniqueand hence renders the count-based episodic bonus meaningless.
The three NovelD variants which usedifferent features extracted from the state for the count-based bonus perform better.
Figure 6 showsaggregate performance across a subset of 13 navigation-based tasks (middle row) and 9 skill-basedtasks (bottom row) (see Appendix C.1.3 for the task breakdown).
On the navigation-based tasks,
N OVEL D- POSITION has excellent performance, since visiting a large number of different ( x, y )locations is closely aligned with the true task reward.
However, N OVEL D- POSITION fails on allthe skill-based tasks, resulting in poor performance overall.
We observe an opposite trend for the
N OVEL D- MESSAGE variant, which performs fairly well on the skill-based tasks, but poorly on thenavigation-based ones.
This highlights that although certain inductive biases can help for certain tasks, it is difﬁcult toﬁnd one which performs well across all of them.
Our method, E3B, performs well on both thenavigation-based tasks and the skill-based tasks, resulting in superior performance overall.
It is worthnoting that E3B does not require any task-speciﬁc engineering: the exact same algorithm is run forall tasks, and it uses the unprocessed states.
Results for individual tasks, along with more analysisand discussion, can be found in Appendix D.4.
5.1.3 Ablation Experiments
We next report results for ablation experiments measuring the effects of different algorithmiccomponents of E3B, namely the feature encoding  and the episodic nature of the bonus.
Results are shown in Figure 7.
random indicates E3B where  is a randomly initialized network which is kept ﬁxed throughouttraining.
E3B (policy enc.) indicates E3B wherethe weights of  are tied to those of the policynetwork (with the last layer producing actionprobabilities removed).
E3B (non-episodic) indicates E3B where the elliptical bonus is computed using observations across all timesteps inthe lifetime of the agent, and not just the current episode.
All three variants perform signiﬁcantly worse than E3B, which uses an inverse dynamics model encoding for  and an episodic bonus.
This highlights that both the inverse dynamics model and episodic bonus are important for success.
5.2 Pixel-Based VizDoom
As our second evaluation testbed, we used the sparse reward, pixel-based VizDoom [36] environmentsused in prior work [51, 55].
Although these are singleton MDPs, they still constitute challengingexploration problems and probe whether our method scales to continuous high-dimensional pixelbased observations.
Results comparing E3B to RIDE, ICM and IMPALA on three versions of the taskare shown in Figure 8 (hyperparameters can be found in Appendix C.2).
IMPALA succeeds on thedense
Related Work
Exploration in RL.
Exploration remains a long-standing problem in RL.
Common approachesinclude ✏ -greedy [66], count-based exploration [64, 8, 47, 44, 68, 42], curiosity-based exploration [58,62, 63, 11], and other types of intrinsic motivation [48, 49, 61, 1, 12].
These methods were largelydesigned for singleton MDPs, where the environment remains the same across episodes.
As a result,they measure novelty over the agent’s lifetime in a static environment rather than on an episodic basisunder a distribution of diverse contexts, as necessary in CMDPs.
Other intrinsic motivation methodshave recently been developed for exploration in CMDPs [55, 70, 25, 73].
As our experiments show,they critically rely on episodic count-based bonuses, which E3B generalizes.
Another class of methods automatically generate curricula over variations of the CMDP to encourageefﬁcient learning, effectively performing a form of curiosity-driven exploration in the context space.
Goal-conditioned automatic curriculum methods provide the goal subset of the context to the agent[27, 26, 24, 54, 19, 14, 22], while goal-free variants can explore contexts without a priori disclosingany knowledge about the context [65, 53, 32, 21].
Unlike our method, these methods assume theability to actively conﬁgure the environment context.
In principle, E3B can be combined with theseapproaches, whereby E3B explores at an episodic level, while the automatic curriculum methodexplores at a context level, making our method complementary to this body of work.
Elliptical Bonuses.
The use of elliptical bonuses has a long history in the contextual bandit literature[6, 20, 41], which corresponds to the RL setting with a single time step.
More recently, severalworks have begun to explore the use of elliptical bonuses in the context of multi-step RL.
The PC-PGalgorithm [2] considers MDPs with linear dynamics and uses an elliptical bonus based on a policycover to explore in a provably efﬁcient manner.
The ACB algorithm [5] also uses an elliptical bonus,approximated using linear regressors trained on random noise, while FLAMBE [3] uses an ellipticalbonus inside a learned dynamics model.
All of these algorithms operate on singleton MDPs, whereasours is designed for contextual MDPs and constructs elliptical bonuses at the episode level rather thanacross episodes.
We also use different feature learning methods, namely inverse dynamics models,instead of the policy encoders, random networks or kernel methods used in the aforementioned works.
Conclusion
In this work, we identiﬁed a fundamental limitation of existing methods for exploration in CMDPs:their performance relies heavily on an episodic count-based term, which is not meaningful wheneach state is unique.
This is a common scenario in realistic applications, and it is difﬁcult to alleviatethrough feature engineering.
To remedy this limitation, we introduce a new method, E3B, whichextends episodic count-based methods to continuous state spaces using an elliptical episodic bonus, aswell as an inverse dynamics model to automatically extract useful features from states.
E3B achievesa new state-of-the-art on a wide range of complex tasks from the MiniHack suite, without the need forfeature engineering.
Our approach also scales to high-dimensional pixel-based environments such as VizDoom, matching the results of other top exploration methods.
Future research directions includeapplying E3B to complex embodied AI tasks, experimenting with more advanced feature learningmethods, and investigating ways to integrate within-episode and across-episode novelty bonuses.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We discuss how our method (as
well as existing methods) are not able to solve MiniHack tasks with large action spaces
in Section 5 as well as Appendix C.1.4.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix A.(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We included
the code with instructions in the supplement, and will release the code on GitHub upon
acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix C.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We use the rliable library to compute error bars using
stratiﬁed bootstrapping in Figure 6.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.1.5.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited all the
codebases we built upon, see Section C.3.
(b) Did you mention the license of the assets?
[Yes] We mentioned all licenses associated with the codebases we used, see Section C.3.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] All of our new code is included in our code release.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]