Exploration via Elliptical Episodic Bonuses
Abstract
In recent years, a number of reinforcement learning (RL) methods have been proposed to explore complex environments which differ across episodes.
In this work, we show that the effectiveness of these methods critically relies on a count-based episodic term in their exploration bonus.
As a result, despite their success in relatively simple, noise-free settings, these methods fall short in more realistic scenarios where the state space is vast and prone to noise.
To address this limitation, we introduce E xploration via E lliptical E pisodic B onuses ( E3B ), a new method which extends count-based episodic bonuses to continuous state spaces and encourages an agent to explore states that are diverse under a learned embedding within each episode.
The embedding is learned using an inverse dynamics model in order to capture controllable aspects of the environment.
Our method sets a new state-of-the-art across 16 challenging tasks from the MiniHack suite, without requiring task-speciﬁc inductive biases.
E3B also matches existing methods on sparse reward, pixel-based Vizdoom environments, and outperforms existing methods in reward-free exploration on Habitat, demonstrating that it can scale to high-dimensional pixel-based observations and realistic environments.
Introduction
Exploration in environments with sparse rewards is a fundamental challenge in reinforcement learning (RL).
In the tabular setting, provably optimal algorithms have existed since the early 2000s [36, 10].
More recently, exploration has been studied in the context of deep RL, and a number of empirically successful methods have been proposed, such as pseudocounts [9], intrinsic curiosity modules (ICM) [52] and random network distillation (RND) [13].
These methods rely on intrinsically generated exploration bonuses that reward the agent for visiting states that are novel according to some measure.
Different measures of novelty have been proposed, such as the likelihood of a state under a learned density model, the error of a forward dynamics model, or the loss on a random prediction task.
These approaches have proven effective on hard exploration problems, as exempliﬁed by the Atari games Montezuma’s Revenge and PitFall [22].
The approaches above are, however, designed for singleton RL tasks, where the agent is spawned in the same environment in every episode.
Recently, several studies have drawn attention to the fact that RL agents exhibit poor generalization across environments, and that even minor changes to the environment can lead to substantial degradation in performance [75, 35, 74, 18, 38].
This has motivated the creation of benchmarks in the Contextual Markov Decision Process (CMDP) framework, where different episodes correspond to different environments that nevertheless share certain characteristics.
Examples of CMDPs include procedurally generated (PCG) environments 36th Conference on Neural Information Processing Systems (NeurIPS 2022). [15, 58, 41, 34, 17, 7, 29, 53] or embodied AI tasks where the agent must generalize its behavior to unseen physical spaces at test time [59, 61, 28, 72].
A number of methods have been proposed which have shown promising performance in PCG environments with sparse rewards, such as RIDE [56],
AGAC [25] and NovelD [76].
These methods propose different intrinsic reward functions, such as the change in representation in a latent space, the divergence between the predictions of a policy and an adversary, or the difference between random network prediction errors at two consecutive states.
Although not presented as a central algorithmic feature, these methods also include a count-based bonus which is computed at the episode level.
In this work, we take a closer look at exploration in CMDPs, where each episode corresponds to a different environment context.
We ﬁrst show that, surprisingly, the count-based episodic bonus that is often included as a heuristic is in fact essential for good performance, and current methods fail if it is omitted.
Furthermore, due to this dependence on a count-based term, existing methods fail on more complex tasks with irrelevant features or dynamic entities, where each observation is rarely seen more than once.
We ﬁnd that performance can be improved by counting certain features extracted from the observations, rather than the observations themselves.
However, different features are useful for different tasks, making it difﬁcult to design a feature extractor that performs well across all tasks.
To address this fundamental limitation, we propose a new method, E3B, which uses an elliptical bonus [6, 20, 42] at the episode level that can be seen as a natural generalization of a count-based episodic bonus to continuous state spaces, and that is paired with a self-supervised feature learning method using an inverse dynamics model.
Our algorithm is simple to implement, scalable to large or inﬁnite state spaces, and achieves state-of-the-art performance across 16 challenging tasks from the MiniHack suite [58], without the need for task-speciﬁc prior knowledge.
It also matches existing methods on hard exploration tasks from the VizDoom environment [37], and signiﬁcantly outperforms existing methods in reward-free exploration on the Habitat embodied AI environment [59, 69]. This demonstrates that E3B scales to rich, high dimensional pixel-based observations and real-world scenarios.
Our code is available at https://github.com/facebookresearch/e3b .
Background 2.1 Contextual MDPs
We consider a Contextual Markov Decision Process 1 (CMDP [30]) given by M = ( S , A , C , P, r, µ C , µ S ) where S is the state space, A is the action space, C is the context space, P is the transition function, r is the reward function, µ C is the distribution over contexts and µ S is the conditional initial state distribution.
At each episode, we sample a context c ⇠ µ C , an initial {+state s 
+} ⇠ µ ( ·| c ) , and subsequent states in the episode are sampled according to s t +1 ⇠ P ( ·| s t , a t , c ) .
Let d c ⇡ denote the distribution over states induced by following policy ⇡ in context c .
The goal is to learn a policy ⇡ which maximizes the expected return over all contexts, i.e. R = E c ⇠ µ C ,s ⇠ d c ⇡ ,a ⇠ ⇡ ( s ) [ r ( s, a )] .
Examples of CMDPs include procedurally-generated environments, such as ProcGen [17], MiniGrid [15], NetHack [41], or MiniHack [58], where each context c corresponds to the random seed used to generate the environment; in this case, the number of contexts |C| is effectively inﬁnite.
Other examples include embodied AI environments [59, 69, 28, 61, 72], where the agent is placed in different simulated houses and must navigate to a location or ﬁnd an object.
In this setting, each context c 2 C represents a house identiﬁer and the number of houses |C| is typically between 20 and 1000 .
For an in-depth review of the literature on CMDPs and generalization in RL, see [39].
2.2 Exploration Bonuses
If the environment rewards are sparse, learning a policy using simple ✏ -greedy exploration may require intractably many samples.
We therefore consider methods that augment the external reward function r with an intrinsic reward bonus b .
A number of intrinsic bonuses that encourage exploration in singleton (non-contextual) MDPs have been proposed, including pseudocounts [9], intrinsic curiosity modules (ICM) [52] and random network distillation (RND) error [13].
At a high level, these methods deﬁne an intrinsic reward bonus that is high if the current state is different from the previous states visited by the agent, and low if it is similar (according to some measure). [Table] [Equation]
More recently, several methods have been proposed for and evaluated on procedurally-generated
MDPs.
All use different exploration bonuses, which are summarized in Table 1.
RIDE [56] deﬁnes a bonus based on the distance between the embeddings of two consecutive observations, NovelD [76] uses a bonus based on the difference between two consecutive RND bonuses, and AGAC [25] uses the KL divergence between the predictions of the agent’s policy and those of an adversarial policy trained to mimic it (see the original works for details).
In addition, all three methods have a term in their bonus (marked in blue) which depends on N e ( s t +1 ) – the number of times s t +1 has been encountered during the current episode .
Although presented as a heuristic, below we will show that without this count-based episodic term, all three methods fail to learn.
This in turn limits their effectiveness in more complex, dynamic, and noisy environments.
Importance and Limitations of Count-Based Episodic Bonuses
We now discuss in more detail the importance and limitations of the count-based episodic bonuses used in RIDE, AGAC and NovelD, which depend on N e ( s t ) .
Figure 1 shows results for the three methods with and without their respective count-based episodic terms, on one of the MiniGrid environments used in prior work.
When the count-based terms are removed, all three methods fail to learn.
Similar trends apply for other MiniGrid environments (see Appendix D.1).
This shows that the episodic bonus is in fact essential for good performance.
However, the count-based episodic bonus suffers from a fundamental limitation, which is similar to that faced by count-based approaches in general: if each state is unique, then N e ( s t ) will always beand the episodic bonus is no longer meaningful.
This is the case for many real-world applications.
For example, a household robot’s state as recorded by its camera might include moving trees outside the window, clocks showing the time or images on a television screen which are not relevant for its tasks, but nevertheless make each state unique.
Previous works [56, 25, 76] have used the MiniGrid test suite [15] for evaluation, where observations are less noisy and do not typically contain irrelevant information.
Thus, methods relying on episodic counts have been effective in these scenarios.
However, in more complex environments such as MiniHack [58] or with high-dimensional pixel-based observations, episodic count-based approaches can cease to be viable.
A possible alternative could be to design a function to extract relevant features from each state and feed them to the count-based episodic bonus.
Speciﬁcally, instead of deﬁning a bonus using
N e ( s t ) , we could deﬁne the bonus using N e (  ( s t )) , where  is a hand-designed feature extractor.
For example, in the paper introducing the MiniHack suite [58], the RIDE implementation uses  ( s t ) =
( x t , y t ) , where ( x t , y t ) is the spatial location of the agent at time t .
However, this approach relies heavily on task-speciﬁc knowledge.
Figure 2 shows results on two tasks from the MiniHack suite for three NovelD variants (we decided to focus our study on variants of NovelD, since it was previously shown to outperform competing methods on MiniGrid [76]).
The ﬁrst, N OVEL D, denotes the standard formulation which uses the bonus I [ N e ( s t ) = 1] .
The second, N OVEL D- POSITION , uses the positional feature encoding described above, i.e. the episodic bonus is deﬁned as I [ N e (  ( s t ))
= 1] with  ( s t ) =
( x t , y t ) .
The third, N OVEL D- MESSAGE , uses a feature encoding where  ( s t ) extracts the message portion of the state s t similarly to [47] (both encodings are explained in more detail in Section 5).
In contrast to the MiniGrid environments, here standard N OVEL D fails completely due to the presence of a time counter feature in the MiniHack observations, which makes each observation in the episode unique.
Using the positional encoding enables N OVEL D- POSITION to solve the MultiRoom task, but this method fails on the Freeze task.
On the other hand, using the message encoding enables N OVEL D- MESSAGE to succeed on the Freeze task, but it fails on the MultiRoom one.
This illustrates that different feature extractors are effective on different tasks, and that designing one which is broadly effective is challenging.
Therefore, robust new methods which do not require task-speciﬁc engineering are needed.
Elliptical Episodic Bonuses
In this section we describe Exploration via Elliptical Episodic Bonuses , ( E3B ), our algorithm for exploration in contextual MDPs.
It is designed to address the shortcomings of count-based episodic bonuses described above, with two aims in mind.
First, we would like an episodic bonus that can be used with continuous state representations, unlike the count-based bonus which requires discrete states.
Second, we would like a representation learning method that only captures information about the environment that is relevant for the task at hand.
The ﬁrst requirement is met by using an elliptical bonus [6, 20, 42], which provides a continuous analog to the count-based bonus, while the second requirement is met by using a representation learned with an inverse dynamics model [52, 56].
A summary of the method is shown in Figure 3.
We deﬁne an intrinsic reward based on the position of the current state’s embedding with respect to an ellipse ﬁt on the embeddings of previous states encountered within the same episode.
This bonus is then combined with the environment reward and used to update the agent’s policy.
The next two sections describe the elliptical bonus and embedding method in detail.
4.1 Elliptical Episodic Bonus
Given a feature encoding  , at each time step t in the episode the elliptical bonus b is deﬁned as follows:
[Equation]
Here  I is a regularization term to ensure that the matrix C t  1 is non-singular, where  is a scalar coefﬁcient and I is the identity matrix.
The reward optimized by the algorithm is then deﬁned as
[Figure] [Figure] [Figure] ¯ r ( s t , a t ) =
r ( s t , a t ) +  · b ( s t ) , where r ( s t , a t ) is the extrinsic reward provided by the environment and  is a scalar term balancing the tradeoff between exploration and exploitation.
Intuition.
One perspective is that the elliptical bonus is a natural generalization of a count-based episodic bonus [2].
To see this, observe that if the problem is tabular and  is a one-hot encoding of the state, then C t  1 will be a diagonal matrix whose entries contain the counts corresponding to each state encountered in the episode.
Its inverse C  1 t  1 will also be a diagonal matrix whose entries are inverse state visitation counts, and the bilinear form  ( s t ) > C  1 t  1  ( s t ) reads off the entry corresponding to the current state s t , yielding a bonus of 1 /N e ( s t ) .
For a more general geometric interpretation, if  ( s 0 ) , ...,  ( s t  1 ) are roughly centered at zero, then
C t  1 can be viewed as their unnormalized covariance matrix.
Now consider the eigendecomposition C t  1 =
U >
⇤ U , where ⇤ is the diagonal matrix whose entries are the eigenvalues  1 , ...,  n (these are real since C t  1 is symmetric).
Letting z
= U  ( s t ) =
( z 1 , ..., z n ) be the set of coordinates of  ( s t ) in the eigenspace of C t  1 , we can rewrite the elliptical bonus as:
[Equation]
The bonus increases the more  ( s t ) is aligned with the eigenvectors corresponding to smaller eigenvalues of C t  1 (directions of low data density), and decreases the more it is aligned with eigenvectors corresponding to larger eigenvalues (directions of high data density).
An illustration for n = 2 is shown in Figure 10 of Appendix B.
In our experiments, n is typically between 256 and 1024 .
Efﬁcient Computation.
The matrix C t  1 needs to be inverted at each step t in the episode, an operation which is cubic in the dimension of  and may thus be expensive.
To address this, we use the Sherman-Morrison matrix identity [62] to perform fast rank- 1 updates in quadratic time:
[Equation]
This results in an approximately 3 ⇥ speedup over naïve matrix inversion (details in Appendix D.2).
4.2 Learned Feature Encoder
Any feature learning method could in principle be used to learn  .
Here we use the inverse dynamics model approach proposed in [52], which trains a model g along with  to map each pair of consecutive embeddings  ( s t ) ,  ( s t +1 ) to a distribution over actions a t linking them.
In our setup,  is separate from the policy network.
The g model is trained jointly with  using the following per-sample loss:
[Equation]
The motivation is that the mapping  will discard information about the environment which is not useful for predicting the agent’s actions.
Previous work [52] has shown that this can make learning more robust to random noise or other parts of the state which are not controllable by the agent. 4.3 Full Algorithm
Putting all of these together, the full algorithm is given below. Algorithm 1 Exploration via Episodic Elliptical Bonuses (E3B) Initialize policy ⇡ , feature encoder  and inverse dynamics model f . while not converged do Sample context c ⇠ µ C and initial state s 0 ⇠ µ S ( ·| c )
Initialize inverse covariance matrix: C  1= 1  I for t = 0 , ..., T do a t ⇠ ⇡ ( ·| s t ) // Sample action s t +1 , r t +1 ⇠ P ( ·| s t , a t ) // Step through environment b t +1 =  ( s t +1 ) >
C  1 t  ( s t +1 ) // Compute bonus u = C  1 t  ( s t +1 ) C  1 t +1 = C  1 t  1 1+ b t +1 uu > // Update inverse covariance matrix ¯ r t +1 =
r t +1 +  b t + end for Perform policy gradient update on ⇡ using rewards ¯ r 1 , ..., ¯ r T . Update  and end while  Experiments 5.1 MiniHack Suite
In order to probe the capabilities of existing methods and evaluate E3B, we seek CMDP environments which exhibit challenges associated with realistic scenarios, such as sparse rewards, noisy or irrelevant features, and large state spaces.
For our ﬁrst experimental testbed, we opted for the procedurally generated tasks from the MiniHack suite [58], which is itself based on the NetHack Learning
Environment [41].
NetHack is a notoriously challenging roguelike video game where the agent must navigate through procedurally generated dungeons to recover a magical amulet.
The MiniHack tasks contain numerous challenges such as ﬁnding and using magical objects, navigating through levels while avoiding lava, and ﬁghting monsters.
Furthermore, rewards are sparse and as detailed below, the state representation contains a large amount of information, only some of which is relevant for a given task.
For all our experiments we use the Torchbeast [40] implementation of IMPALA [23] as our base RL algorithm.
For certain skill-based tasks, we restricted the action space to the necessary actions for solving the task at hand, since we found that none of the methods were able to make progress with the full action space (see Appendix C.1.4).
See Appendix C.1.3 for environment details and C.1 for other experiment details.
5.1.1 Modalities for Episodic Bonus
The MiniHack environments provide an observation at each step that includes three different modalities: i) a symbolic image, which indicates the location of the agent, monsters or other entities, objects and different types of terrain (such as walls, lava, water); ii) a statistics vector which indicates the agent’s current ( x, y ) location, hit points, time step t , and other features such as strength, dexterity and constitution, and iii) a textual message which gives different types of feedback about the environment ("you see a key of master thievery", "the wall is solid rock").
These are illustrated in Figure 4.
[Figure]
We now draw attention to some important subtleties regarding how existing methods implement count-based episodic bonuses—as we will see, these can have a large impact on performance.
The works of [56, 25, 76], which use the MiniGrid suite [15] for evaluation, use a hash table whose keys are the full observations (in this case symbolic images) encountered by the agent.
The occurrences of these observations are then counted to compute the episodic bonus.
The baselines run in the MiniHack tasks [58], on the other hand, use a hash table whose keys are the ( x, y ) positions of the agent, which are extracted from the observation and are then similarly counted to compute the episodic bonus.
In order to understand the effect of these different input modalities on count-based episodic bonuses, we consider three variants of the NovelD algorithm, in addition to the original formulation.
These methods all have a count-based episodic bonus of the form
I [ N e (  ( s t )) = 1] , with different choices of  detailed below.
N OVEL D: in the original version of the algorithm,  is the identity.
Here the input to the count-based episodic bonus is the full state, namely the concatenation of the symbol image, the message and the stats vector.
This makes no assumptions about which part of the state is most useful.
N OVEL D- POSITION : in this variant,  extracts the ( x, y ) position of the agent from the statistics vector, which reﬂects a strong inductive bias that the task is navigation-based.
N OVEL D- IMAGE : in this variant,  extracts the symbol image only.
This reﬂects an inductive bias that the message and the stats vector are not useful since they are discarded.
N OVEL D-
MESSAGE : in this variant,  extracts the message only.
This reﬂects an inductive bias that the messages are important but the stats and symbolic images are not.
For E3B, we feed the full state to the algorithm and do not make any assumptions about which part is useful for the task at hand.
Despite the lack of task-speciﬁc inductive biases, our method is still able to extract the relevant features for each task, thus outperforming the other exploration approaches (or matching their performance when prior knowledge of the task is used).
5.1.2 Results on MiniHack
Aggregate results for IMPALA, RND, RIDE, ICM, NovelD (with the three variants described above) and E3B over 16 sparse reward tasks from the MiniHack suite are shown in Figure 5.
Out ofenvironments, 8 are based on the MiniGrid suite, but use the MiniHack interface and observation space (environment details are included in Appendix C.1.3).
We used the performance metrics and bootstrapping protocol from [4] to compute conﬁdence intervals, which are more informative than simple point estimates.
We see that over all tasks (top row), E3B outperforms all other methods by a signiﬁcant margin across all three performance metrics.
[Figure] Standard N OVEL D performs poorly due to the fact that the MiniHack observations contain a time counter in the statistics vector (see Figure 4), which makes each observation in the episode unique and hence renders the count-based episodic bonus meaningless.
The three NovelD variants which use different features extracted from the state for the count-based bonus perform better.
Figure 5 shows aggregate performance across a subset of 9 navigation-based tasks (middle row) and 7 skill-based tasks (bottom row) (see Appendix C.1.3 for the task breakdown).
On the navigation-based tasks,
N OVEL D- POSITION has excellent performance, since visiting a large number of different ( x, y ) locations is closely aligned with the true task reward.
However, N OVEL D- POSITION fails on all the skill-based tasks, resulting in poor performance overall.
We observe an opposite trend for the
N OVEL D- MESSAGE variant, which performs very well on the skill-based tasks, but poorly on the navigation-based ones.
This highlights that although certain inductive biases can help for certain tasks, it is difﬁcult to ﬁnd one which performs well across all of them.
Our method, E3B, performs well on both the navigation-based tasks and the skill-based tasks, resulting in superior performance overall.
It is worth noting that E3B does not require any task-speciﬁc engineering: the exact same algorithm is run for all tasks, and it uses the unprocessed states.
Results for individual tasks, along with more analysis and discussion, can be found in Appendix D.4.
5.1.3 Ablation Experiments
We next report results for ablation experiments measuring the effects of different algorithmic components of E3B, namely the feature encoding  and the episodic nature of the bonus.
Results are shown in Figure 6.
E3B (random enc.) indicates E3B where  is a randomly initialized network which is kept ﬁxed throughout training.
E3B (policy enc.) indicates E3B where the weights of  are tied to those of the policy network (with the last layer producing action probabilities removed).
E3B (non-episodic) indicates E3B where the elliptical bonus is computed using observations across all timesteps in the lifetime of the agent, and not just the current episode.
All three variants perform signiﬁcantly worse than E3B, which uses an inverse dynamics model encoding for  and an episodic bonus.
This highlights that both the inverse dynamics model and episodic bonus are important for success.
5.2 Pixel-Based VizDoom
As our second evaluation testbed, we used the sparse reward, pixel-based VizDoom [37] environments used in prior work [52, 56]. Although these are singleton MDPs, they still constitute challenging exploration problems and probe whether our method scales to continuous high-dimensional pixelbased observations. Results comparing E3B to RIDE, ICM and IMPALA on three versions of the task are shown in Figure 7 (hyperparameters can be found in Appendix C.2).
IMPALA succeeds on the dense reward task but fails on the two sparse reward ones.
E3B is able to solve both versions of the sparse reward task, similar to RIDE and ICM.
We emphasize that these are singleton MDPs, where the environment does not change from one episode to the next.
Therefore, it is unsurprising that ICM, which was designed for singleton MDPs, succeeds in this task.
RIDE is also able to solve the task, consistent with results from prior work [56].
The fact that E3B also succeeds provides evidence of its robustness and its applicability to settings with pixel-based observations.
5.3 Reward-free Exploration on Habitat
As our third experimental setting, we investigate reward-free exploration in Habitat [59, 69].
Habitat is a platform for embodied AI research which provides an interface for agents to navigate and act in photorealistic simulations of real indoor environments.
At each episode during training, the agent is initialized in a different environment, and it is tested on a set of held-out environments not used during training.
These experiments are designed to evaluate exploration of realistic CMDPs with visually rich observations.
We use the HM3D dataset [57], which contains high-quality renditions of 1000 different indoor spaces.
As our base RL algorithm we use DD-PPO [71] and train ICM, RND, NovelD and E3B agents using the intrinsic reward alone.
We then evaluate each agent (as well as two random agents) on unseen test environments by measuring how much of each environment has been revealed by the agent’s line of sight over the course of the episode.
Full details on the experimental setup can be found in Appendix C.3.
Quantitative results are shown in Figure 8.
The E3B agent reveals signiﬁcantly more of the test maps than any of the other agents.
Trajectories for E3B, ICM, RND and NovelD on one of the test maps are shown in Figure 9, which illustrate how E3B explores a large portion of the space whereas the others do not.
These results provide evidence for E3B’s scalability to high-dimensional pixel-based observations, and reinforce its broad applicability.
[Figure]
Related Work
Exploration in RL.
Exploration remains a long-standing problem in RL.
Common approaches include ✏ -greedy [68], count-based exploration [66, 8, 48, 45, 70, 43], curiosity-based exploration [60, 64, 65, 11], and other types of intrinsic motivation [49, 50, 63, 1, 12].
These methods were largely designed for singleton MDPs, where the environment remains the same across episodes.
As a result, they measure novelty over the agent’s lifetime in a static environment rather than on an episodic basis under a distribution of diverse contexts, as necessary in CMDPs.
Other intrinsic motivation methods have recently been developed for exploration in CMDPs [56, 73, 25, 76].
As our experiments show, they critically rely on episodic count-based bonuses, which E3B generalizes.
Another class of methods automatically generate curricula over variations of the CMDP to encourage efﬁcient learning, effectively performing a form of curiosity-driven exploration in the context space.
These include goal-conditioned [27, 26, 24, 55, 19, 14, 22] and goal-free variants [67, 54, 33, 21].
Unlike our method, these methods assume the ability to actively conﬁgure the environment context.
In principle, E3B can be combined with these approaches, whereby E3B explores at an episodic level, while the automatic curriculum method explores at a context level, making the two complementary.
Elliptical Bonuses.
The use of elliptical bonuses has a long history in the contextual bandit literature [6, 20, 42], which corresponds to the RL setting with a single time step.
More recently, several works have begun to explore the use of elliptical bonuses in the context of multi-step RL.
The PC-PG algorithm [2] considers MDPs with linear dynamics and uses an elliptical bonus based on a policy cover to explore in a provably efﬁcient manner.
The ACB algorithm [5] also uses an elliptical bonus, approximated using linear regressors trained on random noise, while FLAMBE [3] uses an elliptical bonus inside a learned dynamics model.
All of these algorithms operate on singleton MDPs, whereas ours is designed for contextual MDPs and constructs elliptical bonuses at the episode level rather than across episodes.
We also use different feature learning methods, namely inverse dynamics models, instead of the policy encoders, random networks or kernel methods used in the aforementioned works.
Conclusion
In this work, we identiﬁed a fundamental limitation of existing methods for exploration in CMDPs: their performance relies heavily on an episodic count-based term, which is not meaningful when each state is unique.
This is a common scenario in realistic applications, and it is difﬁcult to alleviate through feature engineering.
To remedy this limitation, we introduce a new method, E3B, which extends episodic count-based methods to continuous state spaces using an elliptical episodic bonus, as well as an inverse dynamics model to automatically extract useful features from states.
E3B achieves a new state-of-the-art on a wide range of complex tasks from the MiniHack suite, without the need for feature engineering.
Our approach also scales to high-dimensional pixel-based environments, demonstrated by the fact that it matches top exploration methods on VizDoom and outperforms them in reward-free exploration on Habitat.
Future research directions include experimenting with more advanced feature learning methods, and investigating ways to integrate within-episode and across-episode novelty bonuses.