Paper: Improving Intrinsic Exploration by Creating Stationary Objectives (ICLR-2024)

Reviewer comment (Novelty):
The technical contribution is not significant enough. It is straightforward to add state visitation count as state input. When it comes to high-dimensional observation space, the proposed method is fully built on E3B and relies on its encoding of the distribution of observed embeddings. So the technical contribution is incremental upon E3B.

Author response:
Regarding the contribution of the paper, we have taken steps to better express the research value of our proposed method, SOFE. We have now added experiments that show that SOFE outperforms recent algorithms like DeRL [1], which are more complex and still aim to tackle the same objective of stabilizing the training with intrinsic rewards. SOFE allows for the training of a single policy in an end-to-end fashion, introducing minimal additional complexity to the RL loop. In contrast, DeRL trains decoupled exploration and exploitation policies to attempt to stabilize the joint objective, which has several limitations: the exploitation policy must be trained from off-policy data only, the exploration policy still faces a complex optimization problem as it optimizes for non-stationary intrinsic rewards, and two completely different policies must be trained in the same loop. With this, we highlight that the simplicity of SOFE is a feature of our contribution.

Please see lines 40-48, 103-106, 256-263, 310-314, and Table 1.

It is true that a version of SOFE uses E3B as a way to estimate the true state distribution. In fact, this is part of the main contribution of SOFE, finding and using the best distributions to estimate the visitation frequencies that are easy for deep models to understand. In larger continuous spaces we use E3B to help estimate the state distribution. Furthermore, E3B achieved SOTA results in partially observable, high-dimensional environments like MiniHack and Habitat (**which we have improved with SOFE**), further outperforming popular baselines like RND and ICM. To the best of our knowledge, there is no published article that reports better exploration performance than E3B in such challenging environments.

Please see the improved Section 3.2.2. Please see Section 4.1 - Intuition here

Reviewer comment (Novelty):
Is E3B the best approach to solve any hard-exploration problems with high-dimensional observation space? If not, could the proposed method be general enough to improve other exploration approaches for hard-exploration problems with high-dimensional observations?

Author response:
It is true that a version of SOFE uses E3B as a way to estimate the true state distribution. In fact, this is part of the main contribution of SOFE, finding and using the best distributions to estimate the visitation frequencies that are easy for deep models to understand. In larger continuous spaces we use E3B to help estimate the state distribution. Furthermore, E3B achieved SOTA results in partially observable, high-dimensional environments like MiniHack and Habitat (**which we have improved with SOFE**), further outperforming popular baselines like RND and ICM. To the best of our knowledge, there is no published article that reports better exploration performance than E3B in such challenging environments.

Please see the improved Section 3.2.2. Please see Section 4.1 - Intuition here

Regarding RND, we are now working to add a discussion section that motivates future work that extends SOFE to other exploration bonuses like RND and ICM, as we had discussed this before and believe it is possible.

Reviewer comment (Experiments):
The proposed method is specifically constrained to the count-based exploration approach. As for other exploration methods with intrinsic motivations, such as RND (random network distillation), the proposed method is not compatible with RND and cannot be used to improve the exploration result.

Author response:
We have disentangled SOFE from count-based bonuses, and have shown that SOFE provides orthogonal gains across exploration objectives of different nature like state-entropy maximization and pseudo-counts.

Please see lines 13-16, 54-58, 92-93, and the improved Section 3.2.

Importantly, SOFE is designed to overcome the non-stationarity of intrinsic rewards by finding a simple representation of the state visitation-frequencies or their representative embedding. RND and ICM **do not** estimate the state-visitation frequencies, as these methods use prediction-error as a heuristic. It is possible to combine SOFE with RND or ICM. However, to model RND or ICM, SOFE needs the parameters of the distribution that is computing the intrinsic rewards. In the case of RND and ICM that distribution is defined by the complete set of parameters of the neural networks being trained (i.e. the predictor network in RND and the forward/inverse dynamics models in ICM). It is very difficult to describe the sufficient statistics for such high-dimensional and non-linear distributions, and this makes the non-stationarity of the RND and ICM rewards practically unsolvable. In our work, we put focus on more principled exploration methods that aim to use the state-visitation frequencies (e.g. counts, state-entropy maximization) or estimate them (e.g. E3B). When using these methods, SOFE strikes a middle ground where the policy can be conditioned on a representation that is used to compute the state-visitation frequencies and leverage it to drive exploration more efficiently as we show in our extensive experiments. 

Additionally, recent works on pseudo-count-based methods have achieved great results in popular benchmark environments by leveraging the estimates of the state-visitation frequencies, and RND and ICM fall short in many challenging environments (e.g. stochastic or procedurally generated). [1, 2, 3]


[1] Henaff, Mikael, et al. "Exploration via elliptical episodic bonuses." Advances in Neural Information Processing Systems 35 (2022): 37631-37646.
[2] Lobel, Sam, Akhil Bagaria, and George Konidaris. "Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning." arXiv preprint arXiv:2306.03186 (2023).
[3] Wang, Kaixin, et al. "Revisiting intrinsic reward for exploration in procedurally generated environments." The Eleventh International Conference on Learning Representations. 2022.

Reviewer comment (Writing):
In Section 3.2., the paper introduces the notation \phi_t without definition. For the context, I think it refers to sufficient statistics for count-based bonuses, but this point is very unclear in Section 3.2. Also, sufficient statistics in the problem setting are not defined.

Author response:
We thank the reviewer for spotting this important issue. We have modified Section 3.2 to improve the notation and better present SOFE. 

Please see the improved Section 3.2.

Reviewer comment (Theory):
In Equation 7, does s_t include sufficient statistics \phi_t or not? I think the state s_t means a fully observable state in the augmentation MDP \hat{M}, so s_t should already contain the information in \phi_t. Then it is weird to have the reward function and policy depending on both s_t and \phi_t in Equation 7.

Author response:
We thank the reviewer for raising this point. We have removed the equation for the optimal policy in the augmented MDP, as it did not provide valuable insights about SOFE. However, we have kept the definition of the augmented MDP where, as you correctly pointed out, the states $\hat{s}_t \in \mathcal{\hat{S}}$ contain both the observations and $\phi_t$.

In the next question, we answer the remaining questions.

Reviewer comment (Theory):
(Motivation) The paper does not clarify that most of the reported considerations are valid when count-based bonuses are the actual learning objective rather than shaping of (sparse) external rewards. The former is arguably not the setting count-based methods have been designed for.

Author response:
**1 - Motivation)** We appreciate the reviewer's attention to the motivation behind our work. While we present SOFE in a reward-free setting to illustrate its core concepts in Section 4, our method extends beyond this setting. In Figure 1, we explicitly present SOFE within a broader context, considering both intrinsic and extrinsic rewards. Initially, our focus on the reward-free setting addresses our first research question, demonstrating that SOFE effectively mitigates the non-stationarity inherent in intrinsic rewards. Next, in our second research question, we show that SOFE better optimizes the joint objective, which is the combination of task and intrinsic rewards. As rightly pointed out by the reviewer, intrinsic rewards are often used to shape sparse task rewards. In this work, we do not contradict this idea, and study both the reward-free and joint objective settings to showcase the empirical performance of SOFE in both settings. Moreover, we have modified the paper to highlight that SOFE is a general framework that works across exploration objectives of different natures (we have unified counts, pseudo-counts, and state-entropy maximization under the same framework).

Please see lines 47-48, the improved Section 3.2, and lines 249-263

Furthermore, in response to the experiments suggested by Reviewer 7vAW (see W2), we augment our evidence by demonstrating SOFE's efficiency in solving more popular sparse-reward tasks. Importantly, our results highlight the efficacy of SOFE beyond purely reward-free settings. We hope this clarification provides a comprehensive understanding of the motivation behind our approach.

Please see lines 310-314, and Table 1.

Reviewer comment (Novelty):
(Novelty and scope) Identifying the non-stationarity of the bonuses as a limitation of count-based methods does not look to be novel (e.g., Schafer et al., Decoupled reinforcement learning to stabilise intrinsically-motivated exploration, 2022) and the paper does not credit previous works on that.

Author response:
**2 - Novelty and Scope)** We appreciate the reviewer's comments regarding the novelty and scope of our work. In response to this feedback and in alignment with Reviewer 7vAW's suggestions (see W1 and W2), we have taken steps to better position our work in the context of related literature. Concretely, we have adjusted the claims of identifying the non-stationarity of rewards but we provide SOFE as a simple and high-performing framework to solve this issue.

Please see lines 4-5, 40-48, 103-106.

As the reviewer pointed out in the work of Schafer et al. [1], we have now included a quantitative comparison between DeRL and SOFE. This comparison serves to highlight the distinct advantages of SOFE in terms of simplicity and performance. Specifically, we demonstrate that SOFE achieves superior performance with significantly less complexity compared to DeRL. SOFE allows for the training of a single policy in an end-to-end fashion, introducing minimal additional complexity to the RL loop. In contrast, DeRL trains decoupled exploration and exploitation policies to attempt to stabilize the joint objective.

Please see lines 310-314, and Table 1.

[1] Schafer et al., Decoupled reinforcement learning to stabilise intrinsically-motivated exploration, 2022

Reviewer comment (Experiments):
(Robustness of the empirical evaluation) The experiments section does not specify how count-based bonuses are implemented. Several strategies exist, and they are known to make a significant difference in the resulting performance, which leaves one wondering how general the reported comparison really is. Moreover, the paper does not compare SOFE against alternative pure exploration methods aside from E3B, ICM, RND in Fig.7.

Author response:
**3 - Robustness of the Empirical Evaluation)** We appreciate the reviewer's observation regarding the implementation of count-based bonuses and the selection of alternative baseline exploration methods for comparison. In response to your feedback and to provide greater clarity to the reader, we have modified Section 3 to provide details of the implementation of count-based bonuses

Please see the improved Section 3.2 and the titles in Figures 4-6.

In terms of alternative exploration methods, we have used RND and ICM, which are the most popular algorithms for long-horizon exploration in deep RL, serving as commonly used baselines in the literature. Additionally, we have introduced E3B and DeRL to broaden the comparisons. 

Finally, we have modified the Appendix section to enhance the clarity of our empirical evaluation and now provide implementation details for the networks and environments used in the paper. This includes hyperparameters, pseudocode, Figures, and curves.

In the next comment, we tackle your questions.

Reviewer comment (Novelty):
Count-based bonuses have been largely employed for hard-exploration tasks, either in terms of regret minimization in theoretical papers or shaping of sparse rewards in empirical literature. From the reviewer's perspective, count-based bonuses are not a great fit for pure exploration: they are not only non-stationary, as the authors noted, but also vanishing, and suffer from other well-known issues (see Ecoffet et al., Go-Explore: A new approach for hard-exploration problems, 2021). However, only in the pure exploration setting do the POMDP argument and optimality of history-based policies seem to make sense. If an external reward is present, it is actually good to converge to a Markovian deterministic policy. The reviewer asks the authors to comment on why they think it is worth studying count-based methods for pure exploration.

Author response:
**Q1) Count-based bonuses as the only learning objective** 

We use the reward-free setting as a strategic choice to illustrate the POMDP argument and the optimality of history-based policies. By studying the non-stationarity of intrinsic rewards and framing it as a POMDP, we emphasize that even in the presence of task rewards, the joint objective formed by the combination of intrinsic and task rewards remains non-stationary.

It is crucial to note that the optimal policy for a joint objective involving both intrinsic and task rewards differs from the optimal policy when considering extrinsic rewards alone. When incorporating the intrinsic signal to shape task rewards, we effectively modify the MDP to facilitate the learning process, although with the trade-off of potentially obtaining policies that may not be optimal in the original task MDP. Importantly, the modified MDP, which integrates the joint objective of intrinsic and task rewards, remains non-stationary. Therefore, SOFE is applicable and beneficial in this context, addressing the challenges posed by non-stationarity and enabling agents to optimize efficiently for the joint objective.

We have modified the paper to better express that SOFE does not only work under the assumptions of count-based rewards being the only learning objective. Concretely, we have unified counts, pseudo-counts, and state-entropy maximization under the same framework and have provided more empirical evidence that SOFE provides performance gains when training on a joint objective of intrinsic+task rewards.

Please see lines 40-48, 256-263, 291-295, 310-314 and Table 1.

Reviewer comment (Novelty):
Count-based bonuses have been used in recent reward-free RL literature as well (e.g., Jin et al., Reward-free exploration for reinforcement learning, 2020). In this setting, the bonuses are frozen at the start of the episode, so that a Markovian policy optimizing a stationary objective is always deployed to collect data.

Author response:
**Q2) Freezing counts to stabilize the bonus** 

We acknowledge that this method can facilitate optimizing for the non-stationary count-based rewards.  However, this approach diverges from the exact count-based exploration objective. In contrast, SOFE does not require any modifications to the count-based objective. Importantly, the optimal policies obtained with SOFE remain optimal for the original count-based objectives, as argued in Section 4 of the paper.

Please see lines 229-231.

Reviewer comment (Novelty):
To address pure exploration settings, other stationary objectives have been designed, e.g., state entropy maximization. The reviewer asks the authors to relate their contributions with this stream of works:  
- Hazan et al., Provably efficient maximum entropy exploration, 2019;  
- Mutti et al., Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate, 2021;  
- Liu & Abbeel, Behavior from the void: Unsupervised active pre-training, 2021;  
- Seo et al., State entropy maximization with random encoders for efficient exploration, 2021;  
- Yarats et al., Reinforcement learning with prototypical representations, 2021;  
- and many others.  
Moreover, the reviewer asks if state entropy maximization is a meaningful baseline for the reported experimental evaluation.

Author response:
**Q3) SOFE & State-Entropy maximization**

We thank the reviewer for raising this point. Even though the comparison between state-entropy maximization and count-based bonuses remains out of the scope of this paper (i.e. in terms of how much they facilitate state coverage and exploration for sparse reward tasks), we believe that SOFE can be easily applied to state-entropy maximization algorithms and provide orthogonal gains. We have now taken steps to show this for the Surprise-Maximizing algorithm [1] which fits a generative model over the state visitation distribution in order to compute its entropy and maximize it. We have modified several sections of the paper to better highlight the generality of SOFE, and have obtained empirical results that support our claims.

Please see lines 13-16, 54-58, 71-73, 92-93, Section 3.2.3, lines 291-295, Figure 5.

[1] Berseth, Glen, et al. "Smirl: Surprise minimizing reinforcement learning in unstable environments." arXiv preprint arXiv:1912.05510 (2019).

Reviewer comment (Reproducibility):
The reviewer asks how the pseudocounts are implemented in the experiments. This seems to make a huge difference. For instance, Bellemare et al. (2016), Ostrovski et al., Count-based exploration with neural density models (2017), Tang et al. (2017), Machado et al., Count-based exploration with the successor representation (2020) all make different design choices with varying results. They ask if the authors think their results are general for every implementation of count-based bonuses or for just one.

Author response:
**Q5) Implementation of pseudo-counts**
 
We thank the reviewer for raising this important point. We use the E3B algorithm to obtain an ellipsoid that represents the pseudo-state-visitation frequencies in a latent space, and not only the pseudo-count for a single state. By using SOFE, we augment the agents’ observations with the approximate state-visitation frequencies over the complete state space and show great performance gains. We have modified the paper to better express this very important point. 

Please see Section 3.2.2

Reviewer comment (Reproducibility):
The reviewer asks how many seeds are considered in the experiments and what is the meaning of the shaded areas. This crucial information seems to be missing for some of the results.

Author response:
**Q6) Details on Figures** 

We thank the reviewer for raising this point. We have now added the necessary information for all figures.

Reviewer comment (Experiments):
In the ProcGen Maze experiment, the paper compares SOFE with E3B, ICM, RND. These do not seem to be the state of the art for procedurally generated tasks. The reviewer asks if the authors considered also tailored methods, such as Ghosh et al., Why generalization in RL is difficult, 2021; Zisselman et al., Explore to generalize in zero-shot RL, 2023.

Author response:
**Q7) SOTA baselines** 

E3B achieved SOTA results in procedurally generated environments. ICM and RND remain the most popular deep exploration algorithms. We thank the reviewer for mentioning these 2 works. While related, they do not provide open-source implementations of their methods. However, we have added them in the Related Work.

We again thank the reviewer for the valuable feedback and invite him/her to re-evaluate the improved version of the paper. We have significantly improved the presentation, broadened the contribution, and provided a more sound empirical evaluation.

Reviewer comment (Theory):
The formulation in E3B doesn't really cover the action which is also important for count-based exploration for particular tasks (e.g. deterministic dynamics will be okay but not stochastic). Consequently, this is not particularly a sufficient statistic but only for environments tested.

Author response:
Regarding the formulation in E3B, our choice aligns with the original implementation (and with the typical literature in exploration bonuses), which measures novelty over the state space. While count-based methods can be defined over state-action pairs, many popular novelty-seeking objectives, including RND, ICM, E3B, and CoinFlip Networks, aim to maximize exploration over the state space.

See Page 2, Column 2, Paragraph 1, last sentence [here](https://arxiv.org/abs/2306.03186)

Reviewer comment (Experiments):
The paper only focuses on navigation tasks, whereas there are many other difficult exploration tasks (e.g. Montezuma's Revenge on Atari, Minecraft, etc.) that could be considered.

Author response:
Minihack, Procgen, and Habitat are widely recognized benchmarks for evaluating exploration in deep RL. Both Minihack and Habitat were intentionally selected because they allow for a direct comparison with existing methods (e.g. E3B). While Montezuma could be defined as a navigation environment, it has other mechanics that go beyond conventional navigation tasks  (e.g. avoiding enemies/climbing ladders). Similarly, in our experiments in Section 5.1, we introduce a 3D world with challenges analogous to Montezuma. Here, the agent must efficiently use jump pads and avoid lava and water pools while still aiming to cover the entire map successfully. Additionally, we designed the 3D world to mirror the complexity of open-world navigation seen in Minecraft. Minecraft has large computing requirements, and it takes more than a week to train an agent in Montezuma (see [here](https://docs.cleanrl.dev/rl-algorithms/ppo-rnd/#implementation-details)) (2 billion samples) and is often not included in papers for this reason.  With this, we believe that our proposed mazes, together with DeepSea, the 3D world, Minihack, Procgen-Maze and Habitat, are representative of the hard-exploration settings widely studied in the literature of intrinsic rewards for exploration in deep RL. 

Please see lines 284-289.

Reviewer comment (Theory):
In section 4, the paper states "we consider that the only unobserved components in the POMDP are the parameters of the reward distribution." It is unclear why this is a good assumption, especially if $\phi_t$ is only a sufficient statistic for the exploration bonus and may not be sufficient for deriving the state $s_t$ (with observation $o_t$).

Author response:
This assumption holds true in many cases.  In particular, it holds in the maze environments used in the paper, or any grid world environment in general, where the agent has access to sufficient statistics of the transition dynamics. Even in Atari environments like Breakout, where the assumption might not hold because a stack of observations is required to infer the true state of the game (e.g. direction of the bullets), SOFE mitigates the non-stationary optimization, yielding performance gains. Importantly, this is evident in our experiments on the 3D world, Habitat, Procgen-Maze, and Minihack, where the agent has only a partially observable view of the environment and yet benefits from SOFE.

Reviewer comment (Theory):
By adding $\phi_t$ to the state space, the state space potentially expands to infinity from a finite state space (e.g. the maze example in experimentation). There is a need for intuition or explanation as to how the algorithm is still able to tractably find an optimal policy in this case.

Author response:
While it is true that we are augmenting the state space, we add features that provide information about the hidden dynamics of the rewards and help improve policy convergence. As you note, there are more subtle interactions occurring between the size of the state space and training dynamics, and thanks to deep learning, increasing the size of the state space is less problematic for learning than having non-stationary objectives.

Please see lines 49-54.

Reviewer comment (Theory):
In continuous state-action space, the counting mechanism depends on quantization. There is a question of how stationary reward is ensured in this case, as it is believed to be non-stationary when states within the same bin cannot be differentiated. Otherwise, the global timestep could be used as the count.

Author response:
By discretizing the continuous state space into bins, we define an auxiliary MDP, instantiating the exploration problem within this discrete framework. Since the exploration objective, observations, and state-visitation frequencies are computed over this auxiliary state space, the true continuous space remains untouched and hence does not induce any non-stationarity on the reward function in the auxiliary MDP.

Regarding the use of the time step $t$ to augment the states, we argue that it does not have an effect on the non-stationarity of the reward distribution. However, it does affect the stationarity of the transition function. Moreover, it has been shown in [1] that including the time step information in the states can lead to degenerate policies. Please see our discussion with the Reviewer 7vAW in **Q1**

[1] Pardo, Fabio, et al. "Time limits in reinforcement learning." International Conference on Machine Learning. PMLR, 2018.

Reviewer comment (Writing):
There is some mention of $\phi_t$ in the paper, but in experimentation only $N_t$ (i.e. count) and $C_t$ (i.e. elliptical bonus) are mentioned. It is unclear if these are the $\phi_t$.

Author response:
Correct, SOFE is a general framework applicable to any sufficient statistics $\phi_t$. In this paper, we apply SOFE to count-based methods, E3B,  and state-entropy maximization. We have modified the paper to make the notation more clear.

Please see the improved Section 3.2.

Reviewer comment (Writing):
Under section 5.1, it is unclear what $N_0$ represents. Is it a binary mask over the map in the maze with only the $j$'th cell as 0? Is the observation the "state"?

Author response:
Correct,  it is a binary mask over the map in the maze with only the j'th cell as 0 and all the other cells as 1's. For this analysis, we design $N_0$, marking all states as visited except for a single one. The SOFE agent, when provided with the maze observation and binary mask, has learned to pay attention to the minutest details in $N_t$, and directs exploration efficiently toward the single unvisited state.

Reviewer comment (Presentation):
It is unclear what "salesman reward" and "√-reward" refer to in figure 5. The reviewer believes they correspond to Eq. 3 and Eq. 2, respectively.

Author response:
The $\sqrt{}$ and salesman rewards are shown in Equations 1 and 2, respectively. To enhance clarity, we've added a sentence to explicitly connect these reward formulations.

Please see Section 3.2.1

Reviewer comment (Novelty):
The paper should adjust the claim that it identifies that intrinsic reward functions induce a non-stationary RL objective. While solutions may be under-explored, the issue is generally known and noted in previous papers (e.g., [1]). Other works have chosen not to tackle the non-stationarity in intrinsic reward as it slowly varies and could be tracked [2]. Also, see the references in W2.

Author response:
We thank the reviewer for the insightful comments. We have addressed your points and improved the paper. We now better highlight the generality of SOFE, providing results across multiple exploration objectives. Importantly, and in line with your comments, we have adjusted the claims of our paper, and better positioned our work with respect to previous related research. We have also included relevant experiments that allow for the necessary comparisons to previous methods, as you rightfully suggested. We have significantly improved the presentation, broadened the contribution, and provided a more sound empirical evaluation. 

**1) Adjusting the claims about the non-stationary intrinsic rewards**
We acknowledge the reviewer's observation and have adjusted our claim in the paper. The modifications emphasize the non-stationarity of intrinsic rewards as a recognized issue, drawing connections to observations from various previous works. The core motivation for our work remains to provide a novel solution to address this challenge.

Please see lines 4-5, 40-48, 103-106 and 109.

Reviewer comment (Experiments):
A crucial weakness of the paper is that it misses comparisons with previous works that have proposed decoupling exploration and exploitation policies [3, 4] to tackle the non-stationarity introduced due to intrinsic bonuses. The paper would significantly benefit from comparing their approach with baselines based on decoupling exploration and exploitation.

Author response:
We appreciate the reviewer's observation and have taken steps to address this concern. We have run experiments on the DeepSea environment (as used in the DeRL paper [1]). Results indicate that SOFE, which simplifies the learning process by training a single policy end-to-end with stationary intrinsic and extrinsic rewards, outperforms DeRL in the harder exploration variations of the environment (see Table 1 in the paper).  We again thank the reviewer since these new results provide further evidence which strengthens our paper by showing the superior performance of SOFE in challenging exploration tasks.

Please see lines 40-44, 103-106, 256-263, 310-314, and Table 1.

[1] Schäfer, L., Christianos, F., Hanna, J. P., & Albrecht, S. V. (2021). Decoupled reinforcement learning to stabilise intrinsically-motivated exploration. arXiv preprint arXiv:2107.08966.