Paper: DEXR: A Unified Approach Towards Environment Agnostic Exploration (ICLR-2024)

Reviewer comment (Novelty):
The method is incremental compared to work like Go-Explore.

Author response:
Concern 1: The method is incremental compared to work like Go-Explore.

Response 1: We regret any misunderstanding regarding the relationship between DEXR and Go-Explore. While there is a superficial similarity, DEXR embodies a fundamentally different design philosophy and operates within a markedly distinct setting.

Firstly, Go-Explore's methodology hinges on the environment being deterministic or the availability of a local model (resettable environment) that can relocate the agent to a previously visited state. Contrarily, DEXR operates independently of these requirements.

Secondly, Go-Explore relocates the agent to regions that are rarely visited previously, whereas DEXR relocates the agent to regions that are believed to be fruitful according to the extra exploitation policy. Despite these two paradigms being similar in form, DEXR and Go-Explore hold drastically different ideas and intuition. To be specific, Go-Explore enables more efficient **over-exploration**, by leveraging the special assumptions and privilege accesses to accurately relocate the agent to “regions not well covered”, so that the agent can fully cover the whole MDP, including regions that might not be worth exploring. In contrast, DEXR prevents **over-exploration** and does not need any special assumptions, by employing an extra exploitation policy to relocate the agent to regions that are “worth exploring”, which largely mitigates the hyperparameter sensitivity, as discussed in Section 1 and Section 4 in detail. 

The effectiveness of DEXR, while straightforward and intuitive, is underscored by our comprehensive empirical analysis. This analysis examines the behaviors of the exploitation and exploration policies in navigation and evaluates the capacity of the exploitation policy to effectively guide the exploration policy.

We have already put a discussion on the relationship between DEXR and the Go-Explore algorithm in the related work section, the reviewer can refer to the revised manuscript for a clearer context.

Reviewer comment (Experiments):
There are missing baselines. Specifically, the method is never compared to the same method but without exploration, i.e. the TD3 baseline.

Author response:
Concern 2: "There are missing baselines ... TD3 baseline."

Weakness 2: We acknowledge the inquiry regarding the comparison with TD3. For your reference, this comparison can be found in the context of the PointMaze experiment. Additionally, we provide visual representations of the behavior of various algorithms, including TD3, within the PointMaze environment, detailed on page 8 of our document.

We apologize for the omission of the split plots and the comparative plot against TD3 in the appendix. We have since rectified this oversight, and these plots are now included in the updated version of the manuscript and can be found in Appendix A.

Reviewer comment (Evaluation):
Comparing the results to the original TD3 paper seems that the proposed method does not outperform TD3. Furthermore, the baseline exploration methods reported all perform worse than TD3. Why does this happen? Since these methods are implemented on top of TD3 it would be strange if they had worse performance than the backbone RL method without any extrinsic rewards.

Author response:
Q1: "Comparing the results ... does not outperform TD3."

A1: In our PointMaze experiments, DEXR consistently outperforms TD3 across various layouts, achieving significantly better results. Specifically, we observed that TD3 requires at least twice as long to reach the goal point compared to DEXR. This demonstrates the superior efficiency and effectiveness of DEXR in these scenarios.

Regarding the MuJoCo Locomotion tasks, our results indicate a more nuanced performance comparison. While DEXR does not show a substantial advantage over TD3 in environments with dense rewards, it notably surpasses TD3's performance in most environments characterized by sparse rewards. This distinction highlights DEXR's enhanced adaptability and proficiency in more challenging settings with limited feedback.

Q2: Furthermore, the baseline exploration methods reported all perform worse than TD3. Why does this happen? Since these methods are implemented on top of TD3 it would be strange if they had worse performance than the backbone RL method without any extrinsic rewards.

A2: As discussed extensively in Sections 1 and 2, methods like Exp, which rely on intrinsic rewards, tend to be distracted and exhibit unstable performance, failing to exploit efficiently when the hyperparameter is not carefully tuned. This is the precise challenge DEXR aims to address.

As detailed in Section 5, all algorithms, including TD3, are trained using extrinsic rewards. Notably, the TD3 agent is trained without the addition of intrinsic rewards.

Reviewer comment (Experiments):
Appendix B is still missing, which contains important empirical results to validate the paper's central claim. The paper claims to find an algorithm that has a reduced sensitivity to the exploration hyperparameter. Then, how the algorithm (and baselines) reacts to different values of the exploration hyperparameter should be shown. Also, it would be better if this content is included in the main text.

Author response:
We again apologize for the oversight concerning the absence of split plots in the appendix of our manuscript. To rectify this, we have now included a separate plot in Appendix A.

Reviewer comment (Experiments):
The effect of the extra hyperparameter, the truncation probability \( p \), is not discussed adequately. The paper only mentions that it is set to \( 1 - \gamma \). In addition, there is an annealing of this parameter, which is not mentioned in the main text nor investigated thoroughly. The paper does not provide justifications for these design choices.

Author response:
We have addressed this concern in Q2, the reviewer can refer to the details in our answer A2.

Reviewer comment (Writing):
There are still a lot of typos and reference format issues in the paper. These should be carefully fixed.

Author response:
We appreciate your suggestion and would like to inform you that the necessary corrections have been made.

Reviewer comment (Experiments):
How does \( \beta \) influence the performance of each algorithm?

Author response:
We apologize for the oversight concerning the absence of split plots in the appendix of our manuscript. We have already put the split results in Appendix A, which describes the influence of $\beta$ on each algorithm.

Reviewer comment (Experiments):
Are there any justifications for setting \( p \) to \( 1 - \gamma \)? Also, why is \( p \) annealed to \( \frac{1}{H} \) during the course of the training?

Author response:
Let me walk through the key choices in detail:

Using $p=1-\gamma$ for truncation probability initially:

- The discount factor $\gamma$ determines the effective horizon $H^e=1/(1-\gamma)$
- This horizon suggests the steps needed to propagate rewards back to early states
- Therefore, letting the exploitation policy run for H steps on average via p=1-γ allows it to reach promising states
- Formally, if $T \sim Geom(p)$, then $\mathbb E[T]=1/p=1/(1-γ)=H^e$

Decaying $p$ over time:

- As exploitation policy improves, less exploration is needed near the initial point
- Reducing p increases $\mathbb E[T]$ since $\mathbb E[T]=1/p$
- This lets exploitation run longer to fine-tune on higher-value regions
- $p$ is decayed gradually to $1/H$, to gradually increase $\mathbb E[T]$ to $H$

Avoiding excessive pointless exploitation:

- Despite higher $\mathbb E[T]$ later, $T$ still follows a geometric distribution
- So there are always probabilistic chances of truncating early
- Formally, $\mathbb P(T=t)=(1-p)^{t-1}p$ still enables occasional exploration in region near the initial point
- Experiments confirm this prevents issues of over-exploitation

Per your feedback, we have already added some discussion on this design choice in Section 4 for more information.

Reviewer comment (Theory):
In Eq. 3 and Eq. 4 on page 4, \( V(s, a) \) should be \( V(s) \).

Author response:
We extend our apologies for any inconvenience caused by these errors. Additionally, we want to clarify that in Equations 3 and 4, the term $\mathbb{P}V(s, a)$ is defined as $\mathbb{E}_{s’ \sim \mathbb{P}(\cdot|s, a)} V(s')$, as elaborated in the paragraph subsequent to Equation 2.

Reviewer comment (Writing):
At the bottom of page 8, “for for” should be “for.”

Author response:
We appreciate your suggestion and would like to inform you that the necessary corrections have been made.

Reviewer comment (Writing):
In the first paragraph of page 13, a verb is missing between “then \( \frac{1}{1-\gamma}.”

Author response:
We appreciate your suggestion and would like to inform you that the necessary corrections have been made.

Reviewer comment (Experiments):
While the proposed approach shows robustness to choices of the intrinsic reward coefficient ($\beta$) in the considered environments, it is currently unclear if this approach would hinder existing curiosity-based approaches in the settings where they have shown significant benefits.

Author response:
We would like to acknowledge that harder exploration problems exist, such as Montezuma’s Revenge. However, we believe our experiment, along with our theoretical analysis, supports our technical claim about DEXR well, as our focus is to enhance to stability of curiosity-driven exploration methods under different reward sparsity settings.

Reviewer comment (Experiments):
The environments are not the best representation of when curiosity-based approaches succeed. The environments considered are low-dimensional, without a hard-exploration challenge.

Author response:
The reason why we chose to work with MuJoCo is two-fold:
1. The aim of DEXR is not to outperform the state-of-the-art in the most popular benchmark but to enhance the stability of existing curiosity-driven exploration algorithms.
2. As Suggested in [3], vision representation can affect performance heavily in pixel-based environments. We evaluate DEXR on MuJoCo to eliminate other factors.

PointMaze is a good testbed for exploration:
1. Its observation space contains clear location information, which allows for monitoring visualizing, and analyzing the behavior of the agents.
2. The hardness of an environment should not be defined by whether the random exploration can get the reward, but defined as “how many samples are needed to find the near-optimal policy”. For reference, without structured exploration, PPO can obtain a considerable amount of reward in Montezuma’s Revenge.

Despite TD3 reaching the goal location a few times in the training process in Large-Maze, still fails to escape the suboptimality. And TD3 gets stuck with its sub-optimal policy, it takes at least 2x longer for TD3 to reach the goal compared to DEXR in all tasks.

Reviewer comment (Experiments):
While the maze navigation environment is technically a sparse-reward setting, the considered mazes are ‘simple’ to explore. Even TD-3, without any intrinsic bonus, reliably achieves rewards in three of the four settings and reaches the goal sometimes in the largest maze. Many intrinsic bonus approaches have enabled success in settings where naive exploration (as in standard TD-3) is far slower and (almost) never succeeds in the considered interactions (e.g., mazes in MiniGrid [1] or hard-exploration Atari environments [2]).

Author response:
The reason why we chose to work with MuJoCo is two-fold:
1. The aim of DEXR is not to outperform the state-of-the-art in the most popular benchmark but to enhance the stability of existing curiosity-driven exploration algorithms.
2. As Suggested in [3], vision representation can affect performance heavily in pixel-based environments. We evaluate DEXR on MuJoCo to eliminate other factors.

PointMaze is a good testbed for exploration:
1. Its observation space contains clear location information, which allows for monitoring visualizing, and analyzing the behavior of the agents.
2. The hardness of an environment should not be defined by whether the random exploration can get the reward, but defined as “how many samples are needed to find the near-optimal policy”. For reference, without structured exploration, PPO can obtain a considerable amount of reward in Montezuma’s Revenge.

Despite TD3 reaching the goal location a few times in the training process in Large-Maze, still fails to escape the suboptimality. And TD3 gets stuck with its sub-optimal policy, it takes at least 2x longer for TD3 to reach the goal compared to DEXR in all tasks.

Reviewer comment (Experiments):
To properly evaluate the sensitivity of approaches to the intrinsic reward coefficient, further details are needed for how the range for $\beta$ was selected. For the navigation tasks, only two values of $\beta_s=1$ and $\beta_l=10000$ were selected, and they are extremely wide apart. I have similar concerns for the Mujoco experiments, which use five values of $\beta$ in the same range. Is there a reason why lower values of $\beta$ were not considered?

Author response:
For PointMaze, the intention was to investigate the impact of small v.s. large intrinsic reward coefficients on the performance and the behaviors of different algorithms.

For MuJoCo locomotion, to evaluate the robustness of DEXR in MuJoCo locomotion tasks, we test various hyperparameters. The exploration & exploitation balance with a higher intrinsic reward coefficient is crucial: higher values encourage exploration in unknown environments but can lead to distractions and unstable optimization. We assess DEXR's effectiveness with coefficients ranging from the most natural choice 1.0 and increase up to 1000.0, examining its ability to handle these challenges.

Reviewer comment (Experiments):
It would be useful to understand DEXR’s sensitivity to truncation probability p, as no results are presented regarding this.

Author response:
We have also added some discussion on this design choice in Section 4 for more information, and provide a set of experiments on evaluating how DEXR would respond to different initial truncation probability $p$ and different decay schedules, which can be found in Appendix A.

Reviewer comment (Theory):
The reason why p is chosen as 1-gamma should be explained.

Author response:
We have also added some discussion on this design choice in Section 4 for more information, and provide a set of experiments on evaluating how DEXR would respond to different initial truncation probability $p$ and different decay schedules, which can be found in Appendix A.

Reviewer comment (Novelty):
The paper misses discussions regarding Bayesian RL approaches (see [3] and references within), which can elegantly balance exploration and exploitation without ugly coefficients to balance intrinsic and extrinsic rewards. There also exist some scalable variants (e.g., [4]).

Author response:
We have already added some discussion on the Bayesian RL approach in the related work section.

Reviewer comment (Novelty):
The related work section covers important papers, but I felt a key section was missing, the Go-explore family of algorithms. [1] proposed Go-Explore, which I think should be discussed in related work. [2] actually has quite a similar proposition to this paper; there are two policies learnt, one that learns only to exploit and one that learns only to explore. While that is in the meta-RL setup, I think it should be mentioned that there is existing work that has suggested similar ideas, but not from an intrinsic motivation perspective.

Author response:
Response 1: Thanks for your suggestion to include a discussion on the Go-Explore work in the Related Work section. In response, we have added a discussion on the connection between our method and the Go-Explore method in the related work section.

Reviewer comment (Novelty):
It may also be worth covering temporal abstraction for exploration in RL in related work. For instance, [3] performs exploration at the level of options i.e. at a higher level of temporal abstraction.

Author response:
Response 2: Thank you for the insightful point. Indeed this is quite relevant to our paper, and it shares some similar approach with DEXR. We have added a discussion concerning this work to our related work section.

Reviewer comment (Presentation):
In the results, the paper mentions that DEXR enjoys notably smaller variances compared to DeRL and Exp but that does not seem to be the case (or at least does not seem to be obvious) in Figure 6. Could the authors provide cleaner plots (with only one DEXR, one Exp, and one DeRL variant, instead of the full set of plots)?

Author response:
Response 3: We have put cleaner results in the main texts, and added the detailed split plots in Appendix A.

Reviewer comment (Theory):
If you have the truncation probability, why do you need the horizon $H$ in the algorithm? Is it an implementation detail to ensure that the exploitation phase does not cover a majority of the episode?

Author response:
A1: Note that H is an environment parameter of the finite horizon setting. The algorithm takes this parameter and splits the horizon into two parts randomly (via the truncation probability). The algorithm performs exploitation in the first part and exploration in the second part. Indeed, if the environment is infinite-horizon, we can set another truncation probability $1-\gamma$ for the exploration policy.

Reviewer comment (Experiments):
In Figure 6, the results are averaged over 5 random seeds and 5 values of $\beta$. It is not standard practice to average over hyperparameters, so I am curious to know why the authors went with this particular decision. Also, the paper mentions that the split over $\beta$ is in the appendix but I was unable to find it. Could the authors clarify where I can find these results?

Author response:
A2: We have already restored the split plots in Appendix A to provide more detailed information on the experimental results, we apologize for missing these plots in the previous version. Regarding the averaging performance across both hyperparameters and seeds, our rationale was to use such averaged results over the hyperparameters to show the robustness of our method, as the performance is promising, and the standard deviation is much lower than Exp and DeRL. Per your feedback, we move the aggregated plots to Appendix A, and in the main text, we only keep the results with Disagreement intrinsic reward with $\beta=1.0$ and $\beta=1000.0$ for better clarity.