Paper: Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
The experiments are a bit small-scale (only ~400k environment steps at most in the Atari domains)

Author response:
The Atari experiments are actually run for 4 Million frames instead of 400K. The notation 1e6 is meant to demonstrate 10^6, we will make this more clear in the updated paper.

Reviewer comment (Experiments):
There is no experiment in the main text on the larger domains that runs to completion, only the early exploration behavior; while early exploration behavior is more informative for our understanding, it would be good to have an example of how more frames change behavior in more difficult settings (currently in the Appendix Figure 7, but doesn't show baselines as well)

Author response:
Thank you for your comment. We have updated the paper and have incorporated your feedback! The Atari result in the main text (Section-5.6, Figure-4) is now on 12 million steps (previously 4 million). Also we make the distinction between the PPO-based ACB/RND and DDQN-based ACB/RND (VB ACB/RND) early on and more clearly in the experiment section-5. We also added a new result in the Appendix E where we compare VBE with VB ACB/RND on the six Atari environments. The reviewer was right in noting that DDQN-based version would be more sample-efficient compared to their PPO-based counterparts. However, VBE still performs better in general, outperforming in 3/6 environments. In Qbert VBE performs better than ACB, and in Pitfall VBE performs better than RND. We also note that beyond 5 million frames on Pitfall, VBE surpasses VB ACB too (will update the paper). It is only in Gravitar where both VB ACB/RND are better than VBE, as VBE is slow but continues to improve beyond 5 million frames. We will update the paper with results on more frames in the final version, and perhaps even add it to the main text as soon as we have the results for 12 million frames for consistency. 

We thank the reviewer for giving thorough feedback on how to improve the paper. We hope that we have successfully incorporated the feedback in our revision and have satisfied important concerns raised by the reviewer!

Reviewer comment (Evaluation):
There is little discussion on the relatively worse performance in Pitfall and Gravitar in Figure 4; some closer qualitative analysis is merited

Author response:
You bring up a reasonable point about the worse performance of VBE in Pitfall and Gravitar. In Pitfall our understanding is that VBE takes longer to converge to the policy of always going left in Pitfall, as the agent still tries to explore novel states. In Figure 7 in the Appendix, both variants of VBE continue to improve in Pitfall after 4 Million frames. VBE-CNN fully converges to the policy of always going left and outperforms all baselines. 

For Gravitar, both ACB and RND start off at a better policy than VBE. This may be due to different architectures, since ACB and RND use PPO, since that is originally how they were specified. We cannot say for sure, of course, but could put some discussion on our speculations and label them clearly as speculations (One of the reasons we chose to do a more careful study in classic control environments is precisely because we can better understand the algorithms.) It is interesting to note, though, that VBE does slowly improve, whereas ACB/RND do not. This is especially evident in Figure 7 in Appendix where both variants of VBE continue to improve beyond 4 Million frames and eventually arrive at a similar mean performance to ACB/RND in Gravitar.

Reviewer comment (Experiments):
It would be nice to have a showcase result on Montezuma's Revenge, or perhaps one of the Antmaze tasks for a more difficult sparse-reward setting

Author response:
We did not include Montezuma’s Revenge because initial experiments indicated no baseline does well in this environment within the given frame budget (also reported in [6]). It is note-worthy that [6] also characterizes Montezuma’s Revenge as obfuscating the exploration abilities of reward-bonus based exploration methods. In the original RND work, the agent gets to use multiple parallel copies of the environment and sees much more frames (as the reviewer correctly points out), which is much much more interaction. For the sparse reward setting, we do have Sparse Mountain Car, which we have found to be surprisingly challenging for deep RL agents and actually highlights differences between methods. We will keep Antmaze in mind for future work!

Reviewer comment (Reproducibility):
There is no explicit objective given for q_w in Algorithm 1, even though the reward has been relabeled in line 12; when we aren't in the pure exploration setting are we using the original reward? It would be worthwhile in the main text to give the exact objectives for all components in one place

Author response:
For Algorithm1, we can see how it was unclear how we updated q_w. Line 10 says we update using Equation 1, with the comment in line 9 saying this update is the action-values without explicitly mentioning q_w. We will fix this and update the paper soon. The q_w is updated using the standard DDQN update using the environment reward (namely, Equation 1), with no change from the DDQN update. The primary change is the behavior policy, which now uses value bonuses, and how we update the ensemble to define the value bonus.

Reviewer comment (Reproducibility):
Algorithm 1, define objective for q_w as well as for f_{w_i}

Author response:
For Algorithm1, we can see how it was unclear how we updated q_w. Line 10 says we update using Equation 1, with the comment in line 9 saying this update is the action-values without explicitly mentioning q_w. We will fix this and update the paper soon. The q_w is updated using the standard DDQN update using the environment reward (namely, Equation 1), with no change from the DDQN update. The primary change is the behavior policy, which now uses value bonuses, and how we update the ensemble to define the value bonus.

Reviewer comment (Writing):
Section 5.1, "hard exploration environments" is a subjective claim and probably deserves a citation; I don't consider these to be hard exploration environments, but maybe I should and I don't know why

Author response:
Riverswim and Deepsea [1, 2, 3, 4 ,5] are considered hard exploration environments, and are commonly used in literature to test exploration algorithms.  

References: 

[1] Kumaraswamy, R., Schlegel, M., White, A., & White, M. (2018). Context-dependent upper-confidence bounds for directed exploration. Advances in Neural Information Processing Systems, 31.

[2] Ishfaq, H., Cui, Q., Nguyen, V., Ayoub, A., Yang, Z., Wang, Z., ... & Yang, L. (2021, July). Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning (pp. 4607-4616). PMLR.

[3] Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., ... & Van Hasselt, H. (2019). Behaviour suite for reinforcement learning. arXiv preprint arXiv:1908.03568.

[4] Ian Osband, John Aslanides, and Albin Cassirer. Randomized Prior Functions for Deep Reinforcement Learning. NeurIPS, 2018. 

[5] Ian Osband, Benjamin Van Roy, Daniel J Russo, Zheng Wen, et al. Deep exploration via randomized value functions. J. Mach. Learn. Res., 20(124):1–62, 2019.

[6] Taiga, A. A., Fedus, W., Machado, M. C., Courville, A., & Bellemare, M. G. (2021). On bonus-based exploration methods in the arcade learning environment. arXiv preprint arXiv:2109.11052.

Reviewer comment (Theory):
See Weaknesses above — mostly I want some more clarity around the exact objective for q_w vs. f_{w_i}

Author response:
For Algorithm1, we can see how it was unclear how we updated q_w. Line 10 says we update using Equation 1, with the comment in line 9 saying this update is the action-values without explicitly mentioning q_w. We will fix this and update the paper soon. The q_w is updated using the standard DDQN update using the environment reward (namely, Equation 1), with no change from the DDQN update. The primary change is the behavior policy, which now uses value bonuses, and how we update the ensemble to define the value bonus.

Reviewer comment (Experiments):
Also in Weaknesses — how does performance vs. RND and ACB change with more frames

Author response:
RND and ACB are much more expensive than VBE and take a lot longer to run. But, we are currently completing results for these algorithms for 10 Million frames and will update the paper with these complete results.

Reviewer comment (Writing):
Clarity and Presentation: The paper tends to obfuscate what could be explained more simply. While there is value in rigorous mathematical explanations, these should be accompanied by intuitive explanations and clearer definitions for broader accessibility. For example, the distinction between equation 2 and the actual bonus used in algorithm 1 are not clearly demarcated, leading to potential confusion.

Author response:
We summarize and highlight some advantages of VBE over existing reward bonus methods: 

1- Defining the value bonuses directly allows us to incorporate first-visit optimism.

2- Single step optimization problem compared to solving two optimization problems in conventional reward bonus methods.

3- The reward bonus in existing methods is non-stationary, whereas the reward function for RQFs defined in Equation-3 is stationary. 

4- Easier to combine with value-based approaches, as it is completely independent. Whereas in reward bonus methods there is a choice to be made on how to combine the reward bonus with the main value function. For example ACB and RND use a separate value function because the intrinsic value function needs to be non-episodic.

We ask the reviewer to go through this response and identify if this explanation helps with the clarity. We are more than happy to incorporate clarifications in the paper, if you can provide more specific issues given this additional explanation. Please note  that we will fix the highlighted discrepancy with the definition of Equation 2 and the use in Algorithm 1, thank you for the pointer. 

References:

[1] Ian Osband, John Aslanides, and Albin Cassirer. Randomized Prior Functions for Deep Reinforcement Learning. NeurIPS, 2018.

Reviewer comment (Novelty):
Novelty Concerns: Upon close examination, the proposed method seems to be essentially a variant of the classical UCB exploration strategy, replacing the count based bonus to ensemble error based bonus; as well as a variant to RND, with difference in ensembles. Moreover, there are clear parallels with the bootstrapped DQN approach, which already encourages first-visit optimism. While the paper does acknowledge the connection to bootstrapped DQN, the explanation is not well-presented, and readers may find it challenging to discern the true novelty of the proposed method.

Author response:
We thank the reviewer for their review, and appreciate suggestions to make the work clearer, since we are always striving to present the idea in the simplest way possible. We agree with the reviewer that our proposed methodology is an extension of UCB-style exploration methods to the RL setting. However, our proposed methodology is novel and different from existing attempts to extend UCB to the RL setting – which we refer to as reward bonus methods – like count-based methods, RND, ACB, ICM, etc. We will attempt to further highlight this distinction here. First, it is important to note that UCB-style exploration methods originate from the bandit literature, where the agent maintains and updates its reward estimates for each arm and UCB-based exploration methods provide reward bonuses to encourage selecting arms with higher uncertainty. In RL, an agent maintains and updates its value estimates to determine its behavior policy, thus an exploration bonus is required in the value space to encourage visiting uncertain states. The existing algorithms that attempt to extend such UCB-style exploration methods to the RL setting first approximate a reward bonus using counts, density-based counts, prediction errors [RND, ACB, ICM], etc., and then propagate these reward bonuses (or local estimates of uncertainty) either by adding this reward bonus to the environment reward or by updating a separate value function as in RND, ACB. These existing reward bonus methods attempt to extend UCB-style methods to RL by doing two steps: 1- approximating reward bonus or local uncertainty via supervised learning or self-supervised learning, 2- propagating these reward bonuses, say using TD-learning. 

Our proposed method VBE attempts to extend UCB-style methods to the RL setting by directly approximating bonuses in the value space (or value bonuses). It avoids the two step process of reward bonus approaches, that require estimating local uncertainties and then propagating. VBE directly updates the value functions in the ensemble to get a value bonus. Further, directly defining the bonuses in the value space provides first-visit optimism, which the existing reward bonus methods fail to provide. This is evident from our pure exploration experiment in Section-5.3 and Figure-1. 

To further understand how our value bonus method provides optimism by implicitly conserving uncertainty in the value estimates consider this example: Consider the following transition sequence: $(s, a, s’, a’)$, $f_{rqf}(s,a)$ is the predictor RQF, $f_{trg}(s,a)$ is the target RQF, and reward function for the predictor RQF is defined as $r_{rqf}(s,a,s',a') = f_{trg}(s,a) - \gamma f_{trg}(s’,a’)$ (Equation-3). Now suppose that the state-action pair $(s’,a’)$ has been less frequently visited and thus has high prediction error. This would mean that the error or the $bonus(s’,a’) = |f_{rqf}(s’,a’) - f_{trg}(s’,a’)|$ would be high, but how does this high uncertainty affect  $bonus(s,a)$? Notice that we update $f_{rqf}(s,a)$ by bootstrapping the prediction value of $f_{rqf}(s’,a’)$ (TD learning update), that is the target for $f_{rqf}(s,a)$ is $r_{rqf}(s,a,s’,a’) + \gamma f_{rqf}(s’,a’)$. This bootstrap target can only be the “true target”, $f_{trg}(s,a)$ if  $\gamma f_{rqf}(s’,a’) == \gamma f_{trg}(s’,a’)$, that is $f_{rqf}(s’,a’)$ is error-free. This means that prediction error for $(s,a)$ or the $bonus(s,a)$ will only go down to zero when the prediction error for all subsequent state-action pairs is zero. Proposition 1 reflects this property. 

Connection to BDQN explained: BDQN is not a UCB-style exploration method, rather BDQN aims to approximate Thompson Sampling for exploration in RL. The ensemble in BDQN aims to approximate the posterior distribution over action-value functions. Statistical bootstrapping alone however can only approximate the epistemic uncertainty of observed state-action pairs as noted in [1]. Adding a fixed additive prior can provide the first-visit optimism. Notice this explanation from the BDQN paper “The crucial element is that when a new state s’ is reached there is some ensemble member that estimates $\max_{a’} Q_{k}(s’, a’)$ is sufficiently positive to warrant visiting, even if it causes some negative reward along the way” [1]. The ensemble in BDQN is thus responsible for approximating the posterior distribution and providing first-visit optimism. 

VBE on the other hand is less sensitive to the size of its ensemble. This is especially prominent from Table-1 in Appendix B which shows that in the pure exploration experiment (Section-5.3, Figure-1) VBE only requires 1 RQF to cover the entire state-space for all grid-sizes, whereas BDQN requires 20 heads in the ensemble and still does not manage to cover the state-space for grid-sizes bigger than 35x35. This further highlights the efficacy of VBE over BDQN as it requires smaller ensembles even on hard exploration tasks and also highlights the difference in usage of the ensemble.

Reviewer comment (Novelty):
Significance of Contribution: Given the above, one might argue that the paper's primary contribution is an amalgamation of previously explored ideas. The ensemble error as an exploration bonus, though interesting, might not be substantial enough to warrant a separate methodology, especially given the similarities to existing methods.

Author response:
We thank the reviewer for their review, and appreciate suggestions to make the work clearer, since we are always striving to present the idea in the simplest way possible. We agree with the reviewer that our proposed methodology is an extension of UCB-style exploration methods to the RL setting. However, our proposed methodology is novel and different from existing attempts to extend UCB to the RL setting – which we refer to as reward bonus methods – like count-based methods, RND, ACB, ICM, etc. We will attempt to further highlight this distinction here. First, it is important to note that UCB-style exploration methods originate from the bandit literature, where the agent maintains and updates its reward estimates for each arm and UCB-based exploration methods provide reward bonuses to encourage selecting arms with higher uncertainty. In RL, an agent maintains and updates its value estimates to determine its behavior policy, thus an exploration bonus is required in the value space to encourage visiting uncertain states. The existing algorithms that attempt to extend such UCB-style exploration methods to the RL setting first approximate a reward bonus using counts, density-based counts, prediction errors [RND, ACB, ICM], etc., and then propagate these reward bonuses (or local estimates of uncertainty) either by adding this reward bonus to the environment reward or by updating a separate value function as in RND, ACB. These existing reward bonus methods attempt to extend UCB-style methods to RL by doing two steps: 1- approximating reward bonus or local uncertainty via supervised learning or self-supervised learning, 2- propagating these reward bonuses, say using TD-learning. 

Our proposed method VBE attempts to extend UCB-style methods to the RL setting by directly approximating bonuses in the value space (or value bonuses). It avoids the two step process of reward bonus approaches, that require estimating local uncertainties and then propagating. VBE directly updates the value functions in the ensemble to get a value bonus. Further, directly defining the bonuses in the value space provides first-visit optimism, which the existing reward bonus methods fail to provide. This is evident from our pure exploration experiment in Section-5.3 and Figure-1. 

To further understand how our value bonus method provides optimism by implicitly conserving uncertainty in the value estimates consider this example: Consider the following transition sequence: $(s, a, s’, a’)$, $f_{rqf}(s,a)$ is the predictor RQF, $f_{trg}(s,a)$ is the target RQF, and reward function for the predictor RQF is defined as $r_{rqf}(s,a,s',a') = f_{trg}(s,a) - \gamma f_{trg}(s’,a’)$ (Equation-3). Now suppose that the state-action pair $(s’,a’)$ has been less frequently visited and thus has high prediction error. This would mean that the error or the $bonus(s’,a’) = |f_{rqf}(s’,a’) - f_{trg}(s’,a’)|$ would be high, but how does this high uncertainty affect  $bonus(s,a)$? Notice that we update $f_{rqf}(s,a)$ by bootstrapping the prediction value of $f_{rqf}(s’,a’)$ (TD learning update), that is the target for $f_{rqf}(s,a)$ is $r_{rqf}(s,a,s’,a’) + \gamma f_{rqf}(s’,a’)$. This bootstrap target can only be the “true target”, $f_{trg}(s,a)$ if  $\gamma f_{rqf}(s’,a’) == \gamma f_{trg}(s’,a’)$, that is $f_{rqf}(s’,a’)$ is error-free. This means that prediction error for $(s,a)$ or the $bonus(s,a)$ will only go down to zero when the prediction error for all subsequent state-action pairs is zero. Proposition 1 reflects this property. 

Connection to BDQN explained: BDQN is not a UCB-style exploration method, rather BDQN aims to approximate Thompson Sampling for exploration in RL. The ensemble in BDQN aims to approximate the posterior distribution over action-value functions. Statistical bootstrapping alone however can only approximate the epistemic uncertainty of observed state-action pairs as noted in [1]. Adding a fixed additive prior can provide the first-visit optimism. Notice this explanation from the BDQN paper “The crucial element is that when a new state s’ is reached there is some ensemble member that estimates $\max_{a’} Q_{k}(s’, a’)$ is sufficiently positive to warrant visiting, even if it causes some negative reward along the way” [1]. The ensemble in BDQN is thus responsible for approximating the posterior distribution and providing first-visit optimism. 

VBE on the other hand is less sensitive to the size of its ensemble. This is especially prominent from Table-1 in Appendix B which shows that in the pure exploration experiment (Section-5.3, Figure-1) VBE only requires 1 RQF to cover the entire state-space for all grid-sizes, whereas BDQN requires 20 heads in the ensemble and still does not manage to cover the state-space for grid-sizes bigger than 35x35. This further highlights the efficacy of VBE over BDQN as it requires smaller ensembles even on hard exploration tasks and also highlights the difference in usage of the ensemble.