Paper: $\beta$-DQN: Diverse Exploration via Learning a Behavior Function (ICLR-2024)

Reviewer comment (Experiments):
The toy example is presented well with interesting observations. For example, this makes intuitive sense: "At the beginning, the exploration policies for data coverage (πcov) are chosen more frequently. After a good coverage of the whole state space, the exploration policies for bias correction (πcor) are chosen more frequently in this sparse reward environment." The pattern however does not hold and becomes more complicated in Section 5.3.1 (Figure 4). Could the authors perform more experiments on diverse environments to see if any insights can be obtained collectively with respect to the environment characteristics?

Author response:
- The pattern however does not hold and becomes more complicated in Section 5.3.1 (Figure 4).

In our current experiments, we find that in simple (LavaCrossing-Easy) or dense reward (Asterix) environments, exploring for bias correction plays a more important role. In hard exploration environments (RedBlue-Doors, LavaCrossing-Hard), the two kinds of policies interleave and result in a more intricate selection pattern. 
We distill these insights as follows: In simpler environments, the emphasis is on efficiently discovering low-hanging-fruit rewards, prioritizing exploitation over extensive exploration. In contrast, in challenging exploration environments, the importance of state-novelty exploration is heightened, resembling the principles of depth-first search (DFS) and breadth-first search (BFS). Our approach adapts to this context: upon encountering positive rewards, it adopts a depth-first exploration strategy, delving deeper into discovered areas for further exploration. Conversely, in the absence of immediate rewards, the strategy shifts towards a breadth-first approach, exploring widely to uncover promising areas.

Reviewer comment (Evaluation):
In Section 5.3.2, "The policy argmax Q takes greedy actions among the whole action space, it may take overestimated actions thus the performance is not as stable as argmax Qmask." Figure 5 doesn't seem to suggest any clear difference in terms of stability, though argmax Q_mask does perform better.

Author response:
- argmax Q is not as stable as argmax Qmask in Figure 5.

Thanks for pointing out this. A more precise description would be argmax Q does not perform as well as argmax Qmask because it may select overestimated actions. We will revise the description to accurately reflect this distinction.

Reviewer comment (Novelty):
High-level idea: the key insight behind exploration is not simply to visit the actions that are infrequently visited, but to add frequency measure (like the count-based methods) to the value, to perform an exploration-exploitation trade-off. The idea of learning a behaviour density estimator is interesting, but integrating this idea with a count-based MAB exploration algorithm makes the algorithm no longer elegant, and hard to identify the performance improvement.

Author response:
Integrating our idea with a count-based Multi-Armed Bandit (MAB) exploration algorithm is a plausible approach. Notably, there exist various methods that learn a population of policies and leverage a MAB meta-controller for policy selection [1,2,3].

Reviewer comment (Writing):
The presentation of the paper can be improved. There are many misleading terminology usages. e.g., the *space coverage*, *bias correction*, what does those terms mean in this context?

Author response:
We appreciate the feedback and will refine the presentation to make the definitions clearer.

Reviewer comment (Experiments):
Is the MAB trained concurrently with the Q networks? How does the choice of hyper-parameters affect learning stability (it was partially revealed in Figure 6). In practice, is there a golden standard for selecting those hyper-parameters?

Author response:
MAB does not require training. Instead, it performs statistical analysis on past data and continuously updates its statistics as more data is collected. This unique characteristic sets the MAB simple and efficient, allowing it to adapt and refine its policy selection strategy dynamically with the accumulation of experience.

Reviewer comment (Evaluation):
It would be helpful to have a standard deviation in results (table 1.)

Author response:
We will add it in the next version.

Reviewer comment (Reproducibility):
In Equation (1), it’s not clear what \beta exactly is and how it is parameterised and optimised. Is it a supervised / behaviour cloning policy?

Author response:
$\beta$ serves as an estimation of the frequency of state-action pairs within the replay memory. This estimation is akin to count-based methods, but $\beta$ does not rely on an explicit count. Instead, it operates as a supervised policy, dynamically adapting to the changing environment by providing a probabilistic measure of the likelihood of selecting different state-action pairs.

Reviewer comment (Novelty):
Some related works are missing: 

Ensemble:
- https://arxiv.org/pdf/1611.01929.pdf discussed ensemble methods in DQN.
- https://arxiv.org/pdf/2209.07288.pdf discussed the ensemble methods using different hyperparameter settings.

Curiosity-Driven Exploration:
- https://arxiv.org/abs/2206.08332 BYOL-Explore is one of the state-of-the-art exploration algorithms for discrete and continuous control
- https://arxiv.org/abs/2211.10515
- https://openreview.net/forum?id=_ptUyYP19mP further improves the RND

Author response:
We appreciate your suggestion. We will discuss this related work in the next version.

Reviewer comment (Novelty):
The main concern is that the proposed method has a large overlap with a prior work: BAC [1], but there are neither discussion nor comparison in this paper. For example, the idea of learning in-sample state-action pairs in Eq. (2) is similar to Eq. (4.1) in BAC. The idea of making trade-off between standard and conservative Q-value functions in Eq. (5) is similar to Eq. (4.3) in BAC. Apart from being overlapped with BAC, the idea of constructing a policy set has been extensively studied in literature, e.g. [2].

Author response:
- Related work and baselines

We will discuss these related work and add more baselines in the next version. 
BAC [1] is a concurrent work which is also submitted to ICLR2024 (https://openreview.net/forum?id=VPx3Jw2MSk). We will compare with it in the next paper version. While both works consider in-distribution learning, which makes them similar at the high level, the methods used are very different.  BAC blends two operators to trade off exploitation (IQL-style) and exploration (SAC-style), while our method uses a behavior function $\beta$ as a mask to switch the mode between exploitation and exploration. In this way we can explicitly generate a group of policies ranging from pure exploitation to exploration with Q and $\beta$, while BAC cannot. 
Further, the BAC method tunes parameters for each task individually to get high performance, while we used a single set of parameters for all tasks. This reduces the burden of hyper-parameter tuning.

The idea of constructing a policy set has been extensively studied in the literature [2,3,4,5]. All these methods maintain a group of policies with independent parameters. The computational cost blows up and is unaffordable for most research communities. In contrast, our method constructs a set of diverse policies from only one additionally learned behavior function, which is
straightforward to implement and computationally efficient.

Reviewer comment (Experiments):
There is a lack of baseline algorithms to compare with. The curiosity-driven exploration algorithms should be included as baselines.

Author response:
- Related work and baselines

We will discuss these related work and add more baselines in the next version. 
BAC [1] is a concurrent work which is also submitted to ICLR2024 (https://openreview.net/forum?id=VPx3Jw2MSk). We will compare with it in the next paper version. While both works consider in-distribution learning, which makes them similar at the high level, the methods used are very different.  BAC blends two operators to trade off exploitation (IQL-style) and exploration (SAC-style), while our method uses a behavior function $\beta$ as a mask to switch the mode between exploitation and exploration. In this way we can explicitly generate a group of policies ranging from pure exploitation to exploration with Q and $\beta$, while BAC cannot. 
Further, the BAC method tunes parameters for each task individually to get high performance, while we used a single set of parameters for all tasks. This reduces the burden of hyper-parameter tuning.

The idea of constructing a policy set has been extensively studied in the literature [2,3,4,5]. All these methods maintain a group of policies with independent parameters. The computational cost blows up and is unaffordable for most research communities. In contrast, our method constructs a set of diverse policies from only one additionally learned behavior function, which is
straightforward to implement and computationally efficient.

Reviewer comment (Theory):
The DQN-style formulation constrains the action space to be discrete. Therefore, the proposed algorithm cannot fit to continuous control tasks.

Author response:
- The DQN-style formulation constrains the action space to be discrete.

The DQN-style formulation, by design, restricts the action space to be discrete. Our method is an improvement for DQN, it cannot be applied directly to continuous action space. Addressing this limitation and extending our method to accommodate continuous action spaces is an intriguing avenue for future work.

Reviewer comment (Experiments):
The proposed algorithm is limited to the discrete action domain while most other exploration algorithms do not have such a limitation, and there is no obvious way to directly apply it to the continuous action domains.

Author response:
Yes. Currently our method is an improvement for DQN, and cannot be applied directly to continuous action space. We will extend it to continuous action space in future work.

Reviewer comment (Experiments):
The baselines are not tuned, and the selection of the baseline does not align with the main motivation of the experiment.

Author response:
We did tune each baseline and achieved higher performance than in the original paper. For example, untuned RND and LESSON by Kim et al. achieve success rates of around 0.1 and 0.2 on LavaCrossingS9N1 (LavaCrossing-Easy in our paper). In our paper, these baseline methods are tuned to achieve 0.63 and 0.75, much higher than the original paper. We found that these intrinsic reward based methods are sensitive to hyperparameters.

Reviewer comment (Experiments):
The purpose of this paper is to propose a computationally efficient, hyperparameter-insensitive, and well-performing exploration algorithm. However, \(\beta\)-DQN has two hyperparameters, \(L\) and policy set size. Policy set size controls the hardness of the MAB algorithm, whereas \(L\) affects the process of the learning of MAB. The authors provide an ablation study on the analysis of the sensitivity of the size of the policy set but do not provide the ablation study on \(L\), which is not enough to convince the reader that \(\beta\)-DQN is not sensitive to hyperparameters.

Author response:
We appreciate the systematic suggestions provided. In our next version, we will conduct an ablation study and undertake a more comprehensive systematic hyperparameter tuning of baselines.

Regarding the ablation study on $L$, a rough search among $L$ in [100, 500, 1000, 2000] on SimpleCrossing-Easy and Asterix found no obvious differences between 500, 1000 and 2000, and lower performance for $L=100$. Based on this preliminary experiment, we fixed $L= 1000$ for all environments. We can do a grid search for $L$ for all environments.

Reviewer comment (Experiments):
Most exploration algorithms are sensitive to hyperparameters. For intrinsic-reward based ones, they are particularly sensitive to the bonus scale factor \(\alpha\). Specifically, the implementation of RND is from the LESSON paper by Kim et al., but the authors use a different \(\alpha\) in this paper and achieved better performance compared to the RND in LESSON paper, which further shows that hyperparameter selection can be crucial for the performance of exploration algorithms like RND. Without the systematic tuning of hyperparameters for the baselines, it is hard to convince the readers that \(\beta\)-DQN is preferred. Additionally, the ablation study of the size of the policy set does not show an insensitivity of \(\beta\)-DQN to this hyperparameter.

Author response:
We did tune each baseline and achieved higher performance than in the original paper. For example, untuned RND and LESSON by Kim et al. achieve success rates of around 0.1 and 0.2 on LavaCrossingS9N1 (LavaCrossing-Easy in our paper). In our paper, these baseline methods are tuned to achieve 0.63 and 0.75, much higher than the original paper. We found that these intrinsic reward based methods are sensitive to hyperparameters.

We appreciate the systematic suggestions provided. In our next version, we will conduct an ablation study and undertake a more comprehensive systematic hyperparameter tuning of baselines.

Reviewer comment (Experiments):
Overall, the experiment setup, including the choice of environments, the hyperparameter selections for baselines, and the ablation study on hyperparameters of \(\beta\)-DQN, does not provide enough information for the reader to decide whether or not \(\beta\)-DQN is more hyperparameter insensitive, or performs better than other exploration methods.

Author response:
We did tune each baseline and achieved higher performance than in the original paper. For example, untuned RND and LESSON by Kim et al. achieve success rates of around 0.1 and 0.2 on LavaCrossingS9N1 (LavaCrossing-Easy in our paper). In our paper, these baseline methods are tuned to achieve 0.63 and 0.75, much higher than the original paper. We found that these intrinsic reward based methods are sensitive to hyperparameters.

We appreciate the systematic suggestions provided. In our next version, we will conduct an ablation study and undertake a more comprehensive systematic hyperparameter tuning of baselines.

Regarding the ablation study on $L$, a rough search among $L$ in [100, 500, 1000, 2000] on SimpleCrossing-Easy and Asterix found no obvious differences between 500, 1000 and 2000, and lower performance for $L=100$. Based on this preliminary experiment, we fixed $L= 1000$ for all environments. We can do a grid search for $L$ for all environments.

Reviewer comment (Experiments):
The authors propose a novel algorithm that smartly integrates the MAB algorithm in the RL setting to help reduce the computation complexity; however, the claims of "hyperparameter-insensitivity" and "effective exploration" are not sufficiently backed up by the experiment. It is suggested to tune the hyperparameters of RND and Bootstrapped DQN in a systematic manner, and show that even the best hyperparameters would not yield better performance compared to \(\beta\)-DQN, thereby demonstrating that \(\beta\)-DQN is indeed favorable.

Author response:
We appreciate the systematic suggestions provided. In our next version, we will conduct an ablation study and undertake a more comprehensive systematic hyperparameter tuning of baselines.

Reviewer comment (Experiments):
It is suggested to provide a more systematic ablation study for the hyperparameter of \(\beta\)-DQN.

Author response:
We appreciate the systematic suggestions provided. In our next version, we will conduct an ablation study and undertake a more comprehensive systematic hyperparameter tuning of baselines.

Regarding the ablation study on $L$, a rough search among $L$ in [100, 500, 1000, 2000] on SimpleCrossing-Easy and Asterix found no obvious differences between 500, 1000 and 2000, and lower performance for $L=100$. Based on this preliminary experiment, we fixed $L= 1000$ for all environments. We can do a grid search for $L$ for all environments.

Reviewer comment (Novelty):
The paper does not have enough technical contribution. The way to construct diverse policies is elementary. It only concerns learning some function beta that models the data coverage and then doing some policy combination. The way to construct meta controller borrows the MAB framework and does not contain enough novel investigations.

Author response:
- Technical contribution. 

Our paper is motivated by the fact that while many complex methods have been proposed for efficient exploration, the most commonly used ones are still simple methods such as $\epsilon$-greedy and entropy regularization [1,2]. Previous literature attributes this to two key challenges of other methods: their increased complexity and lack of generality [3,4]. In response, our contribution is a simple yet effective exploration method which enhances the performance of DQN. Our idea is simple and straightforward to implement, which we consider an advantage.

Reviewer comment (Theory):
The paper has no theoretical results. The proposed method is not built based on foundational theory but instead some intuitions. This makes the methodology not very trustworthy.

Author response:
- No theoretical results. 

Our paper is experimental, focused on empirical results rather than theoretical analysis. Other well-known simple yet effective practical exploration approaches include $\epsilon z$-greedy [4] and RND [5]. Our work builds on and extends this empirical tradition, and contributes to the understanding of exploration strategies through a series of experiments.

Reviewer comment (Experiments):
The experiments are performed only on synthetic data and very simple environments. It remains not clear how well the method generalizes to more complicated scenarios such as continuous environment.

Author response:
- Not clear how well .. generalizes to more complicated scenarios such as continuous environments. 

As an improvement for DQN, our method does not apply directly to continuous action spaces. However, it is suitable for state spaces that are continuous/high-dimensional. Our current experiments are all on image-based domains. While MinAtar is simpler than Atari it is also high-dimensional, in 4 out of 5 games (all except Breakout) trajectories do not overlap in practice [6]. Even for minigrid, the tasks are not simple for current algorithms. For example, in our results for LavaCrossing-Hard, no algorithm can learn to reliably reach a goal state within 5 million steps. This is still a very challenging testbed worth deeper research.