Paper: The Curse of Diversity in Ensemble-Based Exploration (ICLR-2024)

Reviewer comment (Theory):
I’m lost in the motivation of using ensemble in **policy learning**. As has been demonstrated in [EDAC] and [REDQ], I acknowledge that using ensemble learning for the **value function** could lead to improved performance, as the value can be more accurate, with uncertainty. But what is the motivation for having **multiple policies** for ensemble (because they are sample generators, rather than learners). Should not those samplers aim at more efficiently decreasing the uncertainty in the value function?

Author response:
Thank you for liking our presentation and providing detailed feedback and suggestions! Please see our response below

## On high-level motivation

> As has been demonstrated in [EDAC] and [REDQ], I acknowledge that using ensemble learning for the value function could lead to improved performance, as the value can be more accurate, with uncertainty. But what is the motivation for having multiple policies for ensemble

We emphasize that even though both (1) ensemble-based exploration methods such as Bootstrapped DQN and Ensemble SAC and (2) methods such as EDAC and REDQ use ensembles, they use ensembles in completely different ways and purposes. Bootstrapped DQN and Ensemble SAC use ensembles *purely for better exploration*.  In contrast, methods such as EDAC and REDQ use ensembles *purely for improving value estimation*. The concrete differences are detailed below, which should clear up your confusion:

* The number of policies:
  * Both Bootstrapped DQN and Ensemble SAC maintain $N$ policies. Note that for Bootstrapped DQN the $N$ policies are *implicitly defined by the $N$ value functions*. Most importantly, all $N$ policies are used to collect samples. Concretely, for each episode, one policy is sampled for interaction. **The motivation for having $N$ policies in Ensemble SAC is the same as that of having $N$ different value functions (and hence $N$ *implicit* policies) in Bootstrapped DQN: to explore the state space with a diverse set of policies (explicit or implicit) concurrently in a temporally coherent manner**.
  * EDAC and REDQ are simply not designed for exploration. They only have one policy and it is the only one used to interact with the environment.
* How the value functions are used:
  * In Bootstrapped DQN and Ensemble SAC, *each value function $Q_i$  evaluates one  policies $\pi_i$*. These value functions are *independent* (though they may share networks) in the sense that the TD regression target for each ensemble member is computed independently with their target network. Concretely, the target for $Q_i(s, a)$ is $r(s, a) + \gamma \bar{Q}_i(s’, a_i’)$, where $a_i’\sim\pi_i(a_i’|s’)$. They are not designed to produce more accurate value estimations than single-agent Double DQN or SAC. There is no explicit mechanism that encourages more accurate value estimations such as penalizing Q-value based on uncertainty (though the diverse data may help implicitly).
  * In EDAC and REDQ, *all the $N$ value functions $Q_{1:N}$ evaluate the same single policy $\pi$*. The $N$ value functions **share a single TD regression target** that is aggregated from all target networks. For examples, in EDAC, the target for $Q_i(s, a)$ -- regardless $i$ -- is $r(s, a) + \gamma\min_j \bar Q_j(s’, a’)$ (ignoring the entropy term), where $a' \sim\pi(a'|s')$. Note how the target values are aggregated with the $\min$ operator to counter overestimation.

In summary, the goal of having multiple policies in Ensemble SAC is to promote better exploration. This is the exactly same motivation as having multiple value functions (which implicitly define multiple policies) in Bootstrapped DQN. In contrast, EDAC and REDQ are not designed for better exploration; there is only one policy in them and it is the only policy interacting with the environment.

Reviewer comment (Experiments):
In BootstrapDQN, different samples are assigned different weights to analog bootstrap sampling. It would not be surprising to see individual BootstrapDQN perform worse than DDQN. To verify the claim that *BootstrapDQN’s performance improvement is purely based on voting* made by the authors, a comparison between Bootstrap Sampling/ normal sampling; and a comparison between majority voting and [text incomplete in original].

Author response:
## On BootstrapDQN

> In BootstrapDQN, different samples are assigned different weights to analog bootstrap sampling. It would not be surprising to see individual BootstrapDQN perform worse than DDQN. To verify the claim that BootstrapDQN’s performance improvement is purely based on voting made by the authors, a comparison between Bootstrap Sampling/ normal sampling; and a comparison between majority voting and

As mentioned in Appendix A, we do *not* use data bootstrapping in
Bootstrapped DQN. We just compute minibatch losses as usual without any masking or weighting. Note the original Bootstrapped DQN paper does not find bootstrapping to be useful in Atari games (see Figure 5b in the Bootstrapped DQN paper). As a result, they do not use bootstrapping in their Atari experiment. This is likely the standard way to implement Bootstrapped DQN for Atari.

In our revision, we also mentioned this point in Section 2 of the main text to avoid confusion for future readers. 

For completeness, we also tested Bootstrapped DQN with data bootstrapping (with masking probability $p=0.75$). The results are shown in Figure 20 of Appendix D.3.7. As you suggested, it does damage performance. This is not surprising because masking out samples essentially reduces the number of transitions each member has access to. It also effectively reduces the batch size.

We also emphasize that we do not claim “BootstrapDQN’s performance improvement is purely based on voting” (even though our result does suggest that voting is much more important than previously attributed). We have revised Section 3.1 to avoid this misinterpretation.

Reviewer comment (Reproducibility):
Figure 2: The reported results of SAC can not match the open-source implementations. This might be due to the usage of a different size of buffer (200k). Could the authors please explain more about this?

Author response:
> Figure 2: The reported results of SAC can not match the open-source implementations. This might be due to the usage of a different size of buffer (200k). Could the authors please explain more about this?

This is correct. As mentioned in the **Environments and algorithms** paragraph of Section 2 this is done for analysis purposes. Results with larger replay buffers are presented in Figure 4 (right) and Figure 7 (right), which match the performance of open-source implementation, except on HalfCheetah where our implementation slightly underperforms. Note that we use the v4 MuJoCo environments which have some technical differences from the v2 environments the original JAXRL repository used to report results. We also slightly adjusted the implementation of the actor (tanh instead of hard clipping for the `log_std` parameter) for better stability. Therefore, some minor differences in performance are expected.

Reviewer comment (Evaluation):
The performance of CERL seems marginal in Figure 7. Also in Figure 7, BootstrapDQN seems not helpful at all.

Author response:
## On Performance

> The performance of CERL seems marginal in Figure 7. 

Figure 7 only shows aggregate performance over 55 games, which can sometimes be misleading. We invite the reviewer to look at Figure 13 of Appendix D.2.3, which provides a per-game performance summary and should better reflect the benefits of CERL. Also, in MuJoCo tasks, CERL provides significant improvements when the curse of diversity is severe (i.e., when the replay buffer is small), which is exactly what CERL is proposed to fix. On the other hand, we emphasize that our work is the first to identify the curse of diversity phenomenon and CERL is only a first step towards mitigating it.

> Also in Figure 7, BootstrapDQN seems not helpful at all.

As mentioned above, aggregate performance can be misleading. Please see Figure 2 top right, where we provide a per-game performance summary. In this plot, we see that Bootstrapped DQN behaves very differently from Double DQN, and whether it is helpful is highly environment-dependent.

Reviewer comment (Novelty):
This work [https://arxiv.org/pdf/2209.07288.pdf] discussed the ensemble learning that trades off between exploration and exploitation using different reward values may also be worth a discussion.

Author response:
## On related work

Thanks for pointing out these related works and we have included them in the related work section. 

We hope our response has addressed your concerns. Please let us know if you have any further questions!

Reviewer comment (Novelty):
This work [https://openreview.net/forum?id=NOApNZTiTNU] and [REDQ] discussed the usage of the ensemble in Q-learning. REDQ has been considered to be state-of-the-art since 2021.

Author response:
## On related work

Thanks for pointing out these related works and we have included them in the related work section. 

We hope our response has addressed your concerns. Please let us know if you have any further questions!

Reviewer comment (Experiments):
"First, it challenges some previous work’s claims that the improved performance of approaches such as Bootstrapped DQN in some tasks comes from improved exploration" (p.3): as mentioned earlier in the paper, ensemble methods are widely used to drive exploration into unseen regions of the state-action space. However, the authors claim that "Surprisingly, simply aggregating the learned policies at test-time provides a huge performance boost in many environments" (p.3), and proceed to downplay the effect of exploration in favor of an explanation based on "aggregation". This is a problem in statements like "the two cases[, DQN(indiv.) and Double DQN,] have access to the same amount of data and have the same network capacity" (p.3), as the change in exploration means that the two algorithms are based on different data distribution and will therefore behave differently, irrespective of the on- and off-policy-ness of the data. This potential interaction between exploration (which part of the state-space is covered) and on-policy-ness (which member got to sample it) cannot be distinguished with the presented experiments, but the text reads as if the latter is the only obvious conclusion.

Author response:
Thank you for raising the point regarding exploration/coverage. We have included a detailed discussion on this in the “Remark on data coverage” paragraph at the end of Section 3.2. The main points are summarized as follows:

1. State-action space coverage is *purely a property of the data distribution*, but off-policy-ness depends on *both the data distribution and the policies*. So indeed they are different.
2. Our p%-tandem experiment disentangles these two aspects: the active and passive agents are trained on the same data (thus same coverage), but experience different degrees of “off-policy-ness”.
3. As a result, our experiment indicates that “off-policy-ness” is detrimental and sufficient to cause the performance degradation we see. At the same time, we recognize that it *does not* imply whether wider state-action space coverage is beneficial (more likely and intuitive) or harmful.
4. On the other hand, as you mentioned, the fact that BootDQN (indiv.) performs better than the passive agent indicates that wider coverage is *likely* highly beneficial. Reasoning: 
   1. Due to the presence of $N$ different policies, the individual agents in BootDQN *likely* suffer from more severe off-policy-ness than the passive agent.
   2. Therefore, if the wider coverage in BootDQN was not beneficial, due to more severe off-policy-ness BootDQN (indiv.) should have performed significantly worse than the passive agent.
5. However, unlike the comparison between the active and passive agents which is perfectly controlled, the comparison between BootDQN (indiv.) and the passive agent may have other (unexpected) confounders. Therefore we avoid making conclusive claims about the effect of state-action space coverage in our paper.

We have also added “even though they are trained on different data distributions and thus are expected to behave differently,” before the statement “the agents in these two cases have access to the same amount of data and have the same network capacity” in Section 3.1 to emphasize that the data distributions in these two cases are different.

To avoid misinterpretation, we have revised the second paragraph in Section 3.1 to emphasize that even though we show that majority voting plays a more important role than previously attributed, it does not indicate that exploration/state-action space coverage is not important or beneficial.

Reviewer comment (Evaluation):
"These results confirm our hypothesis regarding the cause of the observed performance degradation" (p.5). This is dangerous abductive reasoning: experimental evidence that is consistent with your hypothesis does not make your hypothesis true (one can only falsify a hypothesis). This may sound like a nitpick, but the paper does not discuss the exploration effect of ensembles enough. For example, in Figure 3 (middle), the BootDQN (indiv.) curve is significantly above the Passive (10%) curve. If indeed Algo 2 and Algo 3 are exactly equivalent except for how often they sample the other (non-passive) ensemble members, and *only* this affects the performance due to off-policy sampling, then both curves (which have seen the same fraction of off-policy samples) should perform identical. The fact that they do not shows that there must be another effect at work here. This could be the exploration, but the experiments do not allow to distinguish these potential effects. To be fair, the reviewer also does not know how to distinguish exploration and on-policy-ness with the given setup, but another setup might be able to (e.g. with intrinsic-reward methods where the exploration can be separated from who acts). The presented analysis is strong enough for publication (a new setup is not needed), but these ambiguities must be discussed.

Author response:
Thank you for raising the point regarding exploration/coverage. We have included a detailed discussion on this in the “Remark on data coverage” paragraph at the end of Section 3.2. The main points are summarized as follows:

1. State-action space coverage is *purely a property of the data distribution*, but off-policy-ness depends on *both the data distribution and the policies*. So indeed they are different.
2. Our p%-tandem experiment disentangles these two aspects: the active and passive agents are trained on the same data (thus same coverage), but experience different degrees of “off-policy-ness”.
3. As a result, our experiment indicates that “off-policy-ness” is detrimental and sufficient to cause the performance degradation we see. At the same time, we recognize that it *does not* imply whether wider state-action space coverage is beneficial (more likely and intuitive) or harmful.
4. On the other hand, as you mentioned, the fact that BootDQN (indiv.) performs better than the passive agent indicates that wider coverage is *likely* highly beneficial. Reasoning: 
   1. Due to the presence of $N$ different policies, the individual agents in BootDQN *likely* suffer from more severe off-policy-ness than the passive agent.
   2. Therefore, if the wider coverage in BootDQN was not beneficial, due to more severe off-policy-ness BootDQN (indiv.) should have performed significantly worse than the passive agent.
5. However, unlike the comparison between the active and passive agents which is perfectly controlled, the comparison between BootDQN (indiv.) and the passive agent may have other (unexpected) confounders. Therefore we avoid making conclusive claims about the effect of state-action space coverage in our paper.

We have also added “even though they are trained on different data distributions and thus are expected to behave differently,” before the statement “the agents in these two cases have access to the same amount of data and have the same network capacity” in Section 3.1 to emphasize that the data distributions in these two cases are different.

We agree that there could be other factors at play (e.g., state-action space coverage, as discussed), even though we do believe that our results provide strong evidence that off-policy-ness plays a major role in the observed performance degradation. We have changed our statement to “These results offer strong evidence of a connection between the off-policy learning challenges in ensemble-based exploration and the observed performance degradation” as a more accurate conclusion of our results.

Reviewer comment (Experiments):
While the reviewer liked most experiments (e.g. Figure 3 and 4 are great!), Figure 5 seems to test DQN with a very small replay buffer, where even the aggregated ensemble does not perform as well as Double DQN. This makes some conclusions suspect. For example, does the aggregated performance improves when more layers are shared? Or does the ensemble become indistinguishable from the baseline? If the authors would have picked a larger replay buffer (e.g. 4M), the performance would improve in the first and worsen in the second case. This would have an impact, as it shows that sharing layers in the ensemble reduces the positive effect it can have (e.g. by exploration).

Author response:
In Figure 5 we use a replay buffer of 1M transitions. This is the “default size” used in most DQN variant papers (e.g., Rainbow, QR-DQN, M-IQN). For analysis purposes, we select the four environments where the curse of diversity is obvious based on preliminary experiments, and that’s why both their `indiv.` and `agg.` performance is worse than Double DQN. For completeness, we have performed the “layer sharing” experiment using a replay buffer size of 4M transitions. The results are shown in Figure 18 of Appendix D.3.5. This does allow BootDQN (agg.) to outperform Double DQN in 3 environments, but the trade-off between the advantages and disadvantages of diversity remains: as we increase the number of shared layers and hence reduce diversity,

1. The curse of diversity, i.e. the gap between Double DQN and BootDQN (indiv.), reduces.
2. The performance gain we get from majority voting, i.e. the gap between BootDQN (agg.) and BootDQN (indiv.), also reduces. An exception is Space Invaders, where the gap seems to slightly increase when L increases from 0 to 3, which might be related to certain properties of this game (that we do not understand at this point).

The Nature DQN architecture we use in this work has 5 layers, so BootDQN (L=5) is equivalent to Double DQN. Therefore, further increasing the number of shared layers will not improve performance. 

Given the above results, we realize that the statement “Most importantly, neither of these techniques allows Bootstrapped DQN (agg.) to outperform Double DQN in a statistically significant way” in our initial submission can be environment- and hyperparameter-dependent. Besides, this statement may obfuscate the main points we are making (i.e., the trade-off between the advantages and disadvantages of diversity) with Figure 5, so we have removed it in our revision.

Reviewer comment (Theory):
The main hypothesis of the paper is that "the curse of diversity" hurts individual member performance due the large number of off-policy samples. This makes sense, as an individual member can make 90% of all decisions correctly and still have poor performance, whereas a majority vote of the ensemble will agree with high certainty on the correct action. Ultimately this is the reason why BootDQN explores well. However, CERL seems to have almost as much diversity as the L=0 ensemble. CERL is a representation learning algorithm, which seems to be at odds with the main hypothesis. Contrast this with BootDQN which switches ensemble members mid-episode. If this algorithm would improve individual performance, the reviewer would accept it as a fix of the identified problem. With CERL presented as the "solution", the reviewer wonders: is the underlying problem is indeed "diversity"? Please discuss why representation learning fixes the off-policy-ness of the data (there are some attempts, but it is still vague).

Author response:
As mentioned at the beginning of Section 3, two main factors contribute to the curse of diversity: (1) the off-policy-ness of the data w.r.t the agents (which largely depends on ensemble diversity) and (2) the agent’s inefficiency to utilize such off-policy data. **CERL does not fix (1) but tries to fix (2)**. It is easier to understand this point by realizing that a major part of the difficulty of off-policy learning in *deep* RL is about generalization: how well an agent can learn from off-policy data depends a lot on how well the Q network can *generalize* to state-action pairs that have high probability under the current policy but are *under-represented* in the data. This is often studied under the name of “extrapolation error” in off-policy/offline RL [1, 2]. Representation learning can potentially improve the network’s generalization abilities and thus can improve the agent’s capability to learn from off-policy data.

An example of “proper representations” helping off-policy learning is given in the tandem RL paper [1]. They show that if the passive agent uses the active agent’s representation (by sharing the first few layers but not allowing the passive agent to update them), the tandem effect can largely be mitigated. In the context of ensemble RL, the equivalent of this approach is just sharing layers of the Q-networks, which we have shown can compromise diversity and the advantages of ensembling (e.g. exploration and majority voting). The motivation of CERL is just to achieve the similar representation learning effect of network sharing (i.e., learning multiple value functions at the same time) without actually needing to share the networks

We have included a sentence in the first paragraph of Section 4 to further clarify the motivation of CERL.

 We have tested this variant In Figure 17 of Appendix D.3.4 . Specificially, we switch ensemble members at each step instead of only at the start of an episode. As shown in the result it does not mitigate the curse of diversity. This is not surprising because it does not change the proportion of “self-generated” data for each agent in the replay buffer.

Reviewer comment (Experiments):
Perhaps this is out of scope of the paper, but I do feel it's difficult to draw conclusions about deep RL more broadly without investigating distributional RL. For example, this paper (https://openreview.net/pdf?id=ryeUg0VFwr) shows that distributional RL will likely do better with this off-policy data. It would be interesting to investigate the extent of this phenomena in the distributional setting.

Author response:
This is a great suggestion and we have added experiments on four Atari games with QR-DQN. The results are presented in Figure 16 of Appendix D.3.3. The significant performance gap between QR-DQN and Bootstrapped QR-DQN (indiv.) shows that the curse of diversity is also present in distributional RL. Note that due to limited time, we are not able to finish the full 200M frames of training. We will include the full results when the experiments are finished.

Reviewer comment (Experiments):
CERL does seem to scale poorly with the ensemble size, since if the ensemble size is N, there are N x N Q-functions. In practice, this does not seem like a major problem.

Author response:
In practice, CERL scales very well with ensemble size. As mentioned in the “Method” paragraph in Section 4, to implement CERL we only need to increase the output dimension of the *last linear layer* of the network (note this is the same architectural modification involved in distributional RL). So even though CERL requires N x N Q-functions as opposed to N Q-functions in a standard ensemble, the increase in the number of parameters and wall clock time is very small. For example, adding CERL to Bootstrapped DQN (L=0, N=10) increases the number of parameters by no more than 5% and the increase in wall clock time is barely noticeable. Even if we use N=20 which is already much larger than commonly used in the literature, applying CERL increases the number of parameters by no more than 11% and the increase in wall clock time is less than 5%.

Reviewer comment (Presentation):
For the bar plots in Figure 2, it might be helpful to plot the top plot minus the bottom plot to highlight the point.

Author response:
Thanks for your suggestion! We have included this figure in Figure 12 of Appendix D.2.2. We do not include it in the main text to save space for other changes, and also because this plot mainly shows the benefits of majority voting which is not the primary focus of our work.

Reviewer comment (Experiments):
Did you try different levels of passivity? Other than 10%?

Author response:
We have also tried 0%, 5%, and 20% in four Atari games. The results are shown in Figure 19 of Appendix D.3.6. As expected, as the degree of passivity reduces (i.e., larger $p$), the active/passive agents’ performance gap also reduces.