From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Looking Forward to Your Response! 
comment: Dear reviewer c6V8,

Thanks again for taking the time to read our paper and provide feedback. We are glad to see that you agree that not many existing works have explicitly and extensively studied individual learners' performance in an ensemble. With regard to the papers you referenced in your latest review, we have also pointed out that they either do not use data sharing or do not present the same empirical analysis in this paper. Given these discussions, we believe that we can agree that our work brings useful and novel knowledge to the community. We would greatly appreciate it if you can re-assess the novelty of our work based on our responses. Also, any further feedback is welcomed!

Thanks and regards,

Authors

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Looking Forward to Your Response!
comment: Dear reviewer gUQE,

Thanks again for taking the time to read our paper and provide feedback. Since this is the last day of the discussion phase, we would like to remind you of our response to your review. In particular, we emphasize that we clearly indicated in the paper that the ensemble strategy we studied is present in many previous works and this is not our claimed contribution. We would greatly appreciate it if you can re-evaluate our paper based on our claimed contributions detailed in the paper and in our [General Response](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ).

Thank you very much!

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Did Our Response Address Your Concerns?
comment: It has been a while since we post our response to your latest comment so we would like to kindly ask you if our response has addressed your concerns. We are very happy to answer any further questions that you may have.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Did Our Response Address Your Concerns?
comment: Dear reviewer,

It has been a while since we post our response so we would like to kindly ask you if our response has addressed your concerns. We are very happy to answer any further questions that you may have.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Reminder of Our Latest Response
comment: Dear reviewer,

Since this is the last day of the author rebuttal period, we would like to remind you that we've posted [a response to your latest comment](https://openreview.net/forum?id=g4RxrIB52M9&noteId=KcU24SHr27). Please let us know if you have any further concerns or questions. We are happy to have a further discussion and provide more details.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Rebuttal Reminder
comment: Dear reviewer,

Since this is the last day of the author rebuttal period, we would like to ask if you have any further questions or concerns. We hope you can kindly increase your score if our response has sufficiently addressed your concerns (in particular, regarding novelty). We are also happy to engage in further discussions.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Authors' Response
comment: Thank you for your response! Please see our reply below

1. **Regarding** "*a few papers seem to have touched this issue to a certain extent*". Thank you for pointing us to these papers and we've included them in the Related Work section in our revision. Note that "Distributional Reinforcement Learning with Ensembles" uses separate replay buffers for different ensemble members (see their Algorithm A1). **It is not even using data sharing**, let alone other differences. Regarding "Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning", what this paper about MARL shows is "when training multiple agents, using data sharing is better than not", which as we stated is an obvious fact in the context of standard ensemble RL and therefore not even mentioned in our paper. Also, we again emphasize that **showing the usefulness** of data sharing is not equivalent to demonstrating its detailed effect on the ensemble members. Therefore, we believe these two papers do not undermine the novelty of our work.

2. **Regarding ED2**.  Indeed, not many additional techniques are used in ED2. In fact, the only major one is just average policy at test time. But as we have pointed out, this average policy is **crucial** in ED2, as has been shown in the ablation study in Figure 30 in their paper. And by "this means that in their case, data sharing is not enough, and policy aggregation is necessary for good performance" we mean, as shown in their Figure 30, `ED2 (ours)` is much better than `ED2 (single actor eval.)` and  `ED2 (single actor eval.)` is not too different from `SOP`. Note that
   * `ED2 (ours)` is just the basic ensemble strategy with data sharing (as in this paper) + average policy at test time
   * `ED2 (single actor eval)` is just the basic ensemble strategy with data sharing, **without any additional technique**.
   * `SOP` is just `ED2`, but with ensemble size 1.
   
   Therefore,  the fact that  `ED2 (single actor eval)` is roughly the same as `SOP` indicates "data sharing is not enough" in ED2 (to surpass the performance of only training a single agent), and the fact that `ED2 (ours)` is much better than  `ED2 (single actor eval)` indicates "policy aggregation is necessary for good performance" in ED2.

   Again, we emphasize that the existence of papers that use data sharing or only superficially investigate the **usefulness** of data sharing does not undermine the novelty of our work. As we've pointed out, none of our listed contributions in our paper or [General Response](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ) is present in ED2.
3. **Regarding** "*we believe few can really accurately predict how data sharing will impact individual agents when we share the data by merging the buffer,*". Note we originally used this statement in our rebuttal to support the fact that **our analysis experiment** provides novel knowledge and insights to the community. We sure agree that "accurate prediction of the impact of data sharing remains difficult" **in general** because it certainly depends on conditions such as the environment, algorithm, and whether peer learning is stable, but we are not sure how this is relevant to the novelty of the insights offered by the **analysis experiment in our paper**.

Besides, we believe we have provided a very detailed description of the novelty and contributions of this work in our [General Response](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ) and in the specific response to your review in order to address your concern about the novelty of this work. Please let us know if these have sufficiently addressed your concern and if so, we hope you can kindly raise the score. We are also very happy to engage in further discussions.

From: ICLR.cc/2023/Conference/Paper3413/Reviewer_c6V8
title: Thank you for the response
comment: Thank the authors for their response. I agree that not many existing works have explicitly and extensively studied individual learners' performance in an ensemble. However, a few papers seem to have touched this issue to a certain extent. For example, the paper "Distributional Reinforcement Learning with Ensembles" studied the relationship between individual agents and ensembles. The paper "Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning" also studied the importance of experience sharing in a multi-agent context.

I believe ED2 mainly trained a group of agents with data sharing and not much additional techniques were used. The ED2 paper also considered using the action produced by one policy at each time step (not just averaging the action outputs) during testing. Clearly as an ensemble algorithm, the aim of ED2 is to achieve the best possible performance of an ensemble, rather than the performance of each individual agent in the ensemble. Regarding the statement "this means that in their case, data sharing is not enough, and policy aggregation is necessary for good performance", I am not sure what this means. In my opinion, data sharing in ED2 is used to achieve better policy aggregation, the common focus of ensemble learning.

As for the statement "we believe few can really accurately predict how data sharing will impact individual agents when we share the data by merging the buffer, especially how quickly and how drastically the agents will improve", I think this question depends on many factors, some have been covered in this paper but some may not. In view of the experiment results and findings in this paper, accurate prediction of the impact of data sharing remains difficult.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Response to Reviewer c6V8 (2/2)
comment: 5. **Regarding the “data sharing is sufficient for a significant improvement” statement.** We’ve included a comparison with some other algorithms in Table 4 in Appendix E.3 to demonstrate the improvement brought by using an ensemble with data sharing is truly significant. In short, standard SAC reaches an IQM of 569 and the state-of-the-art result is 805, while adding ensembling (either with or without reset) can often improve performance by roughly 40-140 depending on the replay ratio and whether we do reset (Figure 3).

   Also, with this statement we certainly do not mean that other techniques like policy aggregation are not useful (e.g., in ED2). However, we agree that our original account of this point can be a bit extreme and misleading. We’ve made this point clearer in Section 4.2 in our revision. We also changed our narrative to make it less extreme (e.g. “is sufficient” -> “has the potential”).
6. **Lack of theoretical analysis**. this is indeed a limitation of our work. Currently, it is not so obvious how we can approach data sharing from a theoretical point of view. However, we want to emphasize that data sharing can be used in any off-policy deep RL algorithms and the reason why it helps (i.e., peer learning) is quite intuitive and algorithm-agnostic, so we believe the insights presented in this work are transferable to different environments or even to other RL problems other than standard ensemble RL.

Minor concerns:

* We’ve fixed the missing citations and improved the overall clarity of the paper.
* Subsampling: we simply subsample the buffer at uniform intervals (e.g., with an ensemble size of 10, we will take the 1st, 11th, 21st, 31st transitions, etc.) so it is essentially uniform sampling. We want to emphasize that the buffer merging experiment is just for demonstration purposes, and the goal of subsampling is just to prevent unfair advantage over the “separate buffer” baseline (e.g., merging 10 buffers of size 500k creates a buffer with 5M transitions, while the “separate buffer” baseline only has 500k transitions in each agent’s buffer at the same point). It is not an algorithmic design, and thus there is no reason to bias towards any ensemble member. Please refer to Appendix C for more details regarding this set of experiments.

We hope that our response addresses your concern and, in particular, clarifies the novelty of our work. We are happy to have a further discussion and provide more details.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Response to Reviewer c6V8 (1/2)
comment: Thank you for taking the time to write this thorough review. We hope the following can address your concerns:



1. **Overall novelty:** first, we would like to invite you to read our discussion about novelty in our [General Reponse](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ). We will further address some of your specific comments as follows:

2. **About ED2**: first, note that none of our listed contributions in our paper and [General Reponse](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ) is present in ED2. Also, work that shows that using an ensemble is better training a single agent only  superficially shows the usefulness of data sharing rather than its impact on the individual ensemble members. The latter is important for understanding the usefulness and inner workings of data sharing and therefore ensemble methods. In addition, none of previous work has even rigorously demonstrated that using an ensemble **without any additional technique** is better training a single agent (see the fourth point of the outlined contributions in our [General Reponse](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ)).


   Besides, ED2 uses an average policy at test time, so one cannot simply say that its performance improvement comes from data sharing and improved exploration. In fact, the ablation experiment (Figure 30 in their paper) shows that using a single policy instead of the average policy for evaluation can be much worse. This means that in their case, data sharing is not enough, and policy aggregation is necessary for good performance. Note this does not contradict our arguments and results because, as we show, how much one can benefit from data sharing depends on some conditions (e.g., whether the agent can successfully learn from each other’s data). We haven’t looked into why data sharing is insufficient in their case but that is certainly part of our future work.

3. **Regarding** “*it is a common belief in the research community data sharing can help to improve the learning performance of individual agents in an ensemble*”: This looks like a strong claim, so we would like to discuss it in detail below:

   * If what you mean is “when training an ensemble, using data sharing is better than not using it”: it is indeed a common belief. It is actually somewhat obvious, and therefore we did not even mention it in our paper.
   * If what you mean is “training an ensemble with data sharing (without any additional techniques) can be much better than only training a single agent”: this is indeed a claim in our work, but we argue that it is not obvious. The main reason is that when using an ensemble, the total environment interaction budget is shared by all ensemble members. For example, when using an ensemble size of 10 and an interaction budget of 1M steps, each ensemble member only gets to interact with the environment for 100k environment steps. Therefore, unless the benefits of data sharing (and exploration) outweigh this negative aspect, this statement will not be true. 
   * If you are indicating the results in our buffer merging experiments can be expected (Figure 1) and therefore do not provide any new information to the community: the results are intuitive, but not **obvious**. We believe few can really accurately predict how data sharing will impact individual agents when we share the data by merging the buffer, especially how quickly and how drastically the agents will improve. Also, we show the effects are diverse: sometimes it causes all ensemble members to match or outperform the previously best agent (hopper-hop) where there is clear variance within the ensemble, and sometimes it helps even when there is no clear variance within the ensemble (fish-swim). Therefore, we believe this experiment presents useful knowledge and insights that are previously unknown to the community. We’ve also included additional plots of individual ensemble members’ performances in Figures 1 and 2 to highlight these points.
   * Please let us know if you are actually indicating something else. We are very happy to engage in a further discussion about this.

4. **Regarding the Primacy Bias**: we agree that it is not that surprising to see that adding resets to an ensemble can improve performance. However, we argue that they are a particularly interesting pair instead of just some random combination. Please see the first point of the outlined contributions in our [General Reponse](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ) for more details. We've also updated Figure 2 to include additional plots that show individual ensemble's performance and added an extended discussion (highlighted in blue in Section 4.2 in our revision) to help better understand why they are an interesting combination.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Response to Reviewer gUQE (2/2)
comment: References:

[1] Anschel, O., Baram, N., and Shimkin, N. Deep reinforcement learning with averaged target DQN. CoRR, abs/1611.01929, 2016. URL http://arxiv.org/abs/1611.01929.

[2] Peer, O., Tessler, C., Merlis, N., and Meir, R. Ensemble bootstrapping for q-learning. In International Conference on Machine Learning, pp. 8454–8463. PMLR, 2021.

[3] Litian Liang, Yaosheng Xu, Stephen Mcaleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, Roy Fox Proceedings of the 39th International Conference on Machine Learning, PMLR 162:13285-13301, 2022.

[4] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In NIPS, 2016.

[5] Piotr Januszewski, Mateusz Olko, Michal Kr ́olikowski, Jakub Bartlomiej Swiatkowski, MarcinAndrychowicz, Lukasz Kuci’nski, and Piotr Milo’s. Continuous control with ensemble deep deterministic policy gradients. ArXiv, abs/2111.15382, 2021.

[6] Zhenghao Peng, Hao Sun, and Bolei Zhou. Non-local policy optimization via diversity-regularized collaborative exploration. ArXiv, abs/2006.07781, 2020.

[7] Ge Liu, Rui Wu, Heng-Tze Cheng, Jing Wang, Jayden Ooi, Lihong Li, Ang Li, Wai Lok Sibon Li, Craig Boutilier, and Ed H. Chi. Data efficient training for reinforcement learning with adaptivebehavior policy sharing. ArXiv, abs/2002.05229, 2020.

[8] Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-policy actor-critic with shared experience replay. In ICML, 2020.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Response to Reviewer gUQE (1/2)
comment: Thank you for taking the time to read our paper and write the review. We address the outlined concerns as follows:

1. **Regarding novelty:** Please note that the ensemble strategy considered in our work is not our claimed contribution. We would like to invite you to read our [General Reponse](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ) for a more thorough description of the contributions of this work. Also, thank you for pointing us to relevant works in offline RL and multi-task RL. We’ve included these in our Related Work section. 

   As a side note, we would like to mention that the work you referenced [1, 2, 3] (and many more, including REDQ) uses ensembling only for improving value estimation (e.g., reducing overestimation bias), while the strategy considered in this paper is mainly for better exploration. This kind of work [1, 2, 3] uses a single (potentially aggregated) policy for exploration, and all ensemble members are trained to evaluate this policy. Therefore, they are essentially still only training a single agent. With the strategy considered in this work, each ensemble member interacts with its own environment instance and is trained against its own target network. These two approaches are fundamentally different, both in terms of the purpose of introducing ensembles and how the ensemble is used. Nevertheless, we still included [1, 2, 3] alongside existing citations to ensemble methods for value estimation in the Related Work section of our revision.

   Works that truly use this ensemble strategy are [4, 5, 6, 7, 8], and they are already cited in our paper. Again, we’ve never claimed that this ensemble strategy is our contribution, and our actual claimed contributions are listed in our paper and also our [General Reponse](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ).

2. **Comparison with REDQ**. As mentioned above, the strategy considered in this work is fundamentally different from works like REDQ. Specifically, besides the exploration strategy, the most significant difference is that REDQ aggregates the prediction results of the ensemble to obtain a shared value target for all ensemble members to reduce the overestimation bias (which is also their main contribution), while in ours each ensemble member has its own target network and uses the standard clipped double Q-learning. 

   That said, one can still ask for a baseline in which there is an ensemble of critics and a single actor, while each member critic is trained against **its own target network** instead of some aggregate target. On its surface, this may look like a reasonable ablation, but it is not: when doing so, one has to aggregate the prediction results of the critics to provide a learning target for the actor. Therefore, the change is more than just the exploration strategy. A truly reasonable ablation experiment is just to compare with the case in which we only train a single agent (e.g., standard SAC), which is already present in our work.

   Nevertheless, for the sake of completeness, we still included a comparison with REDQ and also some other algorithms in Table 4 in Appendix E.3 in our revision. In short, REDQ is not really competitive on the benchmark we used in this paper, which contains 16 diverse environments from DMC.

3. **Sample efficiency.** Note that even though we use multiple environment instances, we keep the total environment steps the same as when only training a single agent. For example, with a 1M environment interaction budget and an ensemble size of 10, each ensemble member can only interact with the environment for 100k steps. This is listed in the experimental details in Section 5.1 and we’ve made this point clearer in the Preliminaries section in our revision. Also, as recommended, we’ve included an algorithm box in Appendix B in our revision to make the environment interaction and agent update process more explicit.

We hope that our response addresses your concern and clarifies the novelty of our work. We are happy to have a further discussion and provide more details.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Response to Reviewer ypJE
comment: Thank you for taking the time to read our paper and acknowledging the value of our insights. We address your comments as follows:

1. **Regarding novelty:** indeed, the insights are part of our main contributions. We would also like to invite you to read our [General Response](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ) for a more thorough description of the novelty of our work, especially on why introducing resets is not a trivial idea.
2. **Theoretical results:** this is indeed a limitation of our work. Currently, it is not obvious how we can approach data sharing from a theoretical point of view. However, we want to emphasize that data sharing can be used in any off-policy deep RL algorithms and the reason why it helps (i.e., peer learning) is quite intuitive and algorithm-agnostic, so we believe the insights presented in this work are transferable to different environments or even to other RL problems other than standard ensemble RL.
3. **Minor concerns**: we’ve fixed the typos and citations. Also, thank you for raising the issue about the color. We’ve changed to a colorblind-friendly palette in our revision.

We are happy to address any further concerns you may have.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: Response to Reviewer YR9c
comment: Thank you for taking the time to read our paper and write the review. Please note that the ensemble strategy considered in our work is not our contribution, and we’ve never claimed this or even implied this. We would like to invite you to read our [General Response](https://openreview.net/forum?id=g4RxrIB52M9&noteId=ul4Vs2kFFQ) for a thorough description of the contributions of this work.

We also hope you can elaborate on the lack-of-novelty argument and also provide specific references to papers that performed similar work so that we can better address the concern. Please note the existence of works that use ensemble for exploration (which we are more than aware of and are cited in this paper) does not undermine the novelty of our work unless our contributions listed in our General Response are present in their works.

We are happy to clarify any further concerns that you may have, especially about the novelty.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: General Response (3/3)
comment: # Updates to The Paper:

We also updated our paper to address the reviewers’ concerns and to improve overall clarity. **We’ve highlighted our major modifications in blue in our revision.**
Some notable updates are:

* Additional plots of the individual ensemble members’ learning curves to help better visualize the effect of peer learning and its interaction with reset (Section 4, Figures 1 and 2).
* Further elaboration on why ensemble + resets is interesting (Section 4.2). In particular, we added a discussion about the increased off-policyness and hence more severe primacy when using an ensemble.
* Comparison with other algorithms (Appendix E.3, Table 4)An algorithm box for the ensemble strategy used in this work (Appendix B, Algorithm 1)
* Citations to work in multi-task RL and offline RL (Section 2)
* Clarification of some technical details:
  * We emphasize that when using an ensemble the total interaction is kept the same as when we train a single agent (Section 2, the Ensemble Strategy Paragraph)
  * We elaborate on how we subsample the shared replay buffer upon merging (Appendix C)
* For the main results, besides showing scores with the best ensemble member, we also show the scores when we take the average performance of each ensemble (Section 5.1, Figure 3).
* Besides SAC, we also show the performance distribution within the ensemble for DrQ (Section 5.2, Figure 4)
* All figures now use a colorblind-friendly color palette.
* Some contents are moved to the Appendix (e.g., ensemble size ablation)

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: General Response (2/3)
comment: # Regarding novelty (continued):

2. We present a clear demonstration of the effect of data sharing  (i.e., the peer learning process) on the performance of individual ensemble members via the buffer merging experiment (Section 4.1). We believe that it is not well understood in the research community that this sudden merging of the replay buffers would cause most ensemble members to rapidly align themselves to the previously best member or even outperform it in just a few thousand steps (e.g., hopper-hop and fish-swim in Figure 1). 

3. We show the existence of a failure case of peer learning and hence data sharing (e.g., hopper-hop and quadruped-run in Figure 2). This demonstration may encourage future work to explicitly evaluate whether or not peer learning is working when designing and diagnosing ensemble algorithms, for example by plotting the performance distribution of the ensemble as done in Figure 4 of our work.

4. We show that this strategy alone has the potential to provide a significant gain in performance over just training a single agent, **without the use of any additional techniques** (Figure 3). We comment on how previous works’ support for this statement is absent or insufficient:

   1. Bootstrapped DQN [4] uses a voting policy (see Appendix D.1 in [4]) for evaluation yet does not provide an ablation study. 
   2. ED2 [5] uses an average policy for evaluation and provides an ablation study, yet they show that this basic strategy is not sufficient in their case (see Figure 30 in [5])
   3. DiCE [6] uses a bunch of techniques and provides an ablation study. However, all of their ablations still contain additional techniques other than the basic ensemble strategy (e.g., in Figure 6(b) of their paper, w/o DR still has TSC loss).
   4. ABPS [7] and LASER [8] tackle the problem of efficient hyperparameter sweep and use different hyperparameters (potentially highly suboptimal) for different agents. It is thus hard to say how their results can generalize to the standard ensemble case in which all members use the same (tuned) hyperparameters. Besides, [8] uses $N$ times more samples when training $N$ agents.
   5. We also want to point out that there are many works that use ensemble for exploration purposes but do not use the strategy considered in this work (e.g., SUNRISE [9] uses UCB for exploration) so they are much less relevant. Note there are also works that use ensemble but not for exploration (e.g., purely for value estimation [10]) and these works are fundamentally different from what we study in this work.

   Note that this point is more for the purpose of supporting the important role of data sharing in ensemble methods. Also, while previous works’ account of the exploration benefits of ensembling implied that this strategy alone has the potential to provide a significant gain in performance [4], we present a rigorous validation of this statement.



We argue that the above points jointly provide a more complete picture for understanding the effectiveness of ensemble methods and can promote better exploitation of data sharing in deep RL. We believe these contributions are novel and can benefit the RL community.

[1] Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. In NeurIPS, 2021. 

[2] Aviral Kumar, Justin Fu, G. Tucker, and Sergey Levine. Stabilizing off-policy q-learning via boot-strapping error reduction. In NeurIPS, 2019

[3] Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron C. Courville. The primacy bias in deep reinforcement learning. In ICML, 2022.

[4] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In NIPS, 2016.

[5] Piotr Januszewski, Mateusz Olko, Michal Kr ́olikowski, Jakub Bartlomiej Swiatkowski, MarcinAndrychowicz, Lukasz Kuci’nski, and Piotr Milo’s. Continuous control with ensemble deep deterministic policy gradients. ArXiv, abs/2111.15382, 2021.

[6] Zhenghao Peng, Hao Sun, and Bolei Zhou. Non-local policy optimization via diversity-regularized collaborative exploration. ArXiv, abs/2006.07781, 2020.

[7] Ge Liu, Rui Wu, Heng-Tze Cheng, Jing Wang, Jayden Ooi, Lihong Li, Ang Li, Wai Lok Sibon Li, Craig Boutilier, and Ed H. Chi. Data efficient training for reinforcement learning with adaptivebehavior policy sharing. ArXiv, abs/2002.05229, 2020.

[8] Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-policy actor-critic with shared experience replay. In ICML, 2020.

[9] Kimin Lee, Michael Laskin, A. Srinivas, and P. Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In ICML, 2021

[10] Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without a model. ArXiv, abs/2101.05982, 2021.

From: ICLR.cc/2023/Conference/Paper3413/Authors
title: General Response (1/3)
comment: We thank all reviewers for taking the time to read our paper and provide useful feedback. Please see our general response below: 

# Regarding novelty:

Some reviewers (YR9c and gUQE) seem to have understood that the ensemble strategy studied in this work is our claimed contribution. We clarify below that the use of this ensemble strategy in RL is not novel, nor is it our claimed contribution. We would also like to take the chance to further elaborate on the novelty of our work for all reviewers.

First, we emphasize that we clearly indicated in the paper that we are studying an **existing** ensemble strategy that underlies many previous ensemble methods. For example:

1. The first two sentences of this paper: “Amid their popularity in supervised learning, ensemble methods have also proven to be effective in deep reinforcement learning (RL) for a wide range of purposes (Osband et al., 2016; Chua et al., 2018; Chen et al., 2021; Schmitt et al., 2020; Agarwal et al., 2020; Liu et al., 2020). Underlying some of these methods is a simple principle: training a set of independent agents while sharing the data among them.”
2. In the Related Work section: “...the simple ensemble strategy…considered in this work underlies many previous works…”
3. In the Preliminaries section: “...In this work, we consider a simple form of an ensemble strategy that underlies many previous works…”

An accurate characterization of this work will be twofold: **1) an empirical study of the data sharing aspect in this ensemble strategy and 2) the novel introduction of resets into this ensemble strategy**. We outline these points in detail as follows:

1. We first emphasize that introducing resets to this ensemble strategy is one of our main contributions. We agree that both are existing techniques, and also it is not that surprising that ensemble + resets will be better than ensemble alone since resetting also helps when just training one agent. However, we argue that maintaining the ability to learn when using an ensemble is particularly important and therefore this is **an important combination in its own right**:
   * How much we can benefit from using an ensemble strongly depends on how much the agents can benefit from data sharing, which in turn depends on how well they can learn from the diverse data generated by other agents through peer learning. Therefore, maintaining the agents’ ability to learn is more explicitly required when using an ensemble. Furthermore, without our empirical investigation into the nature of the peer learning process that underlies data sharing, one would likely not consider the introduction of resets. We contend that **the peer learning process** and **its failure cases** are currently not well recognized in the RL community. To highlight these points, we’ve added additional plots that show the individual ensemble members’ performance in Figures 1 and 2.
   * In an ensemble, individual agents suffer more from the primacy bias due to the increased off-policyness, and therefore resets are more important. **Note that when using an ensemble, we keep the total interaction steps the same as when training a single agent (e.g., 100k interactions for each of the 10 members when using a 1M step total interaction budget)**. However, due to the natural design choice of keeping the number of gradient updates for each agent unchanged, each agent receives much fewer on-policy samples compared to the number of gradient updates they receive. This increased off-policyness might exacerbate issues like significant overestimation [1, 2], which has been shown to be one of the causes of the primacy bias [3]. We support this statement with the counter-intuitive fact that sometimes using an ensemble without resets can be **much worse** than just training a single agent. For example, in the humanoid-run, humanoid-walk, and humanoid-stand tasks, when using a high replay ratio, SAC + ensemble is much worse than SAC (Figures 11 and 12). Note that we’ve included a more explicit discussion of this point in our revision.

From: ICLR.cc/2023/Conference/Paper3413/Reviewer_YR9c
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The authors present a straight-forward, easy to understand method for RL where an ensemble is created by training each member independently using Q-learning.  The experimentation clearly shows the benefit of the approach.
strength_and_weaknesses: The review of this paper was cut short when this reviewer's literature review found several papers from other conferences/journals which performed similar work.  When removing the novel insights from this paper which were found elsewhere, an insufficient amount of novelty remained for ICLR.  For example, see the following: https://pubmed.ncbi.nlm.nih.gov/34185651/
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written the is the sort of work which leads to reproducible work.
summary_of_the_review: Quality, interesting work with insufficient novelty.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper3413/Reviewer_ypJE
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies the problem of data sharing in ensemble methods, careful empirical analysis is made to understand how data-sharing can help ensemble learning, and then the authors combine data-sharing ensemble learning with periodic reset and show that this combination provides the best performance on DMC tasks. 
strength_and_weaknesses: Strength: 

**Insights** some interesting insights, for example how policy aggregation techniques might not be that important. I believe these insights are really the main contributions of this paper and can be considered somewhat significant in helping us understand these ensemble methods better. 

**Overall quality** quality is good and the writing is clear throughout. 

**Experimental design and convincing claims** experiments are well designed, claims are supported by evidence, great work in disentangle different factors, results are convincing. 

**Technical details** Adequate details provided to understand what is happening

Weaknesses:
- This paper mainly studies existing methods and there is no significant new technical contribution
- No theoretical results to help understand the empirical results

minor issues:
- Not sure if using green in the figures is a good idea as it might be bad for color-blind people. 
- Please go through the paper and check for typos and other small issues e.g. "[CITE]" in page 7

clarity,_quality,_novelty_and_reproducibility: - Clarity: Overall very clear
- Quality: good quality in writing and presentation, claims are well-supported
- Novelty: The methods are not too novel, but the insights are good and can be considered novel contributions
- Reproducibility: Technical details clearly presented and experiements well designed, good reproducibility
summary_of_the_review: Although the technical novelty seems to be a bit limited, the analysis made in the paper provides some interesting insights, and the paper has a good overall quality, and results seem quite reliable. Might be a good paper to accept. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3413/Reviewer_gUQE
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents a new strategy for deep RL algorithms, which is to train an ensemble of agents interacting with each of their individual environment concurrently and share the replay buffers. The authors find that such a data sharing approach can lead to performance improvement on DM Control environments and further more show that using a resetting scheme proposed in prior works can further stabilize the performance.
strength_and_weaknesses: Strength:

1. The idea of using data sharing across an ensemble of individual agents + resetting seems simple yet effective.

2. The empirical results are comprehensive on DM control tasks and can clearly show the benefits of the method.

Weaknesses:

1. While the idea of using ensemble learning with data sharing in deep RL is intuitive, I believe several prior methods have explored this direction e.g. [1, 2, 3], where they study an ensemble version of DQN with a shared replay buffer and show good performance on Atari games. This makes the discovery of the approach less original. Moreover, the idea of data sharing has also been studied in multi-task RL and offline RL works [4, 5], but authors didn't mention these works.

2. I think it would important to compare the method to approaches such as REDQ that only learn an ensemble of Q-functions but not an ensemble of policies and use a high UTD ratio. It seems that the difference between the approach and REDQ is simply that the data is collected separately with different policies. Hence, it is important to compare to REDQ to show the importance of such a design choice.

3. Learning in separate environments would cause higher sample complexity as we are collecting samples concurrently now. It would be important to show that such a method can actually lead to better sample complexity rather than the other way around.

[1] Anschel, O., Baram, N., and Shimkin, N. Deep reinforcement learning with averaged target DQN. CoRR, abs/1611.01929, 2016. URL http://arxiv.org/abs/1611.01929.

[2] Peer, O., Tessler, C., Merlis, N., and Meir, R. Ensemble bootstrapping for q-learning. In International Conference on Machine Learning, pp. 8454–8463. PMLR, 2021.

[3] Litian Liang, Yaosheng Xu, Stephen Mcaleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, Roy Fox Proceedings of the 39th International Conference on Machine Learning, PMLR 162:13285-13301, 2022.

[4] Kalashnikov, D., Varley, J., Chebotar, Y., Swanson, B., Jonschkowski, R., Finn, C., ... & Hausman, K. (2021). Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212.

[5] Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Finn, C., & Levine, S. (2022). How to Leverage Unlabeled Data in Offline Reinforcement Learning. arXiv preprint arXiv:2202.01741.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written though the authors didn't provide an algorithm box to make the methodology more clear. The originality is a bit lacking as the idea has been explored in many prior works.
summary_of_the_review: Based on the discussion above, I think the paper is not novel and the its connection to prior works are not thoroughly discussed. Hence, I would vote for a weak reject.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3413/Reviewer_c6V8
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presented an in depth empirical analysis of the positive role of data sharing in ensemble deep reinforcement learning. Inspired by several recently proposed ensemble deep reinforcement learning algorithms, this paper used specially designed experiments to show the benefits of data sharing. Experiments have also been conducted to understand the influence of primacy bias on ensemble deep reinforcement learning and how the simple resetting mechanism can help to alleviate the impact of primacy bias.
strength_and_weaknesses: Strength:
Ensemble deep reinforcement learning has great potential to improve the sample efficiency and stability of deep reinforcement learning. It is important to understand the impact of data sharing in ensemble deep reinforcement learning.

Weakness:
It is argued in the paper that the impact of data sharing is overlooked by existing studies. However, the ED2 algorithm developed in Januszewski et al. (2021), which is also referenced in the paper, clearly showed experimentally the importance of using data sharing to improve the sample efficiency and performance of deep reinforcement learning algorithms. While existing works may not have used similarly designed experiments to explicitly demonstrate the effectiveness of data sharing, however, it is a common belief in the research community that data sharing can help to improve the learning performance of individual agents in an ensemble. Hence, the experiments in the paper do not seem to bring any significantly new findings to the literature.

Also, it is natural for people to expect that primacy bias can negatively impact the performance of an ensemble, given its negative influence on each individual agent. Hence, if resetting can address this issue for any individual agent, it is likely to work on an ensemble. Again, the relevant experiments in the paper do not seem to bring any significantly new findings to the literature.

Some key statements do not seem to be fully convincing. For example, it is stated on page 7 that "data sharing alone is sufficient to provide a significant improvement in performance without the need of any additional tricks". What kind of performance improvement is considered sufficient? Does this mean that no other tricks or new algorithm design ideas can further improve performance significantly, as long as data sharing is properly used? I am not fully convinced that the experiment results can lead to such strong conclusions.

While this paper presented interesting empirical evidence regarding the positive role of data sharing, there is clearly a lack of theoretical analysis (or insights presented in a more general/theoretical context). Hence, it remains a bit questionable whether the findings from the experiment results can easily generalize to a wide range of reinforcement learning problems.

Some citations are missing, for example the citation for the replay ratio on page 7.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow in general. However, the clarity of some important statements may need to be improved.

For example, from the statement on page 4 "we subsample the replay buffer by N to eliminate the effect of a sudden increase", I find it difficult to understand how subsampling was actually carried out. Did the authors also consider the varied performance of individual agents during subsampling? Why or why not?
summary_of_the_review: Ensemble deep reinforcement learning has great potential to improve the sample efficiency and stability of deep reinforcement learning. It is important to understand the impact of data sharing in ensemble deep reinforcement learning. However, the experimental study in the paper does not seem to bring any significantly new findings to the literature. More in-depth theoretical analysis may also be necessary to significantly improve the technical contribution.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough