U NLEASHING T HE P OTENTIAL OF D ATA S HARING IN E NSEMBLE D EEP R EINFORCEMENT L EARNING
A
BSTRACT
This work studies a crucial but often overlooked element of ensemble methods in deep reinforcement learning: data sharing between ensemble members.
We show that data sharing enables peer learning , a powerful learning process in which individual agents learn from each other’s experience to significantly improve their performance.
When given access to the experience of other ensemble members, even the worst agent can match or outperform the previously best agent, triggering a virtuous circle.
However, we show that peer learning can be unstable when the agents’ ability to learn is impaired due to overtraining on early data.
We thus employ the recently proposed solution of periodic resets and show that it ensures effective peer learning.
We perform extensive experiments on continuous control tasks from both dense states and pixels to demonstrate the strong effect of peer learning and its interaction with resets.
I NTRODUCTION
Amid their popularity in supervised learning ensemble methods have also proven to be effective in deep reinforcement learning (RL) for a wide range of purposes (Osband et al., 2016; Chua et al., 2018; Chen et al., 2021; Schmitt et al., 2020; Agarwal et al., 2020; Liu et al., 2020).
Underlying some of these methods is a simple principle: training a set of independent agents while sharing the data among them.
This simple strategy allows different agents to concurrently explore different policies while sharing their knowledge.
It has been shown to be particularly useful for promoting deep exploration (Osband et al., 2016) and thus improving the learning efficiency.
Intuitively data sharing allows different agents to learn from each other’s experience.
This learning process, which we refer to as peer learning , happens covertly during training and thus is often overlooked, despite being essential for exploiting the diverse data brought by deep exploration.
Moreover, previous work often applies additional techniques (Osband et al., 2018; Lee et al., 2021;
Januszewski et al., 2021) on top of the basic ensemble strategy, which further obscures the role of peer learning in the success of these methods.
As a result, almost no effort has been put into understanding how peer learning affects individual agents, under what conditions is it effective, and how we can better benefit from it.
This work presents a dedicated study of the effect of data sharing in ensemble deep reinforcement learning.
Through carefully designed experiments, we present a clear view of the peer learning process in which agents of different levels learn from each other to improve their own performance.
Specifically, we show that a transition from separate to shared replay buffers during training can often cause all ensemble members to match or outperform the previously best agent in the ensemble, and thus resulting in a significant improvement in the overall performance.
Based on these observations, we show that oftentimes data sharing alone can already produce a significant gain over just training a single agent, without the use of any additional technique such as policy aggregation (Januszewski et al., 2021), bootstrapping (Osband et al., 2016), or UCB exploration (Chen et al., 2018; Lee et al., 2021).
A natural question is whether peer learning is always stable in practice.
Recently, Nikishin et al.
(2022) has shown that deep RL agents can suffer from the primacy bias, i.e., a tendency to overfit early data that impairs the agent’s ability to learn from its own future experience.
We show that, in the context of ensemble methods, the primacy bias can also prevent an agent from learning from other agents, even when the other agents have good performance and are producing high-quality samples.
Fortunately, we find that the simple resetting mechanism proposed in Nikishin et al.
(2022) provides a robust solution and ensures exploitation of the shared data.
Our experiments on continuous control tasks from both dense states and pixels show that ensemble and resets are a particularly robust combination and always significantly improve the performance over the baseline methods while only incurring a minor increase in computation time.
The contribution of this paper can be summarized as follows:
We expose peer learning as a prominent effect when using data sharing in ensemble methods and demonstrate its impact on individual agents’ performance; 2.
We show that data sharing alone has the potential to provide a significant improvement in performance, without the use of any additional technique; 3.
We find that peer learning can be unreliable in the presence of the primacy bias, and show that the recently proposed resetting mechanism provides a robust solution; 4.
We empirically verify the effect of peer learning and its interaction with resets with extensive experiments on continuous control tasks from both dense states and pixels.
R ELATED W ORK
Ensemble methods in RL Ensemble methods have been employed in deep reinforcement learning for a wide range of purposes including promoting deep exploration (Osband et al., 2016; Peng et al., 2020), improving value estimations (Chen et al., 2021;
Lan et al., 2020;
Peer et al., 2021; Liang et al., 2022; Anschel et al., 2016) or model predictions (Chua et al., 2018; Kurutach et al., 2018), and accelerating hyperparameter sweeps (Schmitt et al., 2020; Liu et al., 2020).
While there are various distinct ways to apply an ensemble, the simple ensemble strategy (i.e., training a set of independent agents while sharing the data) considered in this work underlies many previous works.
For example, Osband et al.
(2016) trains a multi-head DQN (Mnih et al., 2015) with all heads sharing a single replay buffer, to improve exploration.
Januszewski et al.
(2021) trains multiple DDPG (Lillicrap et al., 2016) policies with a shared replay buffer, and combine them into an average policy at testtime.
Schmitt et al.
(2020) and Liu et al.
(2020) accelerate hyperparameter sweeps by training agents with different hyperparameters sharing a central replay buffer.
However, most previous work does not investigate the role of data sharing, or only marginally touches on its usefulness, without uncovering its effect on individual agents in detail or analyzing its failure cases.
Data sharing in offline and multi-task RL Besides its wide adoption in ensemble RL methods, data sharing has also been explored in offline RL (Yu et al., 2022; 2021; Dorfman & Tamar, 2020;
Mitchell et al., 2021) and multi-task RL (Andrychowicz et al., 2017; Eysenbach et al., 2020; Li et al., 2020; Liu et al., 2019;
Kalashnikov et al., 2021; Chebotar et al., 2021).
Both paradigms provide natural motivations for data sharing: offline RL involves reusing previously collected data, while multi-task RL involves sharing experience across different tasks.
These scenarios pose various challenges that are very different from the online, single-task setting that we study in this work.
The primacy bias in deep RL The recently studied primacy bias (Nikishin et al., 2022) is a tendency of deep RL agents to overfit early data that impairs the agents’ ability to learn from their own future experience.
Such overfitting can have multiple different causes and various negative effects.
For example, Nikishin et al.
(2022) shows that a high replay ratio can often cause significant overfitting due to the large number of gradient updates relative to the number of environment interactions collected.
Interestingly, Nikishin et al.
(2022) proposes a surprisingly simple but effective solution: periodically reinitializing part of the agents, while preserving the replay buffer.
This simple mechanism, which they refer to as “resetting”, can reliably eliminate the negative effect of the primacy bias and result in significant improvements.
In this work, we show that the primacy bias can damage the peer learning process as well and that resetting also robustly fixes this issue
P RELIMINARIES
The RL formulation We consider the standard RL framework (Sutton & Barto, 2018) that models the environment as a Markov decision process (MDP).
At each timestep, the agent observes the current environment state S t and executes an action A t sampled from its current policy π .
It then receives a reward signal R t and the next state S t +1 from the environment.
The goal of a reinforcement learning algorithm is then to find a policy π that maximizes the expected discounted sum of rewards
E π
[ (cid:80) ∞ t =
γ t R t ] based on its past interactions with the environment.
This work focuses on deep RL algorithms that come with a replay buffer and support off-policy training, as they provide natural infrastructures and mechanisms for data sharing.
Specifically, we use the popular Soft Actor-Critic (SAC) algorithm (Haarnoja et al., 2018)
(for dense states) and the DrQ algorithm (Kostrikov et al., 2021) (for image observations) as the base algorithms in all our experiments.
Ensemble strategy
In this work, we consider a simple form of ensemble that underlies many previous works (Osband et al., 2016;
Schmitt et al., 2020; Januszewski et al., 2021; Peng et al., 2020; Liu et al., 2020).
Given a base RL algorithm and the ensemble size N , we train N independent agents interacting with N environment instances concurrently.
Each agent has their own copy of the parameters (e.g., policy, value, and target networks) and optimizers, and any learning updates of the base algorithm are performed independently for each agent, but in parallel.
However, we allow different agents to share the same image encoder when using images as input.
Unless otherwise stated, the agents share a central replay buffer as well as the training batch.
Note that for fair comparison, whenever we use an ensemble with data sharing, we keep the total interaction steps the same as when we train a single agent.
For example, if the total interaction budget is 1 × 10steps and the ensemble size is 10 , each ensemble member can only interact with the environment for 1 × 10 5 steps.
For evaluation, we do not perform any policy aggregation such as voting (Osband et al., 2016) or averaging (Januszewski et al., 2021).
Instead, we always use the policies of individual ensemble members and report either the best or average performance within ensemble in most cases. A detailed description of this ensemble strategy can be found in Algorithm 1 in our Appendix.
Resetting The resetting mechanism is originally proposed in Nikishin et al.
(2022) to counter the negative effect of the primacy bias.
Similar to Nikishin et al.
(2022), whenever we refer to “resetting an agent”, we mean to reinitialize all or part of the parameters and optimizer states of all components of the agent, while preserving the replay buffer.
Specifically, for SAC, this means reinitializing everything while preserving the buffer.
This is also the case for DrQ except that we retain the parameters and optimizer states of the image encoder.
Replay ratio (RR) An important hyperparameter that deserves special attention is the replay ratio 1 .
It refers to the ratio of the number of gradient updates to the number of environment steps at any point in training.
Crucially, Nikishin et al.
(2022) shows that a high replay ratio can often worsen the negative effect of primacy bias due to overtraining on early data.
Though in the single-agent case replay ratio is well-defined, in the context of ensemble methods there exists some ambiguity because it involves multiple agents.
We therefore give an exact definition of replay ratio that we use throughout this paper in Appendix D.
T HE P EER L EARNING P ROCESS
When training an ensemble of independent data sharing agents, one can expect the following two processes to happen:
A diversification process in which random initialization and stochastic training introduce diversity to the ensemble; 2.
A peer learning process in which different ensemble members learn from each other’s experience to improve themselves.
Intuitively, the first process creates a diverse set of agents and thus diverse data, which peer learning might exploit to improve the performance of individual ensemble members.
These two processes cooperate closely and jointly contribute to the high performance of ensemble methods.
However, most focus and credit have been given to the diversification aspect and its exploration benefits (Osband et al., 2016; Chen et al., 2018; Lee et al., 2021), while the more subtle peer learning process has often been overlooked despite being essential for the exploitation of the diverse data brought by deep exploration.
A better understanding of peer learning can reveal the inner workings behind the success of ensemble methods and guide the design of better ensemble algorithms.
The rest of this section is divided into two parts.
In the first part, we demonstrate the strong effect of peer learning on the performance of individual ensemble members.
In the second part, we demonstrate a failure case of peer learning in which the agents’ ability to learn is impaired due to overtraining on early data, and that the recently proposed solution of periodic resets robustly fixes this issue.
4. T HE E FFECT OF P EER L EARNING
We now demonstrate the effect of peer learning on the individual ensemble members.
Normally, peer learning happens covertly during training and its effect is only indirectly reflected in the overall performance.
In order to make the impact of peer learning visible, we design a simple setup that disentangles the diversification process and the peer learning process.
Specifically, we start training with the N ensemble members with separate replay buffers.
After a certain timestep, we either continue training with separate replay buffers, or merge the separate replay buffers into one shared replay buffer.
We train an ensemble of N = 10 SAC agents with a replay ratio of 1 , and select four environments from the DeepMind Control Suite (Tassa et al., 2018) that are representative of the effects of ensemble and resets. We carefully control the interaction and gradient update schedule as well as the buffer sizes before and after merging to ensure the results are fair and meaningful.
More details and results can be found in Appendix C and Appendix F respectively. [Figure] i [Figure]
t
[Figure]
r n
[Figure]
n
[Figure]
individual learning curves from one run with or without buffer merging.
The individual learning curves are Gaussian smoothed to reduce visual clutter.
Figures 1 shows the effect of buffer merging on the overall performance of the ensemble as well as the individual learning curves of all ensemble members from a single run with or without buffer merging.
Before merging, different agents develop distinct policies and have different performance; upon merging, the overall performance improves significantly, usually with a drastic reduction in the variance within the ensemble.
In all cases, the improvement happens robustly and rapidly, demonstrating the strong effect of peer learning.
Inspecting the individual learning curves reveals the diverse effects of peer learning: on cartpole-swingup_sparse and quadruped-walk , the under-performing agents learn from the data of the optimal agents in the ensemble and quickly converge to the optimal policy.
On fish-swim and hopper-hop , the entire ensemble even outperforms the previously best agents before merging.
Note the this experiment is for demonstration purposes only.
In practice, one would always directly start with a shared replay buffer for best sample efficiency.
Given the above results, we can imagine how peer learning improves performance in that case: starting with random initialization, the stochastic training process continues to diversify the ensemble; concurrently, peer learning selects and integrates the solutions discovered and improves the performance of all agents in the ensemble.
The result of this simultaneous diversification and peer learning is significantly improved sample efficiency relative to just training a single agent throughout the entire training process, as observed in previous work (Osband et al., 2016) and confirmed in Section 5 of this work.
The above results have an important implication.
As long as peer learning is stable, we can expect almost all ensemble members to be more high-performing than if we only train a single agent.
Therefore, one can simply train an ensemble of agents without any additional technique and still benefit from it. This however does not mean that other techniques are not useful or not worth investigating, as previous work has shown that techniques such as policy aggregation and diversity regularization can further improve performance (Januszewski et al., 2021; Peng et al., 2020).
4. A F AILURE C ASE AND THE S OLUTION
Intuitively, successful peer learning requires an agent to maintain the ability to quickly adapt their policy.
One can imagine that this is less likely to happen when the agent is stuck in some suboptimal policy, loses their representation capacity (Lyle et al., 2022), or experiences significant value overestimation (Kumar et al., 2019).
This work investigates one particular effect that might impair an agent’s ability to learn: the primacy bias (Nikishin et al., 2022).
Specifically, Nikishin et al.
(2022) shows that overtraining on early data, e.g. when using a high replay ratio, can significantly impair the agent’s ability to learn from its own future experience.
This creates a vicious circle, as the data generated by a poor agent will also be poor, which further eliminates the possibility of improvement.
We hypothesize that the primacy bias can also damage the agent’s ability to learn from other agents’ data, even if it has access to the high quality samples generated by other high-performing agents .
Notably, Nikishin et al.
(2022) proposes a simple solution to counter the primacy bias: periodically resetting part of the agent, while preserving the replay buffer.
This strategy is shown to be highly effective.
We therefore expect it to also provide a solution that ensures stable peer learning.
To verify these hypotheses, we use the same setup as in the previous experiment, except for a replay ratio of 16 instead of 1 to make the effect of the primacy bias clearly detectable.
To quantify the effect of resets, we include two additional variants: in the first one, we continue training with separate buffers, and reset the agents; in the second one, we merge the buffers and reset the agents.
Results are shown in Figure 2.
Naively merging the buffer is much less effective at a replay ratio of 16 than it is when using a replay ratio of 1 , except for cartpole-swingup_sparse , which has the simplest environment dynamics among the four.
On hopper-hop and quadruped-walk , merging the buffer provides almost no benefit.
The individual learning curves show that there exists a significant variance in the ensemble members’ performance even if the data is shared within the ensemble, indicating that peer learning is less effective in this case.
Specifically, for hopper-hop and quadruped-walk , our inspection shows that the under-performing agents are incapable to learn from the better agents due to significant value overestimation either before or after merging, while for fish-swim the cause is less clear.
Nevertheless, resetting restores the agents’ ability to effectively learn from each other and achieves high performance with a low variance. Note that even though resetting without merging the buffer also provides a reasonable gain (e.g., on quadruped-walk ), it is not sufficient for reaching optimal performance in all cases. [Figure]
[Figure] Besides ensuring effective peer learning, another reason why resetting can be particularly important when using an ensemble is the increased off-policyness and hence more severe primacy bias. Note that when using an ensemble, we keep the total interaction steps the same as when training a single agent (e.g., 1 × 10 5 interactions for each of the 10 members when using a 1 × 10 6 step total interaction budget). Due to the natural design choice of keeping the number of gradient updates for each agent unchanged, each agent receives much fewer on-policy samples compared to the number of gradient updates it experiences.
This increased off-policyness may exacerbate issues such as significant overestimation (Ostrovski et al., 2021; Kumar et al., 2019), which has been shown to be one of the causes of the primacy bias (Nikishin et al., 2022). We support this statement with the counterintuitive fact that sometimes using an ensemble can be much worse than just training a single agent. For example, in Figure 12 in our Appendix, we show that combining SAC with ensemble is much worse than SAC on the three humanoid tasks when using a high replay ratio. Through these experiments and analysis, we learned more about the mechanics of peer learning and how to overcome its limitations.
In the next section, we perform experiments with a standard ensemble strategy (i.e., start training with a shared buffer) to demonstrate the effect of peer learning in practice and its interaction with resets.
E XPERIMENTS
Our experimental results can be summarized as follows.
First, we verify the strong effect of peer learning by showing that data sharing alone has the potential to provide a significant gain over just training a single agent, without the use of any additional technique.
Second, we show that the primacy bias can damage peer learning and weaken the advantages of ensembles, while resetting ensures the exploitation of their benefits.
Then, we present an analysis of the performance distribution within ensemble and show the dominance of ensemble methods in terms of sample efficiency.
5. S ETUP
General setup We focus on continuous control tasks from the DeepMind Control Suite (Tassa et al., 2018).
We select 16 tasks for state-based control and 13 tasks for pixel-based control.
Please refer to Appendix A for the full environment lists.
For state-based tasks, each algorithm is allowed a budget of 1 × 10 6 environment steps; for pixel-based tasks, we use 2 × 10 6 environments steps.
We use SAC (Haarnoja et al., 2018) and DrQ (Kostrikov et al., 2021) as our base algorithms for the state-based and pixel-based tasks respectively.
For each base algorithm X, where X can either be SAC or DrQ, we also consider 3 variants: X + ensemble, X + resets, and X + ensemble + resets.
For all experiments we use 10 random seeds.
We use N = 10 for all our main results with ensemble.
We maintain completely independent ensemble members for SAC, while for DrQ we share the image encoder among ensemble members.
As stated earlier, when using an ensemble we keep the total number of environment interactions the same as when we train a single agent to ensure fair comparison.
For SAC, in order to probe the effect of primacy bias, we experiment with five different replay ratios { 1 , 2 , 4 , 8 , 16 } .
Recently, Anonymous (2022) shows that SAC + resets exhibits scalability with respect to the replay ratio, and thus it would also be interesting to see whether this still applies with the addition of ensemble.
Following a similar strategy in Anonymous (2022), when using resets, we set the reset period of
SAC according to the replay ratio such that we reset every 1 .
92 × 10 6 gradient updates; for DrQ, we reset the whole agent except for the image encoder per 2 × 10 5 environment steps as done in
Nikishin et al.
We always skip the last reset to avoid reporting on undertrained agents.
Evaluation protocol For the main results, we show the performance when using the best agent in each ensemble for evaluation (best), as well when using the average performance of each ensemble as the corresponding run’s performance (avg).
To avoid bias in the reported results, when reporting with the best ensemble members, we perform two independent rounds of evaluation for all ensemble members.
We select the best agent based on the first round of evaluation, and report its performance in the second round.
For our main results, we report the point estimates and 95% confidence intervals of the interquartile mean (IQM) (Agarwal et al., 2021) of the evaluation results of the last timestep across all environments.
All evaluation uses 10 episodes.
Other experimental details can be found in Appendix A
5. M AIN R ESULTS
[Figure]
Our main results for SAC and DrQ are presented in Figure 3.
More results, including an ablation of the ensemble size and comparison with state-of-the-art results, can be found in Appendix E.
Effect of ensemble We first focus on the overall effect of ensemble.
As shown in Figure 3, using an ensemble while sharing data always provides a significant gain over just training a single agent, regardless of the base algorithm, replay ratio, or whether we do resets (e.g., SAC + ensemble + resets is better than SAC + resets for all replay ratios).
This provides strong evidence that data sharing alone has the potential to provide a significant improvement in performance without the need of any additional trick, which also validates the strong effect of peer learning and its important role in ensemble methods.
Effect of the primacy bias and resets From Figure 3 (left) we can see the impact of the replay ratio when using an ensemble.
For low replay ratios such as 1, 2 and 4, SAC + ensemble is able to robustly outperform SAC and SAC + resets.
However, as we step into the high replay ratio regime in which the primacy bias becomes severe, the performance of SAC + ensemble starts to degrade and is outperformed by SAC + resets.
Note SAC + ensemble still dominates SAC even with replay ratio 16 , indicating that the benefits of using an ensemble still exist but are weakened by the primacy bias, likely due to a combination of its damage to the peer learning process as shown in Section 4 and its other the direct negative effects.
As expected, SAC + ensemble + resets does not suffer from the damaged peer learning process even under a high replay ratio (e.g. 16 ) and remains high performing.
Scalability w.r.t.
computation Similar to SAC + resets, we see that SAC + ensemble + resets exhibits the same scalability with respect to the replay ratio (Anonymous, 2022).
However, it is much more high-performing: SAC + ensemble + resets at replay ratio 4 already outperforms SAC + resets at replay ratio 16, despite the fact that the latter is much more expensive.
Besides, it is also more robust, as can be seen from the much narrower confidence intervals.
5. A NALYSIS
Performance distribution within ensemble In Figure 4, we show the results when using the n -th best agent in the ensemble for evaluation, where n ranges from 1 to N .
For SAC, most ensemble members are high-performing and stay close to the best agent in the ensemble.
This is especially true when resets are applied, in which case the performance distribution of agents of different ranks are highly concentrated.
Without resets, the worst and the second worst ensemble members gradually deviate from other members as replay ratio increases. For DrQ, with resets there is almost no variance within the ensembles, while without resets the performance of different ensemble members is clearly spread out. The low variance in performance across ensemble members when using resets validates our claim that resets can ensure effective peer learning.
[Figure]
Sample efficiency In Figure 5 we show the learning curves of the different methods.
For SAC we show the results of an intermediate replay ratio 4, and more results can be found in Appendix E.
From Figure 5 we can see that the dominance of ensemble methods over the single-agent methods often starts from the beginning of the training and continues until the end, consistent with previous work’s observation (Osband et al., 2016).
This confirms that the interaction between diversification and peer learning is a continual process that happens throughout the entire training process.
[-For SAC, we use replay ratio 
-]
[Figure]
C ONCLUSION
The work studies a crucial element behind the success of many ensemble methods: data sharing between ensemble members.
We highlight the powerful peer learning process enabled by data sharing and demonstrate how it improves individual agents’ performance.
We also identify a failure case of peer learning caused by overtraining on early data, and show that the recently proposed solution of periodic resets robustly alleviates the problem and boosts performance.
Our experiments on continuous control tasks from both dense states and pixels demonstrate the strong effect of peer learning and its interaction with resets.
We hope our work will encourage the RL community to better recognize the effectiveness of data sharing and fully exploit its potential.
A G ENERAL E XPERIMENTAL D ETAILS
[Table]
Our selection of the environments for state-based and pixel-based control are listed in Table 1.
We try to include as many commonly used environments as possible while excluding the overly simple environments (e.g., cartpole-balance ).
The implementation of all our SAC and DrQ variants are based on an open source JAX implementation (Kostrikov, 2021).
We adopt all their default hyperparameters except for the replay buffer size of DrQ, for which we use 1 × 10 6 instead of 1 × 10 5 as we find it to work much better.
For the implementation of resets, we follow the the official implementation from the authors 2 .
We implement the parallel computation of the ensembles using the nn.vmap function of the Flax (Heek et al., 2020) library.
B D ETAILS OF THE B ASIC E NSEMBLE S TRATEGY
Please see Algorithm 1 for a detailed description of the basic ensemble strategy considered in this work.
C D ETAILS FOR THE B UFFER M ERGING E XPERIMENT
For all experiments we train for a total of 1 × 10 6 steps.
For the first 5 × 10 5 steps where we perform training with separate buffers, each agent will interact with the environment for 5 × 10 5 steps.
For the last 5 × 10 5 steps, if we continue training with separate buffers, then again each agent will interact with the environment for 5 × 10 5 steps.
However, if we merge buffer, the 5 × 10 5 interaction steps will be shared across the N agents, i.e., each agent only gets to interact with the environment for 5 × 10 5 /N steps, but in a round-robin way.
We control the gradient update schedule according to the definition of replay ratio we give in Appendix D, so that all agents receive the equal amount of gradient updates at the same x-axis position in Figure 1 and Figure 2.
Also, upon merging we subsample the shared replay buffer by N to avoid the unfair advantage of a much larger replay buffer.
For example, with N = 10 the shared buffer will contain 10 × 5 × 10 5 = 5 × 10 6 samples (merged from the 10 separate replay buffers), which we will downsample to 5 × 10samples. We uniformly sample the transitions at constant intervals from each replay buffer, and therefore each separate replay buffer will have an equal portion in the merged replay buffer. a ∼ π θ i (
| states */ agents For each environment and setting we use 5 seeds and N = 10 agents in the ensemble.
Reported results is thus average over 50 agents.
D D EFINITION OF R EPLAY R ATIO
Here we give an exact definition of replay ratio that we use throughout this paper: 1) when training a single agent or an ensemble of agents with separate replay buffers, replay ratio refers to the ratio of the number of gradient updates that each agent will experience to the number of their own interaction steps; 2) when training an ensemble with a shared replay buffer, replay ratio refers to the number of gradient updates that each agent will experience to the total number of interaction steps of the entire ensemble.
We find this definition to make most sense because it considers the actual number of environment transitions that the agent has access to.
It also reflects how previous work typically sets this hyperparameter in practice (Osband et al., 2016; Peng et al., 2020).
E A DDITIONAL R ESULTS FOR THE M AIN E XPERIMENT E. N UMERICAL R ESULTS
Please see Table 2 for results for SAC and Table 3 for results for DrQ.
We highlight the row with the highest IQM.
E. E NSEMBLE SIZE ABLATION
In Figure 6 we presents an ablation of the ensemble size N .
For SAC, we use {+replay ratio 
 and thus we do not perform resets. For DrQ, we perform resets as it reaches better performance. As shown in+} the figure, most benefits of using an ensemble can be gained from a small ensemble size of 5 , while going to 10 also provides additional, albeit marginal, improvements.
E. C OMPARISON WITH O THER A LGORITHMS
In Table 4 we show a comparison with SR-SAC (Anonymous, 2022), REDQ (Chen et al., 2021), and
DDPG (Lillicrap et al., 2016).
Note that this comparison is on the DMC15-1M benchmark used in
Anonymous (2022), which contains one less environment ( cartpole-swingup_sparse ) than the set of 16 environments used in this work.
Therefore all our results in this table are re-calculated
[Table]
[Table]
N =
N =
N =
N =
SAC + ensemble
IQM
N =
N =
N =
DrQ + ensemble + resets
IQM without cartpole-swingup_sparse .
Note that SAC + ens.
+ resets with replay ratioalmost approaches the performance of SR-SAC with replay ratio 128, which is the state of art algorithm on the DeepMind Control Suite.
SR-SAC is essentially just SAC + resets but with a different reset frequency and a much higher high replay ratio and therefore much more computationally expensive.
The results of SR-SAC and are
Anonymous which uses the same
(Kostrikov et al., 2021) ours All other this in this we it give also [Table] SAC + ens.
(avg, RR=4) 700 (682, 717) 738 (684, 794) 639 (626, 650) SAC (RR=2) 644 (604, 679) 648 (570, 755) 582 (558, 605)
SAC (RR=1) 569 (524, 613) 696 (515, 739) 531 (505, 557) E. L EARNING CURVES For all sample efficiency curves, 1) for SAC, we use the average of all ensemble members within the ensemble as the ensemble’s reported performance 2) for DrQ, we use the first indexed agent for evaluation
SAC In Figure 7 we show the aggregate sample efficiency curves for each replay ratio.
In Figure 8, 9, 10, 11, 12 we show the per-environment results for different replay ratios.
DrQ In Figure 13 we show the per-environment results for DrQ.
F A DDITIONAL R ESULTS FOR THE B UFFER M ERGING E XPERIMENT
We show results for all 16 environments for the buffer merging environment in Figure 14 and Figure 15 for replay ratio 1 and replay ratio 16, respectively.
Note that the sudden merging can have negative impacts on some environments, with the most obvious being the humanoid tasks.
We hypothesize that this is due to negative effect of the sudden exposure to off-policy data.
Note however these experiments are for demonstration purposes only.
In practice people will use a shared replay buffer throughout training so this issue will not exist.
[Figure]
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]
e
R e t u r n
[Figure]
[Figure]
p i s od e R e t u r n
[Figure]