Original: Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning
Revised: Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We present a novel multi-agent RL approach, Selective Multi-Agent PER , in whichagents share with other agents a limited number of transitions they observe duringtraining.
Revised: We present a novel multi-agent RL approach, Selective Multi-Agent PER , in which agents share with other agents a limited number of transitions they observe during training.

Original: They follow a similar heuristic as is used in (single-agent)
Revised: They follow a similar heuristic as is used in (single-agent)

Original: Prioritized
Revised: Prioritized

Original: Experience Replay and choose those transitions based on their td-error.
Revised: Experience Replay, and choose those transitions based on their td-error.

Original: Theintuition behind this is that even a small number of relevant experiences fromother agents could help each agent learn.
Revised: The intuition behind this is that even a small number of relevant experiences from other agents could help each agent learn.

Original: Unlike many other multi-agent RLalgorithms, this approach allows for largely decentralized training, requiring onlya limited communication channel between agents.
Revised: Unlike many other multi-agent RL algorithms, this approach allows for largely decentralized training, requiring only a limited communication channel between agents.

Original: We show that our approachoutperforms baseline no-sharing decentralized training.
Revised: We show that our approach outperforms baseline no-sharing decentralized training and state-of-the art multi-agent RL algorithms.

Original: Further, sharing only asmall number of experiences outperforms sharing all experiences between agents,and the performance uplift from selective experience sharing is robust across arange of hyperparameters 
Revised: Further, sharing only a small number of experiences outperforms sharing all experiences between agents, and the performance uplift from selective experience sharing is robust across a range of hyperparameters and DQN variants.

Original:  Introduction
Revised:  Introduction

Original: Multi-Agent Reinforcement Learning (RL) is often considered a hard problem: The environmentdynamics and returns depend on the joint actions of all agents, leading to significant variance andnon-stationarity in the experiences of each individual agent.
Revised: Multi-Agent Reinforcement Learning (RL) is often considered a hard problem: The environment dynamics and returns depend on the joint actions of all agents, leading to significant variance and non-stationarity in the experiences of each individual agent.

Original: Much recent work [21, 15] in multi-agent
Revised: Much recent work [24, 18] in multi-agent

Original: RL has focused on mitigating the impact of these.
Revised: RL has focused on mitigating the impact of these.

Original: Our work goes into aslightly different direction: Leveraging the presence of other agents to collaboratively explore the environment more quickly.
Revised: Our work goes in a different direction: leveraging the presence of other agents to collaboratively explore the environment more quickly.

Original: We present a novel multi-agent RL approach that allows agents to share a small number of experienceswith other agents.
Revised: We present a novel multi-agent RL approach that allows agents to share a small number of experiences with other agents.

Original: The intuition is that if one agent discovers something important in the environment,then sharing this with the other agents should help them learn faster.
Revised: The intuition is that if one agent discovers something important in the environment, then sharing this with the other agents should help them learn faster.

Original: However, it is crucial that onlyimportant experiences are shared - we will see that sharing all experiences indiscriminately will notimprove learning.
Revised: However, it is crucial that only important experiences are shared - we will see that sharing all experiences indiscriminately will not improve learning.

Original: To this end, we take inspiration from a well-established technique in single-agent RL, prioritized experience replay (PER).
Revised: To this end, we take inspiration from a well-established technique in single-agent RL, prioritized experience replay (PER) [25].

Original: With PER an off-policy algorithm such as DQN  will sampleexperiences not uniformly, but proportionally to “how far off” the current policy’s predictions are ineach state, formally the temporal difference (td) error .
Revised: With PER an off-policy algorithm such as DQN [19] will sample experiences not uniformly, but proportionally to “how far off” the current policy’s predictions are in each state, formally the temporal difference (td) error .

Original: We use this same metric to prioritize whichexperiences to share with other agents.
Revised: We use this same metric to prioritize which experiences to share with other agents.

Original: We dub the resulting multiagent RL approach “selective multiagent PER” or “suPER”.
Revised: We dub the resulting multiagent RL approach “selective multiagent PER” or “suPER”.

Original: In this, agentsindependently use an off-policy RL algorithm to learn, but with a twist: Each agent relays its highesttd-error experiences to the other agents, who insert them directly into their replay buffer, which theyuse for learning.
Revised: In this, agents independently use a DQN algorithm to learn, but with a twist: Each agent relays its highest-td-error experiences to the other agents, who insert them directly into their replay buffer, which they use for learning.

Original: This approach has several advantages:
Revised: This approach has several advantages:

Original: It consistently leads to faster learning and higher eventual performance, across hyperparameter settings.
Revised: It consistently leads to faster learning and higher eventual performance, across hyperparameter settings.

Original: Unlike many “centralized training, decentralized execution” approaches, the suPER learningparadigm allows for (semi-) decentralized training, requiring only a limited bandwidthcommunication channel between agents.3.
Revised: Unlike many “centralized training, decentralized execution” approaches, the suPER learning paradigm allows for (semi-) decentralized training, requiring only a limited bandwidth communication channel between agents.

Original: The paradigm is agnostic to the underlying decentralized training algorithm, and can enhanceany existing DQN variation including modern algorithms such as Rainbow [8] and otheroff-policy algorithms.
Revised: The paradigm is agnostic to the underlying decentralized training algorithm, and can enhance many existing DQN variations. In addition to the specific algorithm we develop, this work also introduces two key conceptual novelties. We show that communication can improve multi-agent RL even during training.

Revised: Most prior work consider "learning to communicate", also known as emergent communication, which is a difficult problem but may improve coordination at convergence.

Revised: We show that "communicate to learn" can also drastically improve performance during training.

Revised: Related to this, we introduce the paradigm of “decentralized training with communication”.

Original:  In the remainder of the paper, we will discuss related literature and technical preliminaries; introduceour novel algorithm in detail; describe experimental evaluation and results; and suggest avenues forfuture work.
Revised: This is a ‘middle ground between established approaches of decentralized and centralized training (including “centralized training, decentralized execution”). In the remainder of the paper, we will discuss related literature and technical preliminaries; introduce our novel algorithm in detail; describe experimental evaluation and results; and suggest avenues for future work.

Original:  Related Work
Revised:  Related Work

Original: Multi-agent RL approaches in the literature can be broadly categorized according to the degree ofawareness of each learning agent of the other agents in the system [28].
Revised: Multi-Agent RL approaches in the literature can be broadly categorized according to the degree of awareness of each learning agent of the other agents in the system [32].

Original: On one end of this spectrumis independent or decentralized learning where multiple learning agents (in the same environment)are unaware or agnostic to the presence of other agents in the system [27, 12].
Revised: On one end of this spectrum is independent or decentralized learning where multiple learning agents (in the same environment) are unaware or agnostic to the presence of other agents in the system [31, 15].

Original: From the perspectiveof each agent, learning regards the other agents as part of the non-stationary environment.
Revised: From the perspective of each agent, learning regards the other agents as part of the non-stationary environment.

Original: At theopposing end of the spectrum, in centralized control a single policy controls all agents.
Revised: At the opposing end of the spectrum, in centralized control a single policy controls all agents.

Original: In betweenthese two, a number of related strands of research have emerged.
Revised: In between these two, a number of related strands of research have emerged.

Original: Approaches nearer the decentralized end of the spectrum often use communication between theagents [35, 3, 7].
Revised: Approaches nearer the decentralized end of the spectrum often use communication between the agents [39, 6, 10].

Original: Several forms of cooperative communication have been formulated, by whichagents can communicate various types of messages, either to all agents or to specific agent groupsthrough dedicated channels [11, 24, 18, 19].
Revised: Several forms of cooperative communication have been formulated, by which agents can communicate various types of messages, either to all agents or to specific agent groups through dedicated channels [14, 28, 21, 22].

Original: While communication among agents could help withcoordination, training emergent communication protocols also remains a challenging problem; recentempirical results underscore the difficulty of learning meaningful emergent communication protocols,even when relying on centralized training [10].
Revised: While communication among agents could help with coordination, training emergent communication protocols also remains a challenging problem; recent empirical results underscore the difficulty of learning meaningful emergent communication protocols, even when relying on centralized training [13].

Original: Related to this, and often used in conjunction, ismodeling other agents [10, 4, 33], which equips agent with some model of other agent’s behavior.
Revised: Related to this, and often used in conjunction, is modeling other agents [2, 13, 7, 37], which equips agent with some model of other agent’s behavior.

Original: This in turn is related to a number of recent approaches using centralized training but decentralizedexecution.
Revised: This in turn is related to a number of recent approaches using centralized training but decentralized execution.

Original: A prevailing paradigm within this line of work assumes a training stage during whicha shared network (such as a critic) can be accessed by all agents to learn decentralised (locallyexecutable) agent policies [15, 1, 20, 4, 21].
Revised: A prevailing paradigm within this line of work assumes a training stage during which a shared network (such as a critic) can be accessed by all agents to learn decentralised (locallyexecutable) agent policies [18, 1, 23, 7, 24].

Original: These approaches successfully reduce variance duringtraining, e.g. through a shared critic accounting for other agents’ behavior, but rely on joint observations and actions of all agents.
Revised: These approaches successfully reduce variance during training, e.g. through a shared critic accounting for other agents’ behavior, but rely on joint observations and actions of all agents.

Original: Within this paradigm, and perhaps closest related to our own work,[2] introduces an approach in which agents share (all of their) experiences with other agents.
Revised: Within this paradigm, and perhaps closest related to our own work, [5] introduces an approach in which agents share (all of their) experiences with other agents.

Original: Whilethis is based on an on-policy actor-critic algorithm, it uses importance sampling to incorporate theoff-policy data from other agents, and the authors show that this can lead to improved performancein sparse-reward settings.
Revised: While this is based on an on-policy actor-critic algorithm, it uses importance sampling to incorporate the off-policy data from other agents, and the authors show that this can lead to improved performance in sparse-reward settings.

Original: Our work also relies on sharing experiences among agents, but cruciallyrelies on selectively sharing only some experiences.
Revised: Our work also relies on sharing experiences among agents, but crucially relies on selectively sharing only some experiences.

Original:  The approach we present in this paper relies intrinsically on off-policy RL algorithms.
Revised: Off-Policy RL The approach we present in this paper relies intrinsically on off-policy RL algorithms.

Original: Most notablein this class is DQN [16], which achieved human-level performance on a wide variety of Atari 2600games.
Revised: Most notable in this class is DQN [19], which achieved human-level performance on a wide variety of Atari 2600 games.

Original: Various improvements have been made to this algorithm since then, including dueling DQN[31], Double DQN (DDQN) [30], Rainbow [8] and Ape-X [9].
Revised: Various improvements have been made to this algorithm since then, including dueling DQN [35], Double DQN (DDQN) [34], Rainbow [11] and Ape-X [12].

Original: Prioritized Experience Replay [22]improves the performance of many of these, and is closely related to our own work.
Revised: Prioritized Experience Replay [25] improves the performance of many of these, and is closely related to our own work.

Original: Off-policyactor-critic algorithms [6, 14] have also been developed, in part to extend the paradigm to continuouscontrol domains.
Revised: A variant for continuous action spaces is DDPG [27]. Off-policy actor-critic algorithms [9, 17] have also been developed, in part to extend the paradigm to continuous control domains.

Original:  Preliminaries
Revised:  Preliminaries

Original: Reinforcement learning (RL) deals with learning optimal policies for sequential decision makingin environments for which the dynamics are not fully known [25].
Revised: Reinforcement learning (RL) deals with learning optimal policies for sequential decision making in environments for which the dynamics are not fully known [29].

Original: Multi-Agent Reinforcement
Revised: Multi-Agent Reinforcement

Original: Learning extends RL to multi-agent settings.
Revised: Learning extends RL to multi-agent settings.

Original: A common model is a Markov game , or stochasticgame defined as a tuple ⟨ S, A , R , T , γ ⟩ with states S , joint actions 
Revised: A common model is a Markov game , or stochastic game defined as a tuple ⟨ S, A , R , T , γ ⟩ with states S , joint actions A

Original: A = { A i } ni
Revised: = { A i } ni

Original: =1 as a collection ofaction sets A i , one for each of the n agents, R
Revised: =1 as a collection of action sets A i , one for each of the n agents, R

Original: =
Revised: =

Original: { R i } ni
Revised: { R i } ni

Original: =1 as a collection of reward functions R idefining the reward r i ( a t , s t ) that each agent receives when the joint action a t ∈ A is performedat state s t , and 
Revised: =1 as a collection of reward functions R i defining the reward r i ( a t , s t ) that each agent receives when the joint action a t ∈ A is performed at state s t , and T as the probability distribution over next states when a joint action is performed.

Original: In the partially observable case, the definition also includes a joint observation function, definingthe observation for each agent at each state.
Revised: In the partially observable case, the definition also includes a joint observation function, defining the observation for each agent at each state.

Original: In this framework, at each time step an agent has anexperience e = < S t , A t , R t +1 S t +1 > , where the agent performs action A t at state S t after whichreward R t +1 is received and the next state is S t +1 .
Revised: In this framework, at each time step an agent has an experience e = < S t , A t , R t +1 S t +1 > , where the agent performs action A t at state S t after which reward R t +1 is received and the next state is S t +1 .

Original: We focus here on decentralized execution settingsin which each agent follows its own individual policy π i and seeks to maximize its discountedaccumulated return.
Revised: We focus here on decentralized execution settings in which each agent follows its own individual policy π i and seeks to maximize its discounted accumulated return.

Original: In the special case where rewards are shared between all agents a Markov Gameis also called a decentralized POMDP or dec-POMDP.
Revised: In the special case where rewards are shared between all agents a Markov Game is also called a decentralized POMDP or dec-POMDP.

Original: We do not require this assumption, but we dorequire that the Markov game is symmetric or anonymous, meaning all agents have the same actionand observation spaces and the environment reacts identically to their actions.
Revised: We do not require this assumption, but we do require that the Markov game is symmetric or anonymous, meaning all agents have the same action and observation spaces and the environment reacts identically to their actions.

Original: Among the variety of RL solution approaches [26, 25], we focus here on value-based methods thatuse state and action estimates to find optimal policies.
Revised: Among the variety of RL solution approaches [30, 29], we focus here on value-based methods that use state and action estimates to find optimal policies.

Original: Such methods typically use a value function
Revised: Such methods typically use a value function

Original: V π ( s ) to represent the expected value of following policy π from state s onward, and a Q -function
Revised: V π ( s ) to represent the expected value of following policy π from state s onward, and a Q -function

Original: Q π ( s, a ) , to represent for a given policy π the expected rewards for performing action a in a givenstate s and following π thereafter.
Revised: Q π ( s, a ) , to represent for a given policy π the expected rewards for performing action a in a given state s and following π thereafter.

Original: Q-learning [32] is a temporal difference (td) method that considers the difference in Q values between two consecutive time steps.
Revised: Q-learning [36] is a temporal difference (td) method that considers the difference in Q values between two consecutive time steps.

Original: Formally, the update rule is
Revised: Formally, the update rule is

Original: [Equation]
Revised: [Equation]

Original: Where, α is the learning rate or step size setting the amount by which values are updated duringtraining.
Revised: Where, α is the learning rate or step size setting the amount by which values are updated during training.

Original: The learned action-value function, Q, directly approximates q ∗ , the optimal action-valuefunction.
Revised: The learned action-value function, Q, directly approximates q ∗ , the optimal action-value function.

Original: During training, in order to guarantee convergence to an optimal policy, Q-learning typicallyuses an ϵ -greedy selection approach, according to which the best action is chosen with probability1 − ϵ , and a random action, otherwise.
Revised: During training, in order to guarantee convergence to an optimal policy, Q-learning typically uses an ϵ -greedy selection approach, according to which the best action is chosen with probability 1 − ϵ , and a random action, otherwise.

Original: Given an experience e t , the td-error represents the difference between the Q-value estimate and actualreward gained in the transition and the discounted value estimate of the next best actions.
Revised: Given an experience e t , the td-error represents the difference between the Q-value estimate and actual reward gained in the transition and the discounted value estimate of the next best actions.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: In the vanilla version of Q-learning Q-values are stored in a table, which is impractical in manyreal-world problem due to large state space size.
Revised: In the vanilla version of Q-learning Q-values are stored in a table, which is impractical in many real-world problem due to large state space size.

Original: In deep Q-Learning (DQN) [17], the Q-valuetable is replaced by a function approximator typically modeled using a neural network such that
Revised: In deep Q-Learning (DQN) [20], the Q-value table is replaced by a function approximator typically modeled using a neural network such that

Original: Q ( s, a, θ ) ≈ Q ∗ ( s, a ) , where θ denotes the neural network parameters.
Revised: Q ( s, a, θ ) ≈ Q ∗ ( s, a ) , where θ denotes the neural network parameters.

Original: Replay Buffer (RB) and Prioritized RB:
Revised: Replay Buffer (RB) and Prioritized RB:

Original: In their simplest form, RL algorithms use and discardincoming data after updating the policy.
Revised: In their simplest form, RL algorithms use and discard incoming data after updating the policy.

Original: To increase efficiency an experience replay buffer (Lin, 1992)was suggested.
Revised: To increase efficiency an experience replay buffer (Lin, 1992) was suggested.

Original: This is a memory structure that enables online reinforcement learning algorithms tostore and replay past experiences by sampling them from memory.
Revised: This is a memory structure that enables online reinforcement learning algorithms to store and replay past experiences by sampling them from memory.

Original: This allows mixing experiencesfrom different time steps in to a single policy update, allowing rare experiences to be used more thanonce.
Revised: This allows mixing experiences from different time steps in to a single policy update, allowing rare experiences to be used more than once.

Original: Evidence show the replay buffer to stabilize training of the value function for DQN [16, 17] and toreduce the amount of experiences required for an RL-agent to complete the learning process andachieve convergence [22].
Revised: Evidence shows the replay buffer to stabilize training of the value function for DQN [19, 20] and to reduce the amount of experiences required for an RL-agent to complete the learning process and achieve convergence [25].

Original: Initial approaches that used a replay buffer, uniformly sampled experiences from the buffer.
Revised: Initial approaches that used a replay buffer, uniformly sampled experiences from the buffer.

Original: However,some transitions are more effective for the learning process of an RL agents then others [23].
Revised: However, some transitions are more effective for the learning process of an RL agents then others.

Revised: Transitions may be more or less surprising, redundant, or task-relevant.

Revised: Some transitions may not be immediately useful to the agent, but might become so when the agent competence increases [26].

Original: Prioritized Experience Replay (PER) [22] explores the idea that replaying and learning from sometransitions, rather than others, can enhance the learning process.
Revised: Prioritized Experience Replay (PER) [25] explores the idea that replaying and learning from some transitions, rather than others, can enhance the learning process.

Original: PER suggests replacing the standardsampling method, where transitions are replayed according to the frequency they were collected fromthe environment, with a td-error based method, where transitions are sampled according to the valueof their td-error.
Revised: PER suggests replacing the standard sampling method, where transitions are replayed according to the frequency they were collected from the environment, with a td-error based method, where transitions are sampled according to the value of their td-error.

Original: As a further extension, stochastic prioritization balances between strictly greedy prioritization anduniform random sampling.
Revised: As a further extension, stochastic prioritization balances between strictly greedy prioritization and uniform random sampling.

Original: Hence, the probability of sampling transition i is defined as:
Revised: Hence, the probability of sampling transition i is defined as:

Original: [Equation]
Revised: [Equation]

Original: where p i > 0 is the priority associated to transition i and α determines the weight of the priority( α = 0 is uniform sampling).
Revised: where p i > 0 is the priority associated to transition i and α determines the weight of the priority ( α = 0 is uniform sampling).

Original: The value of p i is determined according to the magnitude of the td-error, such that p i = | δ i | + ϵ , δ i is the td-error and ϵ is a small positive constant that guaranteesthat transitions for which the td-error is zero have a non-zero probability of being sampled from thebuffer.
Revised: The value of p i is determined according to the magnitude of the td-error, such that p i = | δ i | + ϵ , δ i is the td-error and ϵ is a small positive constant that guarantees that transitions for which the td-error is zero have a non-zero probability of being sampled from the buffer.

Original:  suPER: Selective Multi-Agent Prioritized Experience Replay
Revised:  suPER: Selective Multi-Agent Prioritized Experience Replay

Original: Our approach is rooted in the same intuition as PER: that not all experiences are equally relevant.
Revised: Our approach is rooted in the same intuition as PER: that not all experiences are equally relevant.

Original: Weuse this insight to help agents learn by sharing between them only a (small) number of their mostrelevant experiences.
Revised: We use this insight to help agents learn by sharing between them only a (small) number of their most relevant experiences.

Original: Our approach builds on standard off-policy algorithms such as DQN, and addsthis experience sharing mechanism between collecting experiences and performing gradient updates,in each iteration of the algorithm:
Revised: Our approach builds on standard DQN algorithms, and adds this experience sharing mechanism between collecting experiences and performing gradient updates, in each iteration of the algorithm:

Original: (DQN) Collect a rollout of experiences, and insert each agent’s experiences into their own replay buffer.
Revised: (DQN) Collect a rollout of experiences, and insert each agent’s experiences into their own replay buffer.

Original: (suPER) Each agent shares their most relevant experiences, which are inserted into all the other agents’ replay buffers.
Revised: (suPER) Each agent shares their most relevant experiences, which are inserted into all the other agents’ replay buffers.

Original: (DQN)
Revised: (DQN)

Original: Each agent samples a minibatch of experiences from their own replay buffer, and performs gradient descent on it.
Revised: Each agent samples a minibatch of experiences from their own replay buffer, and performs gradient descent on it.

Original: Steps 1 and 3 are standard DQN [18]; we merely add an additional step between collecting experiencesand learning on them.
Revised: Steps 1 and 3 are standard DQN; we merely add an additional step between collecting experiences and learning on them.

Original: As a corollary, this same approach works for any standard variants of steps1 and 3, such as dueling DQN [31], DDQN [30], Rainbow [8] and other DQN improvements.
Revised: As a corollary, this same approach works for any standard variants of stepsand 3, such as dueling DQN [35], DDQN [34], Rainbow [11] and other DQN improvements.

Original: Algorithm 1 gives a more detailed listing of this algorithm.
Revised: Beyond

Original: Beyond DQN and its variants, suPER can Algorithm 1 suPER algorithm for DQN for each training iteration do
Revised: DQN and its variants, suPER could in principle apply to any off-policy RL algorithm.

Original: Collect a batch of experiences b ▷ DQN for each agent i do ▷ DQN Insert b i into buffer i ▷ DQN end for ▷ DQN for each agent i do ▷ suPER Select b ∗ i ⊆
Revised: Algorithmgives a more detailed listing of this algorithm.

Original: b i of experiences to share▷ suPER for each agent j ̸ = i do ▷ suPER
Revised: Notice that the only interaction between the agents training algorithms is in the experience sharing step.

Original: Insert b ∗ i into buffer j ▷ suPER end for ▷ suPER end for ▷ suPER for each agent i do ▷ DQN

Original: Sample a train batch b i from buffer i ▷ DQN Learn on train batch b i ▷ DQN end for ▷ DQN end principle to any off-policy RL algorithm. For instance, in Soft Actor Critic (SAC) [6], theresulting algorithm looks exactly like the DQN variant described in Algorithm 1. The only difference is that the “learn on train batch” step is using the actor-critic model and loss from SAC
Revised: This means that the algorithm can easily be implemented in a decentralized manner with a (limited) communications channel.

Original: 4.1 Experience Selection
Revised: 4.1 Experience Selection

Original: We describe here three variants of the suPER algorithm, that differ in how they select experiences tobe shared.
Revised: We describe here three variants of the suPER algorithm, that differ in how they select experiences to be shared.

Original: Deterministic quantile-based experience selection The learning algorithm keeps a list l of the (absolute) td-errors of its last k experiences ( k = 1500 by default).
Revised: Deterministic quantile-based experience selection The learning algorithm keeps a list l of the (absolute) td-errors of its last k experiences ( k = 1500 by default).

Original: For a configured bandwidth β and a new experience e t , the agent shares the experience if its absolute td-error | td( e t ) | is at least as large as the k ∗ β -largest absolute td-error in l .
Revised: For a configured bandwidth β and a new experience e t , the agent shares the experience if its absolute td-error | td( e t ) | is at least as large as the k ∗ β -largest absolute td-error in l .

Original: In other words, the agent aims to share the top β -quantile of its experiences, where the quantile is calculated over a sliding window of recent experiences.
Revised: In other words, the agent aims to share the top β -quantile of its experiences, where the quantile is calculated over a sliding window of recent experiences.

Original: [Equation] 
Revised: [Equation] do

Revised: Collect a b ▷ DQN for i do ▷ DQN

Revised: ▷ DQN end

Revised: ▷ DQN do ▷ suPER

Original:  Deterministic Gaussian experience selection In this, the learning algorithm calculates the mean µand variance σ 2 of the (absolute) td-errors of the k most recent experiences ( k = 1500 by default).
Revised: ▷ suPER do suPER do ▷ DQN ▷ DQN i ▷ DQN end ▷ DQN end Deterministic Gaussian experience selection In this, the learning algorithm calculates the mean µ and variance σ 2 of the (absolute) td-errors of the k most recent experiences ( k = 1500 by default).

Original: Itthen shares an experience e t if
Revised: It then shares an experience e t if

Original: [Equation]
Revised: [Equation]

Original: where c is a constant chosen such that 1 − cdf N ( c )
Revised: where c is a constant chosen such that 1 − cdf N ( c )

Original: = β .
Revised: = β .

Original: In other words, we use the c -quantile ofa normal distribution with the (sample) mean and variance of most recent experiences.
Revised: In other words, we use the c -quantile of a normal distribution with the (sample) mean and variance of most recent experiences.

Original: We includeand benchmark this variant for two reasons.
Revised: We include and benchmark this variant for two reasons.

Original: One, intuitively, we might want to be more sensitive toclusters of outliers, where using a quantile of the actual data might include only part of the cluster,while a Gaussian model might lead to the entire cluster being included.
Revised: One, intuitively, we might want to be more sensitive to clusters of outliers, where using a quantile of the actual data might include only part of the cluster, while a Gaussian model might lead to the entire cluster being included.

Original: Two, mean and variancecould be computed iteratively without keeping a buffer of recent td-errors, and thereby reducingmemory requirements.
Revised: Two, mean and variance could be computed iteratively without keeping a buffer of recent td-errors, and thereby reducing memory requirements.

Original: We aim to benchmark if this approximation impacts performance.
Revised: We aim to benchmark if this approximation impacts performance.

Original: Stochastic weighted experience selection Finally, and most closely related to classical singleagent PER, this variant shares each experience with a probability that’s proportional to its absolute td-error.
Revised: Stochastic weighted experience selection Finally, and most closely related to classical singleagent PER, this variant shares each experience with a probability that’s proportional to its absolute td-error.

Original: In PER, given a train batch size b , we sample b transitions from the replay buffer without replacement, weighted by each transition’s td-error.
Revised: In PER, given a train batch size b , we sample b transitions from the replay buffer without replacement, weighted by each transition’s td-error.

Original: In suPER, we similarly aim to sample a β fraction of experiences, weighted by their td-errors.
Revised: In suPER, we similarly aim to sample a β fraction of experiences, weighted by their td-errors.

Original: However, in order to be able to sample transitions online, we calculate for each experience individually a probability that approximates samplingwithout-replacement in expectation.
Revised: However, in order to be able to sample transitions online, we calculate for each experience individually a probability that approximates samplingwithout-replacement in expectation.

Original: Formally, taking p i = | td( e i ) | , we broadcast experience e t with probability
Revised: Formally, taking p i = | td( e i ) | , we broadcast experience e t with probability

Original: [Equation]
Revised: [Equation]

Original: similarly to equation 2, and taking the sum over a sliding window over recent experiences.
Revised: similarly to equation 2, and taking the sum over a sliding window over recent experiences.

Original: It iseasy to see that if β = 1 / batchsize , this is equivalent to sampling a single experience, weightedby td-error, from the sliding window.
Revised: It is easy to see that if β = 1 / batchsize , this is equivalent to sampling a single experience, weighted by td-error, from the sliding window.

Original: For larger bandwidth β , this approximates sampling multipleexperiences without replacement:
Revised: For larger bandwidth β , this approximates sampling multiple experiences without replacement:

Original: If none of the β · 
Revised: If none of the β · p

Original: [Equation]
Revised: [Equation]

Original:  p αk terms are greater than 1, this is exact.
Revised: k p αk terms are greater than 1, this is exact.

Original: Ifwe have to truncate any of these terms, we slightly undershoot the desired bandwidth.
Revised: If we have to truncate any of these terms, we slightly undershoot the desired bandwidth.

Original: In our current experiments, we share experiences and update all quantiles, means and variancesonce per sample batch, for convenience and performance reasons; However, we could do both online,i.e. after every sampled transition, in real-world distributed deployments.
Revised: In our current experiments, we share experiences and update all quantiles, means and variances once per sample batch, for convenience and performance reasons; However, we could do both online, i.e. after every sampled transition, in real-world distributed deployments.

Original: We do not expect this tomake a significant difference.
Revised: We do not expect this to make a significant difference.

Original:  Experiments and Results
Revised:  Experiments and Results

Original: We evaluate the suPER approach on a number of multiagent benchmark domains.
Revised: We evaluate the suPER approach on a number of multiagent benchmark domains.

Original: 5.1 Algorithm and control benchmarks
Revised: 5.1 Algorithm and control benchmarks

Original: Baseline: Dueling DDQN As a baseline, we use a fully independent DDQN algorithm.
Revised: Baseline: DQN As a baseline, we use a fully independent DQN algorithm.

Original: We usea modern dueling DDQN implementation from RLlib [13], with DQN hyperparameters largelyunchanged.

Original: We ran this with a single environment thread from which experiences are collected.

Original: Thealgorithm samples a sequence of experiences using joint actions from all the agents’ policies, andthen inserts each agent’s observation-action-reward-observation transitions into that agent’s replaybuffer.
Revised: The algorithm samples a sequence of experiences using joint actions from all the agents’ policies, and then inserts each agent’s observation-action-reward-observation transitions into that agent’s replay buffer.

Original: Each agent’s policy is periodically trained using a sample from its own replay buffer, sampledusing PER.
Revised: Each agent’s policy is periodically trained using a sample from its own replay buffer, sampled using PER.

Original: We refer to this baseline as simply “DQN” in the remainder of this section and in figures.
Revised: We refer to this baseline as simply “DQN” in the remainder of this section and in figures.

Original: Baseline: Indiscriminate sharing-all-experiences DQN As a second baseline, we compare insome experiments against a DQN variant that shares all experiences among agents.

Original: In this variant,whenever a batch of experiences is sampled, every agent’s trajectory is inserted into all the agents’replay buffers.

Original: This is meant to test whether sharing experiences in general is helpful (e.g. byintroducing more variety into each agent’s replay buffer, or cycling experiences through the bufferfaster), or if it is selectivity in sharing that is helpful.

Original: Aside from experience sharing, this algorithmis unchanged from the DQN baseline, i.e. we use a dueling DDQN implementation with the samehyperparameters

Original: [Figure]

Original: suPER-DQN We implement suPER on top of the dueling DDQN baseline.
Revised: suPER-DQN We implement suPER on top of the DQN baseline.

Original: Whenever a batch ofexperiences is sampled, each agent shares only its most relevant experiences (according to one ofthe criteria described in the previous section) which are then inserted into all other agents’ replaybuffers.
Revised: Whenever a batch of experiences is sampled, each agent shares only its most relevant experiences (according to one of the criteria described in the previous section) which are then inserted into all other agents’ replay buffers.

Original: As above, we run the suPER experience sharing on whole sample batches.
Revised: As above, we run the suPER experience sharing on whole sample batches.

Original: All DQNhyperparameters are unchanged - the only difference from the baseline is the addition of experiencesharing between experience collection and learning.
Revised: All DQN hyperparameters are unchanged - the only difference from the baseline is the addition of experience sharing between experience collection and learning.

Original: This allows for controlled experiments withlike-for-like comparisons.
Revised: This allows for controlled experiments with like-for-like comparisons.

Revised: Indiscriminate sharing-all-experiences suPER-DQN We also compare against a DQN variant that shares all experiences among agents.

Original: 5.2 Environments
Revised: In this variant, whenever a batch of experiences is sampled, every agent’s trajectory is inserted into all the agents’ replay buffers. This can be seen as both a suPER variant (when testing if experience sharing is helpful in general) or as a baseline (when testing if selectivity in experience sharing is helpful). Parameter-Sharing DQN In parameter-sharing, all agents share the same policy parameters.

Revised: Note that this is different from joint control, in which a single policy controls all agents simultaneously;

Original: SISL: Pursuit is  a semi-cooperative environment, where a group of pursuers has to capture a groupof evaders in  a grid-world with an obstacle.
Revised: In parameter sharing, each agent is controlled independently by a copy of the policy. It is also different from share-all suPER-DQN, in which a different policy is trained for each agent. DQN and Dueling DDQN For all of the above, we consider both variants based on standard DQN, as well as variants based on dueling double DQN (DDQN). In the remainder of the text, whenever we write DDQN we mean dueling double DQN, and similarly for suPER-DDQN, parameter-sharing

Revised: DDQN, etc.

Revised: Whenever we write DQN we mean standard non-dueling, non-double DQN.

Original:  The evaders (blue) move randomly, while the pursuers(red) are controlled by RL agents.
Revised: Multi-Agent Baselines We further compare against standard multi-agent RL algorithms, specifically MADDPG [18], QMIX [24] and SEAC [5]. Like parameter-sharing, these are all considered “centralized training, decentralized execution” approaches. 5.2 Environments SISL: Pursuit is a semi-cooperative environment, where a group of pursuers has to capture a group of evaders in a grid-world with an obstacle, The evaders (blue) move randomly, while the pursuers (red) are controlled by RL agents.

Original: If a group of two or more agents fully surround an evader, theyeach receive a reward, and the evader is removed from the environment.
Revised: If a group of two or more agents fully surround an evader, they each receive a reward, and the evader is removed from the environment.

Original: The episode ends when allevaders have been captured, or after 500 steps, whichever is earlier.
Revised: The episode ends when all evaders have been captured, or after 500 steps, whichever is earlier.

Original: Pursuers also receive a (verysmall) reward for being adjacent to an evader (even if the evader is not fully surrounded), and a(small) negative reward each timestep, to incentivize them to complete episodes early. 
Revised: Pursuers also receive a (very small) reward for being adjacent to an evader (even if the evader is not fully surrounded), and a (small) negative reward each timestep, to incentivize them to complete episodes early. We usepursuers and 30 evaders.

Original: MAgent: Battle is a semi-adversarial environment, where two groups of opposing teams arebattling against each other.
Revised: MAgent: Battle is a semi-adversarial environment, where two groups of opposing teams are battling against each other.

Original: An agent is rewarded 0 .
Revised: An agent is rewarded 0 .

Original: 2 points for attacking agents in the opposite team,and 5 points if the other agent is killed.
Revised: 2 points for attacking agents in the opposite team, and 5 points if the other agent is killed.

Original: All agents start with 10 health points (HP) and lose 2 HP ineach attack received, while regaining 0 .
Revised: All agents start with 10 health points (HP) and lose 2 HP in each attack received, while regaining 0 .

Original: 1 HP in every turn.
Revised: 1 HP in every turn.

Original: Once killed, an agent is removed from theenvironment.
Revised: Once killed, an agent is removed from the environment.

Original: An episode ends when all agents from one team are killed.
Revised: An episode ends when all agents from one team are killed.

Original: The action space, of size 21is identical for all agents, with (8) options to attack, (12) to move and one option to do nothing.
Revised: The action space, of sizeis identical for all agents, with (8) options to attack, (12) to move and one option to do nothing.

Original: Sinceno additional reward is given for collaborating with other agents in the same team, it is considered tobe more challenging to form collaboration between agents in this environment.
Revised: Since no additional reward is given for collaborating with other agents in the same team, it is considered to be more challenging to form collaboration between agents in this environment.

Original: [Figure]
Revised: We use a map of size 18 × 18 and 6 agents per team.

Original: MAgent: Adversarial Pursuit is a predator-prey environment, with two types of agents, prey andpredator.
Revised: MAgent: Adversarial Pursuit is a predator-prey environment, with two types of agents, prey and predator.

Original: The predators navigate through obstacles in the map with the purpose of tagging the prey.
Revised: The predators navigate through obstacles in the map with the purpose of tagging the prey.

Revised: [Figure]

Original: An agent in the predators team is rewarded 1 point for tagging a prey, while a prey is rewarded − 1when being tagged by a predator.
Revised: An agent in the predators team is rewarded 1 point for tagging a prey, while a prey is rewarded −when being tagged by a predator.

Original: Unlike in the Battle environment, prey agents are not removed fromthe game when being tagged.
Revised: Unlike in the Battle environment, prey agents are not removed from the game when being tagged.

Original: Note that prey agents are provided only with a negative or zero reward(when manage to avoid attacks), and their aim is thus to evade predator agents. 
Revised: Note that prey agents are provided only with a negative or zero reward (when manage to avoid attacks), and their aim is thus to evade predator agents. We use 8 prey agents, and 4 predator agents.

Original: 5.3 Performance Evaluation
Revised: 5.3 Experimental Setup

Revised: In Pursuit, we train all agents concurrently using the same algorithm, for each of the algorithms listed.

Revised: Note that only the pursuers are agents in this domain, whereas the evaders move randomly.

Original: Figure 1 shows learning curves of suPER-DQN, with gaussian and quantile experience selection andtarget bandwidth 0.1, compared to standard no-sharing DQN.
Revised: In the standard variants of Battle and Adversarial-Pursuit, we first pre-train a set of agents using independent dueling DDQN, all agents on both teams being trained together and independently. We then take the pre-trained agents of the opposing team (red team in Battle, predator team in Adversarial-Pursuit), and use these to control the opposing team during main training of the blue respectively prey team. In this main training phase, only the blue / prey team agents are trained using each of the suPER and benchmark algorithms, whereas the red / predator team are controlled by the pre-trained policies with no further training. Figures 1 and 3 show learning curves from the main training phase.

Original: We see that both suPER-DQN variantsoutperform no-sharing DQN on  all three domains.
Revised: Figure 1 shows learning curves of suPER implented on dueling DDQN (“suPER DDQN”), with stochastic, gaussian and quantile experience selection and target bandwidth 0.1, compared to standard no-sharing dueling DDQN, share-all suPER-DDQN, parameter-sharing dueling DDQN, as well as MADDPG, QMIX and SEAC. Figure 3 similarly shows suPER implemented on (non-dueling, non-double) DQN (“suPER DQN”) ocmpared to non-sharing DQN, share-all suPER-DQN, parametersharing DQN, and the same multiagent algorithms as above (MADDPG, QMIX, SEAC; learning curves duplicated from Figure 1 for comparison). In both Figures 1 and 3 in Battle and AdversarialPursuit, the opposing team consists of agents pre-trained using non-sharing dueling DDQN (the same set of opposing agents are used for all the algorithms shown).

Original: In Pursuit, suPER-DQN achieves over twice thereward of no-sharing DQN at convergence, increasing from 225.9 (std.dev 22.0) to 457.8 (std.dev55.7) for quantile suPER-DQN measured at 2M training steps.
Revised: In Figure 4 we show a subset of algorithms (suPER-DDQN, DDQN and share-all suPER-DDQN) training against a set of co-evolving agents. In this, the opposing team agents are also randomized at the start of training, and train together with the first team, but the opposing team agents do not utilize suPER experience sharing. Comparison Against Baseline DQN We find that suPER (red curves) consistently outperforms the baseline DQN/DDQN algorithm (solid green curve), often significantly. For instance for DDQN (Fig. 1), in Pursuit suPER-DDQN achieves over twice the reward of no-sharing DDQN at convergence, increasing from 180.7 (std.dev 2.8) to 450.5 (std.dev 9.9) for quantile suPER-DDQN measured at 800k training steps.

Original: In both Battle and Adversarial-Pursuit,we enabled suPER-sharing for only one of the two teams each, and see that this significantly improvedperformance of the sharing team (as measured by sum of rewards of all agents in the team acros eachepisode).
Revised: In both Battle and Adversarial-Pursuit, we enabled suPER-sharing for only one of the two teams each, and see that this significantly improved performance of the sharing team (as measured by sum of rewards of all agents in the team across each episode), especially mid-training.

Original: In Battle, the blue team performance increase from -11.4 (std.dev 4.8) for no-sharing DQNto 16.2 (std.dev 5.4) for quantile suPER-DQN.
Revised: For instance in Battle, the blue team performance increase from -19.0 (std.dev 11.0) for no-sharing DDQN to 5.5 (std.dev 8.7) for quantile suPER-DDQN at 150k timesteps.

Original: In Adversarial-Pursuit, the prey performance increasedfrom -558.7 (std.dev 53.4) to -373.4 (std.dev 77.1).
Revised: In Adversarial-Pursuit, the prey performance increased from -712.3 (std.dev 55.3) to -508.0 (std.dev 34.2) at 300k timesteps.

Original: Interestingly, in both of those environment thisperformance gain did not come purely at the expense of the other, non-sharing team, whose relativeperformance drop was smaller than the sharing teams’ gains in both environments. We hypothesizethat suPER may help agents avoid environmental penalties in addition to improving performanceagainst the opposing team. Surprisingly, in the Adversarial-Pursuit domain, sharing between thepredator agents had a much smaller effect.
Revised: In the Appendix, we similarly see that suPER-DQN outperforms DQN in Figure 3 in both environments, and that suPER-DDQN significantly outperforms DDQN with co-evolving opponent agents especially early in training (Fig. 4).

Original: 5.4 Bandwidth Sensitivity
Revised: Selective and Indiscriminate Sharing

Original: In the Pursuit domain, we performed an analysis of the performance of suPER-DQN for varyingtarget bandwidths ranging from 0.0001 to 1 (sharing all experiences). Figure 2 (left) shows theconverged performance of suPER-DQN with quantile-based experience selection in Pursuit. A labelof “DQN” indicates that no sharing is taking place, i.e. decentralized DQN; a label of “all” indicatesthat all experiences are shared, without any prioritized selection.
Revised: When comparing suPER with bandwidth 0.1 (red lines) against suPER with indiscriminate sharing ("DDQN share-all", dashed orange line), we see that selective experience sharing produces a much greater and more consistent performance improvement than indiscriminate sharing. This is most extreme in Pursuit, where indiscriminate sharing performs similar to, but slightly worse than, the DQN/DDQN baselines at convergence.

Original: Numerical labels give differenttarget bandwidths. Two things stand out to us: First, sharing all experiences indiscriminately does notresult in increased performance. In fact, at convergence, it results in slightly lower performance thanno-sharing DQN. Second, there isa clear peak of performance around a target bandwidth of 0.01- 0.1, which also holds for gaussian and stochastic experience selection (we refer the reader to theappendix for more details).
Revised: It is also visible in Adversarial-Pursuit especially early in training, but not significant in Battle.

Original: We conclude that  sharing experiences selectively is crucial for learning tobenefit from it. [Figure]
Revised: We note that indiscriminate sharing still outperforms baselines early in training even in Pursuit.

Original: Battle, 400k timesteps in Adversarial-Pursuit. Furthermore, it is noteworthy that early in training, even small bandwidths show significantly higherperformance than no-sharing DQN. Figure 2 (right) shows performance at 250k timesteps, whereeven a target bandwidth of 0.0001 shows significantly improved performance over baseline. Thissuggests the applicability of suPER to settings where agents need to adapt rapidly to an unknownenvironment, even if communication is very limited.
Revised: Comparison Against Other Multi-Agent RL Algorithms suPER performs significantly better than SEAC (solid blue), MADDPG (dashed violet) and QMIX (dotted purple).

Original: Indiscriminate sharing of all experiences againdid not show a performance benefit at this stage of training.
Revised: MADDPG and QMIX performed very poorly on all domains despite extensive attempts at hyperparameter tuning.

Original: It did show increased performance(similar to selective sharing) very early intraining, which we attribute to the faster replay buffer fillrate it entails. After replay buffers had been filled this performance improvement vanished.
Revised: SEAC shows some learning in Pursuit, but remains well below even baseline DDQN performance, again despite extensive hyperparameter tuning. We have also run experiments with SEAC in Pursuit for significantly longer (not shown in the figure), and saw that performance eventually settles around 100-200 episode reward depending on hyperparamters and random seed.

Original: We find that at bandwidth 0.1, all three experience selection variants performed similarly at convergence. Stochastic experience selection however consistently converge slower than both deterministicvariants and had lower performance early in training (but still higher than no-sharing baseline).
Revised: One experiment showed signs of additional learning much later on and began to surpass 200 episode reward (baseline DDQN) around 1.7M timesteps.

Original: Gaussian experience selection performed similarly to the quantile selection we designed it to approximate. Its actual used bandwidth was however much less responsive to target bandwidth than the other twovariants (e.g. in Pursuit at target bandwith 0.001, Gaussian suPER consumed 0.018 actual bandwidth). We believe this demonstrates that in principle approximating the actual distribution of td-errors usingmean and standard deviation is feasible, but that more work is needed in determining the optimalvalue of c in equation 3. 5.6 Stability across hyperparameters
Revised: However, we have not seen SEAC approach suPER performance in any experiment.

Original: We evaluate suPER across a range of hyperparameter settings such as differing train batch sizes andexploration configurations. Figure 3 shows performance of no-sharing DQN and suPER-DQN fordifferent hyperparameters. As we can see, suPER-DQN outperforms no-sharing DQN consistentlyacross all the hyperparameter settings considered.
Revised: Comparison Against Parameter-Sharing suPER (red) significantly outperforms parametersharing DQN/DDQN (dotted turquoise line) in Pursuit and performs similar to it in Battle, whereas parameter-sharing performs significantly better in Adversarial-Pursuit. This holds for both the respective DQN and DDQN variants in Pursuit and Adversarial-Pursuit.

Original:   Discussion
Revised:  Conclusion & Discussion

Original:  We present selective multiagent PER, a selective experience-sharing mechanism that can improveoff-policy RL algorithm in multiagent settings.
Revised: Conclusion We present selective multiagent PER, a selective experience-sharing mechanism that can improve DQN-family algorithms in multiagent settings.

Original: Conceptually, our approach is rooted in the sameintuition that Prioritized Experience Replay was based on, which is that td-error is a useful approximation of how much an agent could learn from a particular experience.
Revised: Conceptually, our approach is rooted in the same intuition that Prioritized Experience Replay was based on, which is that td-error is a useful approximation of how much an agent could learn from a particular experience.

Original: Experimental evaluation ondueling DDQN shows improved performance  (as measured in sample efficiency and/or convergedperformance) across a range of hyperparameters of the underlying algorithm, and in multiple benchmark domains.
Revised: Experimental evaluation on DQN and dueling DDQN shows improved performance compared to fully decentralized training (as measured in sample efficiency and/or converged performance) across a range of hyperparameters of the underlying algorithm, and in multiple benchmark domains.

Original: Wegenerally see significantly higher performance  with suPER and a target bandwidthof 0.01-0.1 late in training, suggesting that the approach can help agents learn to coordinate better.
Revised: We see most consistent performance improvements with suPER and a target bandwidth of 0.01-0.1 late in training, more consistent than indiscriminate experience sharing.

Original: Given that this effect appeared consistently across a wide range of hyperparameters and multipleenvironments, the suPER approach may be useful as a general-purpose multi-agent RL technique.
Revised: Given that this effect appeared consistently across a wide range of hyperparameters and multiple environments, as well as on both DQN and dueling DDQN, the suPER approach may be useful as a general-purpose multi-agent RL technique.

Original: Equally noteworthy is a significantly improved performance early in training even at very low bandwidths.
Revised: Equally noteworthy is a significantly improved performance early in training even at very low bandwidths.

Original: We consider this to be a potential advantage in future real-world applications of RL wheresample efficiency and rapid adaption to new environments are crucial.
Revised: We consider this to be a potential advantage in future real-world applications of RL where sample efficiency and rapid adaption to new environments are crucial.

Original: It is noteworthy that deterministic (aka “greedy”) experience selection seems to perform slightlybetter than stochastic experience selection, while in PER the opposite is generally the case [22].
Revised: suPER consistently and significantly outperforms MADDPG, QMIX and SEAC, and outperforms parameter sharing in Pursuit (but underperforms in Adversarial-Pursuit, and shows equal performance in Battle). Discussion Our selective experience approach improves performance of both DQN and dueling DDQN baselines, and does so across a range of environments and hyperparameters.

Revised: It outperforms state-of-the-art multi-agent RL algorithms, in particular MADDPG, QMIX and SEAC.

Revised: The only pairwise comparison that suPER loses is against parameter sharing in Adversarial-Pursuit, in line with a common observation that in practice parameter sharing often outperforms sophisticated multi-agent

Revised: RL algorithms.

Revised: However, we note that parameter sharing is an entirely different, fully centralized training paradigm.

Revised: Furthermore, parameter sharing is limited in its applicability, and does not work well if agents need to take on different roles or behavior to successfully cooperate.

Revised: We see this in the Pursuit domain, where parameter sharing performs poorly, and suPER outperforms it by a large margin.

Revised: The significantly higher performance than QMIX, MADDPG and SEAC is somewhat expected given that baseline non-sharing DQN algorithms often show state-of-the-art performance in practice, especially with regard to sample efficiency.

Revised: It is noteworthy that deterministic (aka “greedy”) experience selection seems to perform slightly better than stochastic experience selection, while in PER the opposite is generally the case [25].

Original: Wehave two hypotheses for why this is the case.
Revised: We have two hypotheses for why this is the case.

Original: One, we note that in PER, the motivation for stochasticprioritization is to avoid low-error experiences never being sampled (nor re-prioritized) in many drawsfrom the buffer.
Revised: One, we note that in PER, the motivation for stochastic prioritization is to avoid low-error experiences never being sampled (nor re-prioritized) in many draws from the buffer.

Original: On the other hand, in suPER we only ever consider each experience once.
Revised: On the other hand, in suPER we only ever consider each experience once.

Original: Thus, ifin stochastic experience selection a high-error experience through random chance is not shared onthis one opportunity, it will never be seen by other agents.
Revised: Thus, if in stochastic experience selection a high-error experience through random chance is not shared on this one opportunity, it will never be seen by other agents.

Original: In a sense, we may prefer deterministicexperience selection in suPER for the same reason we prefer stochastic selection in PER, which is toavoid missing out on potentially valuable experiences.
Revised: In a sense, we may prefer deterministic experience selection in suPER for the same reason we prefer stochastic selection in PER, which is to avoid missing out on potentially valuable experiences.

Original: Two, in all our current experiments we used(stochastic) PER when sampling training batches from the replay buffer of each agent.
Revised: Two, in all our current experiments we used (stochastic) PER when sampling training batches from the replay buffer of each agent.

Original: When usingstochastic suPER, each experience therefore must pass through two sampling steps before beingshared and learned on by another agent.
Revised: When using stochastic suPER, each experience therefore must pass through two sampling steps before being shared and trained on by another agent.

Original: It is possible that this dilutes the probability of a high-errorexperience being seen too much.
Revised: It is possible that this dilutes the probability of a high-error experience being seen too much.

Revised: Our algorithm is different from the “centralized training, decentralized execution” baselines we compare against in the sense that it does not require fully centralized training.

Revised: Rather, it can be implemented in a decentralized fashion with a communications channel between agents.

Revised: We see that performance improvements scale down even to very low bandwidth, making this feasible even with limited bandwidth.

Revised: We think of this scheme as “decentralized training with communication” and hope this might inspire other semi-decentralized algorithms.

Original:  We  also want to point out a slight subtlety in our theoretical motivation for suPER: We use the sendingagent’s td-error as a proxy for the usefulness of an experience for the receiving agent.
Revised: In addition to training, we note that such a “decentralized with communication” approach could potentially be deployed during execution, if agents keep learning. While this is beyond the scope of the current paper, in future work we would like to investigate if this could help when transferring agents to new domains, and in particular with adjusting to a sim-to-real gap. Our work also shows that communication in multi-agent RL can improve training performance. We would also like to point out a slight subtlety in our theoretical motivation for suPER: We use the sending agent’s td-error as a proxy for the usefulness of an experience for the receiving agent.

Original: We believethat this is justified in symmetric settings, and our experimental results support this.
Revised: We believe that this is justified in symmetric settings, and our experimental results support this.

Original: However, forfuture work, we would be interested to see if, first, using the td-error of each (potential) receivingagent could extend this approach to asymmetric settings, and if it could further improve performanceeven in symmetric settings.
Revised: However, we stress that this is merely a heuristic, and one which we do not expect to work in entirely asymmetric domains. For future work, we would be interested to explore different experience selection heuristics. As an immediate generalization to a more theoretically grounded approach, we wonder if using the td-error of each (potential) receiving agent could extend suPER to asymmetric settings, and if it could further improve performance even in symmetric settings.

Original: While this would effectively be a centralized-training approach, if itshowed similar performance benefits as we have seen in symmetric settings for suPER, it couldnevertheless be a promising avenue for further work.
Revised: While this would effectively be a centralized-training approach, if it showed similar performance benefits as we have seen in symmetric settings for suPER, it could nevertheless be a promising avenue for further work.

Original: Secondly, we would be interested to see ifthe sending agent could learn to approximate each receiver’s td-error locally, and thus retain thedecentralized-with-communication training capability of our current approach.
Revised: Beyond this, we would be interested to explore other heuristics for experience selection. For instance, we are curious if the sending agent could learn to approximate each receiver’s td-error locally, and thus retain the decentralized-with-communication training capability of our current approach.

Original: However, given thattd-error is intrinsically linked to current policy and thus highly non-stationary, we expect there wouldbe significant practical challenges to this approach.
Revised: However, given that td-error is intrinsically linked to current policy and thus highly non-stationary, we expect there would be significant practical challenges to this.

Original: Finally, wenote that while we mainly focus on the DQN family of algorithms in this paper, suPERcould be applied to  other off-policy algorithm as well.
Revised: Finally, we focus on the DQN family of algorithms in this paper. In future work, we would like to explore suPER in conjunction with other off-policy RL algorithms such as SAC [9, 17] and DDPG [27].

Revised: The interplay with experience sampling methods other than PER, such as HER [3] would also be interesting.

Original: If the improvements we see in this work holdfor other algorithms and domains as well, this could improve multi-agent RL performance in manysettings.
Revised: If the improvements we see in this work hold for other algorithms and domains as well, this could improve multi-agent RL performance in many settings. A Additional Experiment Analysis A.1 Co-Evolving Team Experiments

Revised: In Battle and Adversarial-Pursuit we further show a variant where the opposing team are co-evolving with the blue / prey team.

Revised: In this variant, all agents start from a randomly initialized policy and train concurrently, using a DDQN algorithm.

Original: A Additional Analysis of Bandwidth Sensitivity
Revised: However, only the blue / prey team share experiences using the suPER mechanism. We only do this for the DDQN baseline as well as discriminate and share-all suPER variants. This is in part because some of the other baseline algorithms do not support concurrently training opposing agents with a different algorithm in available implementations; and in part because we consider this variant more relevant to real-world scenarios where fully centralized training may not be feasible. We aim to show here how sharing even a small number of experiences changes the learning dynamics versus to non-sharing opponents.

Revised: Figure 4 shows this variant.

Revised: A.2 Bandwidth Sensitivity

Revised: In the Pursuit domain, we performed an analysis of the performance of suPER-DDQN for varying target bandwidths ranging from 0.0001 to 1 (sharing all experiences).

Revised: Figure 2 (left) shows the converged performance of suPER-DDQN with quantile-based experience selection in Pursuit.

Revised: A label of “DQN” indicates that no sharing is taking place, i.e. decentralized DDQN; a label of “all” indicates that all experiences are shared, without any prioritized selection.

Revised: Numerical labels give different target bandwidths.

Revised: Two things stand out to us: First, sharing all experiences indiscriminately does not result in increased performance.

Revised: In fact, at convergence, it results in slightly lower performance than no-sharing DDQN.

Revised: Second, there is a clear peak of performance around a target bandwidth of 0.

Revised: - 0.1, which also holds for gaussian and stochastic experience selection (we refer the reader to the appendix for more details).

Revised: We conclude that sharing experiences selectively is crucial for learning to benefit from it.

Revised: Furthermore, it is noteworthy that early in training, even small bandwidths show significantly higher performance than no-sharing DDQN.

Revised: Figure 2 (right) shows performance at 250k timesteps, where even a target bandwidth of 0.0001 shows significantly improved performance over baseline.

Revised: This suggests the applicability of suPER to settings where agents need to adapt rapidly to an unknown environment, even if communication is very limited.

Revised: Indiscriminate sharing of all experiences again did not show a performance benefit at this stage of training.

Revised: It did show increased performance (similar to selective sharing) very early in training, which we attribute to the faster replay buffer fill rate it entails.

Revised: After replay buffers had been filled this performance improvement vanished.

Revised: A.3 Experience Selection

Revised: We see that at target bandwidth 0.1, quantile (solid red lines) and gaussian (dashed red) experience selection perform very similar across all domains.

Revised: Stochastic experience selection (dotted red) performs similar or worse than both other variants, but generally still comparably or better than

Revised: [Figure]

Revised: [Figure]

Revised: baseline DQN/DDQN.

Revised: Gaussian experience selection performed similarly to the quantile selection we designed it to approximate.

Revised: Its actual used bandwidth was however much less responsive to target bandwidth than the other two variants (e.g. in Pursuit at target bandwidth 0.001, Gaussian suPER consumed 0.018 actual bandwidth).

Revised: We believe this demonstrates that in principle approximating the actual distribution of td-errors using mean and standard deviation is feasible, but that more work is needed in determining the optimal value of c in equation 3.

Revised: A.4 Stability Across Hyperparameters

Revised: We evaluate suPER across a range of hyperparameter settings such as differing train batch sizes and exploration configurations, and find that it consistently improves performance over baseline DDQN in al settings.

Revised: Appendix B shows this in more detail.

Revised: B Stability across hyperparameters

Revised: Figure 5 shows performance of no-sharing DDQN and suPER-DDQN for different hyperparameters.

Revised: As we can see, suPER-DDQN outperforms no-sharing DDQN consistently across all the hyperparameter settings considered.

Revised: [Figure]

Revised: otherwise identical to those used in Figure 1.

Revised: Performance measured at 1M timesteps in Pursuit, 300k timesteps in Battle, 400k timesteps in Adversarial-Pursuit.

Original: We present here a more detailed analysis of bandwidth sensitivity of suPER-DQN in the threeexperience selection modes we discuss in the main text.
Revised: We present here a more detailed analysis of bandwidth sensitivity of suPER-DDQN in the three experience selection modes we discuss in the main text.

Original: Figure 4 shows the mean performance acrossfive seeds for gaussian (left), quantile (middle) and stochastic (right) experience selection, at 1-2Mtimesteps (top) and at 250k timesteps (bottom).
Revised: Figure 6 shows the mean performance across five seeds for gaussian (left), quantile (middle) and stochastic (right) experience selection, at 1-2M timesteps (top) and at 250k timesteps (bottom).

Original: We can see that at 1-2M timesteps and a targetbandwidth of 0 .
Revised: We can see that at 1-2M timesteps and a target bandwidth of 0 .

Original: 1 , all three experience selection criteria perform similary.
Revised: 1 , all three experience selection criteria perform similary.

Original: One thing that stands outis that stochastic selection has much lower performance at other target bandwidths, and also muchless performance uplift compared to no-sharing DQN at 250k timesteps at any bandwidth.
Revised: One thing that stands out is that stochastic selection has much lower performance at other target bandwidths, and also much less performance uplift compared to no-sharing DDQN at 250k timesteps at any bandwidth.

Original: Gaussianexperience selection appears to be less sensitive to target bandwidth, but upon closer analysis wefound that it also was much less responsive in terms of how much actual bandwidth it used at differentsettings.
Revised: Gaussian experience selection appears to be less sensitive to target bandwidth, but upon closer analysis we found that it also was much less responsive in terms of how much actual bandwidth it used at different settings.

Original: Figure 5 (left) shows the actual bandwidth used by each selection criterion at different targetbandwidths.
Revised: Figure 7 (left) shows the actual bandwidth used by each selection criterion at different target bandwidths.

Original: We can see that quantile and stochastic experience hit their target bandwidth very well in 
Revised: We can see that quantile and stochastic experience hit their target bandwidth very well in general.

Original: [Figure] general. 1 What stands out, however, is that gaussian selection vastly overshoots the target bandwidthat lower settings, never going significantly below 0.01 actual bandwidth.
Revised: What stands out, however, is that gaussian selection vastly overshoots the target bandwidth at lower settings, never going significantly below 0.01 actual bandwidth.

Original: What is a fairer comparison therefore is to look at performance versus actual bandwidth used for eachof the approaches, which we do in Figure 5 (middle, at 1-2M timesteps, and right, at 250k timesteps).
Revised: What is a fairer comparison therefore is to look at performance versus actual bandwidth used for each of the approaches, which we do in Figure 7 (middle, at 1-2M timesteps, and right, at 250k timesteps).

Original: For these figures, we did the following: First, for each experience selection approach and target bandwidth, we computed the mean performance and mean actual bandwidth across the five seeds.
Revised: For these figures, we did the following: First, for each experience selection approach and target bandwidth, we computed the mean performance and mean actual bandwidth across the five seeds.

Original: Then,for each experience selection mode, we plotted these (meanactualbandwidth , meanperformance)(one for each target bandwidth) in a line plot.
Revised: Then, for each experience selection mode, we plotted these (meanactualbandwidth , meanperformance) (one for each target bandwidth) in a line plot.

Original: The result gives us a rough estimate of how eachapproach’s performance varies with actual bandwidth used.
Revised: The result gives us a rough estimate of how each approach’s performance varies with actual bandwidth used.

Original: We see again that stochastic selectionshows worse performance than quantile at low bandwidths, and early in training.
Revised: We see again that stochastic selection shows worse performance than quantile at low bandwidths, and early in training.

Original: We also see thatgaussian selection very closely approximates quantile selection.
Revised: We also see that [Figure] gaussian selection very closely approximates quantile selection.

Original: Notice that gaussian selection neverhits an exact actual bandwidth of 0.1, and so we cannot tell from these data if it would match quantileselection’s performance at its peak.
Revised: Notice that gaussian selection never hits an exact actual bandwidth of 0.1, and so we cannot tell from these data if it would match quantile selection’s performance at its peak.

Original: However, we can see that at the actual bandwidths that gaussianselection does hit, it shows very similar performance to quantile selection.
Revised: However, we can see that at the actual bandwidths that gaussian selection does hit, it shows very similar performance to quantile selection.

Original: As stated in the maintext, our interpretation of this is that using mean and variance to approximate the exact distributionof absolute td-errors is a reasonable approximation, but that we might need to be more clever inselecting c in equation 3.
Revised: As stated in the main text [Figure] our interpretation of this is that using mean and variance to approximate the exact distribution of absolute td-errors is a reasonable approximation, but that we might need to be more clever in selecting c in equation 3.

Original: B Experiment Hyperparameters & Details
Revised: D Experiment Hyperparameters & Details

Original: We performed all experiments using the open-source library RLlib [13], version 1.13.0, availableunder Apache license.
Revised: We performed all experiments using the open-source library RLlib [16]. Experiments in Figureand 3 were ran using RLlib version 2.0.0; experiments in other figures were run using version 1.13.0.

Original: Environments used are from PettingZoo [29], including SISL [5] and MAgent[34].
Revised: Environments used are from PettingZoo [33], including SISL [8] and MAgent [38].

Original: The suPER algorithm was implemented by modifying RLlib’s standard DQN algorithm to[Figure] perform the suPER experience sharing between rollout and training.
Revised: The suPER algorithm was implemented by modifying RLlib’s standard DQN algorithm to perform the suPER experience sharing between rollout and training.

Original: Table 1 lists all the DQNhyperparameters and environment settings we used for all the experiments.
Revised: Table 3 lists all the algorithm hyperparameters and environment settings we used for all the experiments.

Original: Experiments in the“Stability across hyperparameters” section had hyperparameters set to those listed in Table 1 exceptthose specified in Figure 3.
Revised: Experiments in the “Stability across hyperparameters” section had hyperparameters set to those listed in Table 3 except those specified in Figure 5.

Original: Any parameters not listed were left at their default values from RLlib1.13.0, in particular we used dueling DDQN.
Revised: Any parameters not listed were left at their default values.

Original: Hyperparameters were tuned using a grid search; someof the combinations tested are also discussed in the “Stability across hyperparameters” section.
Revised: Hyperparameters were tuned using a grid search; some of the combinations tested are also discussed in the “Stability across hyperparameters” section.

Revised: For DQN, DDQN and their suPER variants, we found hyperparameters using a grid search on independent DDQN in each environment, and then used those hyperparameters for all DQN/DDQN and suPER variants in that environment.

Revised: For all other algorithms we performed a grid search for each algorithm in each environment.

Revised: For MADDPG and QMIX, we attempted further optimization using the Python HyperOpt package [4], however yielding no significant improvement over our manual grid search.

Original: All experiments were repeated with five seeds, except exploration-configuration experiments in the“Stability across hyperparameters” section, which were repeated with three seeds.
Revised: For SEAC, we performed a grid search in each environment, but found no better hyperparameters than the default. We found a CNN network architecture using manual experimentation in each environment, and then used this architecture for all algorithms except MADDPG and QMIX where we used a fully connected net for technical reasons. We tested all other

Revised: [Figure]

Revised: algorithms using both the hand-tuned CNN as well as a fully connected network, and found that the latter performed significantly worse, but still reasonable (and in particular significantly better than

Revised: MADDPG and QMIX using the same fully connected network, on all domains).

Revised: All experiments were repeated with three seeds.

Original: All plots showthe mean and standard deviation of these seeds at each point in training.
Revised: All plots show the mean and standard deviation of these seeds at each point in training.

Original: For technical reasons,individual experiment runs did not always report data at identical intervals.
Revised: For technical reasons, individual experiment runs did not always report data at identical intervals.

Original: For instance, one runmight report data when it had sampled 51000 environment timesteps, and another run might reportat 53000 environment timesteps.
Revised: For instance, one run might report data when it had sampledenvironment timesteps, and another run might report at 53000 environment timesteps.

Original: In order to still be able to report a meaningful mean and standarddeviation across repeated runs, we rounded down the timesteps reported to the next 10000 steps, i.e.we would take both the data above to represent each run’s performance at 50000 steps. 
Revised: In order to still be able to report a meaningful mean and standard deviation across repeated runs, we rounded down the timesteps reported to the nearest k steps, i.e. taking both the data above to represent each run’s performance at 50000 steps. We set k to the target reporting interval in each domain (8000 timesteps in Pursuit, 6000 timesteps in the other two domains).

Original: Where a runreported more than once in a 10000 step interval, we took the mean of its reports to represent thatrun’s performance in the interval.
Revised: Where a run reported more than once in a 10000 step interval, we took the mean of its reports to represent that run’s performance in the interval.

Original: Mean and standard deviation were calculated across this meanperformance for each of the five seeds.
Revised: Mean and standard deviation were calculated across this mean performance for each of the five seeds.

Revised: To increase legibility, we applied smoothing to Figures 1 and 3 using an exponential window with α = 0 .

Revised: 3 for Pursuit, α = 0 .

Revised: 1 for Battle, and α = 0 .

Revised: 25 for Adversarial-Pursuit.

Revised: This removes some noise from the reported performance, but does not change the relative ordering of any two curves.

Original: C Implementation & Reproducibility
Revised: E Implementation & Reproducibility

Original: All source code is included in the supplementary material and will be made available on publicationunder an open-source license.
Revised: All source code is included in the supplementary material and will be made available on publication under an open-source license.

Original: We refer the reader to the included README file, which containsinstructions to recreate the experiments discussed in this paper. 
Revised: We refer the reader to the included README file, which contains instructions to recreate the experiments discussed in this paper. [Table] [Table] [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: NN

Revised: [Table]

Revised: RLlib default

Revised: [Table]

Revised: [Table]

Revised: NN architecture

Revised: RLlib default

Revised: [Table]