Original: P ROMOTING C OORDINATION THROUGH P OLICY R EGULARIZATION IN M ULTI -A GENT D EEP R EINFORCEMENT L EARNING
Revised: P ROMOTING C OORDINATION THROUGH P OLICY R EGULARIZATION IN M ULTI -A GENT D EEP R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A central challenge in multi-agent reinforcement learning is the induction of coordination between agents of a team.
Revised: A central challenge in multi-agent reinforcement learning is the induction of coordination between agents of a team.

Original: In this work, we investigate how to promote inter-agent coordination using policy regularization and discuss two possible avenues respectively based on inter-agent modelling and synchronized sub-policy selection.
Revised: In this work, we investigate how to promote inter-agent coordination using policy regularization and discuss two possible avenues respectively based on inter-agent modelling and synchronized sub-policy selection.

Original: We test each approach in four challenging continuous control tasks with sparse rewards and compare them against three baselines including
Revised: We test each approach in four challenging continuous control tasks with sparse rewards and compare them against three baselines including

Original: MADDPG, a state-of-the-art multi-agent reinforcement learning algorithm.
Revised: MADDPG, a state-of-the-art multi-agent reinforcement learning algorithm.

Original: To ensure a fair comparison, we rely on a thorough hyper-parameter selection and training methodology that allows a ﬁxed hyper-parameter search budget for each algorithm and environment.
Revised: To ensure a fair comparison, we rely on a thorough hyper-parameter selection and training methodology that allows a ﬁxed hyper-parameter search budget for each algorithm and environment.

Original: We consequently assess both the hyperparameter sensitivity, sample-efﬁciency and asymptotic performance of each learning method.
Revised: We consequently assess both the hyperparameter sensitivity, sample-efﬁciency and asymptotic performance of each learning method.

Original: Our experiments show that the proposed methods lead to signiﬁcant improvements on cooperative problems.
Revised: Our experiments show that the proposed methods lead to signiﬁcant improvements on cooperative problems.

Original: We further analyse the effects of the proposed regularizations on the behaviors learned by the agents.
Revised: We further analyse the effects of the proposed regularizations on the behaviors learned by the agents.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-Agent Reinforcement Learning (MARL) refers to the task of training an agent to maximize its expected return by interacting with an environment that contains other learning agents.
Revised: Multi-Agent Reinforcement Learning (MARL) refers to the task of training an agent to maximize its expected return by interacting with an environment that contains other learning agents.

Original: It represents a challenging branch of Reinforcement Learning (RL) with interesting developments in recent years (Hernandez-Leal et al., 2018).
Revised: It represents a challenging branch of Reinforcement Learning (RL) with interesting developments in recent years (Hernandez-Leal et al., 2018).

Original: A popular framework for MARL is the use of a Centralized Training and a Decentralized Execution (CTDE) procedure (Lowe et al., 2017; Foerster et al., 2018; Iqbal &
Revised: A popular framework for MARL is the use of a Centralized Training and a Decentralized Execution (CTDE) procedure (Lowe et al., 2017; Foerster et al., 2018; Iqbal &

Original: Sha, 2019; Foerster et al., 2019; Rashid et al., 2018).
Revised: Sha, 2019; Foerster et al., 2019; Rashid et al., 2018).

Original: It is typically implemented by training critics that approximate the value of the joint observations and actions, which are used to train actors restricted to the observation of a single agent.
Revised: It is typically implemented by training critics that approximate the value of the joint observations and actions, which are used to train actors restricted to the observation of a single agent.

Original: Such critics, if exposed to coordinated joint actions leading to high returns, can steer the agents’ policies toward these highly rewarding behaviors.
Revised: Such critics, if exposed to coordinated joint actions leading to high returns, can steer the agents’ policies toward these highly rewarding behaviors.

Original: However, these approaches depend on the agents luckily stumbling on these actions in order to grasp their beneﬁt.
Revised: However, these approaches depend on the agents luckily stumbling on these actions in order to grasp their beneﬁt.

Original: Thus, it might fail in scenarios where coordination is unlikely to occur by chance.
Revised: Thus, it might fail in scenarios where coordination is unlikely to occur by chance.

Original: We hypothesize that in such scenarios, coordination-promoting inductive biases on the policy search could help discover coordinated behaviors more efﬁciently and supersede task-speciﬁc reward shaping and curriculum learning.
Revised: We hypothesize that in such scenarios, coordination-promoting inductive biases on the policy search could help discover coordinated behaviors more efﬁciently and supersede task-speciﬁc reward shaping and curriculum learning.

Original: In this work, we explore two different priors for successful coordination and use these to regularize the learned policies.
Revised: In this work, we explore two different priors for successful coordination and use these to regularize the learned policies.

Original: The ﬁrst avenue, TeamReg, assumes that an agent must be able to predict the behavior of its teammates in order to coordinate with them.
Revised: The ﬁrst avenue, TeamReg, assumes that an agent must be able to predict the behavior of its teammates in order to coordinate with them.

Original: The second, CoachReg, supposes that coordinating agents individually recognize different situations and synchronously use different subpolicies to react to them.
Revised: The second, CoachReg, supposes that coordinating agents individually recognize different situations and synchronously use different subpolicies to react to them.

Original: In the following sections we show how to derive practical regularization terms from these premises and meticulously evaluate them 1 .
Revised: In the following sections we show how to derive practical regularization terms from these premises and meticulously evaluate them 1 .

Original: Our contributions are twofold.
Revised: Our contributions are twofold.

Original: First, we propose two novel approaches that aim at promoting coordination in multi-agent systems.
Revised: First, we propose two novel approaches that aim at promoting coordination in multi-agent systems.

Original: Our methods augment CTDE MARL algorithms with additional multi-agent objectives that act as regularizers and are optimized jointly with the main return-maximization objective.
Revised: Our methods augment CTDE MARL algorithms with additional multi-agent objectives that act as regularizers and are optimized jointly with the main return-maximization objective.

Original: Second, we design two new sparse-reward cooperative tasks in the multi-agent particle environment (Mordatch & Abbeel, 2018).
Revised: Second, we design two new sparse-reward cooperative tasks in the multi-agent particle environment (Mordatch & Abbeel, 2018).

Original: We use them along with two standards multi-agent tasks to present a detailed evaluation of our approaches against three different baselines.
Revised: We use them along with two standard multi-agent tasks to present a detailed evaluation of our approaches against three different baselines.

Original: Finally, we validate our methods’ key components by performing an ablation study.
Revised: Finally, we validate our methods’ key components by performing an ablation study.

Original: Our experiments suggest that our TeamReg objective provides a dense learning signal that helps to guide the policy towards coordination in the absence of external reward, eventually leading it to the discovery of high performing team strategies in a number of cooperative tasks.
Revised: Our experiments suggest that our TeamReg objective provides a dense learning signal that helps to guide the policy towards coordination in the absence of external reward, eventually leading it to the discovery of high performing team strategies in a number of cooperative tasks.

Original: Similarly, by enforcing synchronous sub-policy selections, CoachReg enables to ﬁne-tune a sub-behavior for each recognized situation yielding signiﬁcant improvements on the overall performance.
Revised: Similarly, by enforcing synchronous sub-policy selections, CoachReg enables to ﬁne-tune a sub-behavior for each recognized situation yielding signiﬁcant improvements on the overall performance.

Original:  B ACKGROUND 2. M ARKOV G AMES
Revised:  B ACKGROUND 2. M ARKOV G AMES

Original: In this work we consider the framework of Markov Games (Littman, 1994), a multi-agent extension of Markov Decision Processes (MDPs) with N independent agents.
Revised: In this work we consider the framework of Markov Games (Littman, 1994), a multi-agent extension of Markov Decision Processes (MDPs) with N independent agents.

Original: A Markov Game is deﬁned by the tuple (cid:104)S , T , P , {O i , A
Revised: A Markov Game is deﬁned by the tuple (cid:104)S , T , P , {O i , A

Original: i , R i } Ni =1 (cid:105) .
Revised: i , R i } Ni =1 (cid:105) .

Original: S , T , and P respectively are the set of all possible states, the transition function and the initial state distribution.
Revised: S , T , and P respectively are the set of all possible states, the transition function and the initial state distribution.

Original: While these are global properties of the environment, O i , A i and R i are individually deﬁned for each agent i .
Revised: While these are global properties of the environment, O i , A i and R i are individually deﬁned for each agent i .

Original: They are respectively the observation functions, the sets of all possible actions and the reward functions.
Revised: They are respectively the observation functions, the sets of all possible actions and the reward functions.

Original: At each time-step t , the global state of the environment is given by s t ∈ S and every agent’s individual action vector is denoted by
Revised: At each time-step t , the global state of the environment is given by s t ∈ S and every agent’s individual action vector is denoted by

Original: a it ∈
Revised: a it ∈

Original: A i .
Revised: A i .

Original: To select their action, each agent i has only access to its own observation vector o it which is extracted by its observation function O i from the global state s t .
Revised: To select their action, each agent i has only access to its own observation vector o it which is extracted by its observation function O i from the global state s t .

Original: The initial global state s 0 is sampled from the initial state distribution P : S → [0 , 1] and the next states of the environment s t +1 are sampled from the probability distribution over the possible next states given by the transition function T : S ×S ×A 1 × ...
Revised: The initial global state s 0 is sampled from the initial state distribution P : S → [0 , 1] and the next states of the environment s t +1 are sampled from the probability distribution over the possible next states given by the transition function T : S ×S ×A 1 × ...

Original: ×A N → [0 , 1] .
Revised: ×A N → [0 , 1] .

Original: Finally, at each time-step, each agent receives an individual scalar reward r it from its reward function
Revised: Finally, at each time-step, each agent receives an individual scalar reward r it from its reward function

Original: R
Revised: R

Original: i : S × S ×
Revised: i : S × S ×

Original: A 1 × ...
Revised: A 1 × ...

Original: ×
Revised: ×

Original: A N → R .
Revised: A N → R .

Original: Agents aim at maximizing their expected discounted return E (cid:104)(cid:80) Tt =

Revised: Agents aim at maximizing their expected discounted return E (cid:104)(cid:80) Tt =

Original: γ t r it (cid:105) over the time horizon T , where γ ∈ [0 , 1] is a discount factor.
Revised: γ t r it (cid:105) over the time horizon T , where γ ∈ [0 , 1] is a discount factor.

Original: 2. M ULTI -A GENT D EEP D ETERMINISTIC P OLICY G RADIENT (MADDPG)
Revised: 2. M ULTI -A GENT D EEP D ETERMINISTIC P OLICY G RADIENT (MADDPG)

Original: MADDPG (Lowe et al., 2017) is an adaptation of the Deep Deterministic Policy Gradient algorithm (DDPG) (Lillicrap et al., 2015) to the multi-agent setting.
Revised: MADDPG (Lowe et al., 2017) is an adaptation of the Deep Deterministic Policy Gradient algorithm (DDPG) (Lillicrap et al., 2015) to the multi-agent setting.

Original: It allows the training of cooperating and competing decentralized policies through the use of a centralized training procedure.
Revised: It allows the training of cooperating and competing decentralized policies through the use of a centralized training procedure.

Original: In this framework, each agent i possesses its own deterministic policy µ i for action selection and critic Q i for state-action value estimation, which are respectively parametrized by θ
Revised: In this framework, each agent i possesses its own deterministic policy µ i for action selection and critic Q i for state-action value estimation, which are respectively parametrized by θ

Original: i and φ i .
Revised: i and φ i .

Original: All parametric models are trained off-policy from previous transitions ζ t :=
Revised: All parametric models are trained off-policy from previous transitions ζ t :=

Original: ( o t , a t , r t , o t +1 ) uniformly sampled from a replay buffer D .
Revised: ( o t , a t , r t , o t +1 ) uniformly sampled from a replay buffer D .

Original: Note that o t :=
Revised: Note that o t :=

Original: [ o 1 t , ..., o Nt ] is the joint observation vector and a t :=
Revised: [ o 1 t , ..., o Nt ] is the joint observation vector and a t :=

Original: [ a 1 t , ..., a Nt ] is the joint action vector, obtained by concatenating the individual observation vectors o i t and action vectors a i t of all N agents.
Revised: [ a 1 t , ..., a Nt ] is the joint action vector, obtained by concatenating the individual observation vectors o i t and action vectors a i t of all N agents.

Original: Each centralized critic is trained to estimate the expected return for a particular agent i using the Deep Q-Network (DQN) (Mnih et al., 2015) loss:
Revised: Each centralized critic is trained to estimate the expected return for a particular agent i using the Deep Q-Network (DQN) (Mnih et al., 2015) loss:

Original: L i ( φ i ) =
Revised: L i ( φ i ) =

Original: E ζ t ∼D
Revised: E ζ t ∼D

Original: [Equation]
Revised: [Equation]

Original: For a given set of weights w , we deﬁne its target counterpart ¯ w , updated from ¯ w ← τw + (1 − τ ) ¯ w where τ is a hyper-parameter.
Revised: For a given set of weights w , we deﬁne its target counterpart ¯ w , updated from ¯ w ← τw + (1 − τ ) ¯ w where τ is a hyper-parameter.

Original: Each policy is updated to maximize the expected discounted return of the corresponding agent i :
Revised: Each policy is updated to maximize the expected discounted return of the corresponding agent i :

Original: [Equation]
Revised: [Equation]

Original: By taking into account all agents’ observation-action pairs when guiding an agent’s policy, the valuefunctions are trained in a centralized, stationary environment, despite taking place in a multi-agent setting.
Revised: By taking into account all agents’ observation-action pairs when guiding an agent’s policy, the valuefunctions are trained in a centralized, stationary environment, despite taking place in a multi-agent setting.

Original: In addition, this mechanism can allow to implicitly learn coordinated strategies that can then be deployed in a decentralized way.
Revised: In addition, this mechanism can allow to implicitly learn coordinated strategies that can then be deployed in a decentralized way.

Original: However, this procedure does not encourage the discovery of coordinated strategies since high-reward behaviors have to be randomly experienced through unguided exploration.
Revised: However, this procedure does not encourage the discovery of coordinated strategies since high-reward behaviors have to be randomly experienced through unguided exploration.

Original: This work aims at alleviating this limitation.
Revised: This work aims at alleviating this limitation.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Many works in MARL consider explicit communication channels between the agents and distinguish between communicative actions (e.g. broadcasting a given message) and physical actions (e.g. moving in a given direction)
Revised: Many works in MARL consider explicit communication channels between the agents and distinguish between communicative actions (e.g. broadcasting a given message) and physical actions (e.g. moving in a given direction)

Original: (Foerster et al., 2016; Mordatch & Abbeel, 2018; Lazaridou et al., 2016).
Revised: (Foerster et al., 2016; Mordatch & Abbeel, 2018; Lazaridou et al., 2016).

Original: Consequently, they often focus on the emergence of language, considering tasks where the agents must discover a common communication protocol in order to succeed.
Revised: Consequently, they often focus on the emergence of language, considering tasks where the agents must discover a common communication protocol in order to succeed.

Original: Deriving a successful communication protocol can already be seen as coordination in the communicative action space and can enable, to some extent, successful coordination in the physical action space (Ahilan & Dayan, 2019).
Revised: Deriving a successful communication protocol can already be seen as coordination in the communicative action space and can enable, to some extent, successful coordination in the physical action space (Ahilan & Dayan, 2019).

Original: Yet, explicit communication is not a necessary condition for coordination as agents can rely on physical communication (Mordatch & Abbeel, 2018; Gupta et al., 2017).
Revised: Yet, explicit communication is not a necessary condition for coordination as agents can rely on physical communication (Mordatch & Abbeel, 2018; Gupta et al., 2017).

Original: Approaches to shape RL agents’ behaviors with respect to other agents have also been explored.
Revised: Approaches to shape RL agents’ behaviors with respect to other agents have also been explored.

Original: Strouse et al.
Revised: Strouse et al.

Original: (2018) use the mutual information between the agent’s policy and a goal-independent policy to shape the agent’s behavior towards hiding or spelling out its current goal.
Revised: (2018) use the mutual information between the agent’s policy and a goal-independent policy to shape the agent’s behavior towards hiding or spelling out its current goal.

Original: However, this approach is only applicable for tasks with an explicit goal representation and is not speciﬁcally intended for coordination.
Revised: However, this approach is only applicable for tasks with an explicit goal representation and is not speciﬁcally intended for coordination.

Original: Jaques et al.
Revised: Jaques et al.

Original: (2018) approximate the direct causal effect between agent’s actions and use it as an intrinsic reward to encourage social empowerment.
Revised: (2019) approximate the direct causal effect between agent’s actions and use it as an intrinsic reward to encourage social empowerment.

Original: This approximation relies on an agent modelling the other agents’ policies to predict its effect on them.
Revised: This approximation relies on each agent learning a model of other agents’ policies to predict its effect on them.

Revised: In general, this type of behavior prediction can be referred to as agent modelling (or opponent modelling) and has been used in previous work to enrich representations (Hernandez-Leal et al., 2019), to stabilise the learning dynamics (He et al., 2016) or to classify the opponent’s play style (Schadd et al., 2007).

Original: In our work, agent modelling focuses on maximising predictability between agents.
Revised: In our work, agent modelling is extended to derive a novel incentive toward team-predictable behaviors.

Original: Finally, Barton et al.
Revised: Finally, Barton et al.

Original: (2018) propose convergent cross mapping (CCM) to measure the degree of effective coordination between two agents.
Revised: (2018) propose convergent cross mapping (CCM) to measure the degree of effective coordination between two agents.

Original: Although this may represent an interesting avenue for behavior analysis, it fails to provide a tool for effectively enforcing coordination as CCM must be computed over long time series which makes it an impractical learning signal for singlestep temporal difference methods.
Revised: Although this may represent an interesting avenue for behavior analysis, it fails to provide a tool for effectively enforcing coordination as CCM must be computed over long time series which makes it an impractical learning signal for singlestep temporal difference methods.

Original: In this work, we design two coordination-driven multi-agent approaches that do not rely on the existence of explicit communication channels and allow to carry the learned coordinated behaviors at test time, when all agents act in a decentralized fashion.
Revised: In this work, we design two coordination-driven multi-agent approaches that do not rely on the existence of explicit communication channels and allow to carry the learned coordinated behaviors at test time, when all agents act in a decentralized fashion.

Original:  C OORDINATION AND P OLICY 
Revised:  C OORDINATION AND P OLICY REGULARIZATION

Original: [Figure]

Original: 𝝁𝝁
Revised: 𝝁𝝁

Original: Intuitively, coordination can be deﬁned as an agent’s behavior being informed by the one of another agent, i.e. structure in the agents’ interactions.
Revised: Intuitively, coordination can be deﬁned as an agent’s behavior being informed by the one of another agent, i.e. structure in the agents’ interactions.

Original: Namely, a team where agents act independently of one another would not be coordinated.
Revised: Namely, a team where agents act independently of one another would not be coordinated.

Original: To promote such structure, our proposed methods rely on team-objectives as regularizers of the common policy gradient update.
Revised: To promote such structure, our proposed methods rely on team-objectives as regularizers of the common policy gradient update.

Original: In this regard, our approach is closely related to General Value Functions and Auxiliary tasks (Sutton & Barto, 2018) used in Deep RL to learn efﬁcient representations (Jaderberg et al.,2018; Hernandez-Leal et al., 2019).
Revised: In this regard, our approach is closely related to General Value Functions and Auxiliary tasks (Sutton & Barto, 2018) used in Deep RL to learn efﬁcient representations (Jaderberg et al., 2019).

Original: However, this work’s novelty lies in the explicit bias of agents’ policy towards either predictabily for their teammates or synchronous sub-policy selection.
Revised: However, this work’s novelty lies in the explicit bias of agents’ policy towards either predictability for their teammates or synchronous sub-policy selection.

Original: Pseudocodes of our implementations are provided in Appendix C (see Algorithms 1 and 2).
Revised: Pseudocodes of our implementations are provided in Appendix C (see Algorithms 1 and 2).

Original: 4. T EAM REGULARIZATION
Revised: 4. T EAM REGULARIZATION

Original: The structure of coordinated interactions can be leveraged to attain a certain degree of predictability of one agent’s behavior with respect to its teammate(s).
Revised: The structure of coordinated interactions can be leveraged to attain a certain degree of predictability of one agent’s behavior with respect to its teammate(s).

Original: We hypothesize that the reciprocal also holds i.e. that promoting agents’ predictability could foster such team structure and lead to more coordinated behaviors.
Revised: We hypothesize that the reciprocal also holds i.e. that promoting agents’ predictability could foster such team structure and lead to more coordinated behaviors.

Original: This assumption is cast into the decentralized framework by training agents to predict their teammates’ actions given only their own observation.
Revised: This assumption is cast into the decentralized framework by training agents to predict their teammates’ actions given only their own observation.

Original: For continuous control, the loss is deﬁned as the Mean Squared Error (MSE) between the predicted and true actions of the teammates, yielding a teammate-modelling secondary objective similar to the models of other agents used in Jaques et al. and Hernandez-Leal et al. and often referred to as agent modelling (Schadd et al.,
Revised: For continuous control, the loss is deﬁned as the Mean Squared Error (MSE) between the predicted and true actions of the teammates, yielding a teammate-modelling secondary objective.

Original: Most previous works such as Hernandez-Leal et al.
Revised: While the previous work of Hernandez-Leal et al.

Original: (2019) focus on stationary, non-learning teammates and exclusively use this approach to improve the richness of the learned representations. In our case however, the same objectiveis also used to drive the teammates’ behaviors closer to the prediction by leveraging a differentiable action selection mechanism.
Revised: (2019) focus on stationary, non-learning teammates and exclusively use this approach to learn richer internal representations, we propose to extend this objective to drive the teammates’ behaviors closer to the prediction by leveraging a differentiable action selection mechanism.

Original: Therefore, we callthis the team-spirit  objective J i,jTS between agents i and j :
Revised: We call team-spirit this novel objective J i,jTS between agents i and j :

Original: [Equation]
Revised: [Equation]

Original: where ˆ µ i,j is the policy head of agent i trying to predict the action of agent j .
Revised: where ˆ µ i,j is the policy head of agent i trying to predict the action of agent j .

Original: The total gradient for a given agent i becomes:
Revised: The total gradient for a given agent i becomes:

Original: [Equation]
Revised: [Equation]

Original: where λ 1 and λ 2 are hyper-parameters that respectively weight how well an agent should predict its teammates’ actions, and how predictable an agent should be for its teammates.
Revised: where λ 1 and λ 2 are hyper-parameters that respectively weight how well an agent should predict its teammates’ actions, and how predictable an agent should be for its teammates.

Original: We call TeamReg this dual regularization from team-spirit objectives.
Revised: We call TeamReg this dual regularization from team-spirit objectives.

Original: Figure 1 summarizes these interactions.
Revised: Figure 1 summarizes these interactions.

Original: 4. C OACH REGULARIZATION
Revised: 4. C OACH REGULARIZATION

Original: [Figure]
Revised: [Figure]

Original:  agents’ observations as input and outputs the current mode (policy mask).
Revised: all agents’ observations as input and outputs the current mode (policy mask).

Original: Agents are regularized to predict the same mask from their local observations only and optimize the corresponding sub-policy.
Revised: Agents are regularized to predict the same mask from their local observations only and optimize the corresponding sub-policy.

Original: In order to foster structured agents interactions, this method aims at teaching the agents to recognize different situations and synchronously select corresponding sub-behaviors.
Revised: In order to foster structured agents interactions, this method aims at teaching the agents to recognize different situations and synchronously select corresponding sub-behaviors.

Original: 4.2. S UB - POLICY SELECTION
Revised: 4.2. S UB - POLICY SELECTION

Original: Firstly, to enable explicit sub-behavior selection, we propose policy masks that modulate the agents’ policy.
Revised: Firstly, to enable explicit sub-behavior selection, we propose policy masks that modulate the agents’ policy.

Original: A policy mask u j is a one-hot vector of size K with its j th component set to one.
Revised: A policy mask u j is a one-hot vector of size K with its j th component set to one.

Original: In practice, we use policy masks to perform dropout (Srivastava et al., 2014) in a structured manner on ˜ h 1 ∈ R M , the pre-activations of the ﬁrst hidden layer hof the policy network π .
Revised: In practice, we use policy masks to perform dropout (Srivastava et al., 2014) in a structured manner on ˜ h 1 ∈ R M , the pre-activations of the ﬁrst hidden layer hof the policy network π .

Original: To do so, we construct the vector u j , which is the concatenation of C copies of u j , in order to reach the dimensionality M = C ∗ K .
Revised: To do so, we construct the vector u j , which is the concatenation of C copies of u j , in order to reach the dimensionality M = C ∗ K .

Original: The element-wise product u j (cid:12) ˜
Revised: The element-wise product u j (cid:12) ˜

Original: h 1 is then performed and only the units of ˜ hat indices m modulo K =
Revised: h 1 is then performed and only the units of ˜ hat indices m modulo K =

Original: j are kept for m = 0 , . . .
Revised: j are kept for m = 0 , . . .

Original: , M − 1 .
Revised: , M − 1 .

Original: In our contribution, each agent i generates e i t , its own policy mask, from its observation o i t .
Revised: In our contribution, each agent i generates e i t , its own policy mask, from its observation o i t .

Original: Here, a simple linear layer l i is used to produce a categorical probability distribution p
Revised: Here, a simple linear layer l i is used to produce a categorical probability distribution p

Original: i ( e it | o it ) from which the one-hot vector is sampled:
Revised: i ( e it | o it ) from which the one-hot vector is sampled:

Original: [Equation]
Revised: [Equation]

Original: To our knowledge, while this method draws similarity to the options and hierarchical frameworks (Sutton & Barto, 2018; Ahilan & Dayan, 2019) and to policy dropout for exploration (Xie et al., 2018), it is the ﬁrst to introduce an agent induced modulation of the policy network by a structured dropout that is decentralized at evaluation and without an explicit communication channel.
Revised: To our knowledge, while this method draws similarity to the options and hierarchical frameworks (Sutton & Barto, 2018; Ahilan & Dayan, 2019) and to policy dropout for exploration (Xie et al., 2018), it is the ﬁrst to introduce an agent induced modulation of the policy network by a structured dropout that is decentralized at evaluation and without an explicit communication channel.

Original: Although the policy masking mechanism enables the agent to swiftly switch between sub-policies it does not encourage the agents to synchronously modulate their behavior.
Revised: Although the policy masking mechanism enables the agent to swiftly switch between sub-policies it does not encourage the agents to synchronously modulate their behavior.

Original: 4.2. S YNCHRONOUS SUB - POLICY SELECTION
Revised: 4.2. S YNCHRONOUS SUB - POLICY SELECTION

Original: To promote synchronization we introduce the coach entity, parametrized by ψ , which learns to produce policy-masks e ct from the joint observations, i.e. p c ( e ic | o t ; ψ ) .
Revised: To promote synchronization we introduce the coach entity, parametrized by ψ , which learns to produce policy-masks e ct from the joint observations, i.e. p c ( e ic | o t ; ψ ) .

Original: The coach is used at training time only and drives the agents toward synchronously selecting the same behavior mask.
Revised: The coach is used at training time only and drives the agents toward synchronously selecting the same behavior mask.

Original: In other words, the coach is trained to output masks that (1) yield high returns when used by the agents and (2) are predictable by the agents.
Revised: In other words, the coach is trained to output masks that (1) yield high returns when used by the agents and (2) are predictable by the agents.

Original: Similarly, each agent is regularized so that (1) its private mask matches the coach’s mask and (2) it derives efﬁcient behavior when using the coach’s mask.
Revised: Similarly, each agent is regularized so that (1) its private mask matches the coach’s mask and (2) it derives efﬁcient behavior when using the coach’s mask.

Original: At evaluation time, the coach is removed and the agents only rely on their own policy masks.
Revised: At evaluation time, the coach is removed and the agents only rely on their own policy masks.

Original: The policy gradient loss when agent i is provided with the coach’s mask is given by:
Revised: The policy gradient loss when agent i is provided with the coach’s mask is given by:

Original: [Equation]
Revised: [Equation]

Original: The difference between the mask of agent i and the coach’s one is measured from the Kullback–Leibler divergence:
Revised: The difference between the mask of agent i and the coach’s one is measured from the Kullback–Leibler divergence:

Original: [Equation]
Revised: [Equation]

Original: The total gradient for agent i is:
Revised: The total gradient for agent i is:

Original: [Equation]
Revised: [Equation]

Original: with λ 1 and λ 2 the regularization coefﬁcients.
Revised: with λ 1 and λ 2 the regularization coefﬁcients.

Original: Similarly, the coach is trained with the following dual objective, weighted by the λ 3 coefﬁcient
Revised: Similarly, the coach is trained with the following dual objective, weighted by the λ 3 coefﬁcient

Original: [Equation]
Revised: [Equation]

Original: In order to propagate gradients through the sampled policy mask we reparametrized the categorical distribution using the Gumbel-softmax trick (Jang et al., 2016) with a temperature of 1.
Revised: In order to propagate gradients through the sampled policy mask we reparametrized the categorical distribution using the Gumbel-softmax trick (Jang et al., 2017) with a temperature of 1.

Original: We call this coordinated sub-policy selection regularization CoachReg and illustrate it in Figure 2.
Revised: We call this coordinated sub-policy selection regularization CoachReg and illustrate it in Figure 2.

Original:  T RAINING ENVIRONMENTS
Revised:  T RAINING ENVIRONMENTS

Original: All of our tasks are based on the OpenAI multi-agent particle environments (Mordatch & Abbeel, 2018).
Revised: All of our tasks are based on the OpenAI multi-agent particle environments (Mordatch & Abbeel, 2018).

Original: SPREAD and CHASE were introduced by (Lowe et al., 2017).
Revised: SPREAD and CHASE were introduced by (Lowe et al., 2017).

Original: We use SPREAD as is but with sparse rewards only.
Revised: We use SPREAD as is but with sparse rewards only.

Original: CHASE is modiﬁed with a prey controlled by repulsion forces and only the predators are learnable, as we wish to focus on coordination in cooperative tasks.
Revised: CHASE is modiﬁed with a prey controlled by repulsion forces and only the predators are learnable, as we wish to focus on coordination in cooperative tasks.

Original: Finally we introduce COMPROMISE and BOUNCE where agents are explicitly tied together.
Revised: Finally we introduce COMPROMISE and BOUNCE where agents are explicitly tied together.

Original: While nonzero return can be achieved in these tasks by selﬁsh agents, they all beneﬁt from coordinated strategies and optimal return can only be achieved by agents working closely together.
Revised: While nonzero return can be achieved in these tasks by selﬁsh agents, they all beneﬁt from coordinated strategies and optimal return can only be achieved by agents working closely together.

Original: Figurepresents visualizations and a brief description of all four tasks.
Revised: Figurepresents visualizations and a brief description of all four tasks.

Original: A detailed description is provided in
Revised: A detailed description is provided in

Original: Appendix A.
Revised: Appendix A.

Original: In all tasks, agents receive as observation their own global position and velocity as well as the relative position of other entities.
Revised: In all tasks, agents receive as observation their own global position and velocity as well as the relative position of other entities.

Original: Note that work showcasing experiments on this environment often use discrete action spaces and (dense) reward shaping (e.g. the proximity with the objective)
Revised: Note that work showcasing experiments on this environment often use discrete action spaces and (dense) reward shaping (e.g. the proximity with the objective)

Original: (Iqbal & Sha, 2019; Lowe et al., 2017; Jiang & Lu, 2018).
Revised: (Iqbal & Sha, 2019; Lowe et al., 2017; Jiang & Lu, 2018).

Original: However, in our experiments, agents learn with continuous action spaces and from sparse rewards.
Revised: However, in our experiments, agents learn with continuous action spaces and from sparse rewards.

Original:  R ESULTS AND D ISCUSSION
Revised:  R ESULTS AND D ISCUSSION

Original: The proposed methods offer a way to incorporate new inductive biases in CTDE multi-agent policy search algorithms.
Revised: The proposed methods offer a way to incorporate new inductive biases in CTDE multi-agent policy search algorithms.

Original: In this work, we evaluate them by extending MADDPG, a state of the art algorithm widely used in the MARL litterature.
Revised: In this work, we evaluate them by extending MADDPG, a state of the art algorithm widely used in the MARL litterature.

Original: We compare against vanilla MADDPG as well as two of its variations in the four cooperative multi-agent tasks described in Section 5.
Revised: We compare against vanilla MADDPG as well as two of its variations in the four cooperative multi-agent tasks described in Section 5.

Original: The ﬁrst variation (DDPG) is the single-agent counterpart of MADDPG (decentralized training).
Revised: The ﬁrst variation (DDPG) is the single-agent counterpart of MADDPG (decentralized training).

Original: The second (MADDPG + sharing) shares the policy and value-function models across agents.
Revised: The second (MADDPG + sharing) shares the policy and value-function models across agents.

Original: To offer a fair comparison between all methods, the hyper-parameter search routine is the same for each algorithm and environment (see Appendix D.1).
Revised: To offer a fair comparison between all methods, the hyper-parameter search routine is the same for each algorithm and environment (see Appendix D.1).

Original: For each search-experiment (one per algorithm per environment), 50 randomly sampled hyper-parameter conﬁgurations each using 3 training seeds (total of 150 runs) are used to train the models for 15 , 000 episodes.
Revised: For each search-experiment (one per algorithm per environment), 50 randomly sampled hyper-parameter conﬁgurations each using 3 training seeds (total of 150 runs) are used to train the models for 15 , 000 episodes.

Original: For each algorithm-environment pair, we then select the best hyper-parameter conﬁguration for the ﬁnal comparison and retrain them on 10 seeds for twice as long.
Revised: For each algorithm-environment pair, we then select the best hyper-parameter conﬁguration for the ﬁnal comparison and retrain them on 10 seeds for twice as long.

Original: We give more details about the training setup and model selection in
Revised: We give more details about the training setup and model selection in

Original: Appendix B and D.2.
Revised: Appendix B and D.2.

Original: The results of the hyperparameter searches are given in Appendix D.5.
Revised: The results of the hyperparameter searches are given in Appendix D.5.

Original: 6. A SYMPTOTIC P ERFORMANCE
Revised: 6. A SYMPTOTIC P ERFORMANCE

Original: From the average learning curves reported in Figure 4 we observe that CoachReg signiﬁcantly improves performance on three environments (SPREAD, BOUNCE and COMPROMISE) and performs on par with the baselines on the last one (CHASE).
Revised: From the average learning curves reported in Figure 4 we observe that CoachReg signiﬁcantly improves performance on three environments (SPREAD, BOUNCE and COMPROMISE) and performs on par with the baselines on the last one (CHASE).

Original: The same can be said for TeamReg, except on COMPROMISE, the only task with an adversarial component, where it signiﬁcantly underperforms compared to the other algorithms.
Revised: The same can be said for TeamReg, except on COMPROMISE, the only task with an adversarial component, where it signiﬁcantly underperforms compared to the other algorithms.

Original: We discuss this speciﬁc case in Section 6.3.
Revised: We discuss this speciﬁc case in Section 6.3.

Original: Finally
Revised: Finally

Original: [Figure]
Revised: [Figure]

Original: parameter sharing is the best performing choice on CHASE
Revised: parameter sharing is the best performing choice on CHASE

Original: [Figure]
Revised: [Figure]

Original: yet this superiority is restricted to this task where the optimal play is to move symmetrically and squeeze the prey into a corner.
Revised: yet this superiority is restricted to this task where the optimal play is to move symmetrically and squeeze the prey into a corner.

Original: [Figure]
Revised: [Figure]

Original: 6. A BLATION STUDY
Revised: 6. A BLATION STUDY

Original: Additionally to our two proposed algorithms and the three baselines
Revised: Additionally to our two proposed algorithms and the three baselines

Original: [Figure]
Revised: [Figure]

Original: we present results for two ablated versions of our methods.
Revised: we present results for two ablated versions of our methods.

Original: The ﬁrst ablation (MADDPG + agent modelling) is similar to
Revised: The ﬁrst ablation (MADDPG + agent modelling) is similar to

Original: TeamReg but with λ 2 = 0 , which results in only enforcing agent modelling (i.e. agent predictability is not encouraged).
Revised: TeamReg but with λ 2 = 0 , which results in only enforcing agent modelling (i.e. agent predictability is not encouraged).

Original: The second ablation (MADDPG + policy mask) is structurally equivalent to
Revised: The second ablation (MADDPG + policy mask) is structurally equivalent to

Original: CoachReg, but with λ 1 , 2 , 3 = 0 , which means that agents still predict and apply a mask to their own policy, but synchronicity is not encouraged.
Revised: CoachReg, but with λ 1 , 2 , 3 = 0 , which means that agents still predict and apply a mask to their own policy, but synchronicity is not encouraged.

Original: Figure 12 and 13 (Appendix D.6) present the results of the corresponding hyper-parameter search and Figure 5 shows the learning curves for our full regularization approaches, their respective ablated versions and MADDPG.
Revised: Figure 12 and 13 (Appendix D.6) present the results of the corresponding hyper-parameter search and Figure 5 shows the learning curves for our full regularization approaches, their respective ablated versions and MADDPG.

Original: [Figure]
Revised: [Figure]

Original: The use of unsynchronized policy masks might result in swift and unpredictable behavioral changes and make it difﬁcult for agents to perform together and coordinate.
Revised: The use of unsynchronized policy masks might result in swift and unpredictable behavioral changes and make it difﬁcult for agents to perform together and coordinate.

Original: Experimentally, “MADDPG + policy mask” performs similarly or worse than MADDPG on all but one environment, and never outperforms the full CoachReg approach.
Revised: Experimentally, “MADDPG + policy mask” performs similarly or worse than MADDPG on all but one environment, and never outperforms the full CoachReg approach.

Original: However, policy masks alone seem enough to succeed on SPREAD, which is about selecting a landmark from a set.
Revised: However, policy masks alone seem enough to succeed on SPREAD, which is about selecting a landmark from a set.

Original: Regarding “MADDPG + agent modelling”, it does not drastically improve on MADDPG apart  on the SPREAD environment, and the full TeamReg approach shows improvement over its ablated version except on the
Revised: Regarding “MADDPG + agent modelling”, it does not drastically improve on MADDPG apart from on the SPREAD environment, and the full TeamReg approach shows improvement over its ablated version except on the

Original: COMPROMISE task, which we discuss in Section 6.3.
Revised: COMPROMISE task, which we discuss in Section 6.3.

Original: 6. E FFECTS OF ENFORCING PREDICTABLE BEHAVIOR
Revised: 6. E FFECTS OF ENFORCING PREDICTABLE BEHAVIOR

Original: First, we investigate the reason for TeamReg’s poor performance on COMPROMISE.
Revised: First, we investigate the reason for TeamReg’s poor performance on COMPROMISE.

Original: Then, we analyse how TeamReg might be helpful in other environments.
Revised: Then, we analyse how TeamReg might be helpful in other environments.

Original: [Figure]
Revised: [Figure]

Original:  associated with high values of λ 2 (right).
Revised: agents associated with high values of λ 2 (right).

Original: COMPROMISE is the only task with a competitive component (and the only one in which agents do not share their rewards).
Revised: COMPROMISE is the only task with a competitive component (and the only one in which agents do not share their rewards).

Original: The two agents being linked, a good policy has both agents reach their landmark successively (maybe by simply having both agents navigate towards the closest landmark).
Revised: The two agents being linked, a good policy has both agents reach their landmark successively (maybe by simply having both agents navigate towards the closest landmark).

Original: However, if one agent never reaches for its landmark, the optimal strategy for the other one becomes to drag it around and always go for its own, leading to a strong imbalance in the return cumulated by both agents.
Revised: However, if one agent never reaches for its landmark, the optimal strategy for the other one becomes to drag it around and always go for its own, leading to a strong imbalance in the return cumulated by both agents.

Original: While this scenario very rarely occurs for the other algorithms, we found TeamReg to often lead to such domination cases (see Figure 14 in Appendix E).
Revised: While this scenario very rarely occurs for the other algorithms, we found TeamReg to often lead to such domination cases (see Figure 14 in Appendix E).

Original: Figure 6 depicts the agents’ performance difference for every 150 runs of the hyperparameter search for TeamReg and the baselines, and shows that (1)
Revised: Figure 6 depicts the agents’ performance difference for every 150 runs of the hyperparameter search for TeamReg and the baselines, and shows that (1)

Original: TeamReg is the only algorithm that does lead to large imbalances in performance between the two agents and (2) that these cases where one agent becomes dominant are all associated with high values of λ 2 , which drives the agents to behave in a predictable fashion to one another.
Revised: TeamReg is the only algorithm that does lead to large imbalances in performance between the two agents and (2) that these cases where one agent becomes dominant are all associated with high values of λ 2 , which drives the agents to behave in a predictable fashion to one another.

Original: However, the dominated agent eventually gets exposed more and more to sparse reward gathered by being dragged (by chance) onto its own landmark, picks up the goal of the task and starts pulling in its own direction, which causes the average return over agents to drop as we see in Figure 4.
Revised: However, the dominated agent eventually gets exposed more and more to sparse reward gathered by being dragged (by chance) onto its own landmark, picks up the goal of the task and starts pulling in its own direction, which causes the average return over agents to drop as we see in Figure 4.

Original: This experiment demonstrates that using a predictability-based team-regularization in a competitive task can be harmful; quite understandably, you might not want to optimize an objective that aims at making your behavior predictable to your opponent.
Revised: This experiment demonstrates that using a predictability-based team-regularization in a competitive task can be harmful; quite understandably, you might not want to optimize an objective that aims at making your behavior predictable to your opponent.

Original: On SPREAD and BOUNCE, TeamReg signiﬁcantly improves the performance over the baselines.
Revised: On SPREAD and BOUNCE, TeamReg signiﬁcantly improves the performance over the baselines.

Original: We aim to analyze here the effects of λ 2 on cooperative tasks and investigate if it does make the agent modelling task more successful (by encouraging the agent to be predictable).
Revised: We aim to analyze here the effects of λ 2 on cooperative tasks and investigate if it does make the agent modelling task more successful (by encouraging the agent to be predictable).

Original: To this end, we compare the best performing hyper-parameter conﬁguration for TeamReg on the SPREAD environment with its ablated versions.
Revised: To this end, we compare the best performing hyper-parameter conﬁguration for TeamReg on the SPREAD environment with its ablated versions.

Original: The average return and team-spirit loss deﬁned in Section 4.
Revised: The average return and team-spirit loss deﬁned in Section 4.

Original: are presented in Figure 7 for these three experiments.
Revised: are presented in Figure 7 for these three experiments.

Original: Initially, due to the weight initialization, the predicted and actual actions both have relatively small norms yielding small values of team-spirit loss.
Revised: Initially, due to the weight initialization, the predicted and actual actions both have relatively small norms yielding small values of team-spirit loss.

Original: As training goes on ( ∼ 1000 episodes), the norms of the action-vector increase and the regularization loss becomes more important.
Revised: As training goes on ( ∼ 1000 episodes), the norms of the action-vector increase and the regularization loss becomes more important.

Original: As expected, λ 2 OFF | λ 1 OFF leads to the highest team-spirit loss as it is not trained to predict the actions of other agents correctly.
Revised: As expected, λ 2 OFF | λ 1 OFF leads to the highest team-spirit loss as it is not trained to predict the actions of other agents correctly.

Original: When using only the agent-modelling objective ( λ 1 ON ), the agents signiﬁcantly decrease the team-spirit loss, but it never reaches values as low as when using the full TeamReg objective.
Revised: When using only the agent-modelling objective ( λ 1 ON ), the agents signiﬁcantly decrease the team-spirit loss, but it never reaches values as low as when using the full TeamReg objective.

Original: Finally, when also pushing agents to be predictable ( λ 2 ON ), the agents best predict each others’ actions and performance is also improved.
Revised: Finally, when also pushing agents to be predictable ( λ 2 ON ), the agents best predict each others’ actions and performance is also improved.

Original: We also notice that the team-spirit loss increases when performance starts to improve i.e. when agents start to master the task ( ∼episodes).
Revised: We also notice that the team-spirit loss increases when performance starts to improve i.e. when agents start to master the task ( ∼episodes).

Original: Indeed, once the reward maximisation signals becomes stronger, the relative importance of the second task is reduced.
Revised: Indeed, once the reward maximisation signals becomes stronger, the relative importance of the second task is reduced.

Original: We hypothesize that being predictable with respect to one-another may push agents to explore in a more structured and informed manner in the absence of reward signal, as similarly pursued by intrinsic motivation approaches (Chentanez et al., 2005).
Revised: We hypothesize that being predictable with respect to one-another may push agents to explore in a more structured and informed manner in the absence of reward signal, as similarly pursued by intrinsic motivation approaches (Chentanez et al., 2005).

Original: 6. A NALYSIS OF SYNCHRONOUS SUB - POLICY SELECTION
Revised: 6. A NALYSIS OF SYNCHRONOUS SUB - POLICY SELECTION

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: In this section we aim at experimentally verifying that CoachReg yields the desired behavior: agents synchronously alternating between varied sub-policies.
Revised: In this section we aim at experimentally verifying that CoachReg yields the desired behavior: agents synchronously alternating between varied sub-policies.

Original: A special attention is given when the sub-policies are interpretable.
Revised: A special attention is given when the sub-policies are interpretable.

Original: To this end we record and analyze the agents’ policy masks on 100 different episodes for each task.
Revised: To this end we record and analyze the agents’ policy masks on 100 different episodes for each task.

Original: From the collected masks, we reconstructed the empirical mask distribution of each agent (see Figure 15 in Appendix F.1) whose entropy provides an indication of the mask diversity used by a given agent.
Revised: From the collected masks, we reconstructed the empirical mask distribution of each agent (see Figure 15 in Appendix F.1) whose entropy provides an indication of the mask diversity used by a given agent.

Original: Figure 9 (a) shows the mean entropy for each environment compared to the entropy of Categorical Uniform Distributions of size k ( k -CUD).
Revised: Figure 9 (a) shows the mean entropy for each environment compared to the entropy of Categorical Uniform Distributions of size k ( k -CUD).

Original: It shows that, on all the environments, agents use at least two distinct masks by having non-zero entropy.
Revised: It shows that, on all the environments, agents use at least two distinct masks by having non-zero entropy.

Original: In addition, agents tend to alternate between masks with more variety (close to uniformly switching between 3 masks) on SPREAD (where there are 3 agents and 3 goals) than on the other environments (comprised of 2 agents).
Revised: In addition, agents tend to alternate between masks with more variety (close to uniformly switching between 3 masks) on SPREAD (where there are 3 agents and 3 goals) than on the other environments (comprised of 2 agents).

Original: To test if agents are synchronously selecting the same policy mask at test time (without a coach), we compute the Hamming proximity between the agents’ mask sequences with 1 − D h where D h is the Hamming distance, i.e. the number of timesteps where the two sequences are different divided by the total number of timesteps.
Revised: To test if agents are synchronously selecting the same policy mask at test time (without a coach), we compute the Hamming proximity between the agents’ mask sequences with 1 − D h where D h is the Hamming distance, i.e. the number of timesteps where the two sequences are different divided by the total number of timesteps.

Original: From Figure 9 (b) we observe that agents are producing similar mask sequences.
Revised: From Figure 9 (b) we observe that agents are producing similar mask sequences.

Original: Notably, their mask sequences are signiﬁcantly more similar that the ones of two agent randomly choosing between two masks at each timestep.
Revised: Notably, their mask sequences are signiﬁcantly more similar that the ones of two agent randomly choosing between two masks at each timestep.

Original: Finally, we observe that some settings result in the agents coming up with interesting strategies, like the one depicted in Figure 8 where the agents alternate between two subpolicies depending on the position of the target.
Revised: Finally, we observe that some settings result in the agents coming up with interesting strategies, like the one depicted in Figure 8 where the agents alternate between two subpolicies depending on the position of the target.

Original: More cases where the agents change sub-policies during an episode are presented in
Revised: More cases where the agents change sub-policies during an episode are presented in

Original: Appendix F.1.
Revised: Appendix F.1.

Original: These results indicate that, in addition to improving the performance on coordination tasks, CoachReg indeed yields the expected behaviors.
Revised: These results indicate that, in addition to improving the performance on coordination tasks, CoachReg indeed yields the expected behaviors.

Original: An interesting following work would be to use entropy regularization to increase the mask usage variety and mutual information to further disentangle sub-policies.
Revised: An interesting following work would be to use entropy regularization to increase the mask usage variety and mutual information to further disentangle sub-policies.

Original: 6. R OBUSTNESS TO HYPER - PARAMETERS
Revised: 6. R OBUSTNESS TO HYPER - PARAMETERS

Original: Stability across hyper-parameter conﬁgurations is a recurring challenge in Deep RL.
Revised: Stability across hyper-parameter conﬁgurations is a recurring challenge in Deep RL.

Original: The average performance for each sampled conﬁguration allows to empirically evaluate the robustness of an algorithm w.r.t.
Revised: The average performance for each sampled conﬁguration allow to empirically evaluate the robustness of an algorithm w.r.t.

Original: its hyper-parameters.
Revised: its hyper-parameters.

Original: We share the full results of the hyper-parameter searches in Figures 10, 11, 12 and 13 in Appendix D.5 and D.6.
Revised: We share the full results of the hyper-parameter searches in Figures 10, 11, 12 and 13 in Appendix D.5 and D.6.

Original: Figure 11 shows that while most algorithms can perform reasonably well with the correct conﬁguration, our proposed coordination regularizers can improve robustness to hyper-parameter despite the fact that they have more hyper-parameters to search over.
Revised: Figure 11 shows that while most algorithms can perform reasonably well with the correct conﬁguration, our proposed coordination regularizers can improve robustness to hyper-parameter despite the fact that they have more hyper-parameters to search over.

Original: Such robustness can be of great value with limited computational budgets.
Revised: Such robustness can be of great value with limited computational budgets.

Revised: 6. R OBUSTNESS TO THE NUMBER OF AGENTS

Revised: To assess how the proposed methods perform when using an greater number of agents, we present additional experiments for which the number of agents in the SPREAD task is gradually increased from three to six agents.

Revised: The results presented in Figure 18 (Appendix G) show that the performance beneﬁts provided by our methods hold when the number of agents is increased.

Revised: Strikingly, we also note how quickly the performance of all methods drop when the number of agents rises.

Original:   C ONCLUSION
Revised: Indeed, with each new agent, the coordination problem becomes more and more difﬁcult, and that might explain why our methods that promote coordination maintain a higher degree of performance in the case of 4 agents. Nonetheless, estimating the value function also becomes increasingly challenging as the input space grows exponentially with the number of agents. In the sparse reward setting, the complexity of the task soon becomes too difﬁcult and none of the algorithms is able to solve it with six agents.  C ONCLUSION

Original: In this work we introduced two policy regularization methods to promote multi-agent coordination within the CTDE framework: TeamReg, which is based on inter-agent action predictability and
Revised: In this work we introduced two policy regularization methods to promote multi-agent coordination within the CTDE framework: TeamReg, which is based on inter-agent action predictability and

Original: CoachReg that relies on synchronized behavior selection.
Revised: CoachReg that relies on synchronized behavior selection.

Original: A thorough empirical evaluation of these methods showed that they signiﬁcantly improve asymptotic performances on cooperative multiagent tasks.
Revised: A thorough empirical evaluation of these methods showed that they signiﬁcantly improve asymptotic performances on cooperative multiagent tasks.

Original: Interesting avenues for future work would be to study the proposed regularizations on other policy search methods as well as to combine both incentives and investigate how the two coordinating objectives interact.
Revised: Interesting avenues for future work would be to study the proposed regularizations on other policy search methods as well as to combine both incentives and investigate how the two coordinating objectives interact.

Original: Finally, a limitation of the current formulation is that it relies on single-step metrics, which simpliﬁes off-policy learning but also limits the longer-term coordination opportunities.
Revised: Finally, a limitation of the current formulation is that it relies on single-step metrics, which simpliﬁes off-policy learning but also limits the longer-term coordination opportunities.

Original: A promising direction is thus to explore model-based planning approaches to promote long-term multi-agent interactions.
Revised: A promising direction is thus to explore model-based planning approaches to promote long-term multi-agent interactions.

Original: A T ASKS DESCRIPTIONS
Revised: A T ASKS DESCRIPTIONS

Original: SPREAD (Figure 3a):
Revised: SPREAD (Figure 3a):

Original: In this environment, there are 3 agents (small orange circles) and 3 landmarks (bigger gray circles).
Revised: In this environment, there are 3 agents (small orange circles) and 3 landmarks (bigger gray circles).

Original: At every timestep, agents receive a team-reward r t = n − c where n is the number of landmarks occupied by at least one agent and c the number of collisions occurring at that timestep.
Revised: At every timestep, agents receive a team-reward r t = n − c where n is the number of landmarks occupied by at least one agent and c the number of collisions occurring at that timestep.

Original: To maximize their return, agents must therefore spread out and cover all landmarks.
Revised: To maximize their return, agents must therefore spread out and cover all landmarks.

Original: Initial agents’ and landmarks’ positions are random.
Revised: Initial agents’ and landmarks’ positions are random.

Original: Termination is triggered when the maximum number of timesteps is reached.
Revised: Termination is triggered when the maximum number of timesteps is reached.

Original: BOUNCE (Figure 3b): In this environment, two agents (small orange circles) are linked together with a spring that pulls them toward each other when stretched above its relaxation length.
Revised: BOUNCE (Figure 3b): In this environment, two agents (small orange circles) are linked together with a spring that pulls them toward each other when stretched above its relaxation length.

Original: At episode’s mid-time a ball (smaller black circle) falls from the top of the environment.
Revised: At episode’s mid-time a ball (smaller black circle) falls from the top of the environment.

Original: Agents must position correctly so as to have the ball bounce on the spring towards the target (bigger beige circle), which turns yellow if the ball’s bouncing trajectory passes through it.
Revised: Agents must position correctly so as to have the ball bounce on the spring towards the target (bigger beige circle), which turns yellow if the ball’s bouncing trajectory passes through it.

Original: They receive a team-reward of r t = 0 .
Revised: They receive a team-reward of r t = 0 .

Original: 1 if the ball reﬂects towards the side walls, r t = 0 .
Revised: 1 if the ball reﬂects towards the side walls, r t = 0 .

Original: 2 if the ball reﬂects towards the top of the environment, and r t = 10 if the ball reﬂects towards the target.
Revised: 2 if the ball reﬂects towards the top of the environment, and r t = 10 if the ball reﬂects towards the target.

Original: At initialisation, the target’s and ball’s vertical position is ﬁxed, their horizontal positions are random.
Revised: At initialisation, the target’s and ball’s vertical position is ﬁxed, their horizontal positions are random.

Original: Agents’ initial positions are also random.
Revised: Agents’ initial positions are also random.

Original: Termination is triggered when the ball is bounced by the agents or when the maximum number of timesteps is reached.
Revised: Termination is triggered when the ball is bounced by the agents or when the maximum number of timesteps is reached.

Original: COMPROMISE (Figure 3c):
Revised: COMPROMISE (Figure 3c):

Original: In this environment, two agents (small orange circles) are linked together with a spring that pulls them toward each other when stretched above its relaxation length.
Revised: In this environment, two agents (small orange circles) are linked together with a spring that pulls them toward each other when stretched above its relaxation length.

Original: They both have a distinct assigned landmark (light gray circle for light orange agent, dark gray circle for dark orange agent), and receive a reward of r t = 10 when they reach it.
Revised: They both have a distinct assigned landmark (light gray circle for light orange agent, dark gray circle for dark orange agent), and receive a reward of r t = 10 when they reach it.

Original: Once a landmark is reached by its corresponding agent, the landmark is randomly relocated in the environment.
Revised: Once a landmark is reached by its corresponding agent, the landmark is randomly relocated in the environment.

Original: Initial positions of agents and landmark are random.
Revised: Initial positions of agents and landmark are random.

Original: Termination is triggered when the maximum number of timesteps is reached.
Revised: Termination is triggered when the maximum number of timesteps is reached.

Original: CHASE (Figure 3d):
Revised: CHASE (Figure 3d):

Original: In this environment, two predators (orange circles) are chasing a prey (turquoise circle).
Revised: In this environment, two predators (orange circles) are chasing a prey (turquoise circle).

Original: The prey moves with respect to a scripted policy consisting of repulsion forces from the walls and predators.
Revised: The prey moves with respect to a scripted policy consisting of repulsion forces from the walls and predators.

Original: At each timestep, the learning agents (predators) receive a teamreward of r t = n where n is the number of predators touching the prey.
Revised: At each timestep, the learning agents (predators) receive a teamreward of r t = n where n is the number of predators touching the prey.

Original: The prey has a greater max speed and acceleration than the predators.
Revised: The prey has a greater max speed and acceleration than the predators.

Original: Therefore, to maximize their return, the two agents must coordinate in order to squeeze the prey into a corner or a wall and effectively trap it there.
Revised: Therefore, to maximize their return, the two agents must coordinate in order to squeeze the prey into a corner or a wall and effectively trap it there.

Original: Termination is triggered when the maximum number of time steps is reached.
Revised: Termination is triggered when the maximum number of time steps is reached.

Original: B T RAINING DETAILS
Revised: B T RAINING DETAILS

Original: In all of our experiments, we use the Adam optimizer (Kingma & Ba, 2014) to perform parameter updates.
Revised: In all of our experiments, we use the Adam optimizer (Kingma & Ba, 2014) to perform parameter updates.

Original: All models (actors and critics) are parametrized by feedforward networks containing two hidden layers of 128 units.
Revised: All models (actors, critics and coach) are parametrized by feedforward networks containing two hidden layers of 128 units.

Original: We use the Rectiﬁed Linear Unit (ReLU) (Nair & Hinton, 2010) as activation function and layer normalization (Ba et al., 2016) on the pre-activations unit to stabilize the learning.
Revised: We use the Rectiﬁed Linear Unit (ReLU) (Nair & Hinton, 2010) as activation function and layer normalization (Ba et al., 2016) on the pre-activations unit to stabilize the learning.

Original: We use a buffer-size of 10 6 entries and a batch-size of 1024 .
Revised: We use a buffer-size of 10 6 entries and a batch-size of 1024 .

Original: We collect 100 transitions by interacting with the environment for each learning update.
Revised: We collect 100 transitions by interacting with the environment for each learning update.

Original: For all tasks in our hyper-parameter searches, we train the agents for 15 , 000 episodes of 100 steps and then re-train the best conﬁguration for each algorithm-environment pair for twice as long ( 30 , 000 episodes) to ensure full convergence for the ﬁnal evaluation.
Revised: For all tasks in our hyper-parameter searches, we train the agents for 15 , 000 episodes of 100 steps and then re-train the best conﬁguration for each algorithm-environment pair for twice as long ( 30 , 000 episodes) to ensure full convergence for the ﬁnal evaluation.

Original: The scale of the exploration noise is kept constant for the ﬁrst half of the training time and then decreases linearly to 0 until the end of training.
Revised: The scale of the exploration noise is kept constant for the ﬁrst half of the training time and then decreases linearly to 0 until the end of training.

Original: We use a discount factor γ of 0 .
Revised: We use a discount factor γ of 0 .

Original: 95 and a gradient clipping threshold of 0 .
Revised: 95 and a gradient clipping threshold of 0 .

Original: 5 in all experiments.
Revised: 5 in all experiments.

Original: Finally for CoachReg, we ﬁxed k to 4 meaning that agents could choose between 4 sub-policies.
Revised: Finally for CoachReg, we ﬁxed K to 4 meaning that agents could choose between 4 sub-policies.

Revised: Since policies’ hidden layers are of size 128 the corresponding value for C is 32.

Original: C
Revised: C

Original: A LGORITHMS
Revised: A LGORITHMS

Original: Algorithm 1 Team
Revised: Algorithm 1 Team

Original: Randomly initialize N critic networks Q i and actor networks µ i
Revised: Randomly initialize N critic networks Q i and actor networks µ i

Original: Initialize the target weights
Revised: Initialize the target weights

Original: Initialize one replay buffer D for episode from 0 to number of episodes do
Revised: Initialize one replay buffer D for episode from 0 to number of episodes do

Original: Initialize random processes N i for action exploration
Revised: Initialize random processes N i for action exploration

Original: Receive initial joint observation ofor timestep t from 0 to episode length do
Revised: Receive initial joint observation ofor timestep t from 0 to episode length do

Original: Select action a i = µ i ( o it )
Revised: Select action a i = µ i ( o it )

Original: + N it for each agent
Revised: + N it for each agent

Original: Execute joint action a t and observe joint reward r t and new observation o t +
Revised: Execute joint action a t and observe joint reward r t and new observation o t +

Original: Store transition ( o t , a t , r t , o t +1 ) in D end for
Revised: Store transition ( o t , a t , r t , o t +1 ) in D end for

Original: Sample a random minibatch of M transitions from D for each agent i do
Revised: Sample a random minibatch of M transitions from D for each agent i do

Original: Evaluate L i and J iPG from Equations (1) and (2) for each other agent ( j (cid:54) = i ) do
Revised: Evaluate L i and J iPG from Equations (1) and (2) for each other agent ( j (cid:54) = i ) do

Original: Evaluate J i,jTS from Equations (3)
Revised: Evaluate J i,jTS from Equations (3)

Original: Update actor j with θ j ← θ j + α θ ∇ θ j λ 2 J i,jTS end for
Revised: Update actor j with θ j ← θ j + α θ ∇ θ j λ 2 J i,jTS end for

Original: Update critic with φ i ← φ
Revised: Update critic with φ i ← φ

Original: i − α φ ∇ φ i L i
Revised: i − α φ ∇ φ i L i

Original: Update actor i with θ i ← θ i + α θ ∇ θ i (cid:16) J iPG + λ 1 (cid:80) Nj =1 J i,jTS (cid:17) end for
Revised: Update actor i with θ i ← θ i + α θ ∇ θ i (cid:16) J iPG + λ 1 (cid:80) Nj =1 J i,jTS (cid:17) end for

Original: Update all target weights end for
Revised: Update all target weights end for

Original: Algorithm 2 Coach
Revised: Algorithm 2 Coach

Original: Randomly initialize N critic networks Q i , actor networks µ i and one coach network p c
Revised: Randomly initialize N critic networks Q i , actor networks µ i and one coach network p c

Original: Initialize N target networks Q i (cid:48) and µ i (cid:48)
Revised: Initialize N target networks Q i (cid:48) and µ i (cid:48)

Original: Initialize one replay buffer D for episode from 0 to number of episodes do
Revised: Initialize one replay buffer D for episode from 0 to number of episodes do

Original: Initialize random processes N i for action exploration
Revised: Initialize random processes N i for action exploration

Original: Receive initial joint observation ofor timestep t from 0 to episode length do
Revised: Receive initial joint observation ofor timestep t from 0 to episode length do

Original: Select action a i = µ i ( o it )
Revised: Select action a i = µ i ( o it )

Original: + N it for each agent
Revised: + N it for each agent

Original: Execute joint action a t and observe joint reward r t and new observation o t +
Revised: Execute joint action a t and observe joint reward r t and new observation o t +

Original: Store transition ( o t , a t , r t , o t +1 ) in D end for
Revised: Store transition ( o t , a t , r t , o t +1 ) in D end for

Original: Sample a random minibatch of M transitions from D for each agent i do
Revised: Sample a random minibatch of M transitions from D for each agent i do

Original: Evaluate L i and J iPG from Equations (1) and (2)
Revised: Evaluate L i and J iPG from Equations (1) and (2)

Original: Update critic with φ i ← φ
Revised: Update critic with φ i ← φ

Original: i − α φ ∇ φ
Revised: i − α φ ∇ φ

Original: D H YPER - PARAMETER SEARCH D. H YPER - PARAMETER SEARCH RANGES
Revised: D H YPER - PARAMETER SEARCH D. H YPER - PARAMETER SEARCH RANGES

Original: We perform searches over the following hyper-parameters: the learning rate of the actor α θ , the learning rate of the critic ω
Revised: We perform searches over the following hyper-parameters: the learning rate of the actor α θ , the learning rate of the critic ω

Original: φ relative to the actor ( α φ = ω φ ∗ α θ ), the target-network soft-update parameter τ and the initial scale of the exploration noise η noise for the Ornstein-Uhlenbeck noise generating process (Uhlenbeck & Ornstein, 1930) as used by Lillicrap et al. (2015).
Revised: φ relative to the actor ( α φ = ω φ ∗ α θ ), the target-network soft-update parameter τ and the initial scale of the exploration noise η noise for the Ornstein-Uhlenbeck noise generating process (Uhlenbeck & Ornstein, 1930) as used by Lillicrap et al. (2015).

Original: When using
Revised: When using

Original: TeamReg and CoachReg, we additionally search over the regularization weights λ 1 , λ 2 and λ 3 .
Revised: TeamReg and CoachReg, we additionally search over the regularization weights λ 1 , λ 2 and λ 3 .

Original: The learning rate of the coach is always equal to the actor’s learning rate (i.e. α θ = α ψ ), motivated by their similar architectures and learning signals and in order to reduce the search space.
Revised: The learning rate of the coach is always equal to the actor’s learning rate (i.e. α θ = α ψ ), motivated by their similar architectures and learning signals and in order to reduce the search space.

Original: Table 1 shows the ranges from which values for the hyper-parameters are drawn uniformly during the searches.
Revised: Table 1 shows the ranges from which values for the hyper-parameters are drawn uniformly during the searches.

Original: [Table]
Revised: [Table]

Original: D. M ODEL SELECTION
Revised: D. M ODEL SELECTION

Original: During training, a policy is evaluated on a set of 10 different episodes every 100 learning steps.
Revised: During training, a policy is evaluated on a set of 10 different episodes every 100 learning steps.

Original: At the end of the training, the model at the best evaluation iteration is saved as the best version of the policy for this training, and is re-evaluated on 100 different episodes to have a better assessment of its ﬁnal performance.
Revised: At the end of the training, the model at the best evaluation iteration is saved as the best version of the policy for this training, and is re-evaluated on 100 different episodes to have a better assessment of its ﬁnal performance.

Original: The performance of a hyper-parameter conﬁguration is deﬁned as the average performance (across seeds) of the policies learned using this set of hyper-parameter values.
Revised: The performance of a hyper-parameter conﬁguration is deﬁned as the average performance (across seeds) of the policies learned using this set of hyper-parameter values.

Original: D. S ELECTED HYPER - PARAMETERS
Revised: D. S ELECTED HYPER - PARAMETERS

Original: Tables 2, 3,
Revised: Tables 2, 3,

Original: [Table]
Revised: [Table]

Original: and 5 shows the best hyper-parameters found by the random searches for each of the environments and each of the algorithms.
Revised: and 5 shows the best hyper-parameters found by the random searches for each of the environments and each of the algorithms.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D. S ELECTED HYPER - PARAMETERS ( ABLATIONS )
Revised: D. S ELECTED HYPER - PARAMETERS ( ABLATIONS )

Original: Tables 6, 7,
Revised: Tables 6, 7,

Original: [Table]
Revised: [Table]

Original: and 9 shows the best hyper-parameters found by the random searches for each of the environments and each of the ablated algorithms.
Revised: and 9 shows the best hyper-parameters found by the random searches for each of the environments and each of the ablated algorithms.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D. H YPER - PARAMETER SEARCH RESULTS
Revised: D. H YPER - PARAMETER SEARCH RESULTS

Original: The performance of each parameter conﬁguration is reported in Figure 10 yielding the performance distribution across hyper-parameters conﬁgurations for each algorithm on each task.
Revised: The performance of each parameter conﬁguration is reported in Figure 10 yielding the performance distribution across hyper-parameters conﬁgurations for each algorithm on each task.

Original: The same distributions are depicted in Figure 11 using box-and-whisker plot.
Revised: The same distributions are depicted in Figure 11 using box-and-whisker plot.

Original: It can be seen that TeamReg and
Revised: It can be seen that TeamReg and

Original: CoachReg both boost the performance of the third quartile, suggesting an increase in the robustness across hyper-parameter
Revised: CoachReg both boost the performance of the third quartile, suggesting an increase in the robustness across hyper-parameter

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D. H YPER - PARAMETER SEARCH RESULTS ( ABLATIONS )
Revised: D. H YPER - PARAMETER SEARCH RESULTS ( ABLATIONS )

Original: From Figure 13 it seems that the “policy mask” or the “agent modelling” additions respectively provide nearly the same robustness boosts as CoachReg and TeamReg
Revised: From Figure 13 it seems that the “policy mask” or the “agent modelling” additions respectively provide nearly the same robustness boosts as CoachReg and TeamReg

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: E T HE EFFECTS OF ENFORCING PREDICTABILITY ( ADDITIONAL RESULTS )
Revised: E T HE EFFECTS OF ENFORCING PREDICTABILITY ( ADDITIONAL RESULTS )

Original: [Figure]
Revised: [Figure]

Original: F A NALYSIS OF SUB - POLICY SELECTION ( ADDITIONAL RESULTS ) F. M ASK DENSITIES
Revised: F A NALYSIS OF SUB - POLICY SELECTION ( ADDITIONAL RESULTS ) F. M ASK DENSITIES

Original: We depict on Figure 15 the mask distribution of each agent for each (seed, environment) experiment.
Revised: We depict on Figure 15 the mask distribution of each agent for each (seed, environment) experiment.

Original: Firstly, in most of the experiments, agents use at least 2 different masks.
Revised: Firstly, in most of the experiments, agents use at least 2 different masks.

Original: Secondly, for a given experiments, agents’ distributions are very similar, suggesting that they are using the same masks in the same situations and that they are therefore synchronized.
Revised: Secondly, for a given experiments, agents’ distributions are very similar, suggesting that they are using the same masks in the same situations and that they are therefore synchronized.

Original: Finally, agents collapse more to using only one mask on CHASE, where they also display more dissimilarity between one another.
Revised: Finally, agents collapse more to using only one mask on CHASE, where they also display more dissimilarity between one another.

Original: This may explain why CHASE is the only task where CoachReg does not improve performance.
Revised: This may explain why CHASE is the only task where CoachReg does not improve performance.

Original: Indeed, on CHASE, agents do not seem synchronized nor leveraging multiple sub-policies which are the priors to coordination behind CoachReg.
Revised: Indeed, on CHASE, agents do not seem synchronized nor leveraging multiple sub-policies which are the priors to coordination behind CoachReg.

Original: In brief, we observe that CoachReg is less effective in enforcing those priors to coordination of CHASE, an environment where it does not boost nor harm performance.
Revised: In brief, we observe that CoachReg is less effective in enforcing those priors to coordination of CHASE, an environment where it does not boost nor harm performance.

Original: F. E PISODES ROLL - OUTS
Revised: F. E PISODES ROLL - OUTS

Original: We render here some episodes roll-outs, the agents synchronously switch between policy masks during an episode.
Revised: We render here some episodes roll-outs, the agents synchronously switch between policy masks during an episode.

Original: In addition, the whole group selects the same mask as the one that would have been suggested by the coach.
Revised: In addition, the whole group selects the same mask as the one that would have been suggested by the coach.

Original: t = 0, C = t = 5, C = t = 20, C = t = 10, C = t = 15, C = t
Revised: t = 0, C = t = 5, C = t = 20, C = t = 10, C = t = 15, C = t

Original: = 25, C = t = 30, C = t = 35, C
Revised: = 25, C = t = 30, C = t = 35, C

Original: = (a) SPREAD t = 0, C
Revised: = (a) SPREAD t = 0, C

Original: = t = 3, C
Revised: = t = 3, C

Original: = t = 17, C
Revised: = t = 17, C

Original: = t = 6, C
Revised: = t = 6, C

Original: = t = 7, C = t = 22, C
Revised: = t = 7, C = t = 22, C

Original: = t = 32, C = t = 33, C = t = 34, C = t = 37, C = t = 40, C = t = 42, C
Revised: = t = 32, C = t = 33, C = t = 34, C = t = 37, C = t = 40, C = t = 42, C

Original: = t = 43, C = t = 48, C = t
Revised: = t = 43, C = t = 48, C = t

Original: = 51, C = t = 52, C
Revised: = 51, C = t = 52, C

Original: = t = 53, C = t = 54, C = t = 64, C
Revised: = t = 53, C = t = 54, C = t = 64, C

Original: = t = 65, C = t = 66, C = t
Revised: = t = 65, C = t = 66, C = t

Original: = 67, C = t = 68, C = t = 73, C = (b) COMPROMISE
Revised: = 67, C = t = 68, C = t = 73, C = (b) COMPROMISE

Original: F. M ASK DIVERSITY AND SYNCHRONICITY ( ABLATION )
Revised: F. M ASK DIVERSITY AND SYNCHRONICITY ( ABLATION )

Original: As in Subsection 6.4 we report the mean entropy of the mask distribution and the mean Hamming proximity for the ablated “MADDPG + policy mask” and compare it to the full CoachReg.
Revised: As in Subsection 6.4 we report the mean entropy of the mask distribution and the mean Hamming proximity for the ablated “MADDPG + policy mask” and compare it to the full CoachReg.

Original: With “MADDPG + policy mask” agents are not incentivized to use the same masks.
Revised: With “MADDPG + policy mask” agents are not incentivized to use the same masks.

Original: Therefore, in order to assess if they synchronously change policy masks, we computed, for each agent pair, seed and environment, the Hamming proximity for every possible masks equivalence (mask 3 of agentcorresponds to mask 0 of agent 2, etc.) and selected the equivalence that maximised the Hamming proximity between the two sequences.
Revised: Therefore, in order to assess if they synchronously change policy masks, we computed, for each agent pair, seed and environment, the Hamming proximity for every possible masks equivalence (mask 3 of agentcorresponds to mask 0 of agent 2, etc.) and selected the equivalence that maximised the Hamming proximity between the two sequences.

Original: We can observe that while “MADDPG + policy mask” agents display a more diverse mask usage, their selection is less synchronized than with CoachReg.
Revised: We can observe that while “MADDPG + policy mask” agents display a more diverse mask usage, their selection is less synchronized than with CoachReg.

Original: This is easily understandable as the coach will tend to reduce diversity in order to have all the agents agree on a common mask, on the other hand this agreement enables the agents to synchronize their mask selection.
Revised: This is easily understandable as the coach will tend to reduce diversity in order to have all the agents agree on a common mask, on the other hand this agreement enables the agents to synchronize their mask selection.

Original: To this regard, it should be noted that “MADDPG + policy mask” agents are more synchronized that agents independently sampling their masks from k -CUD, suggesting that, even in the absence of the coach, agents tend to synchronize their mask selection.
Revised: To this regard, it should be noted that “MADDPG + policy mask” agents are more synchronized that agents independently sampling their masks from k -CUD, suggesting that, even in the absence of the coach, agents tend to synchronize their mask selection.

Original: [Figure]
Revised: [Figure]

Revised: G R OBUSTNESS TO THE NUMBER OF AGENTS

Revised: We varied the number of agents present in the SPREAD task from three to six.

Revised: For each algorithm we used the best performing hyper-parameter conﬁguration from the hyper-parameter search performed on SPREAD with three agents and trained on ten different random seeds.

Revised: Results are shown in

Revised: Figure 18.

Revised: As expected the task becomes more complicated when the number of agents increases and no algorithm succeeds at the task with six agents.

Revised: This difﬁculty is likely to be exacerbated by the sparse reward setting.

Revised: However, the proposed methods still outperform the baselines showing that they do not disproportionately suffer from the increased regularization pressure of additional agents

Revised: [Figure]