Original: RODE: L EARNING R OLES TO D ECOMPOSE M ULTI -A GENT T ASKS
Revised: RODE: L EARNING R OLES TO D ECOMPOSE M ULTI -A GENT T ASKS

Original: A BSTRACT
Revised: A BSTRACT

Original: Role-based learning holds the promise of achieving scalable multi-agent learning by decomposing complex tasks using roles.
Revised: Role-based learning holds the promise of achieving scalable multi-agent learning by decomposing complex tasks using roles.

Original: However, it is largely unclear how to efﬁciently discover such a set of roles.
Revised: However, it is largely unclear how to efﬁciently discover such a set of roles.

Original: To solve this problem, we propose to ﬁrst decompose joint action spaces into restricted role action spaces by clustering actions according to their effects on the environment and other agents.
Revised: To solve this problem, we propose to ﬁrst decompose joint action spaces into restricted role action spaces by clustering actions according to their effects on the environment and other agents.

Original: Learning a role selector based on action effects makes role discovery much easier because it forms a bi-level learning hierarchy: the role selector searches in a smaller role space and at a lower temporal resolution, while role policies learn in signiﬁcantly reduced primitive action-observation spaces.
Revised: Learning a role selector based on action effects makes role discovery much easier because it forms a bi-level learning hierarchy: the role selector searches in a smaller role space and at a lower temporal resolution, while role policies learn in signiﬁcantly reduced primitive action-observation spaces.

Original: We further integrate information about action effects into the role policies to boost learning efﬁciency and policy generalization.
Revised: We further integrate information about action effects into the role policies to boost learning efﬁciency and policy generalization.

Original: By virtue of these advances, our method (1) outperforms the current state-of-the-art MARL algorithms on 9 of the 14 scenarios that comprise the challenging StarCraft II micromanagement benchmark and (2) achieves rapid transfer to new environments with three times the number of agents.
Revised: By virtue of these advances, our method (1) outperforms the current state-of-the-art MARL algorithms on 9 of the 14 scenarios that comprise the challenging StarCraft II micromanagement benchmark and (2) achieves rapid transfer to new environments with three times the number of agents.

Original: Demonstrative videos can be viewed at https://sites.google.com/view/rode-marl .
Revised: Demonstrative videos can be viewed at https://sites.google.com/view/rode-marl .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Cooperative multi-agent problems are ubiquitous in real-world applications, such as crewless aerial vehicles (Pham et al., 2018; Xu et al., 2018) and sensor networks (Zhang & Lesser, 2013).
Revised: Cooperative multi-agent problems are ubiquitous in real-world applications, such as crewless aerial vehicles (Pham et al., 2018; Xu et al., 2018) and sensor networks (Zhang & Lesser, 2013).

Original: However, learning control policies for such systems remains a major challenge.
Revised: However, learning control policies for such systems remains a major challenge.

Original: Joint action learning (Claus &
Revised: Joint action learning (Claus &

Original: Boutilier, 1998) learns centralized policies conditioned on the full state, but this global information is often unavailable during execution due to partial observability or communication constraints.
Revised: Boutilier, 1998) learns centralized policies conditioned on the full state, but this global information is often unavailable during execution due to partial observability or communication constraints.

Original: Independent learning (Tan, 1993) avoids this problem by learning decentralized policies but suffers from non-stationarity during learning as it treats other learning agents as part of the environment.
Revised: Independent learning (Tan, 1993) avoids this problem by learning decentralized policies but suffers from non-stationarity during learning as it treats other learning agents as part of the environment.

Original: The framework of centralized training with decentralized execution (CTDE) (Foerster et al., 2016;
Revised: The framework of centralized training with decentralized execution (CTDE) (Foerster et al., 2016;

Original: Gupta et al., 2017; Rashid et al., 2018) combines the advantages of these two paradigms.
Revised: Gupta et al., 2017; Rashid et al., 2018) combines the advantages of these two paradigms.

Original: Decentralized policies are learned in a centralized manner so that they can share information, parameters, etc., without restriction during training.
Revised: Decentralized policies are learned in a centralized manner so that they can share information, parameters, etc., without restriction during training.

Original: Although CTDE algorithms can solve many multi-agent problems (Mahajan et al., 2019; Das et al., 2019; Wang et al., 2020d), during training they must search in the joint action-observation space, which grows exponentially with the number of agents.
Revised: Although CTDE algorithms can solve many multi-agent problems (Mahajan et al., 2019; Das et al., 2019; Wang et al., 2020d), during training they must search in the joint action-observation space, which grows exponentially with the number of agents.

Original: This makes it difﬁcult to learn efﬁciently when the number of agents is large (Samvelyan et al., 2019).
Revised: This makes it difﬁcult to learn efﬁciently when the number of agents is large (Samvelyan et al., 2019).

Original: Humans cooperate in a more effective way.
Revised: Humans cooperate in a more effective way.

Original: When dealing with complex tasks, instead of directly conducting a collective search in the full action-observation space, they typically decompose the task and let sub-groups of individuals learn to solve different sub-tasks (Smith, 1937; Butler, 2012).
Revised: When dealing with complex tasks, instead of directly conducting a collective search in the full action-observation space, they typically decompose the task and let sub-groups of individuals learn to solve different sub-tasks (Smith, 1937; Butler, 2012).

Original: Once the task is decomposed, the complexity of cooperative learning can be effectively reduced because individuals can focus on restricted sub-problems, each of which often involves a smaller action-observation space.
Revised: Once the task is decomposed, the complexity of cooperative learning can be effectively reduced because individuals can focus on restricted sub-problems, each of which often involves a smaller action-observation space.

Original: Such potential scalability motivates the use of roles in multi-agent tasks, in which each role is associated with a certain sub-task and a corresponding policy.
Revised: Such potential scalability motivates the use of roles in multi-agent tasks, in which each role is associated with a certain sub-task and a corresponding policy.

Original: The key question in realizing such scalable learning is how to come up with a set of roles to effectively decompose the task.
Revised: The key question in realizing such scalable learning is how to come up with a set of roles to effectively decompose the task.

Original: Previous work typically predeﬁnes the task decomposition and roles (Pav´on & G´omez-Sanz, 2003; Cossentino et al., 2005; Spanoudakis & Moraitis, 2010; Bonjean et al., 2014).
Revised: Previous work typically predeﬁnes the task decomposition and roles (Pav´on & G´omez-Sanz, 2003; Cossentino et al., 2005; Spanoudakis & Moraitis, 2010; Bonjean et al., 2014).

Original: However, this requires prior knowledge that might not be available in practice and may prevent the learning methods from transferring to different environments.
Revised: However, this requires prior knowledge that might not be available in practice and may prevent the learning methods from transferring to different environments.

Original: Therefore, to be practical, it is crucial for role-based methods to automatically learn an appropriate set of roles.
Revised: Therefore, to be practical, it is crucial for role-based methods to automatically learn an appropriate set of roles.

Original: However, learning roles from scratch might not be easier than learning without roles, as directly ﬁnding an optimal decomposition suffers from the same problem as other CTDE learning methods – searching in the large joint space with substantial exploration (Wang et al., 2020c).
Revised: However, learning roles from scratch might not be easier than learning without roles, as directly ﬁnding an optimal decomposition suffers from the same problem as other CTDE learning methods – searching in the large joint space with substantial exploration (Wang et al., 2020c).

Original: To solve this problem, we propose a novel framework for learning ROles to DEcompose (RODE) multi-agent tasks.
Revised: To solve this problem, we propose a novel framework for learning ROles to DEcompose (RODE) multi-agent tasks.

Original: Our key insight is that, instead of learning roles from scratch, role discovery is easier if we ﬁrst decompose joint action spaces according to action functionality.
Revised: Our key insight is that, instead of learning roles from scratch, role discovery is easier if we ﬁrst decompose joint action spaces according to action functionality.

Original: Intuitively, when cooperating with other agents, only a subset of actions that can fulﬁll a certain functionality is needed under certain observations.
Revised: Intuitively, when cooperating with other agents, only a subset of actions that can fulﬁll a certain functionality is needed under certain observations.

Original: For example, in football games, the player who does not possess the ball only needs to explore how to move or sprint when attacking.
Revised: For example, in football games, the player who does not possess the ball only needs to explore how to move or sprint when attacking.

Original: In practice, we propose to ﬁrst learn effect-based action representations and cluster actions into role action spaces according to their effects on the environment and other agents.
Revised: In practice, we propose to ﬁrst learn effect-based action representations and cluster actions into role action spaces according to their effects on the environment and other agents.

Original: Then, with knowledge of effects of available actions, we train a role selector that determines corresponding role observation spaces.
Revised: Then, with knowledge of effects of available actions, we train a role selector that determines corresponding role observation spaces.

Original: This design forms a bi-level learning framework.
Revised: This design forms a bi-level learning framework.

Original: At the top level, a role selector coordinates role assignments in a smaller role space and at a lower temporal resolution.
Revised: At the top level, a role selector coordinates role assignments in a smaller role space and at a lower temporal resolution.

Original: At the low level, role policies explore strategies in reduced primitive action-observation spaces.
Revised: At the low level, role policies explore strategies in reduced primitive action-observation spaces.

Original: In this way, the learning complexity is signiﬁcantly reduced by decomposing a multi-agent cooperation problem, both temporally and spatially, into several short-horizon learning problems with fewer agents.
Revised: In this way, the learning complexity is signiﬁcantly reduced by decomposing a multi-agent cooperation problem, both temporally and spatially, into several short-horizon learning problems with fewer agents.

Original: To further improve learning efﬁciency on the sub-problems, we condition role policies on the learned effect-based action representations, which improves generalizability of role policies across actions.
Revised: To further improve learning efﬁciency on the sub-problems, we condition role policies on the learned effect-based action representations, which improves generalizability of role policies across actions.

Original: We test RODE on StarCraft II micromanagement environments (Samvelyan et al., 2019).
Revised: We test RODE on StarCraft II micromanagement environments (Samvelyan et al., 2019).

Original: Results on this benchmark show that RODE establishes a new state of the art.
Revised: Results on this benchmark show that RODE establishes a new state of the art.

Original: Particularly, RODE has the best performance on 9 out of all 14 maps, including all 5 super hard maps and most hard maps.
Revised: Particularly, RODE has the best performance on 9 out of all 14 maps, including all 5 super hard maps and most hard maps.

Original: Visualizations of learned action representations, factored action spaces, and dynamics of role selections shed further light on the superior performance of RODE.
Revised: Visualizations of learned action representations, factored action spaces, and dynamics of role selections shed further light on the superior performance of RODE.

Original: We also demonstrate that conditioning the role selector and role policies on action representations enables learned RODE policies to be transferred to tasks with different numbers of actions and agents, including tasks with three times as many agents.
Revised: We also demonstrate that conditioning the role selector and role policies on action representations enables learned RODE policies to be transferred to tasks with different numbers of actions and agents, including tasks with three times as many agents.

Original:  RODE L EARNING F RAMEWORK
Revised:  RODE L EARNING F RAMEWORK

Original: In this section, we introduce the RODE learning framework.
Revised: In this section, we introduce the RODE learning framework.

Original: We consider fully cooperative multiagent tasks that can be modelled as a Dec-POMDP (Oliehoek et al., 2016) consisting of a tuple
Revised: We consider fully cooperative multiagent tasks that can be modelled as a Dec-POMDP (Oliehoek et al., 2016) consisting of a tuple

Original: G = (cid:104) I, S, A, P, R, Ω , O, n, γ (cid:105) , where I is the ﬁnite set of n agents, γ ∈ [0 , 1) is the discount factor, and s ∈ S is the true state of the environment.
Revised: G = (cid:104) I, S, A, P, R, Ω , O, n, γ (cid:105) , where I is the ﬁnite set of n agents, γ ∈ [0 , 1) is the discount factor, and s ∈ S is the true state of the environment.

Original: At each timestep, each agent i receives an observation o
Revised: At each timestep, each agent i receives an observation o

Original: i ∈ Ω drawn according to the observation function
Revised: i ∈ Ω drawn according to the observation function

Original: O ( s, i ) and selects an action a i ∈ A , forming a joint action a ∈
Revised: O ( s, i ) and selects an action a i ∈ A , forming a joint action a ∈

Original: A n , leading to a next state s (cid:48) according to the transition function P ( s (cid:48) | s, a ) , and observing a reward r = R ( s, a ) shared by all agents.
Revised: A n , leading to a next state s (cid:48) according to the transition function P ( s (cid:48) | s, a ) , and observing a reward r = R ( s, a ) shared by all agents.

Original: Each agent has local action-observation history τ i ∈ T ≡ (Ω × A ) ∗ .
Revised: Each agent has local action-observation history τ i ∈ T ≡ (Ω × A ) ∗ .

Original: Our idea is to learn to decompose a multi-agent cooperative task into a set of sub-tasks, each of which has a much smaller action-observation space.
Revised: Our idea is to learn to decompose a multi-agent cooperative task into a set of sub-tasks, each of which has a much smaller action-observation space.

Original: Each sub-task is associated with a role, and agents taking the same role collectively learn a role policy for solving the sub-task by sharing their learning.
Revised: Each sub-task is associated with a role, and agents taking the same role collectively learn a role policy for solving the sub-task by sharing their learning.

Original: Formally, we propose the following deﬁnition of sub-tasks and roles.
Revised: Formally, we propose the following deﬁnition of sub-tasks and roles.

Original: Deﬁnition 1 (Role and Sub-Task) .
Revised: Deﬁnition 1 (Role and Sub-Task) .

Original: Given a cooperative multi-agent task G = (cid:104) I, S, A, P, R, Ω , O, n, γ (cid:105) , let Ψ be a set of roles.
Revised: Given a cooperative multi-agent task G = (cid:104) I, S, A, P, R, Ω , O, n, γ (cid:105) , let Ψ be a set of roles.

Original: A role ρ j ∈ Ψ is a tuple (cid:104) g j , π ρ j (cid:105) where g j = (cid:104) I j , S, A j , P, R, Ω j , O, γ (cid:105) is a sub-task and I j ⊂ I , ∪ j I
Revised: A role ρ j ∈ Ψ is a tuple (cid:104) g j , π ρ j (cid:105) where g j = (cid:104) I j , S, A j , P, R, Ω j , O, γ (cid:105) is a sub-task and I j ⊂ I , ∪ j I

Original: j
Revised: j

Original: = I , and I j ∩ I k
Revised: = I , and I j ∩ I k

Original: = ∅ , j (cid:54) =
Revised: = ∅ , j (cid:54) =

Original: k .
Revised: k .

Original: A j is the action space of role j ,
Revised: A j is the action space of role j ,

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 𝒛
Revised: 𝒛

Original: [Figure]
Revised: [Figure]

Original: 𝒛
Revised: 𝒛

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A j ⊂ A , ∪ j A j = A , and we allow action spaces of different roles to overlap: | A j ∩ A k | ≥ 0 , j (cid:54) =
Revised: A j ⊂ A , ∪ j A j = A , and we allow action spaces of different roles to overlap: | A j ∩ A k | ≥ 0 , j (cid:54) =

Original: k .
Revised: k .

Original: π ρ j : T × A j
Revised: π ρ j : T × A j

Original: → [0 , 1] is a role policy for the sub-task g j associated with the role.
Revised: → [0 , 1] is a role policy for the sub-task g j associated with the role.

Original: In our deﬁnition, a given set of roles speciﬁes not only a factorization of the task but also policies for sub-tasks.
Revised: In our deﬁnition, a given set of roles speciﬁes not only a factorization of the task but also policies for sub-tasks.

Original: This role-based formulation is effectively a kind of dynamic grouping mechanism where agents are grouped by roles.
Revised: This role-based formulation is effectively a kind of dynamic grouping mechanism where agents are grouped by roles.

Original: The beneﬁt is that we can learn policies for roles instead of agents, providing scalability with the number of agents and reducing search space of target policies.
Revised: The beneﬁt is that we can learn policies for roles instead of agents, providing scalability with the number of agents and reducing search space of target policies.

Original: In our framework, an agent decides which role to take by a role selector β that is shared among agents.
Revised: In our framework, an agent decides which role to take by a role selector β that is shared among agents.

Original: Here, β : T → Ψ is a deterministic function that is conditioned on local action-observation history.
Revised: Here, β : T → Ψ is a deterministic function that is conditioned on local action-observation history.

Original: Our aim is to learn a set of roles Ψ ∗ that maximizes the expected global return Q Ψ ( s t , a t ) =
Revised: Our aim is to learn a set of roles Ψ ∗ that maximizes the expected global return Q Ψ ( s t , a t ) =

Original: E s t +1: ∞ , a t +1: ∞
Revised: E s t +1: ∞ , a t +1: ∞

Original: [ (cid:80) ∞ i =

Revised: [ (cid:80) ∞ i =

Original: γ i r t
Revised: γ i r t

Original: + i | s t , a t , Ψ] .
Revised: + i | s t , a t , Ψ] .

Original: Learning roles involves two questions: learning sub-tasks and role policies.
Revised: Learning roles involves two questions: learning sub-tasks and role policies.

Original: For learning sub-tasks, we need to learn the role selector, β , and to determine role action spaces, A j .
Revised: For learning sub-tasks, we need to learn the role selector, β , and to determine role action spaces, A j .

Original: These two components determine the observation and action spaces of roles, respectively.
Revised: These two components determine the observation and action spaces of roles, respectively.

Original: We now introduce the framework to learn these components, which is shown in Fig.
Revised: We now introduce the framework to learn these components, which is shown in Fig.

Original: 2. D ETERMINING R OLE A CTION S PACES BY L EARNING A CTION R EPRESENTATIONS
Revised: 2. D ETERMINING R OLE A CTION S PACES BY L EARNING A CTION R EPRESENTATIONS

Original: Restricting the action space of each role can signiﬁcantly reduce the policy search space.
Revised: Restricting the action space of each role can signiﬁcantly reduce the policy search space.

Original: The key to our approach is to factor the action space according to actions’ properties and thereby let roles focus on actions with similar effects, such as attacking enemy units of the same type in StarCraft II.
Revised: The key to our approach is to factor the action space according to actions’ properties and thereby let roles focus on actions with similar effects, such as attacking enemy units of the same type in StarCraft II.

Original: To this end, we learn action representations that can reﬂect the effects of actions on the environment and other agents.
Revised: To this end, we learn action representations that can reﬂect the effects of actions on the environment and other agents.

Original: The effect of an action can be measured by the induced reward and the change in local observations.
Revised: The effect of an action can be measured by the induced reward and the change in local observations.

Original: Therefore, we formulate an objective for learning action representations that incentivizes including enough information such that the next observations and rewards can be predicted when given the actions of other agents and current observations.
Revised: Therefore, we formulate an objective for learning action representations that incentivizes including enough information such that the next observations and rewards can be predicted when given the actions of other agents and current observations.

Original: In practice, we use the predictive model shown in Fig. 1(a) to learn an action encoder f e ( · ; θ e ) :
Revised: In practice, we use the predictive model shown in Fig. 1(a) to learn an action encoder f e ( · ; θ e ) :

Original: R | A | → R d .
Revised: R | A | → R d .

Original: This encoder, parameterized by θ e , maps one-hot actions to a d -dimensional representation space.
Revised: This encoder, parameterized by θ e , maps one-hot actions to a d -dimensional representation space.

Original: We denote the representation of an action a in this space by z a , i .
Revised: We denote the representation of an action a in this space by z a , i .

Original: e ., z a = f e ( a ; θ e ) , which is then used to predict the next observation o (cid:48)
Revised: e ., z a = f e ( a ; θ e ) , which is then used to predict the next observation o (cid:48)

Original: i and the global reward r , given the current observation o i of an agent i , and the one-hot actions of other agents, a - i .
Revised: i and the global reward r , given the current observation o i of an agent i , and the one-hot actions of other agents, a - i .

Original: This model can be interpreted as a forward model, which is trained by minimizing the following loss function:
Revised: This model can be interpreted as a forward model, which is trained by minimizing the following loss function:

Original: [Equation]
Revised: [Equation]

Original: where p o and p r are predictors for observations and rewards, respectively, and parameterized by ξ e .
Revised: where p o and p r are predictors for observations and rewards, respectively, and parameterized by ξ e .

Original: λ e is a scaling factor, D is an replay buffer, and the sum is carried out over all the agents.
Revised: λ e is a scaling factor, D is an replay buffer, and the sum is carried out over all the agents.

Original: In the beginning, we initialize K roles, each with the full action space.
Revised: In the beginning, we initialize K roles, each with the full action space.

Original: After collecting samples and training the predictive model for t e timesteps, we cluster the actions according to their latent representations and set the action space of each role to contain one of the clusters.
Revised: After collecting samples and training the predictive model for t e timesteps, we cluster the actions according to their latent representations and set the action space of each role to contain one of the clusters.

Original: Outliers are added to all the clusters to avoid action sets consisting of only one action.
Revised: Outliers are added to all the clusters to avoid action sets consisting of only one action.

Original: After this update, training begins and action representations and action spaces of each role are kept ﬁxed during training.
Revised: After this update, training begins and action representations and action spaces of each role are kept ﬁxed during training.

Original: In this paper, we simply use k -means clustering based on Euclidean distances between action representations, but it can be easily extended to other clustering methods.
Revised: In this paper, we simply use k -means clustering based on Euclidean distances between action representations, but it can be easily extended to other clustering methods.

Original: In Appendix E, we investigate the inﬂuence of different k values and how to determine the number of clusters automatically.
Revised: In Appendix E, we investigate the inﬂuence of different k values and how to determine the number of clusters automatically.

Original: In practice, we ﬁnd that the forward model converges quickly.
Revised: In practice, we ﬁnd that the forward model converges quickly.

Original: For example, a well-formed representation space can be learned using 50 K samples on StarCraft II micromanagement tasks, while the training usually lasts for 2 M timesteps.
Revised: For example, a well-formed representation space can be learned using 50 K samples on StarCraft II micromanagement tasks, while the training usually lasts for 2 M timesteps.

Original: 2. L EARNING THE R OLE S ELECTOR AND R OLE P OLICIES
Revised: 2. L EARNING THE R OLE S ELECTOR AND R OLE P OLICIES

Original: Learning action representations to cluster actions based on their effects leads to a factorization of the action space, where each subset of actions can fulﬁll a certain functionality.
Revised: Learning action representations to cluster actions based on their effects leads to a factorization of the action space, where each subset of actions can fulﬁll a certain functionality.

Original: To make full use of such a factorization, we use a bi-level hierarchical structure to coordinate the selection of roles and primitive actions.
Revised: To make full use of such a factorization, we use a bi-level hierarchical structure to coordinate the selection of roles and primitive actions.

Original: At the top level, a role selector assigns a role to each agent every c timesteps.
Revised: At the top level, a role selector assigns a role to each agent every c timesteps.

Original: After a role is assigned, an agent explores in the corresponding restricted role action space to learn the role policy.
Revised: After a role is assigned, an agent explores in the corresponding restricted role action space to learn the role policy.

Original: We now describe how to design efﬁcient, transferable, and lightweight structures for the role selector and role policies.
Revised: We now describe how to design efﬁcient, transferable, and lightweight structures for the role selector and role policies.

Original: For the role selector, we can simply use a conventional Q -network whose input is local actionobservation history and output is Q -values for each role.
Revised: For the role selector, we can simply use a conventional Q -network whose input is local actionobservation history and output is Q -values for each role.

Original: However, this structure may not be efﬁcient because it ignores information about action space of different roles.
Revised: However, this structure may not be efﬁcient because it ignores information about action space of different roles.

Original: Intuitively, selecting a role is selecting a subset of actions to execute for the next c timesteps, so the Q -value of selecting a role is closely related to its role action space.
Revised: Intuitively, selecting a role is selecting a subset of actions to execute for the next c timesteps, so the Q -value of selecting a role is closely related to its role action space.

Original: Therefore, we propose to base the role selector on the average representation of available actions.
Revised: Therefore, we propose to base the role selector on the average representation of available actions.

Original: Fig. 1(b) shows the proposed role selector structure.
Revised: Fig. 1(b) shows the proposed role selector structure.

Original: Speciﬁcally, we call
Revised: Speciﬁcally, we call

Original: [Equation]
Revised: [Equation]

Original: the representation of role ρ j , where A j is its restricted action space.
Revised: the representation of role ρ j , where A j is its restricted action space.

Original: To select roles, agents share a linear layer and a GRU, parameterized by θ τ β , to encode the local action-observation history τ into a ﬁxed-length vector h τ .
Revised: To select roles, agents share a linear layer and a GRU, parameterized by θ τ β , to encode the local action-observation history τ into a ﬁxed-length vector h τ .

Original: The role selector, a fully connected network f β ( h τ ; θ β ) with parameters θ β , then maps h τ to z τ ∈ R d , and we estimate the expected return of agent i selecting role ρ j as:
Revised: The role selector, a fully connected network f β ( h τ ; θ β ) with parameters θ β , then maps h τ to z τ ∈ R d , and we estimate the expected return of agent i selecting role ρ j as:

Original: [Equation]
Revised: [Equation]

Original: Agents select their roles concurrently, which may lead to unintended behaviors.
Revised: Agents select their roles concurrently, which may lead to unintended behaviors.

Original: For example, all agents choosing to attack a certain unit type in StarCraft II causes overkill.
Revised: For example, all agents choosing to attack a certain unit type in StarCraft II causes overkill.

Original: To better coordinate role assignments, we use a mixing network to estimate global Q -values, Q β tot , using per-agent utility Q β i so that the role selector can be trained using global rewards.
Revised: To better coordinate role assignments, we use a mixing network to estimate global Q -values, Q β tot , using per-agent utility Q β i so that the role selector can be trained using global rewards.

Original: In this paper, we use the mixing network introduced by QMIX (Rashid et al., 2018) for its monotonic approximation.
Revised: In this paper, we use the mixing network introduced by QMIX (Rashid et al., 2018) for its monotonic approximation.

Original: The parameters of the mixing network are conditioned on the global state s and are generated by a hypernetwork (Ha et al., 2016) parameterized by ξ β .
Revised: The parameters of the mixing network are conditioned on the global state s and are generated by a hypernetwork (Ha et al., 2016) parameterized by ξ β .

Original: Then we minimize the following TD loss to update the role selector f β :
Revised: Then we minimize the following TD loss to update the role selector f β :

Original: [Equation]
Revised: [Equation]

Original: where ¯ Q βtot is a target network, ρ = (cid:104) ρ 1 , ρ 2 , . .
Revised: where ¯ Q βtot is a target network, ρ = (cid:104) ρ 1 , ρ 2 , . .

Original: . , ρ n (cid:105) is the joint role of all agents, and the expectation is estimated with uniform samples from a replay buffer D .
Revised: . , ρ n (cid:105) is the joint role of all agents, and the expectation is estimated with uniform samples from a replay buffer D .

Original: After a role is assigned, an agent follows it for the next c timesteps, during which it can only take actions in the corresponding role action space.
Revised: After a role is assigned, an agent follows it for the next c timesteps, during which it can only take actions in the corresponding role action space.

Original: Each role ρ j has a role policy π ρ j : T × A j
Revised: Each role ρ j has a role policy π ρ j : T × A j

Original: → [0 , 1] that is deﬁned in the restricted action space.
Revised: → [0 , 1] that is deﬁned in the restricted action space.

Original: Similar to the role selector, the most straightforward approach to learning role policies is with a conventional deep Q -network that directly estimates the
Revised: Similar to the role selector, the most straightforward approach to learning role policies is with a conventional deep Q -network that directly estimates the

Original: Q -values of each action.
Revised: Q -values of each action.

Original: However, basing Q -values on action representations makes full use of the effect information of actions and can generalize better across actions.
Revised: However, basing Q -values on action representations makes full use of the effect information of actions and can generalize better across actions.

Original: Therefore, we use the framework shown in Fig.
Revised: Therefore, we use the framework shown in Fig.

Original: 1(c) for learning role policies.
Revised: 1(c) for learning role policies.

Original: Speciﬁcally, agents again use a shared linear layer and a GRU which together encode a local action-observation history τ into a vector h τ .
Revised: Speciﬁcally, agents again use a shared linear layer and a GRU which together encode a local action-observation history τ into a vector h τ .

Original: We denote the parameters of these two networks by θ τ ρ .
Revised: We denote the parameters of these two networks by θ τ ρ .

Original: Each role policy is a fully connected network f ρ j ( h τ ; θ ρ j ) with parameters θ ρ j , whose input is h τ .
Revised: Each role policy is a fully connected network f ρ j ( h τ ; θ ρ j ) with parameters θ ρ j , whose input is h τ .

Original: f ρ j maps h τ to z τ , which is a vector in R d .
Revised: f ρ j maps h τ to z τ , which is a vector in R d .

Original: Using action representations z a k and z τ , we estimate the value of agent i choosing a primitive action a k as:
Revised: Using action representations z a k and z τ , we estimate the value of agent i choosing a primitive action a k as:

Original: [Equation]
Revised: [Equation]

Original: For learning the Q i using global rewards, we again feed local Q -values into a QMIX-style mixing network to estimate the global action-value, Q tot ( s, a ) .
Revised: For learning the Q i using global rewards, we again feed local Q -values into a QMIX-style mixing network to estimate the global action-value, Q tot ( s, a ) .

Original: The parameters of the mixing network are denoted by ξ ρ .
Revised: The parameters of the mixing network are denoted by ξ ρ .

Original: This formulation gives the TD loss for learning role policies:
Revised: This formulation gives the TD loss for learning role policies:

Original: [Equation]
Revised: [Equation]

Original: Here, ¯ Q tot is a target network, θ ρ is the parameters of all role policies, and the expectation is estimated with uniform samples from the same replay buffer D as the one for the role selector.
Revised: Here, ¯ Q tot is a target network, θ ρ is the parameters of all role policies, and the expectation is estimated with uniform samples from the same replay buffer D as the one for the role selector.

Original: Moreover, two mixing networks for training the role selector and role policies are only used during training.
Revised: Moreover, two mixing networks for training the role selector and role policies are only used during training.

Original: By virtue of effect-based action representations, our framework is lightweight – in practice, we use a single linear layer without activation functions for role policies and a two-layer fully-connected network for the role selector.
Revised: By virtue of effect-based action representations, our framework is lightweight – in practice, we use a single linear layer without activation functions for role policies and a two-layer fully-connected network for the role selector.

Original: Basing the role selector and role policies on action representations also enable RODE policies to be rapidly transferred to new tasks with different numbers of agents and actions, which we discuss in detail in Sec. 4.3.
Revised: Basing the role selector and role policies on action representations also enable RODE policies to be rapidly transferred to new tasks with different numbers of agents and actions, which we discuss in detail in Sec. 4.3.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Hierarchical MARL To realize efﬁcient role-based learning, RODE adopts a hierarchical decisionmaking structure.
Revised: Hierarchical MARL To realize efﬁcient role-based learning, RODE adopts a hierarchical decisionmaking structure.

Original: Hierarchical reinforcement learning (HRL) (Sutton et al., 1999; Al-Emran, 2015) has been extensively studied to address the sparse reward problem and to facilitate transfer learning.
Revised: Hierarchical reinforcement learning (HRL) (Sutton et al., 1999; Al-Emran, 2015) has been extensively studied to address the sparse reward problem and to facilitate transfer learning.

Original: Single-agent HRL focuses on learning temporal decomposition of tasks, either by learning subgoals (Nachum et al., 2018b;a; Levy et al., 2018; Ghosh et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019;
Revised: Single-agent HRL focuses on learning temporal decomposition of tasks, either by learning subgoals (Nachum et al., 2018b;a; Levy et al., 2018; Ghosh et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019;

Original: Dilokthanakul et al., 2019) or by discovering reusable skills (Daniel et al., 2012; Gregor et al., 2016; Warde-Farley et al., 2018; Shankar & Gupta, 2020; Thomas et al., 2018; Sharma et al., 2020).
Revised: Dilokthanakul et al., 2019) or by discovering reusable skills (Daniel et al., 2012; Gregor et al., 2016; Warde-Farley et al., 2018; Shankar & Gupta, 2020; Thomas et al., 2018; Sharma et al., 2020).

Original: In multi-agent settings, many challenges, such as efﬁcient communication (Ossenkopf et al., 2019) and labor division (Wang et al., 2020c) in large systems, necessitate hierarchical learning along the second dimension – over agents (Zhang et al., 2010).
Revised: In multi-agent settings, many challenges, such as efﬁcient communication (Ossenkopf et al., 2019) and labor division (Wang et al., 2020c) in large systems, necessitate hierarchical learning along the second dimension – over agents (Zhang et al., 2010).

Original: Ahilan & Dayan (2019) propose a hierarchy in FeUdal (Dayan & Hinton, 1993;
Revised: Ahilan & Dayan (2019) propose a hierarchy in FeUdal (Dayan & Hinton, 1993;

Original: Vezhnevets et al., 2017) style for cooperative multi-agent tasks, where managers and workers are predeﬁned and workers are expected to achieve the goal generated by managers.
Revised: Vezhnevets et al., 2017) style for cooperative multi-agent tasks, where managers and workers are predeﬁned and workers are expected to achieve the goal generated by managers.

Original: Similar hierarchical organizations among agents have been exploited to control trafﬁc lights (Jin & Ma, 2018).
Revised: Similar hierarchical organizations among agents have been exploited to control trafﬁc lights (Jin & Ma, 2018).

Original: Lee et al.
Revised: Lee et al.

Original: Yang et al.
Revised: Yang et al.

Original: (2019) propose to use bi-level hierarchies to train (or discover) and coordinate individual skills.
Revised: (2019) propose to use bi-level hierarchies to train (or discover) and coordinate individual skills.

Original: Vezhnevets et al.
Revised: Vezhnevets et al.

Original: (2020) extend hierarchical MARL to Markov Games, where a high-level policy chooses strategic responses to opponents.
Revised: (2020) extend hierarchical MARL to Markov Games, where a high-level policy chooses strategic responses to opponents.

Original: RODE proposes to learn multi-agent coordination skills in restricted action spaces and is thus complementary to these previous works.
Revised: RODE proposes to learn multi-agent coordination skills in restricted action spaces and is thus complementary to these previous works.

Original: Dealing with Large Discrete Action Spaces In single-agent settings, being able to reason with a large number of discrete actions is essential to bringing reinforcement learning to a larger class of problems, such as natural language processing (He et al., 2016) and recommender systems (Ayush et al., 2020).
Revised: Dealing with Large Discrete Action Spaces In single-agent settings, being able to reason with a large number of discrete actions is essential to bringing reinforcement learning to a larger class of problems, such as natural language processing (He et al., 2016) and recommender systems (Ayush et al., 2020).

Original: Sallans & Hinton (2004); Pazis & Parr (2011) represent actions using binary code to reduce the size of large action spaces.
Revised: Sallans & Hinton (2004); Pazis & Parr (2011) represent actions using binary code to reduce the size of large action spaces.

Original: Cui & Khardon (2016; 2018) address scalability issues of planning problems by conducting a gradient-based search on a symbolic representation of the stateaction value function.
Revised: Cui & Khardon (2016; 2018) address scalability issues of planning problems by conducting a gradient-based search on a symbolic representation of the stateaction value function.

Original: Sharma et al.
Revised: Sharma et al.

Original: (2017) show that factoring actions into their primary categories can improve performance of deep RL algorithms on Atari 2600 games (Bellemare et al., 2013).
Revised: (2017) show that factoring actions into their primary categories can improve performance of deep RL algorithms on Atari 2600 games (Bellemare et al., 2013).

Original: All aforementioned methods assume that a handcrafted binary decomposition of raw actions is provided.
Revised: All aforementioned methods assume that a handcrafted binary decomposition of raw actions is provided.

Original: Another interesting way to deal with large discrete state-action spaces is by estimating equivalence classes over reward trajectories and learn on quotient MDPs (Mahajan & Tulabandhula, 2017a;b).
Revised: Another interesting way to deal with large discrete state-action spaces is by estimating equivalence classes over reward trajectories and learn on quotient MDPs (Mahajan & Tulabandhula, 2017a;b).

Original: Yet another line of research uses continuous action representations to discover underlying structures of large discrete action spaces, thereby easing training.
Revised: Yet another line of research uses continuous action representations to discover underlying structures of large discrete action spaces, thereby easing training.

Original: Van Hasselt & Wiering (2009) use policy gradients with continuous actions and select the nearest discrete action.
Revised: Van Hasselt & Wiering (2009) use policy gradients with continuous actions and select the nearest discrete action.

Original: Dulac-Arnold et al.
Revised: Dulac-Arnold et al.

Original: (2015) extend this work to large-scale problems but use predeﬁned action embeddings.
Revised: (2015) extend this work to large-scale problems but use predeﬁned action embeddings.

Original: Tennenholtz &
Revised: Tennenholtz &

Original: Mannor (2019); Chandak et al.
Revised: Mannor (2019); Chandak et al.

Original: (2019) avoid pre-deﬁnitions by extracting action representations from expert demonstrations or without prior knowledge.
Revised: (2019) avoid pre-deﬁnitions by extracting action representations from expert demonstrations or without prior knowledge.

Original: Action representations have been used to solve natural language processing problems (He et al., 2016), enable generalization to unseen (Ayush et al., 2020) or under-explored (Chandak et al., 2019) actions in RL, learn linear dynamics that help exploration (Kim et al., 2019), and transfer policy across different tasks (
Revised: Action representations have been used to solve natural language processing problems (He et al., 2016), enable generalization to unseen (Ayush et al., 2020) or under-explored (Chandak et al., 2019) actions in RL, learn linear dynamics that help exploration (Kim et al., 2019), and transfer policy across different tasks (

Original: By contrast, we use action representations to factor multi-agent tasks.
Revised: By contrast, we use action representations to factor multi-agent tasks.

Original: Besides learning action representations, Czarnecki et al.
Revised: Besides learning action representations, Czarnecki et al.

Original: Murali et al.
Revised: Murali et al.

Original: ; Farquhar et al.
Revised: ; Farquhar et al.

Original: (2020) propose to use curriculum learning over action spaces of increasing sizes to scale learning to large spaces.
Revised: (2020) propose to use curriculum learning over action spaces of increasing sizes to scale learning to large spaces.

Original: Although the large action space problem is more demanding in multi-agent settings, how to organize effective learning in them is largely unstudied – most algorithms directly search in the full action spaces, including previous works (Wang et al., 2020f) that consider the effects (semantics) of actions like RODE.
Revised: Although the large action space problem is more demanding in multi-agent settings, how to organize effective learning in them is largely unstudied – most algorithms directly search in the full action spaces, including previous works (Wang et al., 2020f) that consider the effects (semantics) of actions like RODE.

Original: This is a key motivation for RODE, where search spaces are reduced by an effect-based action space factorization and a role-based hierarchical structure designed for efﬁcient learning in these factored action spaces.
Revised: This is a key motivation for RODE, where search spaces are reduced by an effect-based action space factorization and a role-based hierarchical structure designed for efﬁcient learning in these factored action spaces.

Original: In Appendix G, we discuss how RODE is related to other role-based learning approaches and multiagent reinforcement learning algorithms in detail.
Revised: In Appendix G, we discuss how RODE is related to other role-based learning approaches and multiagent reinforcement learning algorithms in detail.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We design experiments to answer the following questions: (1) Can the proposed predictive model learn action representations that reﬂect effects of actions?
Revised: We design experiments to answer the following questions: (1) Can the proposed predictive model learn action representations that reﬂect effects of actions?

Original: (Sec. 4.1) (2) Can RODE improve learning efﬁciency?
Revised: (Sec. 4.1) (2) Can RODE improve learning efﬁciency?

Original: If so, which component contributes the most to the performance gains?
Revised: If so, which component contributes the most to the performance gains?

Original: (Sec. 4.2) (3)
Revised: (Sec. 4.2) (3)

Original: Can RODE support rapid transfer to environments with different numbers of actions and agents?
Revised: Can RODE support rapid transfer to environments with different numbers of actions and agents?

Original: (Sec. 4.3) (4) Can the role selector learn interpretable high-level cooperative behaviors?
Revised: (Sec. 4.3) (4) Can the role selector learn interpretable high-level cooperative behaviors?

Original: (Appx. B)
Revised: (Appx. B)

Original: We choose the StarCraft II micromanagement (SMAC) benchmark (Samvelyan et al., 2019) as the testbed for its rich environments and high complexity of control.
Revised: We choose the StarCraft II micromanagement (SMAC) benchmark (Samvelyan et al., 2019) as the testbed for its rich environments and high complexity of control.

Original: The SMAC benchmark requires learning policies in a large action space.
Revised: The SMAC benchmark requires learning policies in a large action space.

Original: Agents can move in four cardinal directions, stop , take noop (do nothing), or select an enemy to attack at each timestep.
Revised: Agents can move in four cardinal directions, stop , take noop (do nothing), or select an enemy to attack at each timestep.

Original: Therefore, if there are n e enemies in the map, the action space for each ally unit consists of n e + 6 discrete actions.
Revised: Therefore, if there are n e enemies in the map, the action space for each ally unit consists of n e + 6 discrete actions.

Original: 4. A CTION R EPRESENTATIONS
Revised: 4. A CTION R EPRESENTATIONS

Original: For learning action representations, on all maps, we set the scaling factor
Revised: For learning action representations, on all maps, we set the scaling factor

Original: λ e to 10 and collect samples and train the predictive model for 50 K timesteps using the loss function described by
Revised: λ e to 10 and collect samples and train the predictive model for 50 K timesteps using the loss function described by

Original: Eq. 1.
Revised: Eq. 1.

Original: The model is trained every episode using a batch of 32 episodes.
Revised: The model is trained every episode using a batch of 32 episodes.

Original: In Fig.
Revised: In Fig.

Original: 2, we show the learned action representations on two maps, 3s5z vs 3s6z and corridor .
Revised: 2, we show the learned action representations on two maps, 3s5z vs 3s6z and corridor .

Original: The map corridor features homogeneous agents and enemies, with 6 Zealots faced with 24 enemy
Revised: The map corridor features homogeneous agents and enemies, with 6 Zealots faced with 24 enemy

Original: Zerglings.
Revised: Zerglings.

Original: In this task, all attack actions have similar effects because the enemies are homogeneous.
Revised: In this task, all attack actions have similar effects because the enemies are homogeneous.

Original: Moreover, due to the symmetry of the map, moving northward and eastward exert similar inﬂuence on the environment – moving the agent towards enemies.
Revised: Moreover, due to the symmetry of the map, moving northward and eastward exert similar inﬂuence on the environment – moving the agent towards enemies.

Original: Likewise, moving southward and westward have the same effect because both actions move agents away from enemies.
Revised: Likewise, moving southward and westward have the same effect because both actions move agents away from enemies.

Original: The learned action encoder captures these underlying structures in the action space (Fig. 2 left).
Revised: The learned action encoder captures these underlying structures in the action space (Fig. 2 left).

Original: We can observe three
Revised: We can observe three

Original: [Figure]
Revised: [Figure]

Original: clear clusters in the latent action representation space, which correspond to the aforementioned three types of actions, respectively.
Revised: clear clusters in the latent action representation space, which correspond to the aforementioned three types of actions, respectively.

Original: On the map 3s5z vs 3s6z , 3 Stalkers and 5 Zealots try to beat 3 enemy Stalkers and 6 Enemy Zealots.
Revised: On the map 3s5z vs 3s6z , 3 Stalkers and 5 Zealots try to beat 3 enemy Stalkers and 6 Enemy Zealots.

Original: Unlike corridor , the enemy team consists of heterogeneous units
Revised: Unlike corridor , the enemy team consists of heterogeneous units

Original: [Figure]
Revised: [Figure]

Original: and attacking Stalkers and Zealots have distinct effects.
Revised: and attacking Stalkers and Zealots have distinct effects.

Original: Our predictive model highlights this difference – the attack actions are clustered into two groups (Fig.
Revised: Our predictive model highlights this difference – the attack actions are clustered into two groups (Fig.

Original: 2 right).
Revised: 2 right).

Original: Moreover, the learned action encoder also captures the dissimilarity between moving northward/southward and moving in other directions – these two move actions do not bring agents near to or far away from enemies.
Revised: Moreover, the learned action encoder also captures the dissimilarity between moving northward/southward and moving in other directions – these two move actions do not bring agents near to or far away from enemies.

Original: Although we only show results on two maps, each with one random seed, similar action clusters can be observed on all the scenarios and are robust across different seeds.
Revised: Although we only show results on two maps, each with one random seed, similar action clusters can be observed on all the scenarios and are robust across different seeds.

Original: Therefore, we conclude that our predictive model effectively learns action representations that reﬂect the effects of actions.
Revised: Therefore, we conclude that our predictive model effectively learns action representations that reﬂect the effects of actions.

Original: 4. P ERFORMANCE AND A BLATION S TUDY
Revised: 4. P ERFORMANCE AND A BLATION S TUDY

Original: For evaluation, all experiments in this section are carried out with 8 different random seeds, and the median performance as well as the 25-75% percentiles are shown.
Revised: For evaluation, all experiments in this section are carried out with 8 different random seeds, and the median performance as well as the 25-75% percentiles are shown.

Original: The detailed setting of hyperparameters is described in Appendix A.
Revised: The detailed setting of hyperparameters is described in Appendix A.

Original: We benchmark our method on all 14 scenarios in order to evaluate its performance across the entire
Revised: We benchmark our method on all 14 scenarios in order to evaluate its performance across the entire

Original: SMAC suite, and the results are shown in Fig.
Revised: SMAC suite, and the results are shown in Fig.

Original: We compare with current state-of-the-art valuebased MARL algorithms (QMIX (Rashid et al., 2018), QPLEX (Wang et al., 2020b), Weighted
Revised: We compare with current state-of-the-art valuebased MARL algorithms (QMIX (Rashid et al., 2018), QPLEX (Wang et al., 2020b), Weighted

Original: QMIX (Rashid et al., 2020a), and AI-QMIX (Iqbal et al., 2020)), hierarchical MARL method (HSD, Yang et al.
Revised: QMIX (Rashid et al., 2020a), and AI-QMIX (Iqbal et al., 2020)), hierarchical MARL method (HSD, Yang et al.

Original: (2019)), role-based MARL method (ROMA, Wang et al.
Revised: (2019)), role-based MARL method (ROMA, Wang et al.

Original: (2020c)), and MARL method based on action semantics (ASN, Wang et al.
Revised: (2020c)), and MARL method based on action semantics (ASN, Wang et al.

Original: (2020f)).
Revised: (2020f)).

Original: RODE outperforms all the baselines by at least 1 / 32 on 9 of all 14 scenarios after 2 M training steps.
Revised: RODE outperforms all the baselines by at least 1 / 32 on 9 of all 14 scenarios after 2 M training steps.

Original: [Figure]
Revised: [Figure]

Original: Maps of the SMAC benchmark have been classiﬁed as easy , hard , and super hard .
Revised: Maps of the SMAC benchmark have been classiﬁed as easy , hard , and super hard .

Original: Hard and super hard maps are typically hard-exploration tasks.
Revised: Hard and super hard maps are typically hard-exploration tasks.

Original: Since RODE is designed to help exploration by decomposing the joint action space, we are especially interested in the performance of our method on these maps.
Revised: Since RODE is designed to help exploration by decomposing the joint action space, we are especially interested in the performance of our method on these maps.

Original: Fig.
Revised: Fig.

Original: 3 shows the performance of RODE on all the super hard scenarios and one hard scenario.
Revised: 3 shows the performance of RODE on all the super hard scenarios and one hard scenario.

Original: In Appendix C.1, we further show learning curves on the whole benchmark.
Revised: In Appendix C.1, we further show learning curves on the whole benchmark.

Original: We can see on all super hard maps and most hard maps, RODE has the best performance.
Revised: We can see on all super hard maps and most hard maps, RODE has the best performance.

Original: Moreover, RODE outperforms baselines by the largest margin on the maps that require more exploration: 3s5z vs 3s6z , corridor , and 6h vs 8z .
Revised: Moreover, RODE outperforms baselines by the largest margin on the maps that require more exploration: 3s5z vs 3s6z , corridor , and 6h vs 8z .

Original: In contrast, on 5 easy maps, RODE typically needs more samples to learn a successful strategy.
Revised: In contrast, on 5 easy maps, RODE typically needs more samples to learn a successful strategy.

Original: We hypothesize that this is because
Revised: We hypothesize that this is because

Original: [Figure]
Revised: [Figure]

Original: these maps do not require substantial exploration, and the exploratory beneﬁt brought by restricted action spaces is not obvious.
Revised: these maps do not require substantial exploration, and the exploratory beneﬁt brought by restricted action spaces is not obvious.

Original: Ablations To understand the superior performance of RODE, we carry out ablation studies to test the contribution of its four main components: (A) Restricted role action spaces; (B) Using action effects to cluster actions; (C)
Revised: Ablations To understand the superior performance of RODE, we carry out ablation studies to test the contribution of its four main components: (A) Restricted role action spaces; (B) Using action effects to cluster actions; (C)

Original: The integration of action representations into role policies and the role selector; (D)
Revised: The integration of action representations into role policies and the role selector; (D)

Original: The hierarchical learning structure.
Revised: The hierarchical learning structure.

Original: To test component A and B, when updating role action spaces after 50 K samples, we let each role action space contains all actions or a random subset of actions, respectively, while keeping other parts in the framework unchanged.
Revised: To test component A and B, when updating role action spaces after 50 K samples, we let each role action space contains all actions or a random subset of actions, respectively, while keeping other parts in the framework unchanged.

Original: For component B, we make sure the union of role action spaces equals the entire action space.
Revised: For component B, we make sure the union of role action spaces equals the entire action space.

Original: To test component C, we can simply use conventional Q -networks for learning role policies and the role selector.
Revised: To test component C, we can simply use conventional Q -networks for learning role policies and the role selector.

Original: The testing of component D is a bit different because we cannot ablate it separately while still using different roles.
Revised: The testing of component D is a bit different because we cannot ablate it separately while still using different roles.

Original: Therefore, we test it by ablating components A and C – using conventional deep Q -networks for role policies and the role selector and allowing role policies to choose from all primitive actions.
Revised: Therefore, we test it by ablating components A and C – using conventional deep Q -networks for role policies and the role selector and allowing role policies to choose from all primitive actions.

Original: The results on the three most difﬁcult scenarios from the benchmark are shown in Fig.
Revised: The results on the three most difﬁcult scenarios from the benchmark are shown in Fig.

Original: Generally speaking, performance of RODE using conventional deep Q -networks ( RODE No Action Repr. ) is close to that of RODE, and is signiﬁcantly better than the other two ablations where role action spaces are not restricted.
Revised: Generally speaking, performance of RODE using conventional deep Q -networks ( RODE No Action Repr. ) is close to that of RODE, and is signiﬁcantly better than the other two ablations where role action spaces are not restricted.

Original: Therefore, using action representations in policy learning can help improve learning but the superior performance of RODE is mostly due to the restriction of the role action spaces.
Revised: Therefore, using action representations in policy learning can help improve learning but the superior performance of RODE is mostly due to the restriction of the role action spaces.

Original: RODE with full role action spaces and conventional Q -networks ( RODE Full Action Spaces
Revised: RODE with full role action spaces and conventional Q -networks ( RODE Full Action Spaces

Original: No Action Repr. ) has similar performance to QMIX, which demonstrates that the hierarchical structure itself does not contribute much to the performance gains.
Revised: No Action Repr. ) has similar performance to QMIX, which demonstrates that the hierarchical structure itself does not contribute much to the performance gains.

Original: RODE with random restricted role action spaces also cannot outperform QMIX, highlighting the importance of using action effects to decompose joint action spaces.
Revised: RODE with random restricted role action spaces also cannot outperform QMIX, highlighting the importance of using action effects to decompose joint action spaces.

Original: [Figure]
Revised: [Figure]

Original: where 6 ally Zealots face
Revised: where 6 ally Zealots face

Original: We do not train policies on new maps.
Revised: We do not train policies on new maps.

Original: In summary, by virtue of restricted role action spaces and the integration of action effect information into policies,
Revised: In summary, by virtue of restricted role action spaces and the integration of action effect information into policies,

Original: RODE achieves a new state of the art on the SMAC benchmark.
Revised: RODE achieves a new state of the art on the SMAC benchmark.

Original: In Appendix B, we provide a case study to better understand why restricting role action spaces can improve performance.
Revised: In Appendix B, we provide a case study to better understand why restricting role action spaces can improve performance.

Original: We study the effects of the interval between role selections in Appendix D.1.
Revised: We study the effects of the interval between role selections in Appendix D.1.

Original: 4. P OLICY T RANSFER
Revised: 4. P OLICY T RANSFER

Original: A bonus of basing role policies on action representations is that we can transfer the learned policies to tasks with new actions that have similar effects to old ones.
Revised: A bonus of basing role policies on action representations is that we can transfer the learned policies to tasks with new actions that have similar effects to old ones.

Original: To demonstrate this, given a new task, we ﬁrst collect some samples (50 k ) and train an action encoder.
Revised: To demonstrate this, given a new task, we ﬁrst collect some samples (50 k ) and train an action encoder.

Original: In this way, we identify the old actions that have similar functionality to the new actions.
Revised: In this way, we identify the old actions that have similar functionality to the new actions.

Original: Then we use the average representation of these old actions as representations for new actions and add new actions to role action spaces to which the similar old actions belong.
Revised: Then we use the average representation of these old actions as representations for new actions and add new actions to role action spaces to which the similar old actions belong.

Original: Since role action spaces are still expected to contain actions with similar effects, which can be reﬂected by old role representations, we do not re-calculate the role representations on the new task.
Revised: Since role action spaces are still expected to contain actions with similar effects, which can be reﬂected by old role representations, we do not re-calculate the role representations on the new task.

Original: Moreover, in our framework, it is roles that are interacting with the environment, and agents only need to be assigned with roles.
Revised: Moreover, in our framework, it is roles that are interacting with the environment, and agents only need to be assigned with roles.

Original: Therefore, our learned policies can also be transferred to tasks with different numbers of agents.
Revised: Therefore, our learned policies can also be transferred to tasks with different numbers of agents.

Original: We test the transferability of our method on the SMAC benchmark.
Revised: We test the transferability of our method on the SMAC benchmark.

Original: To keep the length of observations ﬁxed, we sort allies and enemies by their relative distances to an agent and include the information of nearest N a allies and N e enemies in the visible range.
Revised: To keep the length of observations ﬁxed, we sort allies and enemies by their relative distances to an agent and include the information of nearest N a allies and N e enemies in the visible range.

Original: In Fig.
Revised: In Fig.

Original: 5, we show win rates of the policy learned from map corridor on different maps without further policy training.
Revised: 5, we show win rates of the policy learned from map corridor on different maps without further policy training.

Original: In the original task, 6 Zealots face 24 enemy Zerglings.
Revised: In the original task, 6 Zealots face 24 enemy Zerglings.

Original: We set N a to 5 and N e to 24 .
Revised: We set N a to 5 and N e to 24 .

Original: We increase the number of agents and enemies, and the number of actions increases as the number of enemies grows.
Revised: We increase the number of agents and enemies, and the number of actions increases as the number of enemies grows.

Original: The transferability of RODE is attested by observing that the learned policy can still win 50% of the games on unseen maps with three times the number of agents.
Revised: The transferability of RODE is attested by observing that the learned policy can still win 50% of the games on unseen maps with three times the number of agents.

Original: More details of this experiment is described in Appendix C.2.
Revised: More details of this experiment is described in Appendix C.2.

Original: Besides rapid transfer, iteratively training the transferred policies on larger tasks may be a promising future direction to scaling MARL to large-scale problems.
Revised: Besides rapid transfer, iteratively training the transferred policies on larger tasks may be a promising future direction to scaling MARL to large-scale problems.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Coming up with a set of roles that can effectively decompose the task is a long standing problem preventing role-based learning from realizing scalability.
Revised: Coming up with a set of roles that can effectively decompose the task is a long standing problem preventing role-based learning from realizing scalability.

Original: Instead of learning roles from scratch, in this paper, we ﬁnd that role discovery becomes much easier if we ﬁrst decompose joint action spaces according to action effects.
Revised: Instead of learning roles from scratch, in this paper, we ﬁnd that role discovery becomes much easier if we ﬁrst decompose joint action spaces according to action effects.

Original: With a specially designed hierarchical learning framework, we achieve efﬁcient learning over these factored action spaces.
Revised: With a specially designed hierarchical learning framework, we achieve efﬁcient learning over these factored action spaces.

Original: We believe that the scalability and transferability provided by our method are crucial in building ﬂexible and general-purpose multi-agent systems.
Revised: We believe that the scalability and transferability provided by our method are crucial in building ﬂexible and general-purpose multi-agent systems.

Revised: A CKNOWLEDGMENTS

Revised: We would like to thank the anonymous reviewers for their insightful comments and helpful suggestions.

Revised: This work is supported in part by Science and Technology Innovation 2030 – “New Generation Artiﬁcial Intelligence” Major Project (No. 2018AAA0100904), and a grant from the Institute of Guo Qiang, Tsinghua University.

Original: A A RCHITECTURE , H YPERPARAMETERS , AND I NFRASTRUCTURE
Revised: A A RCHITECTURE , H YPERPARAMETERS , AND I NFRASTRUCTURE

Original: In this paper, we use simple network structures for the role selector and role policies.
Revised: In this paper, we use simple network structures for the role selector and role policies.

Original: Each role policy is a simple linear network without hidden layers or activation functions, and the role selector is a two-layer feed-forward fully-connected network with a 64 -dimensional hidden layer.
Revised: Each role policy is a simple linear network without hidden layers or activation functions, and the role selector is a two-layer feed-forward fully-connected network with a 64 -dimensional hidden layer.

Original: Agents share a trajectory encoding network consisting of two layers, a fully-connected layer followed by a GRU layer with a 64-dimensional hidden state.
Revised: Agents share a trajectory encoding network consisting of two layers, a fully-connected layer followed by a GRU layer with a 64-dimensional hidden state.

Original: For all experiments, the length of action representations, d , is set to 20 .
Revised: For all experiments, the length of action representations, d , is set to 20 .

Original: The outputs of role policies and the outputs of role selector are each fed into their own separate QMIX-style mixing networks (Rashid et al., 2020b) to estimate the global action values.
Revised: The outputs of role policies and the outputs of role selector are each fed into their own separate QMIX-style mixing networks (Rashid et al., 2020b) to estimate the global action values.

Original: The two mixing networks use the same architecture, containing a 32-dimensional hidden layer with
Revised: The two mixing networks use the same architecture, containing a 32-dimensional hidden layer with

Original: ReLU activation.
Revised: ReLU activation.

Original: Parameters of the mixing networks are generated by hypernetworks conditioning on global states.
Revised: Parameters of the mixing networks are generated by hypernetworks conditioning on global states.

Original: These settings are the same to QMIX (Rashid et al., 2020b).
Revised: These settings are the same to QMIX (Rashid et al., 2020b).

Original: Moreover, it can be easily extended to other mixing mechanisms, such as QPLEX (Wang et al., 2020b), which may further improve the performance of RODE.
Revised: Moreover, it can be easily extended to other mixing mechanisms, such as QPLEX (Wang et al., 2020b), which may further improve the performance of RODE.

Original: For all experiments, the optimization is conducted using RMSprop with a learning rate of 5 × 10 − 4 , α of 0.99, and with no momentum or weight decay.
Revised: For all experiments, the optimization is conducted using RMSprop with a learning rate of 5 × 10 − 4 , α of 0.99, and with no momentum or weight decay.

Original: For exploration, we use (cid:15) -greedy with (cid:15) annealed linearly from 1 . 0 to 0 .
Revised: For exploration, we use (cid:15) -greedy with (cid:15) annealed linearly from 1 . 0 to 0 .

Original: 05 over 50 K time steps and kept constant for the rest of the training.
Revised: 05 over 50 K time steps and kept constant for the rest of the training.

Original: For three hard exploration maps— 3s5z vs 3s6z , 6h vs 8z , and 27m vs 30m —we extend the epsilon annealing time to 500 K , for both RODE and all the baselines and ablations.
Revised: For three hard exploration maps— 3s5z vs 3s6z , 6h vs 8z , and 27m vs 30m —we extend the epsilon annealing time to 500 K , for both RODE and all the baselines and ablations.

Original: Batches of 32 episodes are sampled from the replay buffer, and the role selector and role policies are trained end-to-end on fully unrolled episodes.
Revised: Batches of 32 episodes are sampled from the replay buffer, and the role selector and role policies are trained end-to-end on fully unrolled episodes.

Original: All experiments on the SMAC benchmark use the default reward and observation settings of the SMAC benchmark (Samvelyan et al., 2019).
Revised: All experiments on the SMAC benchmark use the default reward and observation settings of the SMAC benchmark (Samvelyan et al., 2019).

Original: Experiments are carried out on NVIDIA GTX 

Revised: Experiments are carried out on NVIDIA GTX

Original: Ti GPU.
Revised: Ti GPU.

Original: We use k -means clustering when determining role action spaces.
Revised: We use k -means clustering when determining role action spaces.

Original: The number of clusters, k , is treated as a hyperparameter.
Revised: The number of clusters, k , is treated as a hyperparameter.

Original: Speciﬁcally, on maps with homogeneous enemies, we set k to 3, and on maps with heterogeneous enemies, we set k to 5.
Revised: Speciﬁcally, on maps with homogeneous enemies, we set k to 3, and on maps with heterogeneous enemies, we set k to 5.

Original: If the task only involves one enemy, k is set to 2.
Revised: If the task only involves one enemy, k is set to 2.

Original: We can avoid this hyperparameter by using more advanced clustering approaches.
Revised: We can avoid this hyperparameter by using more advanced clustering approaches.

Original: For all baseline algorithms, we use the codes provided by their authors where the hyperparameters have been ﬁne-tuned on the SMAC benchmark.
Revised: For all baseline algorithms, we use the codes provided by their authors where the hyperparameters have been ﬁne-tuned on the SMAC benchmark.

Original: B C ASE S TUDY : R OLE D YNAMICS
Revised: B C ASE S TUDY : R OLE D YNAMICS

Original: In this section, we provide a case study to explore the dynamics of the role selector and give an example to explain why restricting action spaces of roles can improve learning performance.
Revised: In this section, we provide a case study to explore the dynamics of the role selector and give an example to explain why restricting action spaces of roles can improve learning performance.

Original: We carry out our study on the map corridor from the SMAC benchmark.
Revised: We carry out our study on the map corridor from the SMAC benchmark.

Original: We choose this map because it presents a hard-exploration task.
Revised: We choose this map because it presents a hard-exploration task.

Original: An optimal strategy for this scenario requires active state space exploration for the allied Zealots to learn to move to edges or to the choke point on the map, so as to not be surrounded by the enemy army.
Revised: An optimal strategy for this scenario requires active state space exploration for the allied Zealots to learn to move to edges or to the choke point on the map, so as to not be surrounded by the enemy army.

Original: Previous state-of-the-art algorithms all learn a suboptimal strategy, where agents merely damage the enemy units for rewards, instead of ﬁrst moving to edges and then attacking.
Revised: Previous state-of-the-art algorithms all learn a suboptimal strategy, where agents merely damage the enemy units for rewards, instead of ﬁrst moving to edges and then attacking.

Original: In Fig.
Revised: In Fig.

Original: 7, we show the roles assigned by the learned role selector at different timesteps in an episode.
Revised: 7, we show the roles assigned by the learned role selector at different timesteps in an episode.

Original: The ﬁrst row shows game snapshots, and the second row presents the corresponding role assignments.
Revised: The ﬁrst row shows game snapshots, and the second row presents the corresponding role assignments.

Original: Here, the action encoder gives three clusters of actions (see Fig.
Revised: Here, the action encoder gives three clusters of actions (see Fig.

Original: 2left) – move eastward & northward , attack , and move westward & southward .
Revised: 2left) – move eastward & northward , attack , and move westward & southward .

Original: These clusters of actions form the action spaces of three roles.
Revised: These clusters of actions form the action spaces of three roles.

Original: The color of circles indicates the role of an ally unit.
Revised: The color of circles indicates the role of an ally unit.

Original: We see that all agents select Role 2 in the beginning (Fig.
Revised: We see that all agents select Role 2 in the beginning (Fig.

Original: 7 (a))
Revised: 7 (a))

Original: to help them move to the edges, which is the good strategy as discussed above.
Revised: to help them move to the edges, which is the good strategy as discussed above.

Original: The agents also learn an effective cooperative policy – one of the agents (agent A) attracts most of the enemies (enemy group A) so that its teammates have an advantage when facing fewer enemies (enemy group B).
Revised: The agents also learn an effective cooperative policy – one of the agents (agent A) attracts most of the enemies (enemy group A) so that its teammates have an advantage when facing fewer enemies (enemy group B).

Original: After killing agent A, enemy group
Revised: After killing agent A, enemy group

Original: A goes through the choke point and arrives their destination.
Revised: A goes through the choke point and arrives their destination.

Original: In Fig.
Revised: In Fig.

Original: 7 (b), the alive agents start to attack this group of enemies.
Revised: 7 (b), the alive agents start to attack this group of enemies.

Original: However, the enemies are still very powerful.
Revised: However, the enemies are still very powerful.

Original: To win the game, ally agents learn to attract part of enemies and kill them.
Revised: To win the game, ally agents learn to attract part of enemies and kill them.

Original: They achieve this by alternating between Role 0 ( retract in this case) and Role 1 ( attack ), as shown in Fig.
Revised: They achieve this by alternating between Role 0 ( retract in this case) and Role 1 ( attack ), as shown in Fig.

Original: 7 (b-d).
Revised: 7 (b-d).

Original: Why do restricted action spaces work?
Revised: Why do restricted action spaces work?

Original: Our effect-based action clustering decomposes the full action spaces into smaller role action spaces, each of which focuses on a smaller subset of actions having similar effects.
Revised: Our effect-based action clustering decomposes the full action spaces into smaller role action spaces, each of which focuses on a smaller subset of actions having similar effects.

Original: As each role learns on a smaller action space which has similar actions, it enables agents to efﬁciently explore and results in each role being able to learn different strategies, like spread and retracting.
Revised: As each role learns on a smaller action space which has similar actions, it enables agents to efﬁciently explore and results in each role being able to learn different strategies, like spread and retracting.

Original: Moreover, exploration in the much smaller role space makes the coordination of these sub-strategies easily.
Revised: Moreover, exploration in the much smaller role space makes the coordination of these sub-strategies easily.

Original: Exploring in the hierarchically decomposed action spaces also provides a bias in exploration space.
Revised: Exploring in the hierarchically decomposed action spaces also provides a bias in exploration space.

Original: For example, on corridor , Role 0 and Role 2 motivate agents to explore the state space in a certain direction which help them learn several important strategies on this map: 1) Agents ﬁrst move to the edges ( Role 0 ) to avoid being surrounded by the enemies (Fig.
Revised: For example, on corridor , Role 0 and Role 2 motivate agents to explore the state space in a certain direction which help them learn several important strategies on this map: 1) Agents ﬁrst move to the edges ( Role 0 ) to avoid being surrounded by the enemies (Fig.

Original: 7 (a)), and 2) Agents alternate between attacking ( Role 1 ) and retracting ( Role 2 ) to attract and kill part of the enemies, gaining an advantage in numbers (Fig.
Revised: 7 (a)), and 2) Agents alternate between attacking ( Role 1 ) and retracting ( Role 2 ) to attract and kill part of the enemies, gaining an advantage in numbers (Fig.

Original: 7 (b-d)).
Revised: 7 (b-d)).

Original: The fact that Role 0 and Role 2 help exploration is also supported by Fig. 8 – their frequencies ﬁrst improve as the agents explore more, and then decrease when cooperation strategies gradually converge.
Revised: The fact that Role 0 and Role 2 help exploration is also supported by Fig. 8 – their frequencies ﬁrst improve as the agents explore more, and then decrease when cooperation strategies gradually converge.

Original: On other maps, restricted role action spaces may help agents solve the task in other ways.
Revised: On other maps, restricted role action spaces may help agents solve the task in other ways.

Original: For example, on the map 3s5z vs 3s6z , agents learn more efﬁciently because the action encoder decomposes the task into two sub-tasks, one focusing on attacking Zealots and another one focusing on attacking Stalkers.
Revised: For example, on the map 3s5z vs 3s6z , agents learn more efﬁciently because the action encoder decomposes the task into two sub-tasks, one focusing on attacking Zealots and another one focusing on attacking Stalkers.

Original: Learning policies for these restricted sub-problems signiﬁcantly reduces the learning complexity.
Revised: Learning policies for these restricted sub-problems signiﬁcantly reduces the learning complexity.

Original: In summary, action effects provide effective information to decompose joint action spaces.
Revised: In summary, action effects provide effective information to decompose joint action spaces.

Original: Clustering actions according to their effects makes role discovery much easier, and learning cooperative strategies in smaller action spaces makes learning more tractable.
Revised: Clustering actions according to their effects makes role discovery much easier, and learning cooperative strategies in smaller action spaces makes learning more tractable.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C E XPERIMENTAL D ETAILS
Revised: C E XPERIMENTAL D ETAILS

Original: In this section, we provide more experimental results supplementary to those presented in Sec. 4.2.
Revised: In this section, we provide more experimental results supplementary to those presented in Sec. 4.2.

Original: We also discuss the details of experimental settings of rapid policy transfer and analyze the results.
Revised: We also discuss the details of experimental settings of rapid policy transfer and analyze the results.

Original: C. B ENCHMARKING ON S TAR C RAFT II M ICROMANAGEMENT T ASKS
Revised: C. B ENCHMARKING ON S TAR C RAFT II M ICROMANAGEMENT T ASKS

Original: In Sec.
Revised: In Sec.

Original: 4.2, we show the overall performance of RODE on the SMAC benchmark.
Revised: 4.2, we show the overall performance of RODE on the SMAC benchmark.

Original: In this section, we further show the performance of our method on all maps.
Revised: In this section, we further show the performance of our method on all maps.

Original: All experiments in this section are carried out with 8 random seeds.
Revised: All experiments in this section are carried out with 8 random seeds.

Original: Median win rate, together with 25 - 75% percentiles, is shown.
Revised: Median win rate, together with 25 - 75% percentiles, is shown.

Original: The SMAC benchmark (Samvelyan et al., 2019) contains 14 maps that have been classiﬁed as easy, hard, and super hard.
Revised: The SMAC benchmark (Samvelyan et al., 2019) contains 14 maps that have been classiﬁed as easy, hard, and super hard.

Original: In Fig.
Revised: In Fig.

Original: 9, we compare the performance of RODE with baseline algorithms on all super hard maps.
Revised: 9, we compare the performance of RODE with baseline algorithms on all super hard maps.

Original: We can see that RODE outperforms all the baselines by a large margin, especially on those requiring the most exploration: 3s5z vs 3s6z , corridor , and 6h vs 8z .
Revised: We can see that RODE outperforms all the baselines by a large margin, especially on those requiring the most exploration: 3s5z vs 3s6z , corridor , and 6h vs 8z .

Original: These results demonstrate that RODE can help exploration and solve complex tasks, in line with our expectations of it.
Revised: These results demonstrate that RODE can help exploration and solve complex tasks, in line with our expectations of it.

Original: On hard maps, RODE still has superior performance compared to baselines, as shown in Fig.
Revised: On hard maps, RODE still has superior performance compared to baselines, as shown in Fig.

Original: However, on easy maps (Fig. 11), RODE tends to use more samples to achieve similar performance to baselines.
Revised: However, on easy maps (Fig. 11), RODE tends to use more samples to achieve similar performance to baselines.

Original: We hypothesize that this is because easy maps do not require substantial exploration – ally agents can win by merely engaging in ﬁght and damaging the enemy units for reward.
Revised: We hypothesize that this is because easy maps do not require substantial exploration – ally agents can win by merely engaging in ﬁght and damaging the enemy units for reward.

Original: Under these circumstances, RODE still explores a lot, which prevents it from performing better.
Revised: Under these circumstances, RODE still explores a lot, which prevents it from performing better.

Original: [Figure]
Revised: [Figure]

Original: In summary, RODE establishes a new state of the art on the SMAC benchmark by outperforming all baselines on 9 out of 14 scenarios.
Revised: In summary, RODE establishes a new state of the art on the SMAC benchmark by outperforming all baselines on 9 out of 14 scenarios.

Original: Notably, its superiority is particularly obvious on all of the hard and super hard maps.
Revised: Notably, its superiority is particularly obvious on all of the hard and super hard maps.

Original: C. P OLICY T RANSFER
Revised: C. P OLICY T RANSFER

Original: In this section, we describe the setting of the policy transfer experiment introduced in Sec. 4.3.
Revised: In this section, we describe the setting of the policy transfer experiment introduced in Sec. 4.3.

Original: Policy transfer experiments are based on the assumption that new actions have similar functionalities to some old actions.
Revised: Policy transfer experiments are based on the assumption that new actions have similar functionalities to some old actions.

Original: We train an action encoder on the new task to cluster the actions.
Revised: We train an action encoder on the new task to cluster the actions.

Original: In this way, we can distinguish old actions a i 1 , . .
Revised: In this way, we can distinguish old actions a i 1 , . .

Original: , a i m that have similar effects to a new action a n i .
Revised: , a i m that have similar effects to a new action a n i .

Original: We then usem (cid:80) mj
Revised: We then usem (cid:80) mj

Original: =1 z a ij as the representation for a n i , where z a ij is the representation for a i j on the original task, and we add a n i to role action spaces to which a i 1 , . . .
Revised: =1 z a ij as the representation for a n i , where z a ij is the representation for a i j on the original task, and we add a n i to role action spaces to which a i 1 , . . .

Original: , a i k belong.
Revised: , a i k belong.

Original: For each new action, we repeat these operations.
Revised: For each new action, we repeat these operations.

Original: The number of roles does not change during this process.
Revised: The number of roles does not change during this process.

Original: This process enables the role selector and role policies learned in original task to be rapidly transferred to a similar task.
Revised: This process enables the role selector and role policies learned in original task to be rapidly transferred to a similar task.

Original: To test our idea, we train RODE on the map corridor , where 6 ally Zealots face 24 enemy Zerglings, for 5 million timesteps and rapidly transfer the learned policy to similar maps by increasing the number of agents and enemies.
Revised: To test our idea, we train RODE on the map corridor , where 6 ally Zealots face 24 enemy Zerglings, for 5 million timesteps and rapidly transfer the learned policy to similar maps by increasing the number of agents and enemies.

Original: The number of actions also increases because a new attack action is added for every new enemy unit.
Revised: The number of actions also increases because a new attack action is added for every new enemy unit.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: D R OLE I NTERVAL AND R ECURRENT R OLE S ELECTOR D. R OLE I NTERVAL
Revised: D R OLE I NTERVAL AND R ECURRENT R OLE S ELECTOR D. R OLE I NTERVAL

Original: In our framework, the role selector assigns roles to each agent every c timesteps.
Revised: In our framework, the role selector assigns roles to each agent every c timesteps.

Original: We call c the role interval.
Revised: We call c the role interval.

Original: The role interval decides how frequently the action spaces change and may have a critical inﬂuence on the performance.
Revised: The role interval decides how frequently the action spaces change and may have a critical inﬂuence on the performance.

Original: To get a better sense of this inﬂuence, we change the role interval while keeping other parts unchanged and test RODE on several environments.
Revised: To get a better sense of this inﬂuence, we change the role interval while keeping other parts unchanged and test RODE on several environments.

Original: In Fig.
Revised: In Fig.

Original: 12, we show the results on 2 easy maps ( 10m vs 11m and 1c3s5z ), 1 hard map ( 3s vs 5z ), and 1 super hard map ( 6h vs 8z ).
Revised: 12, we show the results on 2 easy maps ( 10m vs 11m and 1c3s5z ), 1 hard map ( 3s vs 5z ), and 1 super hard map ( 6h vs 8z ).

Original: Generally speaking, the role interval has a signiﬁcant inﬂuence on performance, butor 7 can typically generate satisfactory results.
Revised: Generally speaking, the role interval has a signiﬁcant inﬂuence on performance, butor 7 can typically generate satisfactory results.

Original: In this paper, we use a role interval of 5 on most of the maps.
Revised: In this paper, we use a role interval of 5 on most of the maps.

Original: D. R ECURRENT R OLE S ELECTOR AND F ULLY -C ONNECTED R OLE S ELECTOR
Revised: D. R ECURRENT R OLE S ELECTOR AND F ULLY -C ONNECTED R OLE S ELECTOR

Original: The role selector is supposed to learn a decomposition of observation space.
Revised: The role selector is supposed to learn a decomposition of observation space.

Original: However, we base this component on local action-observation history.
Revised: However, we base this component on local action-observation history.

Original: In this section, we discuss the inﬂuence of using
Revised: In this section, we discuss the inﬂuence of using

Original: GRUs in the role selector.
Revised: GRUs in the role selector.

Original: In Fig.
Revised: In Fig.

Original: 13, we compare RODE to RODE with a fully-connected role selector conditioned on local observations.
Revised: 13, we compare RODE to RODE with a fully-connected role selector conditioned on local observations.

Original: As expected, a recurrent role selector performs better
Revised: As expected, a recurrent role selector performs better

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: than a selector based on local observations.
Revised: than a selector based on local observations.

Original: Even though this conclusion is not surprising, we are interested in those environments where a recurrent role selector can not perform better.
Revised: Even though this conclusion is not surprising, we are interested in those environments where a recurrent role selector can not perform better.

Original: For map 10m vs 11m , together with the results presented in Fig.
Revised: For map 10m vs 11m , together with the results presented in Fig.

Original: [Figure]
Revised: [Figure]

Original: we see that RODE with a role interval of 7 has similar performance to a fully-connected role selector
Revised: we see that RODE with a role interval of 7 has similar performance to a fully-connected role selector

Original: [Figure]
Revised: [Figure]

Original: but underperforms when the role interval is set to other values.
Revised: but underperforms when the role interval is set to other values.

Original: This indicates that the role interval has a sophisticated interaction with exploration, optimization
Revised: This indicates that the role interval has a sophisticated interaction with exploration, optimization

Original: [Figure]
Revised: [Figure]

Original: and learning stability.
Revised: and learning stability.

Original: We leave the study of this as future work.
Revised: We leave the study of this as future work.

Original: In the previous section, we give more examples showing the inﬂuence of the role interval.
Revised: In the previous section, we give more examples showing the inﬂuence of the role interval.

Original: E C LUSTERING A LGORITHMS
Revised: E C LUSTERING A LGORITHMS

Original: In our framework, we use k -means to cluster action representations to determine the role action spaces.
Revised: In our framework, we use k -means to cluster action representations to determine the role action spaces.

Original: K -means algorithm requires predeﬁning the number of clusters k .
Revised: K -means algorithm requires predeﬁning the number of clusters k .

Original: In this section, we ﬁrst investigate the inﬂuence of k values on the performance and then discuss how to determine the number of clusters with minimum prior knowledge.
Revised: In this section, we ﬁrst investigate the inﬂuence of k values on the performance and then discuss how to determine the number of clusters with minimum prior knowledge.

Original: E. I NFLUENCE OF K VALUES
Revised: E. I NFLUENCE OF K VALUES

Original: We test the inﬂuence of k values on 7 maps, including 2 easy maps ( 3s5z and 10m vs 11m ), 2 hard maps ( 2c vs 64zg and MMM2 ), and 3 super hard maps ( 3s5z vs 3s6z , 6h vs 8z , and corridor ).
Revised: We test the inﬂuence of k values on 7 maps, including 2 easy maps ( 3s5z and 10m vs 11m ), 2 hard maps ( 2c vs 64zg and MMM2 ), and 3 super hard maps ( 3s5z vs 3s6z , 6h vs 8z , and corridor ).

Original: In Fig.
Revised: In Fig.

Original: 14, we show the performance of RODE with k -values of 3 , 5 , and 7 .
Revised: 14, we show the performance of RODE with k -values of 3 , 5 , and 7 .

Original: Generally speaking, a k -value of 3 or 5 can work well across these maps.
Revised: Generally speaking, a k -value of 3 or 5 can work well across these maps.

Original: In practice, we set k = 3 on maps with homogeneous enemies and k = 5 on maps with heterogeneous enemies.
Revised: In practice, we set k = 3 on maps with homogeneous enemies and k = 5 on maps with heterogeneous enemies.

Original: For outliers, we add them to all other clusters to avoid action sets consisting of only one action, which we ﬁnd can largely stabilize training across the benchmark.
Revised: For outliers, we add them to all other clusters to avoid action sets consisting of only one action, which we ﬁnd can largely stabilize training across the benchmark.

Original: To further understand how different values of k affect the learning performance, we take one super hard map, 6h vs 8z , as an example.
Revised: To further understand how different values of k affect the learning performance, we take one super hard map, 6h vs 8z , as an example.

Original: When k = 3 or 5 , the clustering method (after adding outliers to all other clusters) gives three role action spaces – Move Eastward & Northward , Attack , and
Revised: When k = 3 or 5 , the clustering method (after adding outliers to all other clusters) gives three role action spaces – Move Eastward & Northward , Attack , and

Original: Move Westward & Southward .
Revised: Move Westward & Southward .

Original: Similar to the case of corridor as discussed in Appendix B, these roles can improve exploration and thus boost the performance.
Revised: Similar to the case of corridor as discussed in Appendix B, these roles can improve exploration and thus boost the performance.

Original: However, when k = 7 , all move actions become outliers.
Revised: However, when k = 7 , all move actions become outliers.

Original: The result is that most role action spaces become the original full action space.
Revised: The result is that most role action spaces become the original full action space.

Original: Such a decomposition over action space does not reduce the search space, and thus the performance is close to that of QMIX.
Revised: Such a decomposition over action space does not reduce the search space, and thus the performance is close to that of QMIX.

Original: [Figure]
Revised: [Figure]

Original: To avoid such complicated interactions between the selection of k values and the way of dealing with outliers
Revised: To avoid such complicated interactions between the selection of k values and the way of dealing with outliers

Original: [Figure]
Revised: [Figure]

Original: we propose to use X -means when clustering action representations
Revised: we propose to use X -means when clustering action representations

Original: [Figure]
Revised: [Figure]

Original: as we discuss in detail in the next section
Revised: as we discuss in detail in the next section

Original: [Figure]
Revised: [Figure]

Original: E. O THER CLUSTERING ALGORITHMS
Revised: E. O THER CLUSTERING ALGORITHMS

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we investigate how to avoid predeﬁning k values.
Revised: we investigate how to avoid predeﬁning k values.

Original: X-means (Pelleg et al., 2000) improves k -means by automatically determining the number of clusters.
Revised: X-means (Pelleg et al., 2000) improves k -means by automatically determining the number of clusters.

Original: We use minimum noiseless description length as the splitting criterion for X-means and test
Revised: We use minimum noiseless description length as the splitting criterion for X-means and test

Original: RODE using X-means on all the 14 maps.
Revised: RODE using X-means on all the 14 maps.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: We see that the results are comparable to k -means with predeﬁned k values.
Revised: We see that the results are comparable to k -means with predeﬁned k values.

Original: We thus recommend X-means when facing a new task, while for tasks that users know the number of roles, k -means can be more accurate.
Revised: We thus recommend X-means when facing a new task, while for tasks that users know the number of roles, k -means can be more accurate.

Original: Additionally, we can use density-based clustering algorithms to avoid the predeﬁnition of k , like
Revised: Additionally, we can use density-based clustering algorithms to avoid the predeﬁnition of k , like

Original: DBSCAN (Ester et al., 1996).
Revised: DBSCAN (Ester et al., 1996).

Original: In practice, we ﬁnd that DBSCAN can generate similar role action spaces to X-means.
Revised: In practice, we ﬁnd that DBSCAN can generate similar role action spaces to X-means.

Original: Since the clustering algorithm is used only to split the action spaces, RODE using DBSCAN clustering is expected to have similar learning performance to X-means.
Revised: Since the clustering algorithm is used only to split the action spaces, RODE using DBSCAN clustering is expected to have similar learning performance to X-means.

Original: [Figure]
Revised: [Figure]

Original: The separation of the role selector and role policies is important to make the average of action representations well represent the roles.
Revised: The separation of the role selector and role policies is important to make the average of action representations well represent the roles.

Original: F R OLE R EPRESENTATIONS
Revised: F R OLE R EPRESENTATIONS

Original: In our framework, we use the average representation of actions in the role action space to represent a role.
Revised: In our framework, we use the average representation of actions in the role action space to represent a role.

Original: One concern is that whether such representations can well represent the roles, given that both role policies and the role selector are based on action representations.
Revised: One concern is that whether such representations can well represent the roles, given that both role policies and the role selector are based on action representations.

Original: We ﬁnd that the separation of
Revised: We ﬁnd that the separation of

Original: Q networks for the role selector and role policies (including the GRUs for processing local observations and the network f β (or f ρ j )) is important to ensure such role representations have sufﬁcient expressivity.
Revised: Q networks for the role selector and role policies (including the GRUs for processing local observations and the network f β (or f ρ j )) is important to ensure such role representations have sufﬁcient expressivity.

Original: If we share the GRUs and f β (or f ρ j ), RODE struggles even on easy maps (Fig. 16).
Revised: If we share the GRUs and f β (or f ρ j ), RODE struggles even on easy maps (Fig. 16).

Original: G R ELATED W ORKS
Revised: G R ELATED W ORKS

Original: Role-Based Learning Our approach is an attempt to integrate roles into deep multi-agent reinforcement learning.
Revised: Role-Based Learning Our approach is an attempt to integrate roles into deep multi-agent reinforcement learning.

Original: Many natural systems feature emergent roles, such as ants (Gordon, 1996), bees (Jeanson et al., 2005), and humans (Butler, 2012).
Revised: Many natural systems feature emergent roles, such as ants (Gordon, 1996), bees (Jeanson et al., 2005), and humans (Butler, 2012).

Original: In these systems, roles are closely related to the division of labor and is critical to labor efﬁciency.
Revised: In these systems, roles are closely related to the division of labor and is critical to labor efﬁciency.

Original: These beneﬁts inspired multi-agent system designers, who try to reduce the design complexity by decomposing the task and specializing agents with the same role to certain sub-tasks (Wooldridge et al., 2000; Omicini, 2000; Padgham & Winikoff, 2002; Pav ´ on & G ´ omez-Sanz, 2003; Cossentino et al., 2005; Zhu & Zhou, 2008;
Revised: These beneﬁts inspired multi-agent system designers, who try to reduce the design complexity by decomposing the task and specializing agents with the same role to certain sub-tasks (Wooldridge et al., 2000; Omicini, 2000; Padgham & Winikoff, 2002; Pav ´ on & G ´ omez-Sanz, 2003; Cossentino et al., 2005; Zhu & Zhou, 2008;

Original: Spanoudakis & Moraitis, 2010; DeLoach & Garcia-Ojeda, 2010; Bonjean et al., 2014).
Revised: Spanoudakis & Moraitis, 2010; DeLoach & Garcia-Ojeda, 2010; Bonjean et al., 2014).

Original: However, roles and the associated responsibilities (or subtask-speciﬁc rewards (Sun et al., 2020)) are predeﬁned using prior knowledge in these systems (Lhaksmana et al., 2018).
Revised: However, roles and the associated responsibilities (or subtask-speciﬁc rewards (Sun et al., 2020)) are predeﬁned using prior knowledge in these systems (Lhaksmana et al., 2018).

Original: Although pre-deﬁnition can be efﬁcient in tasks with a clear structure, such as software engineering (Bresciani et al., 2004), it hurts generalization and requires prior knowledge that may not be available in practice.
Revised: Although pre-deﬁnition can be efﬁcient in tasks with a clear structure, such as software engineering (Bresciani et al., 2004), it hurts generalization and requires prior knowledge that may not be available in practice.

Original: To solve this problem, Wilson et al.
Revised: To solve this problem, Wilson et al.

Original: (2010) use Bayesian inference to learn a set of roles and Wang et al.
Revised: (2010) use Bayesian inference to learn a set of roles and Wang et al.

Original: (2020c) design a specialization objective to encourage the emergence of roles.
Revised: (2020c) design a specialization objective to encourage the emergence of roles.

Original: However, these methods search the optimal task decomposition in the full state-action space, resulting in inefﬁcient learning in hard-exploration tasks.
Revised: However, these methods search the optimal task decomposition in the full state-action space, resulting in inefﬁcient learning in hard-exploration tasks.

Original: RODE avoids this shortcoming by ﬁrst decomposing joint action spaces according to action effects, which makes role discovery much easier.
Revised: RODE avoids this shortcoming by ﬁrst decomposing joint action spaces according to action effects, which makes role discovery much easier.

Original: Multi-Agent Reinforcement Learning (MARL)
Revised: Multi-Agent Reinforcement Learning (MARL)

Original: Many challenging tasks (Vinyals et al., 2019;
Revised: Many challenging tasks (Vinyals et al., 2019;

Original: Berner et al., 2019; Samvelyan et al., 2019; Jaderberg et al., 2019) can be modelled as multi-agent learning problems.
Revised: Berner et al., 2019; Samvelyan et al., 2019; Jaderberg et al., 2019) can be modelled as multi-agent learning problems.

Original: Many interesting real-world phenomena, including the emergence of tool usage (Baker et al., 2020), communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2017; Das et al., 2019), social inﬂuence (Jaques et al., 2019), and inequity aversion (Hughes et al., 2018) can also be better explained from a MARL perspective.
Revised: Many interesting real-world phenomena, including the emergence of tool usage (Baker et al., 2020), communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2017; Das et al., 2019), social inﬂuence (Jaques et al., 2019), and inequity aversion (Hughes et al., 2018) can also be better explained from a MARL perspective.

Original: Learning control policies for multi-agent systems remains a challenge.
Revised: Learning control policies for multi-agent systems remains a challenge.

Original: Centralized learning on joint action space can avoid non-stationarity during learning and may better coordinate the learning process of individual agents.
Revised: Centralized learning on joint action space can avoid non-stationarity during learning and may better coordinate the learning process of individual agents.

Original: However, these methods typically can not scale well to large-scale problems, as the joint action space grows exponentially with the number of agents.
Revised: However, these methods typically can not scale well to large-scale problems, as the joint action space grows exponentially with the number of agents.

Original: The coordination graph (Guestrin et al., 2002b;a; B¨ohmer et al., 2020) is a promising approach to achieve scalable centralized learning.
Revised: The coordination graph (Guestrin et al., 2002b;a; B¨ohmer et al., 2020) is a promising approach to achieve scalable centralized learning.

Original: It reduces the learning complexity by exploiting coordination independencies between agents.
Revised: It reduces the learning complexity by exploiting coordination independencies between agents.

Original: Zhang & Lesser (2011); Kok & Vlassis (2006) also utilize the sparsity and locality of coordination.
Revised: Zhang & Lesser (2011); Kok & Vlassis (2006) also utilize the sparsity and locality of coordination.

Original: These methods require the dependencies between agents to be pre-supplied.
Revised: These methods require the dependencies between agents to be pre-supplied.

Original: Value function decomposition methods avoid using such prior knowledge by directly learning centralized but factored global Q -functions.
Revised: Value function decomposition methods avoid using such prior knowledge by directly learning centralized but factored global Q -functions.

Original: They implicitly represent the coordination dependencies among agents by the decomposable structure (Sunehag et al., 2018;
Revised: They implicitly represent the coordination dependencies among agents by the decomposable structure (Sunehag et al., 2018;

Original: Rashid et al., 2018; Son et al., 2019;
Revised: Rashid et al., 2018; Son et al., 2019;

Original: Wang et al., 2020e;b; Rashid et al., 2020a; Iqbal et al., 2020).
Revised: Wang et al., 2020e;b; Rashid et al., 2020a; Iqbal et al., 2020).

Original: Multi-agent policy gradient algorithms enjoy stable theoretical convergence properties compared to value-based methods (Gupta et al., 2017; Wang et al., 2020a) and hold the promise to extend
Revised: Multi-agent policy gradient algorithms enjoy stable theoretical convergence properties compared to value-based methods (Gupta et al., 2017; Wang et al., 2020a) and hold the promise to extend

Original: MARL to continuous control problems.
Revised: MARL to continuous control problems.

Original: COMA (Foerster et al., 2018) and MADDPG (Lowe et al., 2017) propose the paradigm of centralized critic with decentralized actors (CCDA).
Revised: COMA (Foerster et al., 2018) and MADDPG (Lowe et al., 2017) propose the paradigm of centralized critic with decentralized actors (CCDA).

Original: PR2 (Wen et al., 2019) and MAAC (Iqbal & Sha, 2019) extend the CCDA paradigm by introducing the mechanism of recursive reasoning and attention, respectively.
Revised: PR2 (Wen et al., 2019) and MAAC (Iqbal & Sha, 2019) extend the CCDA paradigm by introducing the mechanism of recursive reasoning and attention, respectively.

Original: DOP (Wang et al., 2020g) solves the centralizeddecentralized mismatch problem in the CCDA paradigm and enables deterministic multi-agent policy gradient methods to use off-policy data for learning.
Revised: DOP (Wang et al., 2020g) solves the centralizeddecentralized mismatch problem in the CCDA paradigm and enables deterministic multi-agent policy gradient methods to use off-policy data for learning.

Original: Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018;
Revised: Another line of research focuses on fully decentralized actor-critic learning (Macua et al., 2017; Zhang et al., 2018;

Original: Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).
Revised: Yang et al., 2018; Cassano et al., 2018; Suttle et al., 2019; Zhang & Zavlanos, 2019).