Original: L EARNING S TRUCTURED C OMMUNICATION FOR M ULTI - AGENT R EINFORCEMENT L EARNING
Revised: L EARNING S TRUCTURED C OMMUNICATION FOR M ULTI - AGENT R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning to cooperate is crucial for many practical large-scale multi-agent applications.
Revised: Learning to cooperate is crucial for many practical large-scale multi-agent applications.

Original: In this work, we consider an important collaborative task, in which agents learn to efﬁciently communicate with each other under a multi-agent reinforcement learning (MARL) setting.
Revised: In this work, we consider an important collaborative task, in which agents learn to efﬁciently communicate with each other under a multi-agent reinforcement learning (MARL) setting.

Original: Despite the fact that there has been a number of existing works along this line, achieving global cooperation at scale is still challenging.
Revised: Despite the fact that there has been a number of existing works along this line, achieving global cooperation at scale is still challenging.

Original: In particular, most of the existing algorithms suffer from issues such as scalability and high communication complexity, in the sense that when the agent population is large, it can be difﬁcult to extract effective information for highperformance MARL.
Revised: In particular, most of the existing algorithms suffer from issues such as scalability and high communication complexity, in the sense that when the agent population is large, it can be difﬁcult to extract effective information for highperformance MARL.

Original: In contrast, the proposed algorithmic framework, termed
Revised: In contrast, the proposed algorithmic framework, termed

Original: Learning Structured Communication (LSC), is not only scalable but also learns efﬁciently.
Revised: Learning Structured Communication (LSC), is not only scalable but also learns efﬁciently.

Original: The key idea is to allow the agents to dynamically learn a hierarchical communication structure, while under such a structure the graph neural network (GNN) is used to efﬁciently extract useful information to be exchanged between the neighboring agents.
Revised: The key idea is to allow the agents to dynamically learn a hierarchical communication structure, while under such a structure the graph neural network (GNN) is used to efﬁciently extract useful information to be exchanged between the neighboring agents.

Original: A number of new techniques are proposed to tightly integrate the communication structure learning, GNN optimization and MARL tasks.
Revised: A number of new techniques are proposed to tightly integrate the communication structure learning, GNN optimization and MARL tasks.

Original: Extensive experiments are performed to demonstrate that, the proposed
Revised: Extensive experiments are performed to demonstrate that, the proposed

Original: LSC framework enjoys high communication efﬁciency, scalability, and global cooperation capability.
Revised: LSC framework enjoys high communication efﬁciency, scalability, and global cooperation capability.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has achieved remarkable success in solving single-agent sequential decision problems under interactive and complicated environments, such as games (Mnih et al., 2015;
Revised: Reinforcement learning (RL) has achieved remarkable success in solving single-agent sequential decision problems under interactive and complicated environments, such as games (Mnih et al., 2015;

Original: Silver et al., 2016) and robotics (Lillicrap et al., 2016).
Revised: Silver et al., 2016) and robotics (Lillicrap et al., 2016).

Original: In many real world applications such as intelligent transportation systems (Adler & Blue, 2002) and unmanned systems(Semsar-Kazerooni & Khorasani, 2009), not only one, but usually a large number of agents are involved in the learning tasks.
Revised: In many real world applications such as intelligent transportation systems (Adler & Blue, 2002) and unmanned systems(Semsar-Kazerooni & Khorasani, 2009), not only one, but usually a large number of agents are involved in the learning tasks.

Original: Such a setting naturally leads to the popular multi-agent reinforcement learning (MARL) problems, where the key research challenges include how to design scalable and efﬁcient learning schemes under an unstationary environment (caused by partial observation and/or the dynamics of other agents’ policies), with large and/or dynamic problem dimension, and complicated and uncertain relationship between agents.
Revised: Such a setting naturally leads to the popular multi-agent reinforcement learning (MARL) problems, where the key research challenges include how to design scalable and efﬁcient learning schemes under an unstationary environment (caused by partial observation and/or the dynamics of other agents’ policies), with large and/or dynamic problem dimension, and complicated and uncertain relationship between agents.

Original: Learning to communicate among agents has been regarded as an effective manner to strengthen the inter-agent collaboration and ultimately improve the quality of policies learned by MARL.
Revised: Learning to communicate among agents has been regarded as an effective manner to strengthen the inter-agent collaboration and ultimately improve the quality of policies learned by MARL.

Original: Various communication-based MARL algorithms have been devised recently, e.g., DIAL (Foerster et al., 2016), CommNet (Sukhbaatar et al., 2016), ATOC (Jiang & Lu, 2018), IC3Net (Singh et al., 2019) and TarMAC (Das et al., 2019).
Revised: Various communication-based MARL algorithms have been devised recently, e.g., DIAL (Foerster et al., 2016), CommNet (Sukhbaatar et al., 2016), ATOC (Jiang & Lu, 2018), IC3Net (Singh et al., 2019) and TarMAC (Das et al., 2019).

Original: These schemes aim to improve the inter-agent collaboration by learning communication strategy to exchange information between agents.
Revised: These schemes aim to improve the inter-agent collaboration by learning communication strategy to exchange information between agents.

Original: However, there are still two bottlenecks unresolved, especially when faced a large number of agents.
Revised: However, there are still two bottlenecks unresolved, especially when faced a large number of agents.

Original: One bottleneck lies in that achieving effective communication and global collaboration is difﬁcult with limited resources, such as narrow communication bandwidth and energy.
Revised: One bottleneck lies in that achieving effective communication and global collaboration is difﬁcult with limited resources, such as narrow communication bandwidth and energy.

Original: In particular, DIAL and TarMAC require each agent to communicate with all the other agents, i.e., a fully-connected communication network (Figure 1(a)), which is not feasible for large scale scenarios with geographically apart agents.
Revised: In particular, DIAL and TarMAC require each agent to communicate with all the other agents, i.e., a fully-connected communication network (Figure 1(a)), which is not feasible for large scale scenarios with geographically apart agents.

Original: CommNet and IC3 assume a star network (Figure 1(b)) with a central node coordinating the global collaboration of agents, which again does not allow large scale scenarios with long range communications.
Revised: CommNet and IC3 assume a star network (Figure 1(b)) with a central node coordinating the global collaboration of agents, which again does not allow large scale scenarios with long range communications.

Original: ATOC introduces an interesting attention scheme to build a tree
Revised: ATOC introduces an interesting attention scheme to build a tree

Original: [Figure]
Revised: [Figure]

Original: communication network (Figure 1(c)).
Revised: communication network (Figure 1(c)).

Original: While the tree network can be scaled, global collaboration has to be realized through inefﬁcient multi-hop and sequential communications.
Revised: While the tree network can be scaled, global collaboration has to be realized through inefﬁcient multi-hop and sequential communications.

Original: In a word, improper communication topologies will limit the cooperation ability in large scale scenarios.
Revised: In a word, improper communication topologies will limit the cooperation ability in large scale scenarios.

Original: Another bottleneck is the difﬁculty of extracting essential information to exchange between agents for achieving high-performance MARL, especially when the number of agents grows.
Revised: Another bottleneck is the difﬁculty of extracting essential information to exchange between agents for achieving high-performance MARL, especially when the number of agents grows.

Original: Most of the existing works simply concatenate, take the mean or use the LSTM to extract information to be exchanged.
Revised: Most of the existing works simply concatenate, take the mean or use the LSTM to extract information to be exchanged.

Original: First two lack in considering the inter-relationship between agents, and LSTM assumes that there is a ﬁxed sequence of message passing between agents, that is, the relationship between agents is predeﬁned.
Revised: First two lack in considering the inter-relationship between agents, and LSTM assumes that there is a ﬁxed sequence of message passing between agents, that is, the relationship between agents is predeﬁned.

Original: Recently, TarMAC utilized an attention scheme to aggregate messages by considering the relationship from each agent to all others.
Revised: Recently, TarMAC utilized an attention scheme to aggregate messages by considering the relationship from each agent to all others.

Original: However, the improper communication topology still hinders the information extraction.
Revised: However, the improper communication topology still hinders the information extraction.

Original: The communication structure needs to be jointly designed with the information extraction scheme to achieve further improved learning performance.
Revised: The communication structure needs to be jointly designed with the information extraction scheme to achieve further improved learning performance.

Original: [Figure]
Revised: [Figure]

Original: E
Revised: E

Original: F
Revised: F

Original: [Figure]
Revised: [Figure]

Original: G
Revised: G

Original: H
Revised: H

Original: [Figure]
Revised: [Figure]

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: C:
Revised: C:

Original: E:
Revised: E:

Original: F:2 D:
Revised: F:2 D:

Original: G:
Revised: G:

Original: H:
Revised: H:

Original: A:1 B:
Revised: A:1 B:

Original: [Figure]
Revised: [Figure]

Original: E
Revised: E

Original: F D
Revised: F D

Original: G
Revised: G

Original: H
Revised: H

Original: A
Revised: A

Original: [Figure]
Revised: [Figure]

Original: To address the above two issues, we propose a novel structured communication-based algorithm, called learning structured communication ( LSC ).
Revised: To address the above two issues, we propose a novel structured communication-based algorithm, called learning structured communication ( LSC ).

Original: Our LSC combines a structured communication network module and a communication-based policy module, which aims to establish a scalable hierarchically structured network and information exchange scheme for large scale MARL.
Revised: Our LSC combines a structured communication network module and a communication-based policy module, which aims to establish a scalable hierarchically structured network and information exchange scheme for large scale MARL.

Original: In particular, a hierarchically structured communication network (Figure 1(d)) is dynamically learned based on local partial observations of agents.
Revised: In particular, a hierarchically structured communication network (Figure 1(d)) is dynamically learned based on local partial observations of agents.

Original: In the hierarchically structured network, all agents are grouped into clusters, where global collaboration can be achieved via intra-group and inter-group communications.
Revised: In the hierarchically structured network, all agents are grouped into clusters, where global collaboration can be achieved via intra-group and inter-group communications.

Original: In contrast to the other three types in Figure 1, the proposed hierarchical communication network is more ﬂexible and scalable, with fewer resources needed to achieve long-range and global collaboration.
Revised: In contrast to the other three types in Figure 1, the proposed hierarchical communication network is more ﬂexible and scalable, with fewer resources needed to achieve long-range and global collaboration.

Original: The procedure to establish such a hierarchically structured communication network is shown in
Revised: The procedure to establish such a hierarchically structured communication network is shown in

Original: Figure 2.
Revised: Figure 2.

Original: To better utilize the relationship between agents given the hierarchically structured communication network and obtain more effective information extraction, graph neural network (GNN) (Scarselli et al., 2008) is employed.
Revised: To better utilize the relationship between agents given the hierarchically structured communication network and obtain more effective information extraction, graph neural network (GNN) (Scarselli et al., 2008) is employed.

Original: In GNN, each communication step involves information embedding and aggregation.
Revised: In GNN, each communication step involves information embedding and aggregation.

Original: Beneﬁting from the unordered aggregation power and the dynamic graph adaptability of GNN, the proposed LSC algorithm can extract valuable information effectively.
Revised: Beneﬁting from the unordered aggregation power and the dynamic graph adaptability of GNN, the proposed LSC algorithm can extract valuable information effectively.

Original: The GNN-based information extraction procedure is depicted in Figure 3.
Revised: The GNN-based information extraction procedure is depicted in Figure 3.

Original: Normal Nodes
Revised: Normal Nodes

Original: Central Nodes C
Revised: Central Nodes C

Original: E
Revised: E

Original: F D
Revised: F D

Original: G
Revised: G

Original: H
Revised: H

Original: A
Revised: A

Original: B
Revised: B

Original: C
Revised: C

Original: E
Revised: E

Original: F D
Revised: F D

Original: G
Revised: G

Original: H
Revised: H

Original: A
Revised: A

Original: B
Revised: B

Original: C
Revised: C

Original: E
Revised: E

Original: F D
Revised: F D

Original: G
Revised: G

Original: H
Revised: H

Original: A
Revised: A

Original: B
Revised: B

Original: Local infos
Revised: Local infos

Original: Cluster infos
Revised: Cluster infos

Original: Global infos Intra-Group Aggregation
Revised: Global infos Intra-Group Aggregation

Original: Inter-Group Sharing
Revised: Inter-Group Sharing

Original: Intra-Group Sharing
Revised: Intra-Group Sharing

Original: This paper is devoted to the learning of communication structure among agents.
Revised: This paper is devoted to the learning of communication structure among agents.

Original: To our knowledge, this is the ﬁrst work of hierarchical structured learning to communication for MARL.
Revised: To our knowledge, this is the ﬁrst work of hierarchical structured learning to communication for MARL.

Original: It allows to learn communication structure adaptively instead of using predeﬁned forms.
Revised: It allows to learn communication structure adaptively instead of using predeﬁned forms.

Original: Speciﬁcally: i) To improve scalability for a large number of agents, a hierarchical structure is devised that divides the agents into higher-level central agents and sub-level normal ones.
Revised: Speciﬁcally: i) To improve scalability for a large number of agents, a hierarchical structure is devised that divides the agents into higher-level central agents and sub-level normal ones.

Original: As such, the communication network is sparsiﬁed.
Revised: As such, the communication network is sparsiﬁed.

Original: While it still allows for more effective global cooperation via message passing among the central agents, compared with the star/tree structures.
Revised: While it still allows for more effective global cooperation via message passing among the central agents, compared with the star/tree structures.

Original: ii)
Revised: ii)

Original: For effective communication and global cooperation, the message representation learning is deeply integrated into the information aggregating and permeating through the network, via graph neural network (GNN), which is a natural combination with the hierarchical communication structure.
Revised: For effective communication and global cooperation, the message representation learning is deeply integrated into the information aggregating and permeating through the network, via graph neural network (GNN), which is a natural combination with the hierarchical communication structure.

Original: iii) Extensive experiments on both MAgent and StarCraft2 show our approach achieves state-of-theart scalability and effectiveness on large-scale MARL problems.
Revised: iii) Extensive experiments on both MAgent and StarCraft2 show our approach achieves state-of-theart scalability and effectiveness on large-scale MARL problems.

Original:  R ELATED W ORK AND P RELIMINARIES
Revised:  R ELATED W ORK AND P RELIMINARIES

Original: Many multi-agent reinforcement learning algorithms without communication in the inference procedure have experienced fast development.
Revised: Many multi-agent reinforcement learning algorithms without communication in the inference procedure have experienced fast development.

Original: Recent works like MADDPG (Lowe et al., 2017),
Revised: Recent works like MADDPG (Lowe et al., 2017),

Original: QMIX (Rashid et al., 2018), COMA (Foerster et al., 2018) and MAAC (Iqbal & Sha, 2019) adopt a centralized training and decentralized implementing framework.
Revised: QMIX (Rashid et al., 2018), COMA (Foerster et al., 2018) and MAAC (Iqbal & Sha, 2019) adopt a centralized training and decentralized implementing framework.

Original: All agents’ local observations and actions are considered to improve the learning stability.
Revised: All agents’ local observations and actions are considered to improve the learning stability.

Original: These algorithms are generally not suitable for large-scale case due to explosive growing number of agents.
Revised: These algorithms are generally not suitable for large-scale case due to explosive growing number of agents.

Original: Communication-based MARL algorithms have been showed effective for large-scale agent cooperation.
Revised: Communication-based MARL algorithms have been showed effective for large-scale agent cooperation.

Original: Earlier works assume that all agents need to communicate with each other.
Revised: Earlier works assume that all agents need to communicate with each other.

Original: DIAL (Foerster et al., 2016) learns to communication through back-propagating all other agents’ gradients to the message generator network.
Revised: DIAL (Foerster et al., 2016) learns to communication through back-propagating all other agents’ gradients to the message generator network.

Original: Similarly, CommNet (Sukhbaatar et al., 2016) sends all agents’ hidden states to the shared communication channel and further learns the message based on the average of all other hidden states.
Revised: Similarly, CommNet (Sukhbaatar et al., 2016) sends all agents’ hidden states to the shared communication channel and further learns the message based on the average of all other hidden states.

Original: MFRL (Yang et al., 2018) approximates the inﬂuence of other agents by averaging the actions of surrounding neighbor agents, which could mitigate the dimensional disaster for large-scale cases.
Revised: MFRL (Yang et al., 2018) approximates the inﬂuence of other agents by averaging the actions of surrounding neighbor agents, which could mitigate the dimensional disaster for large-scale cases.

Original: However, this can be considered as a predeﬁned communication pattern, which is unable to adapt to complex large-scale scenarios.
Revised: However, this can be considered as a predeﬁned communication pattern, which is unable to adapt to complex large-scale scenarios.

Original: Communication between all agents will lead to high communication complexity and difﬁculty of useful information extraction.
Revised: Communication between all agents will lead to high communication complexity and difﬁculty of useful information extraction.

Original: DGN (Jiang et al., 2018) employs graph convolution network (GCN) to extract relationships between agents which could result in better collaboration.
Revised: DGN (Jiang et al., 2018) employs graph convolution network (GCN) to extract relationships between agents which could result in better collaboration.

Original: However, it considers all agents equivalently and assumes the communication of each agent has to involve all neighbor agents which limits to adapt to more practical bandwidth-limited environments.
Revised: However, it considers all agents equivalently and assumes the communication of each agent has to involve all neighbor agents which limits to adapt to more practical bandwidth-limited environments.

Original: IC3Net (Singh et al., 2019) uses a communication gate to decide whether to communicate with the center, but adopt the same star structure like CommNet which requires high bandwidth and can hard to extract valuable information with only one center.
Revised: IC3Net (Singh et al., 2019) uses a communication gate to decide whether to communicate with the center, but adopt the same star structure like CommNet which requires high bandwidth and can hard to extract valuable information with only one center.

Original: ATOC (Jiang & Lu, 2018) and TarMAC (Das et al., 2019) introduce the attention mechanism to determine when to communicate and whom to communicate with, respectively.
Revised: ATOC (Jiang & Lu, 2018) and TarMAC (Das et al., 2019) introduce the attention mechanism to determine when to communicate and whom to communicate with, respectively.

Original: TarMAC focuses more on message aggregation rather than the communication structure.
Revised: TarMAC focuses more on message aggregation rather than the communication structure.

Original: SchedNet (Kim et al., 2019) aims to learn a weight-based scheduler to determine the communication sequence and priority.
Revised: SchedNet (Kim et al., 2019) aims to learn a weight-based scheduler to determine the communication sequence and priority.

Original: From the perspective of employing GNN into MARL, MAGNet (Malysheva et al., 2018) that utilizes a relevance graph representation of the environment and a message passing mechanism to help agents learning.
Revised: From the perspective of employing GNN into MARL, MAGNet (Malysheva et al., 2018) that utilizes a relevance graph representation of the environment and a message passing mechanism to help agents learning.

Original: However, it requires heuristic rules to establish the graph which is hard to achieve in complex environments.
Revised: However, it requires heuristic rules to establish the graph which is hard to achieve in complex environments.

Original: RFM (Tacchetti et al., 2019) use graph to represent the relationship between different entities, aiming to provide interpretable representations.
Revised: RFM (Tacchetti et al., 2019) use graph to represent the relationship between different entities, aiming to provide interpretable representations.

Original: Before the main method, we introduce some preliminaries to facilitate the presentation.
Revised: Before the main method, we introduce some preliminaries to facilitate the presentation.

Original: Partial Observable Stochastic Games.
Revised: Partial Observable Stochastic Games.

Original: In stochastic games, agents learn policies by maximizing their cumulative rewards through interacting with the environment and other agents.
Revised: In stochastic games, agents learn policies by maximizing their cumulative rewards through interacting with the environment and other agents.

Original: The partial observable stochastic games (POSG) can be characterized as a tuple (cid:10) I , S , b 0 , A , O , P , P e , R (cid:11) where
Revised: The partial observable stochastic games (POSG) can be characterized as a tuple (cid:10) I , S , b 0 , A , O , P , P e , R (cid:11) where

Original: I denotes the set of agents indexed from 1 to n ; S denotes the ﬁnite set of states; b 0 represents the initial state distribution and A denotes the set of joint actions.
Revised: I denotes the set of agents indexed from 1 to n ; S denotes the ﬁnite set of states; b 0 represents the initial state distribution and A denotes the set of joint actions.

Original: A i is the action space of agent i , a = (cid:104) a 1 , · , a n (cid:105) denotes a joint action; O denotes the joint observations and O i is the observation space for agent i , o = (cid:104)
Revised: A i is the action space of agent i , a = (cid:104) a 1 , · , a n (cid:105) denotes a joint action; O denotes the joint observations and O i is the observation space for agent i , o = (cid:104)

Original: o 1 , · , o n (cid:105) denotes a joint observation; P denotes the Markovian transition distribution with P (cid:0) ˜ s, o (cid:12)(cid:12) s, a (cid:1) as the probability of state s transit to ˜ s and result o after taking action a .
Revised: o 1 , · , o n (cid:105) denotes a joint observation; P denotes the Markovian transition distribution with P (cid:0) ˜ s, o (cid:12)(cid:12) s, a (cid:1) as the probability of state s transit to ˜ s and result o after taking action a .

Original: P e ( o | s ) is the Markovian observation emission probability function.
Revised: P e ( o | s ) is the Markovian observation emission probability function.

Original: R : S ×
Revised: R : S ×

Original: A → R n means the reward function for agents.
Revised: A → R n means the reward function for agents.

Original: The overall task of the MARL problem can be solved by properly objective function modeling, which also indicates the relationship among agents, e.g., cooperation, competition or mixed.
Revised: The overall task of the MARL problem can be solved by properly objective function modeling, which also indicates the relationship among agents, e.g., cooperation, competition or mixed.

Original: Graph Neural Network.
Revised: Graph Neural Network.

Original: Graph neural network (GNN) (Scarselli et al., 2008) is a deep embedding framework to handle graph-based data on a graph G =
Revised: Graph neural network (GNN) (Scarselli et al., 2008) is a deep embedding framework to handle graph-based data on a graph G =

Original: ( V , E ) .
Revised: ( V , E ) .

Original: v i denotes the node feature vector for node v i ∈ V (for N v nodes), e k denotes the edge feature vector for edge e k ∈ E (for N e edges) with r k , s k be the receiver and sender of edge
Revised: v i denotes the node feature vector for node v i ∈ V (for N v nodes), e k denotes the edge feature vector for edge e k ∈ E (for N e edges) with r k , s k be the receiver and sender of edge

Original: e k respectively.
Revised: e k respectively.

Original: The vector u denotes the global feature.
Revised: The vector u denotes the global feature.

Original: The graph network framework in (Battaglia et al., 2018) is employed, which divides computation on graph data to several blocks to gain ﬂexible processing ability.
Revised: The graph network framework in (Battaglia et al., 2018) is employed, which divides computation on graph data to several blocks to gain ﬂexible processing ability.

Original: Each block introduces the aggregation and embedding functions to handle graph data.
Revised: Each block introduces the aggregation and embedding functions to handle graph data.

Original: There are many variants of GNN, like messagepassing neural network (Gilmer et al., 2017) and non local neural networks (Wang et al., 2018).
Revised: There are many variants of GNN, like messagepassing neural network (Gilmer et al., 2017) and non local neural networks (Wang et al., 2018).

Original: By treating every agent as a node and each communication message exchanging as the edge in a graph, the observations and messages as the attributes of nodes and edges, respectively.
Revised: By treating every agent as a node and each communication message exchanging as the edge in a graph, the observations and messages as the attributes of nodes and edges, respectively.

Original: The whole communication process can be formulated to a graph neural network.
Revised: The whole communication process can be formulated to a graph neural network.

Original: The relationships among agents can be effectively extracted to enable efﬁcient communication message learning.
Revised: The relationships among agents can be effectively extracted to enable efﬁcient communication message learning.

Original: Independent Deep Q-Learning.
Revised: Independent Deep Q-Learning.

Original: Deep Q -Network (DQN) (Mnih et al., 2015) is popular in deep reinforcement learning, which is one of the few RL algorithms applicable for large-scale MARL.
Revised: Deep Q -Network (DQN) (Mnih et al., 2015) is popular in deep reinforcement learning, which is one of the few RL algorithms applicable for large-scale MARL.

Original: In each step, each agent observes state s and takes an action a based on policy π .
Revised: In each step, each agent observes state s and takes an action a based on policy π .

Original: It receives reward r and next state ˜ s from environment.
Revised: It receives reward r and next state ˜ s from environment.

Original: To maximize the cumulative reward R = (cid:80) t r t ,
Revised: To maximize the cumulative reward R = (cid:80) t r t ,

Original: DQN learns the action-value function Q π ( s, a ) =
Revised: DQN learns the action-value function Q π ( s, a ) =

Original: E s ∼P ,a ∼ π ( s )
Revised: E s ∼P ,a ∼ π ( s )

Original: [ R t | s t = s, a t = a ] by minimizing
Revised: [ R t | s t = s, a t = a ] by minimizing

Original: L ( θ ) =
Revised: L ( θ ) =

Original: E s,a,r, ˜ s [˜ y − Q ( s, a ; θ )] , where ˜ y
Revised: E s,a,r, ˜ s [˜ y − Q ( s, a ; θ )] , where ˜ y

Original: =
Revised: =

Original: r
Revised: r

Original: + γ max ˜ a Q (˜ s, ˜ a ; θ ) .
Revised: + γ max ˜ a Q (˜ s, ˜ a ; θ ) .

Original: The agent follows (cid:15) -greedy policy, that is, selects the action that maximizes the Q -value with probability 1 - (cid:15) or randomly.
Revised: The agent follows (cid:15) -greedy policy, that is, selects the action that maximizes the Q -value with probability 1 - (cid:15) or randomly.

Original: The
Revised: The

Original: Independent Deep Q -Learning (IDQN) (Tampuu et al., 2017) is an extension of DQN by ignoring the inﬂuence of other agents for multi-agent case.
Revised: Independent Deep Q -Learning (IDQN) (Tampuu et al., 2017) is an extension of DQN by ignoring the inﬂuence of other agents for multi-agent case.

Original: Every agent learns a Q -function Q a ( u a | s ; θ a ) based on its own observation and received reward.
Revised: Every agent learns a Q -function Q a ( u a | s ; θ a ) based on its own observation and received reward.

Original: Our algorithm employs DQN as the basic RL algorithm based on the following two considerations: 1) our algorithm is dedicated to discuss the learning communication mechanism in largescaleMARL scenarios, as a result we can choose a concise and effective basic RL algorithm like thewell-known DQN; 2) data collection in large-scale MARL environments is extremely inefﬁciently,while DQN has excellent data efﬁciency as an ofﬂine RL algorithm.
Revised: Our algorithm employs DQN as the basic RL algorithm based on the following two considerations: 1) our algorithm is dedicated to discuss the learning communication mechanism in largescaleMARL scenarios, as a result we can choose a concise and effective basic RL algorithm like thewell-known DQN; 2) data collection in large-scale MARL environments is extremely inefﬁciently,while DQN has excellent data efﬁciency as an ofﬂine RL algorithm.

Original:  LSC: L EARNING S TRUCTURED C OMMUNICATION
Revised:  LSC: L EARNING S TRUCTURED C OMMUNICATION

Original: Our communication architecture has two key modules: structured communication network module and communication-based policy module , shown in Figure 4.
Revised: Our communication architecture has two key modules: structured communication network module and communication-based policy module , shown in Figure 4.

Original: The ﬁrst module aims to establish the dynamic hierarchical structured communication network in a distributed fashion, while the second module contains the GNN-based communication extraction and Q -network components.
Revised: The ﬁrst module aims to establish the dynamic hierarchical structured communication network in a distributed fashion, while the second module contains the GNN-based communication extraction and Q -network components.

Original: Without loss of generality, we use DQN as the basic reinforcement algorithm, however our approach can incorporate any value-based or actor-critic methods.
Revised: Without loss of generality, we use DQN as the basic reinforcement algorithm, however our approach can incorporate any value-based or actor-critic methods.

Original: The details of LSC is depicted in Algorithm
Revised: The details of LSC is depicted in Algorithm

Original: [Figure]
Revised: [Figure]

Original: ( V tn , V tc , E ); 10: for each agent i do 11:
Revised: ( V tn , V tc , E ); 10: for each agent i do 11:

Original: With probability (cid:15) pick a random action a ti else choose the action that has the largest 12: value in the vector q ti ; 13:
Revised: With probability (cid:15) pick a random action a ti else choose the action that has the largest 12: value in the vector q ti ; 13:

Original: Execute global actions and get global reward r t , next state s t +1 , next observation o t +1 ; 14:
Revised: Execute global actions and get global reward r t , next state s t +1 , next observation o t +1 ; 14:

Original: Get updated position POSs t +1 i for each agent i ; 15:
Revised: Get updated position POSs t +1 i for each agent i ; 15:

Original: Store ( s t , o t , { POSs t 1 , · · · , POSs tn } , a t , r t , o t +1 , { POSs t +11 , · · · , POSs t +1 n } , s t +1 ) to R ; 16: for k = 1 , · · · , K do 17:
Revised: Store ( s t , o t , { POSs t 1 , · · · , POSs tn } , a t , r t , o t +1 , { POSs t +11 , · · · , POSs t +1 n } , s t +1 ) to R ; 16: for k = 1 , · · · , K do 17:

Original: Sample a random mini-batch transitions from R ; 18:
Revised: Sample a random mini-batch transitions from R ; 18:

Original: Update weight generator θ w by minimizing Eq.
Revised: Update weight generator θ w by minimizing Eq.

Original: Update communication based policy module ( θ Q , θ gnn ) by minimizing Eq.
Revised: Update communication based policy module ( θ Q , θ gnn ) by minimizing Eq.

Original: Update the target networks through Eq.
Revised: Update the target networks through Eq.

Original: Speciﬁcally, the CBRP function automatically and distributively establishes the structured communication network based on the learnt importance weights.
Revised: Speciﬁcally, the CBRP function automatically and distributively establishes the structured communication network based on the learnt importance weights.

Original: The HCOMM function denotes the communication-based policy module, which outputs the Q -values based on the GNN-based communication messages.
Revised: The HCOMM function denotes the communication-based policy module, which outputs the Q -values based on the GNN-based communication messages.

Original: Both CBRP and HCOMM are discussed in the following subsections, and the details of CBRP and HCOMM can be found in Appendix.
Revised: Both CBRP and HCOMM are discussed in the following subsections, and the details of CBRP and HCOMM can be found in Appendix.

Original: 3. S TRUCTURED C OMMUNICATION N ETWORK M ODULE
Revised: 3. S TRUCTURED C OMMUNICATION N ETWORK M ODULE

Original: The structured communication network module takes the role of establishing a hierarchical structured communication network which will be employed in communication-based policy module.
Revised: The structured communication network module takes the role of establishing a hierarchical structured communication network which will be employed in communication-based policy module.

Original: Two sub-modules are included, i.e., the weight generator and the Cluster Based Routing Protocol (CBRP).
Revised: Two sub-modules are included, i.e., the weight generator and the Cluster Based Routing Protocol (CBRP).

Original: The weight generator sub-module aims to determine the importance weight for each agent automatically.
Revised: The weight generator sub-module aims to determine the importance weight for each agent automatically.

Original: It is modeled through a neural network f
Revised: It is modeled through a neural network f

Original: wg : o → w , where the weight w can mea-
Revised: wg : o → w , where the weight w can mea-

Original: sure the conﬁdence of an agent to become a center.
Revised: sure the conﬁdence of an agent to become a center.

Original: Further, the CBRP sub-module employs the weights of all agents { w i } to construct the hierarchical structured communication network.
Revised: Further, the CBRP sub-module employs the weights of all agents { w i } to construct the hierarchical structured communication network.

Original: To emphasize, the CBRP sub-module can be implemented in a distributed fashion, as a result, the central agents can be elected distributedly.
Revised: To emphasize, the CBRP sub-module can be implemented in a distributed fashion, as a result, the central agents can be elected distributedly.

Original: This advantage ensures the practicability for large-scale case.
Revised: This advantage ensures the practicability for large-scale case.

Original: The CBRP method (Rezaee & Yaghmaee, 2009) is a typical method for establishing a hierarchical routing structure.
Revised: The CBRP method (Rezaee & Yaghmaee, 2009) is a typical method for establishing a hierarchical routing structure.

Original: The key idea for CBRP is that each agent will check whether central agent or agent that has larger weight w exists in its receptive area.
Revised: The key idea for CBRP is that each agent will check whether central agent or agent that has larger weight w exists in its receptive area.

Original: The agent will become a central agent if no above agent is found, else it will keep its own role.
Revised: The agent will become a central agent if no above agent is found, else it will keep its own role.

Original: With enough checking steps, each agent will either be an central agent or in some central agents’ receptive.
Revised: With enough checking steps, each agent will either be an central agent or in some central agents’ receptive.

Original: All agents can be separated into several groups with each central agent as the group leader.
Revised: All agents can be separated into several groups with each central agent as the group leader.

Original: The overall hierarchical structured communication network further will be established by fully connecting all central agents from different groups and connecting the agents in each group to their central agent.
Revised: The overall hierarchical structured communication network further will be established by fully connecting all central agents from different groups and connecting the agents in each group to their central agent.

Original: There is a strong connection between these two modules in LSC algorithm.
Revised: There is a strong connection between these two modules in LSC algorithm.

Original: Different generated weights will lead to different hierarchical structured communication network, which would cause diverse performance of the communication-based policy.
Revised: Different generated weights will lead to different hierarchical structured communication network, which would cause diverse performance of the communication-based policy.

Original: Some experiment results also have conﬁrmed that the weights have a great inﬂuence on the performance, which motivates us to train these two modules end-to-end.
Revised: Some experiment results also have conﬁrmed that the weights have a great inﬂuence on the performance, which motivates us to train these two modules end-to-end.

Original: However the CBRP sub-module is not differentiable, which means the gradients cannot be back-propagated from communication-based policy module to the weight generator sub-module.
Revised: However the CBRP sub-module is not differentiable, which means the gradients cannot be back-propagated from communication-based policy module to the weight generator sub-module.

Original: Therefore, we introduce another RL task as an auxiliary, i.e., each agent takes its weight as an action by treating the communication-based policy module as an extra unobservable part of the environment, and receiving the same reward as the main RL task in the communication-based policy module discussed below.
Revised: Therefore, we introduce another RL task as an auxiliary, i.e., each agent takes its weight as an action by treating the communication-based policy module as an extra unobservable part of the environment, and receiving the same reward as the main RL task in the communication-based policy module discussed below.

Original: Moreover the weight w is constrained in the integer set { 0 , 1 , 2 , 3 , 4 } .
Revised: Moreover the weight w is constrained in the integer set { 0 , 1 , 2 , 3 , 4 } .

Original: The action space becomes discrete, as a result DQN algorithm can be used again to train the weight generator.
Revised: The action space becomes discrete, as a result DQN algorithm can be used again to train the weight generator.

Original: At this time, the weight generator can be regarded as a Q -value function.
Revised: At this time, the weight generator can be regarded as a Q -value function.

Original: The loss (cid:96) ( θ w ) for the weight generator sub-module becomes clear as follows, with y i
Revised: The loss (cid:96) ( θ w ) for the weight generator sub-module becomes clear as follows, with y i

Original: =
Revised: =

Original: r i + γ max ˜ w i Q θ w (˜ o i , ˜ w i ) .
Revised: r i + γ max ˜ w i Q θ w (˜ o i , ˜ w i ) .

Original: r i denotes the reward received for agent i from environment.
Revised: r i denotes the reward received for agent i from environment.

Original: [Equation]
Revised: [Equation]

Original: 3. C OMMUNICATION - BASED P OLICY M ODULE
Revised: 3. C OMMUNICATION - BASED P OLICY M ODULE

Original: After the structured communication network topology is determined, the communication-based policy module will learn the communication content and generate the ﬁnal global collaboration policy.
Revised: After the structured communication network topology is determined, the communication-based policy module will learn the communication content and generate the ﬁnal global collaboration policy.

Original: The communication-based policy module consists of two sub-modules, i.e., GNN-based communication sub-module and the Q -Net sub-module.
Revised: The communication-based policy module consists of two sub-modules, i.e., GNN-based communication sub-module and the Q -Net sub-module.

Original: The ﬁrst one aims to learn the communication messages and further update overall state perception, while the other sub-module learns the policy based on the new state perceptions after efﬁcient communication.
Revised: The ﬁrst one aims to learn the communication messages and further update overall state perception, while the other sub-module learns the policy based on the new state perceptions after efﬁcient communication.

Original: Different from many existing works (Foerster et al., 2016; Das et al., 2019; Singh et al., 2019), the agents play differently in the
Revised: Different from many existing works (Foerster et al., 2016; Das et al., 2019; Singh et al., 2019), the agents play differently in the

Original: GNN-base communication sub-module.
Revised: GNN-base communication sub-module.

Original: Central agents should guarantee high-level information and dominate the agents in their driven groups respectively.
Revised: Central agents should guarantee high-level information and dominate the agents in their driven groups respectively.

Original: The hope is that such a structure can ensure the effectiveness of communication and the efﬁciency of intra-group and inter-group collaboration.
Revised: The hope is that such a structure can ensure the effectiveness of communication and the efﬁciency of intra-group and inter-group collaboration.

Original: Recall Figure 3, the well-established hierarchical structured communication network can be represented by a tuple ( V , E ) while the edges are directed.
Revised: Recall Figure 3, the well-established hierarchical structured communication network can be represented by a tuple ( V , E ) while the edges are directed.

Original: The node set V contains N v nodes which can be divided into the central node set V c and the normal node set V n .
Revised: The node set V contains N v nodes which can be divided into the central node set V c and the normal node set V n .

Original: For central node i ∈ V c , the node feature vector v
Revised: For central node i ∈ V c , the node feature vector v

Original: i includes the embedding feature v ni , the central role feature v ci and the global feature v gi ; for normal node i ∈ V n , the node feature vector v i only includes the embedding feature v ni .
Revised: i includes the embedding feature v ni , the central role feature v ci and the global feature v gi ; for normal node i ∈ V n , the node feature vector v i only includes the embedding feature v ni .

Original: For each edge ( i → j ) ∈ E with i, j ∈ V , the edge feature vector is denoted as e ij .
Revised: For each edge ( i → j ) ∈ E with i, j ∈ V , the edge feature vector is denoted as e ij .

Original: Functions φ and ρ denote the update embedding function and aggregate function respectively.
Revised: Functions φ and ρ denote the update embedding function and aggregate function respectively.

Original: As shown in
Revised: As shown in

Original: Figure 3, the overall GNN-based communication sub-module consists of three steps, and the GNN operation is detailed in Table 1 and as follows:
Revised: Figure 3, the overall GNN-based communication sub-module consists of three steps, and the GNN operation is detailed in Table 1 and as follows:

Original: Step 1: Intra-group aggregation.
Revised: Step 1: Intra-group aggregation.

Original: In each group, the normal agent embeds their local information and sends it to the associated central agent j ∈ V c ; the central agent aggregates the information from all associated normal agents and updates its central role feature; Step 2: Inter-group sharing.
Revised: In each group, the normal agent embeds their local information and sends it to the associated central agent j ∈ V c ; the central agent aggregates the information from all associated normal agents and updates its central role feature; Step 2: Inter-group sharing.

Original: The central agent communicates with the other central agent with cluster information, further aggregates the received and indicates the global perception; Step 3: Intra-group sharing.
Revised: The central agent communicates with the other central agent with cluster information, further aggregates the received and indicates the global perception; Step 3: Intra-group sharing.

Original: The central agent communicates all its feature with the associated normal agents while the normal agent aggregates the received information from central agents.
Revised: The central agent communicates all its feature with the associated normal agents while the normal agent aggregates the received information from central agents.

Original: Both the embedding feature of central and normal agents will be updated.
Revised: Both the embedding feature of central and normal agents will be updated.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: The GNN-based communication sub-module is modeled as a GNN ( f θ gnn ) with parameter θ gnn , while the following Q -Net of agent i ( Q iθ Q ) is parameterized by shared parameter θ Q .
Revised: The GNN-based communication sub-module is modeled as a GNN ( f θ gnn ) with parameter θ gnn , while the following Q -Net of agent i ( Q iθ Q ) is parameterized by shared parameter θ Q .

Original: The gradient can be back-propagated from Q -Net to the graph neural network, as a result the overall loss of communication based policy module is as follows:
Revised: The gradient can be back-propagated from Q -Net to the graph neural network, as a result the overall loss of communication based policy module is as follows:

Original: [Equation]
Revised: [Equation]

Original: where y i =
Revised: where y i =

Original: r i + γ max ˜ a i Q iθ Q ( f θ gnn (˜ o ) , ˜ a i ) .
Revised: r i + γ max ˜ a i Q iθ Q ( f θ gnn (˜ o ) , ˜ a i ) .

Original: r i denotes the reward received for agent i from environment.
Revised: r i denotes the reward received for agent i from environment.

Original: Some softly updating scheme is further employed to update target network, i.e.,
Revised: Some softly updating scheme is further employed to update target network, i.e.,

Original: [Equation]
Revised: [Equation]

Original: and θ ˜ gnn
Revised: and θ ˜ gnn

Original: [Equation]
Revised: [Equation]

Original: Here we discuss communication efﬁciency from three aspects: the number of message exchanging ( N msg ) among agents; the number of steps during the communication procedure ( N step ); the communication bandwidth and range requirements for each agent ( N b - r ), and n is the total number of agents.
Revised: Here we discuss communication efﬁciency from three aspects: the number of message exchanging ( N msg ) among agents; the number of steps during the communication procedure ( N step ); the communication bandwidth and range requirements for each agent ( N b - r ), and n is the total number of agents.

Original: The details about communication efﬁciency are presented in Table 2 for each stage of the
Revised: The details about communication efﬁciency are presented in Table 2 for each stage of the

Original: MARL algorithms.
Revised: MARL algorithms.

Original: For DIAL, each agent communicates with all other agents based on the fullyconnected network, which results in O ( n 2 ) message exchanging complexity.
Revised: For DIAL, each agent communicates with all other agents based on the fullyconnected network, which results in O ( n 2 ) message exchanging complexity.

Original: DIAL need only one communication step, however the communication bandwidth and range requirement for each agent is high and in the order of O ( n ) .
Revised: DIAL need only one communication step, however the communication bandwidth and range requirement for each agent is high and in the order of O ( n ) .

Original: Different from DIAL, CommNet and IC3 both employ the star communication network, as a result the number of message exchanging is in the order of O ( n ) .
Revised: Different from DIAL, CommNet and IC3 both employ the star communication network, as a result the number of message exchanging is in the order of O ( n ) .

Original: N step and N b - r are the same as DIAL.
Revised: N step and N b - r are the same as DIAL.

Original: The communication complexity of ATOC and our proposed
Revised: The communication complexity of ATOC and our proposed

Original: LSC depends on the number of groups (denoted as k < n , which is automatically determined in the algorithms) and the maximum output degree of the communication network (denoted as b < n ).
Revised: LSC depends on the number of groups (denoted as k < n , which is automatically determined in the algorithms) and the maximum output degree of the communication network (denoted as b < n ).

Original: For ATOC, the communication network it tree-type, so that it only need to exchange O ( kb ) messages.
Revised: For ATOC, the communication network it tree-type, so that it only need to exchange O ( kb ) messages.

Original: However, the number of steps is larger for ATOC for its sequential property and is in the order of O ( d ) ( d denotes the depth of the communication network).
Revised: However, the number of steps is larger for ATOC for its sequential property and is in the order of O ( d ) ( d denotes the depth of the communication network).

Original: N b - r will become much smaller to be O ( b ) because communication happens in groups.
Revised: N b - r will become much smaller to be O ( b ) because communication happens in groups.

Original: Furthermore for our LSC, the number of message exchanging is a bit larger than ATOC due to the communication among all elected centers, i.e., O ( k 2 + kb ) .
Revised: Furthermore for our LSC, the number of message exchanging is a bit larger than ATOC due to the communication among all elected centers, i.e., O ( k 2 + kb ) .

Original: However the depth of the hierarchical communication network is only two which results in O (1) communication steps, while N b - r is in the order of O (max( b, k )) .
Revised: However the depth of the hierarchical communication network is only two which results in O (1) communication steps, while N b - r is in the order of O (max( b, k )) .

Original: Overall, our LSC algorithm has advantage in the communication efﬁciency.
Revised: Overall, our LSC algorithm has advantage in the communication efﬁciency.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We compare LSC with state-of-the-art MARL methods in two large-scale battle environments, i.e., the grid world platform MAgent (Zheng et al., 2017) and StarCraft2 (Samvelyan et al., 2019), to evaluate their performances from aspects of both network structure and communication.
Revised: We compare LSC with state-of-the-art MARL methods in two large-scale battle environments, i.e., the grid world platform MAgent (Zheng et al., 2017) and StarCraft2 (Samvelyan et al., 2019), to evaluate their performances from aspects of both network structure and communication.

Original: 4. L ARGE S CALE B ATTLE GAME IN MA GENT
Revised: 4. L ARGE S CALE B ATTLE GAME IN MA GENT

Original: Settings.
Revised: Settings.

Original: In a MAgent battle, agents ﬁghting against enemies in a 40 × 40 grid world.
Revised: In a MAgent battle, agents ﬁghting against enemies in a 40 × 40 grid world.

Original: Each agent only receives its local observation, acts independently and cooperatively, and further gains its reward.
Revised: Each agent only receives its local observation, acts independently and cooperatively, and further gains its reward.

Original: The goal for each agent is to attack its enemies and prevent them from being attacked.
Revised: The goal for each agent is to attack its enemies and prevent them from being attacked.

Original: Each agent
Revised: Each agent

Original: [Figure]
Revised: [Figure]

Original: from both sides has a 6 × 6 visual ﬁeld and can attack its 8 adjacent grids.
Revised: from both sides has a 6 × 6 visual ﬁeld and can attack its 8 adjacent grids.

Original: The speed, attack power and health point for each agent are 1 , 1 and 4 , which are increased to 2 , 2 and 10 for the enemy to increase the difﬁculty.
Revised: The speed, attack power and health point for each agent are 1 , 1 and 4 , which are increased to 2 , 2 and 10 for the enemy to increase the difﬁculty.

Original: The reward is +5 for successfully attacking an enemy, − 2 for being killed and − 0 .
Revised: The reward is +5 for successfully attacking an enemy, − 2 for being killed and − 0 .

Original: 01 for attacking a blank grid.
Revised: 01 for attacking a blank grid.

Original: Baselines.
Revised: Baselines.

Original: To evaluate the effects of communication scheme, three peer methods on learning to communicate for MARL, i.e., CommNet, IC3 and ATOC, are chosen to compete with our proposed LSC.
Revised: To evaluate the effects of communication scheme, three peer methods on learning to communicate for MARL, i.e., CommNet, IC3 and ATOC, are chosen to compete with our proposed LSC.

Original: Considering the weakness of mean aggregation, we replace the aggregation function of
Revised: Considering the weakness of mean aggregation, we replace the aggregation function of

Original: CommNet and IC3 with GNN which is same as LSC.
Revised: CommNet and IC3 with GNN which is same as LSC.

Original: The group radius is 6, the same as the visual ﬁeld, in ATOC and LSC.
Revised: The group radius is 6, the same as the visual ﬁeld, in ATOC and LSC.

Original: All communication messages are embedded to 3 -dimension vectors for cost-effectiveness.
Revised: All communication messages are embedded to 3 -dimension vectors for cost-effectiveness.

Original: Besides, two MARL methods with no communication, i.e., IDQN and MFQ are also compared, since they are widely used in large-scale environments.
Revised: Besides, two MARL methods with no communication, i.e., IDQN and MFQ are also compared, since they are widely used in large-scale environments.

Original: In MAgent, the policy of enemy is pretrained by IDQN.
Revised: In MAgent, the policy of enemy is pretrained by IDQN.

Original: Policy performance.
Revised: Policy performance.

Original: Figure 5(a) and Table 3 show the overall performances of compared algorithm in a 64 vs. 64 battle.
Revised: Figure 5(a) and Table 3 show the overall performances of compared algorithm in a 64 vs. 64 battle.

Original: The learning curves in Figure 5(a) present the average reward of different agents by epoch.
Revised: The learning curves in Figure 5(a) present the average reward of different agents by epoch.

Original: LSC achieves better rewards quickly after 700 epochs and ﬁnally converges to an obvious higher point (about 1 . 15 ) than baselines.
Revised: LSC achieves better rewards quickly after 700 epochs and ﬁnally converges to an obvious higher point (about 1 . 15 ) than baselines.

Original: Table 3 and Table 4 give quantitative comparisons of these methods.
Revised: Table 3 and Table 4 give quantitative comparisons of these methods.

Original: Each algorithm is given 50 trials with its well-trained model.
Revised: Each algorithm is given 50 trials with its well-trained model.

Original: Mean-reward , N kill , N dead ,
Revised: Mean-reward , N kill , N dead ,

Original: Ratio kd in Table 3 denote the mean of average ﬁnal rewards of agents, the number of killed enemies and dead agents, and the ratio N kill (cid:14) N dead , respectively.
Revised: Ratio kd in Table 3 denote the mean of average ﬁnal rewards of agents, the number of killed enemies and dead agents, and the ratio N kill (cid:14) N dead , respectively.

Original: Following Figure 5(a), LSC can obtain a better mean reward stably, with a 30% performance advantage at least.
Revised: Following Figure 5(a), LSC can obtain a better mean reward stably, with a 30% performance advantage at least.

Original: N kill are similar, because all approaches fulﬁll the mission, and beat the pre-trained IDQN.
Revised: N kill are similar, because all approaches fulﬁll the mission, and beat the pre-trained IDQN.

Original: It is achieved by LSC with the least casualty, i.e., the smallest N dead and the highest Ratio kd .
Revised: It is achieved by LSC with the least casualty, i.e., the smallest N dead and the highest Ratio kd .

Original: Table 4 gives the comparisons in terms of the number of epochs to achieve the same reward value from 0 .
Revised: Table 4 gives the comparisons in terms of the number of epochs to achieve the same reward value from 0 .

Original: 7 to 1 . 2 within maximal 1750 epochs in the training procedure.
Revised: 7 to 1 . 2 within maximal 1750 epochs in the training procedure.

Original: One can see that LSC needs fewer epochs to achieve the same reward compared with all the other algorithms, while more reward can be guaranteed within the maximal epochs.
Revised: One can see that LSC needs fewer epochs to achieve the same reward compared with all the other algorithms, while more reward can be guaranteed within the maximal epochs.

Original: These results indicate that LSC can promote collaboration and cooperation, and produce superior policies.
Revised: These results indicate that LSC can promote collaboration and cooperation, and produce superior policies.

Original: Communication effectiveness.
Revised: Communication effectiveness.

Original: As observed from the blue and orange curves in Figure 5(a) and the ﬁrst and ﬁfth columns in Table 3, that LSC outperforms IDQN greatly.
Revised: As observed from the blue and orange curves in Figure 5(a) and the ﬁrst and ﬁfth columns in Table 3, that LSC outperforms IDQN greatly.

Original: Since IDQN is the special case of LSC without communication procedure, this phenomenon demonstrates the usefulness of our proposed communication solution.
Revised: Since IDQN is the special case of LSC without communication procedure, this phenomenon demonstrates the usefulness of our proposed communication solution.

Original: Meanwhile, it can be noted that LSC also surpasses other MARL with communication algorithms, which is the consequence of its advanced structure.
Revised: Meanwhile, it can be noted that LSC also surpasses other MARL with communication algorithms, which is the consequence of its advanced structure.

Original: As mentioned in our experiments, CommNet and IC3 adopt the same message dimension and the same aggregation function as LSC, which leads to better performances than the original versions.
Revised: As mentioned in our experiments, CommNet and IC3 adopt the same message dimension and the same aggregation function as LSC, which leads to better performances than the original versions.

Original: However, the star structure makes the center node need to process all agents’ information in CommNet and IC3.
Revised: However, the star structure makes the center node need to process all agents’ information in CommNet and IC3.

Original: When the agent number increases, the message extraction could be difﬁcult.
Revised: When the agent number increases, the message extraction could be difﬁcult.

Original: Thus, the ﬁnal performances cannot be compared with the LSC.
Revised: Thus, the ﬁnal performances cannot be compared with the LSC.

Original: For ATOC in large scale environments, the message needs to jump multiple times between local circles, and multiple information aggregation and extraction bring in approximation error, which results in policy deterioration.
Revised: For ATOC in large scale environments, the message needs to jump multiple times between local circles, and multiple information aggregation and extraction bring in approximation error, which results in policy deterioration.

Original: Scalability.
Revised: Scalability.

Original: Figure 5(b) shows the total reward curves of team by the agent number ( 10 - 99 ).
Revised: Figure 5(b) shows the total reward curves of team by the agent number ( 10 - 99 ).

Original: Especially when the team has 99 agents, the reward of LSC is 1 .
Revised: Especially when the team has 99 agents, the reward of LSC is 1 .

Original: 75 times of other methods.
Revised: 75 times of other methods.

Original: It can be seen that the structured communication of LSC confers superior performances at different scales, because it utilizes the divide-and-conquer strategy to automatically group local agents and aggregate centers.
Revised: It can be seen that the structured communication of LSC confers superior performances at different scales, because it utilizes the divide-and-conquer strategy to automatically group local agents and aggregate centers.

Original: In Figure 5(b) When the agent number is less than 80 , the communication between agents can help to learn policy, so the orange dashed line is almost bellow other lines.
Revised: In Figure 5(b) When the agent number is less than 80 , the communication between agents can help to learn policy, so the orange dashed line is almost bellow other lines.

Original: When the number is more than 80 , the demand of star-style information processing exceed the ability of aggregation
Revised: When the number is more than 80 , the demand of star-style information processing exceed the ability of aggregation

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: network, thus CommNet and IC3 are inferior to IDQN.
Revised: network, thus CommNet and IC3 are inferior to IDQN.

Original: For ATOC, the jump of message becomes the bottleneck as the agent number increases, as inspected in the analysis above.
Revised: For ATOC, the jump of message becomes the bottleneck as the agent number increases, as inspected in the analysis above.

Original: To sum up, the experiments indicate LSC obviously has better scalability than the baseline.
Revised: To sum up, the experiments indicate LSC obviously has better scalability than the baseline.

Original: Message visualization.
Revised: Message visualization.

Original: To analyze the communication messages learned by GNN, we execute LSC with its well-trained policy 50 times, and visualize the 3 dimensional vector sent by central nodes in
Revised: To analyze the communication messages learned by GNN, we execute LSC with its well-trained policy 50 times, and visualize the 3 dimensional vector sent by central nodes in

Original: Figure 5.
Revised: Figure 5.

Original: Observing Figure 5(c) and 5(d), after receiving messages, the majority of agents choose to move , and the minority choose to attack .
Revised: Observing Figure 5(c) and 5(d), after receiving messages, the majority of agents choose to move , and the minority choose to attack .

Original: This means that agents in LSC is very positive to adjust the team formation and then cooperate to attack.
Revised: This means that agents in LSC is very positive to adjust the team formation and then cooperate to attack.

Original: It is worthy to notice that most messages have a small norm (less than 10 ), and the norm of a large proportion is around 0 .
Revised: It is worthy to notice that most messages have a small norm (less than 10 ), and the norm of a large proportion is around 0 .

Original: From the aspect of optimization, the redundant messages with larger norm will bring in more noise to other agents.
Revised: From the aspect of optimization, the redundant messages with larger norm will bring in more noise to other agents.

Original: In this way, LSC minimizes the impact of redundant messages on the ﬁnal performance.
Revised: In this way, LSC minimizes the impact of redundant messages on the ﬁnal performance.

Original: This similar phenomenon that central nodes is nearly silent in many cases, is also mentioned by CommNet (Sukhbaatar et al., 2016).
Revised: This similar phenomenon that central nodes is nearly silent in many cases, is also mentioned by CommNet (Sukhbaatar et al., 2016).

Original: Therefore, the message representation module via the GNN-based communication module can generate meaningful and efﬁcient messages theoretically and empirically.
Revised: Therefore, the message representation module via the GNN-based communication module can generate meaningful and efﬁcient messages theoretically and empirically.

Original: Behavior pattern.
Revised: Behavior pattern.

Original: Here, we demonstrate the battle tactics evolved for MARL via structured communication.
Revised: Here, we demonstrate the battle tactics evolved for MARL via structured communication.

Original: To analyzed the globally cooperation strategy of LSC, we visualize the progress of the battle.
Revised: To analyzed the globally cooperation strategy of LSC, we visualize the progress of the battle.

Original: For fairness, we start six algorithms from the same state in Figure 6(a).
Revised: For fairness, we start six algorithms from the same state in Figure 6(a).

Original: The typical behavior patterns learnt by LSC are presented in the ﬁgures of the ﬁrst row.
Revised: The typical behavior patterns learnt by LSC are presented in the ﬁgures of the ﬁrst row.

Original: Via the intra-group communication and cooperation, in Figure 6(b), the blue team (LSC) organizes an encirclement to nearby red enemies; the team has a local numerical superiority, and focus agents’ ﬁre to wipe out enemies, denoted by the black attack arrows in Figure 6(c).
Revised: Via the intra-group communication and cooperation, in Figure 6(b), the blue team (LSC) organizes an encirclement to nearby red enemies; the team has a local numerical superiority, and focus agents’ ﬁre to wipe out enemies, denoted by the black attack arrows in Figure 6(c).

Original: These show the intra-group cooperation of LSC.
Revised: These show the intra-group cooperation of LSC.

Original: Upper ﬁgures in the second row of Figure 6 shows the situation of early stage (17 steps) after initialization for six methods, and lower ﬁgures show the states after 50 steps.
Revised: Upper ﬁgures in the second row of Figure 6 shows the situation of early stage (17 steps) after initialization for six methods, and lower ﬁgures show the states after 50 steps.

Original: From these results, we may arrive a conclusion that the team with our LSC can beat the opponent more quickly with a more aggressive policy.
Revised: From these results, we may arrive a conclusion that the team with our LSC can beat the opponent more quickly with a more aggressive policy.

Original: By intra-group and inter-group collaboration, Figure 6(d)
Revised: By intra-group and inter-group collaboration, Figure 6(d)

Original: LSC has carried out encircling and ﬁre-focusing many times, and achieves an enormous advantage within only 17 steps, and wipe out the enemies within 50 steps.
Revised: LSC has carried out encircling and ﬁre-focusing many times, and achieves an enormous advantage within only 17 steps, and wipe out the enemies within 50 steps.

Original: For both IDQN and MFQ, agents tend to cooperate within their visual range, and once the agents get separated out of visual range, they can hardly form global cooperation, which lead to the failure result in Figure 6(h) and 6(i).
Revised: For both IDQN and MFQ, agents tend to cooperate within their visual range, and once the agents get separated out of visual range, they can hardly form global cooperation, which lead to the failure result in Figure 6(h) and 6(i).

Original: Similarly, agents controlled by ATOC communicate only among group range, thus they encounter similar situations.
Revised: Similarly, agents controlled by ATOC communicate only among group range, thus they encounter similar situations.

Original: Agents for
Revised: Agents for

Original: CommNet and IC3 have global communication, however once some agents get far away from the central agents, central agents can hardly understand their messages, making ineffective cooperation.
Revised: CommNet and IC3 have global communication, however once some agents get far away from the central agents, central agents can hardly understand their messages, making ineffective cooperation.

Original: As Figure 6(e) and 6(f), some agents get far away from the majority of agents, thus their results are not ideal.
Revised: As Figure 6(e) and 6(f), some agents get far away from the majority of agents, thus their results are not ideal.

Original: To sum up, in Figure 6(d), agents controlled by LSC form a global encircle strategy by communication in both intra-group and inter-group, thus LSC-based agents can wipe out enemies faster than the baselines.
Revised: To sum up, in Figure 6(d), agents controlled by LSC form a global encircle strategy by communication in both intra-group and inter-group, thus LSC-based agents can wipe out enemies faster than the baselines.

Original: 4. L ARGE S CALE B ATTLE G AME IN S TAR C RAFT
Revised: 4. L ARGE S CALE B ATTLE G AME IN S TAR C RAFT

Original: Battle game in StarCraft2 is a confrontation between two marine teams shown in appendix, which is much more complex than MAgent.
Revised: Battle game in StarCraft2 is a confrontation between two marine teams shown in appendix, which is much more complex than MAgent.

Original: Speciﬁcally, we use a 25 vs. 25 map, i.e., 25m , where the range vision is 9 and the map size is 1920 × 1200 .
Revised: Speciﬁcally, we use a 25 vs. 25 map, i.e., 25m , where the range vision is 9 and the map size is 1920 × 1200 .

Original: To evaluate the cooperation, agents one team are controlled by the individual learned policy.
Revised: To evaluate the cooperation, agents one team are controlled by the individual learned policy.

Original: Here, we compare our LSC with IQDN, Commnet and IC3.
Revised: Here, we compare our LSC with IQDN, Commnet and IC3.

Original: The action space consists of movement to an adjacent grid and shooting with the range 6 .
Revised: The action space consists of movement to an adjacent grid and shooting with the range 6 .

Original: We adopt the same dense reward setting ( 0 .
Revised: We adopt the same dense reward setting ( 0 .

Original: 44 for killing an enemy, 8 . 9 for winning the battle) as SMAC (Samvelyan et al., 2019).
Revised: 44 for killing an enemy, 8 . 9 for winning the battle) as SMAC (Samvelyan et al., 2019).

Original: Agents of the enemy are controlled by the built-in game AI.
Revised: Agents of the enemy are controlled by the built-in game AI.

Original: [Figure]
Revised: [Figure]

Original: In Figure 7, LSC outperforms compared algorithms, where the blue reward curve is much higher than others.
Revised: In Figure 7, LSC outperforms compared algorithms, where the blue reward curve is much higher than others.

Original: Although the agent number ( 25 ) is fewer than MAgent, the observation and action space of StarCraft2 are much larger, leading to the difﬁculty to learn policy.
Revised: Although the agent number ( 25 ) is fewer than MAgent, the observation and action space of StarCraft2 are much larger, leading to the difﬁculty to learn policy.

Original: Therefore, the performance of IDQN  degrades notably in this complex environment, while LSC, IC3,and CommNet  outperform it.
Revised: Therefore, the performance of IDQN and MFQ degrades notably in this complex environment, while LSC, IC3, CommNet and ATOC outperform it.

Original: This is because they entail communication to facilitate cooperation.
Revised: This is because they entail communication to facilitate cooperation.

Original: Moreover, LSC outperforms IC3,and CommNet greatly.
Revised: Moreover, LSC outperforms IC3, CommNet and ATOC.

Original: We do not compare with ATOC because of its poor performance in MAgent.

Original: This demonstrates that ﬂexible hierarchical communication and expressiveGNN-based message extraction make
Revised: This demonstrates that ﬂexible hierarchical communication and expressive

Original:  LSC more qualiﬁed for complex tasks than CommNet and IC3’s star-style communication.
Revised: GNN-based message extraction make LSC more qualiﬁed for complex tasks than CommNet and IC3’s star-style communication.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, a novel learning structured communication (LSC) algorithm is proposed for multiagent reinforcement learning.
Revised: In this paper, a novel learning structured communication (LSC) algorithm is proposed for multiagent reinforcement learning.

Original: The hierarchical structure is self-learned by cluster based routing protocol.
Revised: The hierarchical structure is self-learned by cluster based routing protocol.

Original: The communication message representation is naturally embedded and extracted via a graph neural network.
Revised: The communication message representation is naturally embedded and extracted via a graph neural network.

Original: Experiments in large-scale games (MAgent and StarCraft2) demonstrated that our LSC can outperform existing learning-to-communicate algorithms with better communication efﬁciency, cooperation capability, and scalability.
Revised: Experiments in large-scale games (MAgent and StarCraft2) demonstrated that our LSC can outperform existing learning-to-communicate algorithms with better communication efﬁciency, cooperation capability, and scalability.

Original: In the future, we will improve LSC by considering some practical constraints, such as communication bandwidth and delay.
Revised: In the future, we will improve LSC by considering some practical constraints, such as communication bandwidth and delay.

Original: A. D ETAILS AND D ISCUSSIONS OF A LL H YPERPARAMETERS
Revised: A. D ETAILS AND D ISCUSSIONS OF A LL H YPERPARAMETERS

Original: For both MAgent and StarCraft2 environments, the details of all hyperparameters used for approaches are summarized in the following Table 5 and Table 6.
Revised: For both MAgent and StarCraft2 environments, the details of all hyperparameters used for approaches are summarized in the following Table 5 and Table 6.

Original: All common hyperparameters of all approaches are set to be the same in the same environments.
Revised: All common hyperparameters of all approaches are set to be the same in the same environments.

Original: Except for the ”ATOC”, we change the message dimension to 64 in both enviroments.
Revised: Except for the ”ATOC”, we change the message dimension to 64 in both enviroments.

Original: For the reason that if the dimension of messages is set to be 3 as other algorithms, the necessary GRU embedding in ATOC leads to failure whileseems to be robust.
Revised: For the reason that if the dimension of messages is set to be 3 as other algorithms, the necessary GRU embedding in ATOC leads to failure whileseems to be robust.

Original: For the neural network setting in MAgent enviroment, the node encoder part are all implemented with two convolutional layers with 32 ﬁlters and kernel size 3 and a MLP with 256 units.
Revised: For the neural network setting in MAgent enviroment, the node encoder part are all implemented with two convolutional layers with 32 ﬁlters and kernel size 3 and a MLP with 256 units.

Original: For the Q Encoder part, they are implemented with MLP (128 , 64 , 13) .
Revised: For the Q Encoder part, they are implemented with MLP (128 , 64 , 13) .

Original: The message generators for algorithms except for ATOC are implemented with MLP (64 , 32 , 3) and the aggregation functions are segment sum.
Revised: The message generators for algorithms except for ATOC are implemented with MLP (64 , 32 , 3) and the aggregation functions are segment sum.

Original: While for ATOC, the message generator is MLP (128 , 64) and the aggregation functin isGRU (64) .
Revised: While for ATOC, the message generator is MLP (128 , 64) and the aggregation functin isGRU (64) .

Original: As for neural network setting in Starcraft2, we only tune the input and output layer to adapt to the new environment.
Revised: As for neural network setting in Starcraft2, we only tune the input and output layer to adapt to the new environment.

Original: All these settings can ensure repeatability.
Revised: All these settings can ensure repeatability.

Original: A. B ATTLE S CENARIOS OF MA GENT AND S TAR C RAFTA. D ISCUSSIONS ON W EIGHT G ENERATOR AND G ROUP R ADIUS
Revised: A. B ATTLE S CENARIOS OF MA GENT AND S TAR C RAFTA. D ISCUSSIONS ON W EIGHT G ENERATOR AND G ROUP R ADIUS

Original: Although LSC utilize the reward feedback to establish the weight generator, which will be various with respect to different settings.
Revised: Although LSC utilize the reward feedback to establish the weight generator, which will be various with respect to different settings.

Original: To investigate the improvement brought by our learned importance weight generator, we also compare with a basic random weight generator (randomly chosen the central nodes and separate normal nodes into groups).
Revised: To investigate the improvement brought by our learned importance weight generator, we also compare with a basic random weight generator (randomly chosen the central nodes and separate normal nodes into groups).

Original: As shown in Figure 9(a), our learned weight generator signiﬁcantly outperforms randomly.
Revised: As shown in Figure 9(a), our learned weight generator signiﬁcantly outperforms randomly.

Original: Faster convergence of our learned weight generator
Revised: Faster convergence of our learned weight generator

Original: 1: function HCOMM( V n , V c , E ) 2: (cid:93) Intra-group aggregation 3: for v i in V n do 4: for v j in V c and ( i → j ) in E do 5: e ij = φ enc ( v i ); (cid:46) Generate normal to central messages 6: for v j in V c do ¯ e j = ρ ( { e ij } ( i → j ) ∈E ); (cid:46) Central agents aggregate received messages 7: v cj =
Revised: 1: function HCOMM( V n , V c , E ) 2: (cid:93) Intra-group aggregation 3: for v i in V n do 4: for v j in V c and ( i → j ) in E do 5: e ij = φ enc ( v i ); (cid:46) Generate normal to central messages 6: for v j in V c do ¯ e j = ρ ( { e ij } ( i → j ) ∈E ); (cid:46) Central agents aggregate received messages 7: v cj =

Original: φ (¯ e j , v nj ); (cid:46) Generate cluster perception 8: (cid:93) Inter-group sharing 9: for v j in V c do 10: for v i in V c and ( i → j ) in E do 11: e ij = φ ( v ci , v ni ); (cid:46) Generate central to central messages 12: for v j in V c do 13: ¯ e j = ρ ( { e ij } ( i → j ) ∈E ); (cid:46) Aggregate received central to central messages 14: v gj = φ (¯ e j , v nj ); (cid:46) Obtain global perception 15: (cid:93) Intro-group sharing 16: for v i in V c do 17: for v j in V n
Revised: φ (¯ e j , v nj ); (cid:46) Generate cluster perception 8: (cid:93) Inter-group sharing 9: for v j in V c do 10: for v i in V c and ( i → j ) in E do 11: e ij = φ ( v ci , v ni ); (cid:46) Generate central to central messages 12: for v j in V c do 13: ¯ e j = ρ ( { e ij } ( i → j ) ∈E ); (cid:46) Aggregate received central to central messages 14: v gj = φ (¯ e j , v nj ); (cid:46) Obtain global perception 15: (cid:93) Intro-group sharing 16: for v i in V c do 17: for v j in V n

Original: and ( i → j ) in E do 18: e ij =
Revised: and ( i → j ) in E do 18: e ij =

Original: φ ( v gi , v ci , v ni , e ji ) , ¯ e j = ρ ( { e ij } ( i → j ) ∈E ); (cid:46) Generate central to normal messages 19: for v j in V n ∪ V c do 20: for v i in V c and ( i → j ) in E do 21: ¯ e j = ρ ( { e ij } ( i → j ) ∈E ); (cid:46) Aggregate received central to normal messages 22: v nj = φ (¯ e j , v nj ); (cid:46) Update states
Revised: φ ( v gi , v ci , v ni , e ji ) , ¯ e j = ρ ( { e ij } ( i → j ) ∈E ); (cid:46) Generate central to normal messages 19: for v j in V n ∪ V c do 20: for v i in V c and ( i → j ) in E do 21: ¯ e j = ρ ( { e ij } ( i → j ) ∈E ); (cid:46) Aggregate received central to normal messages 22: v nj = φ (¯ e j , v nj ); (cid:46) Update states

Original: [Table]
Revised: [Table]

Original: q j = Q ( v nj ); return q .
Revised: q j = Q ( v nj ); return q .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: shows the efﬁciency brought by involving the reward guided weight generator.
Revised: shows the efﬁciency brought by involving the reward guided weight generator.

Original: The hierarchical structured communication network guaranteed through the learned weight generator improves the communication efﬁciency.
Revised: The hierarchical structured communication network guaranteed through the learned weight generator improves the communication efﬁciency.

Original: The higher average reward obtained by our learned weight generator also shows the necessity of selecting central nodes based on the learned weight generator rather than randomly chosen
Revised: The higher average reward obtained by our learned weight generator also shows the necessity of selecting central nodes based on the learned weight generator rather than randomly chosen

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: To better investigate the hierarchical structure
Revised: To better investigate the hierarchical structure

Original: [Figure]
Revised: [Figure]

Original: we compare our LSC algorithm with respect to different group radius d , i.e., 3 , 6 , 12 and 60 .
Revised: we compare our LSC algorithm with respect to different group radius d , i.e., 3 , 6 , 12 and 60 .

Original: As shown in Figure 9(b)
Revised: As shown in Figure 9(b)

Original: [Figure]
Revised: [Figure]

Original: LSC with radius 6 outperform other settings.
Revised: LSC with radius 6 outperform other settings.

Original: When the radius increasing, the agents can establish inter-groups cooperation easier.
Revised: When the radius increasing, the agents can establish inter-groups cooperation easier.

Original: However, LSC with radius 60 performs worse than all the other three cases.
Revised: However, LSC with radius 60 performs worse than all the other three cases.

Original: LSC with extremely large radius will downgrade to CommNet which is not an effective way to extract valuable information.
Revised: LSC with extremely large radius will downgrade to CommNet which is not an effective way to extract valuable information.