Original: C ONSCIOUSNESS -A WARE M ULTI -A GENT R EINFORCEMENT L EARNING
Revised: C ONCENTRATED A TTENTION FOR M ULTI -A GENT R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In cooperative multi-agent reinforcement learning, centralized training with decentralized execution (CTDE) shows great promise for a trade-off between independent
Revised: In cooperative multi-agent reinforcement learning, centralized training with decentralized execution (CTDE) shows great promise for a trade-off between independent

Original: Q-learning and joint action learning.
Revised: Q-learning and joint action learning.

Original: However, vanilla CTDE methods assumed a fixed number of agents could hardly adapt to real-world scenarios where dynamic team compositions typically suffer from the dilemma of dramatic partial observability variance.
Revised: However, vanilla CTDE methods assumed a fixed number of agents could hardly adapt to real-world scenarios where dynamic team compositions typically suffer from the dilemma of dramatic partial observability variance.

Original: Specifically, agents with extensive sight ranges are prone to be affected by trivial environmental substrates, dubbed the “attention distraction” issue; ones with limited observability can hardly sense their teammates, hindering the quality of cooperation.
Revised: Specifically, agents with extensive sight ranges are prone to be affected by trivial environmental substrates, dubbed the “attention distraction” issue; ones with limited observability can hardly sense their teammates, hindering the quality of cooperation.

Original: In this paper, we propose a Consciousness-Aware Multi-Agent reinforcement learning (CAMA) approach, which roots in a divide-and-conquer strategy to facilitate stable and sustainable teamwork.
Revised: In this paper, we propose a Concentrated Attention for Multi-Agent reinforcement learning (CAMA) approach, which roots in a divide-and-conquer strategy to facilitate stable and sustainable teamwork.

Original: Concretely, CAMA targets dividing the input entities with controlled observability masks by an Entity Dividing Module (EDM) according to their execution relevance for consciousness learning.
Revised: Concretely, CAMA targets dividing the input entities with controlled observability masks by an Entity Dividing Module (EDM) according to their contributions for attention weights.

Original: To tackle the attention distraction issue, the highly related entities are fed to a Consciousness Enhancement Module (CEM) for consciousness-aware representation extraction via action prediction with an inverse model.
Revised: To tackle the attention distraction issue, the highly contributed entities are fed to an Attention Enhancement Module (AEM) for execution-related representation extraction via action prediction with an inverse model.

Original: For better out-of-sight-range cooperation, the lowly related ones are compressed to brief messages by a Consciousness Replenishment Module (CRM) with a conditional mutual information estimator.
Revised: For better out-of-sight-range cooperation, the lowly contributed ones are compressed to brief messages by a Attention Replenishment Module (ARM) with a conditional mutual information estimator.

Original: Our CAMA outperforms the SOTA methods significantly on the challenging StarCraftII, MPE, and Traffic Junction benchmarks.
Revised: Our CAMA outperforms the SOTA methods significantly on the challenging StarCraftII, MPE, and Traffic Junction benchmarks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Cooperativeagent deep reinforce ment learning (MARL) has gained increasing attention in many areas such as games (Berner et al., 2019; Samvelyan et al., 2019; Kurach et al., 2019), social science (Jaques et al., 2019), sensor networks (Zhang & Lesser, 2013), and autonomous vehicle control (Xu et al., 2018).
Revised: Cooperative reinforce ment learning (MARL) has gained increasing attention in many areas such as games (Berner et al., 2019; Samvelyan et al., 2019; Kurach et al., 2019), social science (Jaques et al., 2019), sensor networks (Zhang & Lesser, 2013), and autonomous vehicle control (Xu et al., 2018).

Original: With practical agent cooperation and scalable deployment capability, centralized training with decentralized execution (CTDE) (Rashid et al., 2018; Gupta et al., 2017) has been widely adopted for
Revised: With practical agent cooperation and scalable deployment capability, centralized training with decentralized execution (CTDE) (Rashid et al., 2018; Gupta et al., 2017) has been widely adopted for

Original: MARL.
Revised: MARL.

Original: Current CTDE methods usually assume a fixed number of agents such as QMIX (Rashid et al., 2018)
Revised: Current CTDE methods usually assume a fixed number of agents such as QMIX (Rashid et al., 2018)

Original: , MADDPG (Lowe et al., 2017), QPLEX (Wang et al., 2020a), etc.
Revised: , MADDPG (Lowe et al., 2017), QPLEX (Wang et al., 2020a), etc.

Original: To adapt to complicated and dynamic real-world scenarios with dynamic team compositions (i.e., the team size varies), researchers extend these methods by introducing the attention mechanism (Vaswani et al., 2017), which usually requires splitting the state of the environment into a series of entities (Yang et al., 2020;
Revised: To adapt to complicated and dynamic real-world scenarios with dynamic team compositions (i.e., the team size varies), researchers extend these methods by introducing the attention mechanism (Vaswani et al., 2017), which usually requires splitting the state of the environment into a series of entities (Yang et al., 2020;

Original: Agarwal et al., 2019; Iqbal et al., 2021).
Revised: Agarwal et al., 2019; Iqbal et al., 2021).

Original: However, attention-based methods can hardly handle the varying partial observability (e.g., the varying sight range of each agent) in multi-agent systems, Fig. 1.
Revised: However, attention-based methods can hardly handle the varying partial observability (e.g., the varying sight range of each agent) in multi-agent systems, Fig. 1.

Original: With severe partial observability, agents usually lose the sight of teammates, leading to the poor coordination quality.
Revised: With severe partial observability, agents usually lose the sight of teammates, leading to the poor coordination quality.

Original: We use a demo in Sec.
Revised: We use a demo in Sec.

Original: 5.1 to verify the phenomenon.
Revised: 5.1 to verify the phenomenon.

Original: With slight partial observability (large sight ranges with near perfect information), these methods exhibit apparent performance degradation that more trivial entities may distract the agents’ attention and interfere with their decision making.
Revised: With slight partial observability (large sight ranges with near perfect information), these methods exhibit apparent performance degradation that more trivial entities may distract the agents’ attention and interfere with their decision making.

Original: See Sec.4.
Revised: See Sec.

Original:  for a detailed analysis.
Revised: 4.1 for a detailed analysis.

Original: Therefore, maintaining agents’ consciousness on potential cooperators and execution-related entities to adapt to the partial observability variation is crucial for MARL in challenging environments.
Revised: Therefore, maintaining agents’ attention on potential cooperators and executionrelated entities to adapt to the partial observability variation is crucial for MARL in challenging environments.

Original: In this paper, we propose a Consciousness-Aware Multi-Agent reinforcement learning (CAMA) approach, which roots in a divide-and-conquer strategy to facilitate stable and sustainable teamwork via consciousness learning.
Revised: In this paper, we propose a Concentrated Attention for Multi-Agent reinforcement learning (CAMA) approach, which roots in a divide-and-conquer strategy to facilitate stable and sustainable teamwork via attention learning.

Original: Specifically, we first  divide the raw entities into two parts  according to their execution relevance with an Entity Dividing Module (EDM).
Revised: Specifically, we first use an Entity Dividing Module (EDM) to divide the raw entities into two parts for each agent according to its attention weights.

Original: For the attention distraction issue in settings with large sight ranges, a Consciousness Enhancement Module (CEM) is applied on thehighly related entities  for consciousness-aware representation extraction via action prediction with an inverse model.
Revised: For the attention distraction issue in settings with large sight ranges, an Attention Enhancement Module (AEM) is applied on the entities with high attention weights for execution-related representation extraction via action prediction with an inverse model.

Original: For out-of-sight-range coordination in low sight ranges, a Consciousness
Revised: For out-of-sight-range coordination in low sight ranges, an Attention

Original: Replenishment Module (CRM) with a novel conditional mutual information estimator is applied to compress the information in the lowly related entities.
Revised: Replenishment Module (ARM) with a novel conditional mutual information estimator is applied to compress the information in entities with low attention weights.

Original: With the above three modules, CAMA empowers agents with purposeful consciousness to deal with dynamic partial observability.
Revised: With the above three modules, agents’ attention can be properly concentrated on execution of local actions and potential teamwork to deal with dynamic partial observability.

Original: We evaluate our method on three commonly used benchmarks: StarCraftII (SC2) (Samvelyan et al., 2019), Multi-agent Particle Environment (MPE) (Lowe et al., 2017), and Traffic Junction (Sukhbaatar et al., 2016).
Revised: We evaluate our method on three commonly used benchmarks: StarCraftII (SC2) (Samvelyan et al., 2019), Multi-agent Particle Environment (MPE) (Lowe et al., 2017), and Traffic Junction (Sukhbaatar et al., 2016).

Original: The proposed CAMA outperforms SOTA methods significantly on all conducted experiments and exhibits remarkable robustness to sight range variation and dynamic team composition.
Revised: The proposed CAMA outperforms SOTA methods significantly on all conducted experiments and exhibits remarkable robustness to sight range variation and dynamic team composition.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: As a popular paradigm for single reward MARL, CTDE is a trade-off between independent Qlearning (Tan, 1993) and joint action learning (Claus & Boutilier, 1998).
Revised: As a popular paradigm for single reward MARL, CTDE is a trade-off between independent Qlearning (Tan, 1993) and joint action learning (Claus & Boutilier, 1998).

Original: Centralized training makes agents cooperate better while decentralized execution benefits the flexible deployment capability.
Revised: Centralized training makes agents cooperate better while decentralized execution benefits the flexible deployment capability.

Original: A series of works concentrate on distributing the team reward to all agents by value function factorization (Sunehag et al., 2018;
Revised: A series of works concentrate on distributing the team reward to all agents by value function factorization (Sunehag et al., 2018;

Original: Rashid et al., 2018), deriving and extending the Individual-GlobalMax (IGM) principle for policy optimality analysis (Son et al., 2019; Wang et al., 2020a; Rashid et al., 2020; Wan et al., 2021).
Revised: Rashid et al., 2018), deriving and extending the Individual-GlobalMax (IGM) principle for policy optimality analysis (Son et al., 2019; Wang et al., 2020a; Rashid et al., 2020; Wan et al., 2021).

Original: To avoid constraints of IGM, some works then delve into applying centralized critics on local policies using the actor-critic paradigm (Lowe et al., 2017; Foerster et al., 2018;
Revised: To avoid constraints of IGM, some works then delve into applying centralized critics on local policies using the actor-critic paradigm (Lowe et al., 2017; Foerster et al., 2018;

Original: Zhou et al., 2020).
Revised: Zhou et al., 2020).

Original: Although CTDEs have achieved great progresses in recent years, with fixed sizes of agents, they are typically impeded by the dynamic team composition issue (Schroeder de
Revised: Although CTDEs have achieved great progresses in recent years, with fixed sizes of agents, they are typically impeded by the dynamic team composition issue (Schroeder de

Original: Witt et al., 2019; Liu et al., 2021) in real-world applications.
Revised: Witt et al., 2019; Liu et al., 2021) in real-world applications.

Original: Dynamic Team Composition .
Revised: Dynamic Team Composition .

Original: When the agent number varies in each episode, the attention mechanism (Vaswani et al., 2017) is commonly adopted to handle the issue (Jiang et al., 2018; Agarwal et al., 2019; Yang et al., 2020;
Revised: When the agent number varies in each episode, the attention mechanism (Vaswani et al., 2017) is commonly adopted to handle the issue (Jiang et al., 2018; Agarwal et al., 2019; Yang et al., 2020;

Original: Hu et al., 2020; Iqbal et al., 2021).
Revised: Hu et al., 2020; Iqbal et al., 2021).

Original: Some works develop a set of curricula to adapt to the increasing team sizes (Baker et al., 2019;
Revised: Some works develop a set of curricula to adapt to the increasing team sizes (Baker et al., 2019;

Original: Long et al., 2020; Wang et al., 2020c) with non-negligible computational costs for training on different team sizes.
Revised: Long et al., 2020; Wang et al., 2020c) with non-negligible computational costs for training on different team sizes.

Original: Iqbal et al.
Revised: Iqbal et al.

Original: (2021) add auxiliary Q-learning tasks to increase the multi-agent system’s robustness by randomly masking out part of agents’ observability, which increases the types of situations encountered by agents.
Revised: (2021) add auxiliary Q-learning tasks to increase the multi-agent system’s robustness by randomly masking out part of agents’ observability, which increases the types of situations encountered by agents.

Original: Although these methods adapt to different team sizes well, they still suffer obvious attention distraction when the sight ranges of agents is large, and are prone to fail in some situations where agents with limited observability must cooperate beyond their sight ranges (Liu et al., 2021).
Revised: Although these methods adapt to different team sizes well, they still suffer obvious attention distraction when the sight ranges of agents is large, and are prone to fail in some situations where agents with limited observability must cooperate beyond their sight ranges (Liu et al., 2021).

Original: Communication Mechanism is a feasible solution to enhance agents’ cooperation.
Revised: Communication Mechanism is a feasible solution to enhance agents’ cooperation.

Original: Recently some works regard the relationship between agents as a proximity-based or fully-connected graph and assume the information can propagate among the graph edges (Foerster et al., 2016; Suttle et al., 2020; Agarwal et al., 2019; Zhang et al., 2018; Sukhbaatar et al., 2016; Liu et al., 2020; Mao et al., 2020a).
Revised: Recently some works regard the relationship between agents as a proximity-based or fully-connected graph and assume the information can propagate among the graph edges (Foerster et al., 2016; Suttle et al., 2020; Agarwal et al., 2019; Zhang et al., 2018; Sukhbaatar et al., 2016; Liu et al., 2020; Mao et al., 2020a).

Original: These methods usually let agents communicate with all neighbors or the whole team, which brings high communication costs.
Revised: These methods usually let agents communicate with all neighbors or the whole team, which brings high communication costs.

Original: Moreover, the communication mode is sensitive to team sizes.
Revised: Moreover, the communication mode is sensitive to team sizes.

Original: Some works assume the existence of a centralized coach to integrate information and send messages to all agents (Liu et al., 2021; Mao et al., 2020b; Niu et al., 2021), which requires centralized execution, with relatively low communication costs.
Revised: Some works assume the existence of a centralized coach to integrate information and send messages to all agents (Liu et al., 2021; Mao et al., 2020b; Niu et al., 2021), which requires centralized execution, with relatively low communication costs.

Original: The communication messages are usually trained by backpropagation of RL loss, sometimes with constraint from mutual information objective to reduce the communication bandwidth (Wang et al., 2020b), help action decision (Yuan et al., 2022) or predict future trajectories (Liu et al., 2021).
Revised: The communication messages are usually trained by backpropagation of RL loss, sometimes with constraint from mutual information objective to reduce the communication bandwidth (Wang et al., 2020b), help action decision (Yuan et al., 2022) or predict future trajectories (Liu et al., 2021).

Original: Unlike these methods, we use communication from a centralized coach to apply consciousness replenishment for agents’ better coordination.
Revised: Unlike these methods, we use communication from a centralized coach to apply attention replenishment for agents’ better coordination.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: MARL Symbols.
Revised: MARL Symbols.

Original: We model a fully collaborative multi-agent task with n agents as a decentralised partially observable Markov decision process (Dec-POMDP) (Oliehoek et al., 2016)
Revised: We model a fully collaborative multi-agent task with n agents as a decentralised partially observable Markov decision process (Dec-POMDP) (Oliehoek et al., 2016)

Original: G = ⟨ S, A, I, P, r, Z, O, n, γ ⟩ , where s ∈ S is the environment’s state.
Revised: G = ⟨ S, A, I, P, r, Z, O, n, γ ⟩ , where s ∈ S is the environment’s state.

Original: At time step t , each agent i ∈ I a ≡ { 1 , ..., n a } chooses an action a i ∈ A , which makes up the joint action
Revised: At time step t , each agent i ∈ I a ≡ { 1 , ..., n a } chooses an action a i ∈ A , which makes up the joint action

Original: a ∈
Revised: a ∈

Original: A ≡ A n a .
Revised: A ≡ A n a .

Original: P ( s t +1 | s t , a t ) : S ×
Revised: P ( s t +1 | s t , a t ) : S ×

Original: A × S → [0 , 1] is the environment’s state transition distribution.
Revised: A × S → [0 , 1] is the environment’s state transition distribution.

Original: All agents share the same reward function r ( s, a ) : S ×
Revised: All agents share the same reward function r ( s, a ) : S ×

Original: A → R .
Revised: A → R .

Original: The discount factor is denoted by γ ∈ [0 , 1) .
Revised: The discount factor is denoted by γ ∈ [0 , 1) .

Original: Each agent i has its local observations o
Revised: Each agent i has its local observations o

Original: i ∈
Revised: i ∈

Original: O drawn from the observation function Z ( s, i ) :
Revised: O drawn from the observation function Z ( s, i ) :

Original: S × I → O and chooses an action by its stochastic policy π i ( a i | ρ i , χ i ) :
Revised: S × I → O and chooses an action by its stochastic policy π i ( a i | ρ i , χ i ) :

Original: Γ × X → ∆([0 , 1] | A | ) , where ρ i ∈ Γ ≡ ( O × A ) l denotes the action-observation history of agent i , and l is the number of stateaction pairs in ρ i .
Revised: Γ × X → ∆([0 , 1] | A | ) , where ρ i ∈ Γ ≡ ( O × A ) l denotes the action-observation history of agent i , and l is the number of stateaction pairs in ρ i .

Original: ρ is the action-observation histories of all agents.
Revised: ρ is the action-observation histories of all agents.

Original: χ
Revised: χ

Original: i ∈
Revised: i ∈

Original: X denotes the additional communication message and π i has no dependence on χ
Revised: X denotes the additional communication message and π i has no dependence on χ

Original: i in CTDE.
Revised: i in CTDE.

Original: The agents’ joint policy π induces a joint action-value function : Q π ( s t , a t ) =
Revised: The agents’ joint policy π induces a joint action-value function : Q π ( s t , a t ) =

Original: E s t +1: ∞ , a t +1: ∞
Revised: E s t +1: ∞ , a t +1: ∞

Original: [ R t | s t , a t ] , where R t = (cid:80) ∞ k =0 γ k r t + k is the discounted accumulated team reward.
Revised: [ R t | s t , a t ] , where R t = (cid:80) ∞ k =0 γ k r t + k is the discounted accumulated team reward.

Original: The goal of MARL is to find the optimal joint policy π ∗ such that Q π ∗ ( s, a ) ≥ Q π ( s, a ) , for all π and ( s, a ) ∈ S × A .
Revised: The goal of MARL is to find the optimal joint policy π ∗ such that Q π ∗ ( s, a ) ≥ Q π ( s

Original: Value Function Factorization.
Revised: Value Function Factorization.

Original: During execution in CTDE, each agent chooses actions by its local
Revised: During execution in CTDE, each agent chooses actions by its local

Original: Q function Q i ( ρ i ) induced by local observation o i .
Revised: Q function Q i ( ρ i ) induced by local observation o i .

Original: During training, since all agents share a common team reward, a global Q function Q tot is calculated from all local Q i conditioned on the global state s by a Mixer module as: Q tot ( ρ , a ) =
Revised: During training, since all agents share a common team reward, a global Q function Q tot is calculated from all local Q i conditioned on the global state s by a Mixer module as: Q tot ( ρ , a ) =

Original: g ( Q 1 ( ρ 1 , a 1 ) , ..., Q n ( ρ n , a n ) , s ) .
Revised: g ( Q 1 ( ρ 1 , a 1 ) , ..., Q n ( ρ n , a n ) , s ) .

Original: To guarantee the global optimality from the local optimality, a Mixer should satisfy the IGM principle (Son et al., 2019): argmax a Q tot ( ρ , a ) = (arg
Revised: To guarantee the global optimality from the local optimality, a Mixer should satisfy the IGM principle (Son et al., 2019): arg

Original: [Equation]
Revised: [Equation]

Original: Q 1 ( ρ 1 , a 1 ) , ..., arg
Revised: tot [Equation] [Equation] arg

Original: [Equation]
Revised: [Equation]

Original: Q n (
Revised: [Equation] [Equation] , arg

Original: [Equation]
Revised: [Equation]

Original: n,
Revised: n

Original: [Equation]
Revised: [Equation]

Original: n )) .
Revised: [Equation]

Original: Multi-Head Attention (MHA) in MARL.
Revised: Multi-Head Attention (MHA) in MARL.

Original: Under the condition of dynamic teams, the observation vector for each agent may have varying sizes during one episode, and therefore we can hardly apply traditional methods which only accept fixed-size input.
Revised: Under the condition of dynamic teams, the observation vector for each agent may have varying sizes during one episode, and therefore we can hardly apply traditional methods which only accept fixed-size input.

Original: In contrast, we use the “entity-wise input” to represent the observation and the multi-head attention (MHA) module to embed the entities with dynamic number into a fixed length vector for each agent.
Revised: In contrast, we use the “entity-wise input” to represent the observation and the multi-head attention (MHA) module to embed the entities with dynamic number into a fixed length vector for each agent.

Original: To feed MHA, the raw state s of the environment is commonly expressed as a series of entities e i , i.e., s e := { e i } , i ∈
Revised: To feed MHA, the raw state s of the environment is commonly expressed as a series of entities e i , i.e., s e := { e i } , i ∈

Original: [1 , n e ] with the same vector length, where n e is the maximum number of entities.
Revised: [1 , n e ] with the same vector length, where n e is the maximum number of entities.

Original: The entities include agents we can or can not control, and other substrates in the multi-agent scenario (e.g. obstacles).
Revised: The entities include agents we can or can not control, and other substrates in the multi-agent scenario (e.g. obstacles).

Revised: We consider a proximity-based observation function Z ( s e , i ) for agent i , i.e., each agent has a sight range SR , and Z ( s e , i ) :=

Revised: { e j | d ( i, j ) < = SR } , j ∈ [1 , n e ] , where d ( i, j ) is the Euclidean (or Manhattan) distance between entity i and j .

Original: Let X ∈ R n e × d be the entity input, in which each row is an entity.
Revised: Let X ∈ R n e × d be the entity input, in which each row is an entity.

Original: Let I a ⊆
Revised: Let I a ⊆

Original: I := { 1 , ..., n e } be the set of indices that selects which entities of the input X are used to compute queries such that X I a ∈ R n a × d (usually i ∈ I a means entity
Revised: I := { 1 , ..., n e } be the set of indices that selects which entities of the input X are used to compute queries such that X I a ∈ R n a × d (usually i ∈ I a means entity

Original: i is an controllable agent, and I a := { 1 , ..., n a } ).
Revised: i is an controllable agent, and I a := { 1 , ..., n a } ).

Original: The attention head is as follows:
Revised: The attention head is as follows:

Original: [Equation]
Revised: [Equation]

Original: The mask( Y , M ) operation takes two matrices with the same size as input, and fills the entries of Y with −∞ where M equals 0 .
Revised: The mask( Y , M ) operation takes two matrices with the same size as input, and fills the entries of Y with −∞ where M equals 0 .

Original: If we set the values in the positions of unseen entities to 0 , this operation blocks the information from certain entities after softmax, to uphold the partial observability for local agents.
Revised: If we set the values in the positions of unseen entities to 0 , this operation blocks the information from certain entities after softmax, to uphold the partial observability for local agents.

Original: W Q , W K , and W V are learnable parameters.
Revised: W Q , W K , and W V are learnable parameters.

Original: Then we can define the mulit-head attention module by concatenating n h attention heads together:
Revised: Then we can define the mulit-head attention module by concatenating n h attention heads together:

Original: [Equation]
Revised: [Equation]

Original: where information not blocked can be integrated across all entities.
Revised: where information not blocked can be integrated across all entities.

Original:  M ETHOD 4. I NTUITION AND O VERVIEW
Revised:  M ETHOD 4. I NTUITION AND O VERVIEW

Original: Intuition.
Revised: Intuition.

Original: The performance of traditional MARL methods is highly affected by the partial observability of the environment.
Revised: The performance of traditional MARL methods is highly affected by the partial observability of the environment.

Original: We use the agents in a game (e.g., StarCraftII) for example.
Revised: We use the agents in a game (e g., StarCraftII) for example.

Original: When the sight range is small, the agents can hardly find the teammates and support them, leading to poor team coordination (a demo in Sec.
Revised: When the sight range is small, the agents can hardly find the teammates and support them, leading to poor team coordination (a demo in Sec.

Original: 5.1 verifies this hypothesis).
Revised: 5.1 verifies this hypothesis).

Original: However, it is counterintuitive that with increasing sight ranges, the agents’ performance typically degrades (this phenomenon is detailed in Sec. 5.1).
Revised: However, it is counterintuitive that with increasing sight ranges, the agents’ performance typically degrades (this phenomenon is detailed in Sec. 5.1).

Original: We argue that the agents’ attention is easily distracted by unrelated scenarios, causing the attention distraction issue.
Revised: We argue that the agents’ attention is easily distracted by unrelated scenarios, causing the attention distraction issue.

Original: To reveal the importance of agents’ attention on their performance, we train the SOTA algorithm REFIL (Iqbal et al., 2021) on different sight ranges (3,9, and ∞ ) and visualize the agents’ attention weights on all entities (i e., the value of the matrix QK ⊤ in MHA module) in Fig.
Revised: To reveal the importance of agents’ attention on their performance, we train the SOTA algorithm REFIL (Iqbal et al., 2021) on different sight ranges (3, and ∞ ) and visualize the agents’ attention weights on all entities (i e., the value of the matrix QK ⊤ in MHA module) in Fig.

Original: It can be seen that when more entities are visible, the agents’ attentions get more dispersed so that more difficulties need to be overcome to win the preset AI.
Revised: It can be seen that when more entities are visible, the agents’ attentions get more dispersed so that more difficulties need to be overcome to win the preset AI.

Original: Existing dynamic team
Revised: Existing dynamic team

Original: MARL methods commonly face such problem (see Appendix E)
Revised: MARL methods commonly face such problem (see Appendix E).

Original: To deal with the dilemma of the dynamic partial observability in MARL, we resort to a divideand-conquerconsciousness-aware learning strategy, dubbed CAMA, for stable performance and sustainable teamwork.
Revised: To deal with the dilemma of the dynamic partial observability in MARL, we resort to a divideand-conquer learning strategy, dubbed CAMA, for stable performance and sustainable teamwork.

Original: Concretely, in low sight ranges we need to improve agents’ consciousness on realizing the potential cooperators by out-of-sight-range information, while in large sight ranges, agents’consciousness of attention concentration on execution-related entities should be kept.
Revised: Concretely, in low sight ranges we need to improve agents’ attention on realizing the potential cooperators by out-of-sight-range information, while in large sight ranges, agents’ attention concentration on execution-related entities should be kept.

Original: Framework and RL Training.
Revised: Framework and RL Training.

Original: CAMA mainly consists of three components, including an EntityDividing Module (EDM) for dynamic partial observability control, a Consciousness Enhancement Module (CEM) for attention concentration on execution-related entities, and a Consciousness Replenishment Module (CRM) for agents’ out of sight range coordination, Fig. 2.
Revised: CAMA mainly consists of three components, including an EntityDividing Module (EDM) for dynamic partial observability control, an Attention Enhancement Module (AEM) for attention concentration on execution-related entities, and an Attention Replenishment Module (ARM) for agents’ out of sight range coordination, Fig. 2.

Original: Specifically, for each agent i , EDM divides and embeds the raw entities as f i and f − i to feed CEM and CRM respectively.
Revised: Specifically, for each agent i , EDM divides and embeds the raw entities as f i and f − i to feed AEM and ARM respectively.

Original: For CEM, an inverse model is applied to resolve the attention distraction issue.
Revised: For AEM, an inverse model is applied to resolve the attention distraction issue.

Original: For CRM, a coach with global sights is introduced to generate a communication message ζ i for agents’ coordination.
Revised: For ARM, a coach with global sights is introduced to generate a communication message ζ i for agents’ coordination.

Original: Following Yuan et al.
Revised: Following Yuan et al.

Original: (2022), we generate a Q i ( local ) from f i and agent i ’s observation-action history
Revised: (2022), we generate a Q i ( local ) from f i and agent i ’s observation-action history

Original: ρ
Revised: ρ

Original: i (which is the output of a gated recurrent unit (GRU) cell), and a Q i ( global ) from ζ i .
Revised: i (which is the output of a gated recurrent unit (GRU) cell), and a Q i ( global ) from ζ i .

Original: Q i is computed by their summation, i.e., Q i =
Revised: Q i is computed by their summation, i.e., Q i =

Original: Q i ( local ) + Q i ( global ) .
Revised: Q i ( local ) + Q i ( global ) .

Original: All local Q i s are fed into a mixing network (Yang et al., 2020) to calculate a Q tot .
Revised: All local Q i s are fed into a mixing network (Yang et al., 2020) to calculate a Q tot .

Original: The RL loss can be formulated as follows:
Revised: The RL loss can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: where ˆ Q tot is the target network, and D is the replay buffer.
Revised: where ˆ Q tot is the target network, and D is the replay buffer.

Original: 4. E NTITY D IVIDING M ODULE
Revised: 4. E NTITY D IVIDING M ODULE

Original: An EDM divides raw entities into a consciousness enhancement part and a consciousness replenishment part to adapt to the dynamic partial observability.
Revised: An EDM divides raw entities into an attention enhancement part and an attention replenishment part by their ranking of attention weights.

Original: For the former one, we wish to constrain the maximum of observed entities to avoid the attention distraction.
Revised: For the former one, we wish to constrain the maximum of observed entities to avoid the attention distraction.

Original: And the latter one should contain enough out-of-sight-range information for team coordination.
Revised: And the latter one should contain enough outof-sight-range information for team coordination.

Original: We first deal with the consciousness enhancement part.
Revised: We first deal with the attention enhancement part.

Original: Recall that in MHA module, M ∈ { 0 , 1 } n a × n e is a binary mask applied on the entity embeddings which is generated by the environment.
Revised: Recall that in MHA module, M ∈ { 0 , 1 } n a × n e is a binary mask applied on the entity embeddings which is generated by the environment.

Original: To uphold each agent’s partial observability, a more sparse mask M s is introduced to replace M which satisfies the following constraints:
Revised: To uphold each agent’s partial observability, a more sparse mask M s is introduced to replace M which satisfies the following constraints:

Original: [Equation]
Revised: [Equation]

Original: where n e is the maximum number of entities, α ∈ (0 , 1] is a hyper-parameter, and ⊙ means the element-wise multiplication operation.
Revised: where n e is the maximum number of entities, α ∈ (0 , 1] is a hyper-parameter, and ⊙ means the element-wise multiplication operation.

Original: The negation of M is defined as ¬M := 1 − M , whereis an all 1 matrix with the same shape as M .
Revised: The negation of M is defined as ¬M := 1 − M , whereis an all 1 matrix with the same shape as M .

Original: The left side of Eq.
Revised: The left side of Eq.

Original: (4) ensures that the percent of observable entities is less than α , while the right side makes the agent observe the entities available in the original mask M only.
Revised: (4) ensures that the percent of observable entities is less than α , while the right side makes the agent observe the entities available in the original mask M only.

Original: We can assign a low value for α (e.g., 0.4) to limit each agent’s visible entities in complicated environments.
Revised: We can assign a low value for α (e.g., 0.4) to limit each agent’s visible entities in complicated environments.

Original: To get M s , we define a B i ( W ) operation to get the indices of the top i values in each row of W .
Revised: To get M s , we define a B i ( W ) operation to get the indices of the top i values in each row of W .

Original: M s can be calculated as follows:
Revised: M s can be calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: where ⌊·⌋ means to round down the value, and Q , K are matrices of queries and keys in MHA module, respectively.
Revised: where ⌊·⌋ means to round down the value, and Q , K are matrices of queries and keys in MHA module, respectively.

Original: Under Eq.
Revised: Under Eq.

Original: (5), each agent only remains the sight on at most ⌊ αn e ⌋ entities with the highest attention weight, and improves itsconsciousness via attention concentration.
Revised: (5), each agent only remains the sight on at most ⌊ αn e ⌋ entities with the highest attention weight, and improves its attention concentration.

Original: We prove in Appendix A that M s obtained by Eq.
Revised: We prove in Appendix A that M s obtained by Eq.

Original: (5) is adequate for Eq.
Revised: (5) is adequate for Eq.

Original: After getting M s , we can compute f i for the consciousness enhancement part, which is the output of MHA( I, X , M s ) .
Revised: After getting M s , we can compute f i for the attention enhancement part, which is the output of MHA( I, X , M s ) .

Original: Since the consciousness replenishment part should contain all the information not involved in the former one for better out-of-sight-range coordination, we transmit it with f − i , which is the embedding of the complement entities of M s on s with the same MHA module as MHA( I, X , ¬M s ) .
Revised: Since the attention replenishment part should contain all the information not involved in the former one for better out-of-sight-range coordination, we transmit it with f − i , which is the embedding of the complement entities of M s on s with the same MHA module as MHA( I, X , ¬M s ) .

Original: 4. C ONSCIOUSNESS E NHANCEMENT FOR L OCAL A GENT
Revised: 4. A TTENTION E NHANCEMENT FOR L OCAL A GENT

Original: When knowing what will happen when a specific action is taken, the learned agents can hardly be distracted.
Revised: When knowing what will happen when a specific action is taken, the learned agents can hardly be distracted.

Original: Thereby, we aim to raise the agents’ consciousness of their states and ability by distilling execution-related information in the high-dimensional state space.
Revised: Thereby, we aim to concentrate agents’ attention on execution-related information distilled from the high-dimensional state space.

Original: Specifically, we resort to the inverse model (Pathak et al., 2017), a two-layer MLP, that uses the local observation o it and o it +1 to predict the agent i ’s action a i .
Revised: Specifically, we resort to the inverse model (Pathak et al., 2017), a two-layer MLP, that uses the local observation o it and o it +1 to predict the agent i ’s action a i .

Original: In the prediction module, the local observation o it and o it +1 are first fed into the same EDM to get the features f it and f it +1 .
Revised: In the prediction module, the local observation o it and o it +1 are first fed into the same EDM to get the features f it and f it +1 .

Original: Then, the probability of each action can be predicted as p (ˆ a it ) =
Revised: Then, the probability of each action can be predicted as p (ˆ a it ) =

Original: IM ( f it , f it +1 ; θ ) .
Revised: IM ( f it , f it +1 ; θ ) .

Original: The learning loss can be defined as:
Revised: The learning loss can be defined as:

Original: [Equation]
Revised: [Equation]

Original: where CE means the cross entropy.
Revised: where CE means the cross entropy.

Original: f i will contain the necessary information for predicting a i by optimizing Eq.
Revised: f i will contain the necessary information for predicting a i by optimizing Eq.

Original: (6), which encourages EDMs to discard irrelevant distracting information for the consciousness enhancement embedding f i .
Revised: (6), which encourages EDMs to discard irrelevant distracting information for the consciousness enhancement embedding f i .

Original: With the auxiliary representation learning task, the learned embedding f i can be used to calculate each agent’s local Q function Q i ( local ) .
Revised: With the auxiliary representation learning task, the learned embedding f i can be used to calculate each agent’s local Q function Q i ( local ) .

Original: 4. C ONSCIOUSNESS R EPLENISHMENT BY G LOBAL C OACH
Revised: 4. A TTENTION R EPLENISHMENT BY G LOBAL C OACH

Original: To equip agents with the ability of out-of-sight-range coordination, we use a centralized coach equipped with global states to generate a message ζ i from f − i and compute a Q i ( global ) for each agent i at each time step (Liu et al., 2021;
Revised: To equip agents with the ability of out-of-sight-range coordination, we use a centralized coach equipped with global states to generate a message ζ i from f − i and compute a Q i ( global ) for each agent i at each time step (Liu et al., 2021;

Original: Niu et al., 2021; Mao et al., 2020b).
Revised: Niu et al., 2021; Mao et al., 2020b).

Original: The message plays the role of consciousness replenishment.
Revised: The message plays the role of attention replenishment when agents facing difficulty to cooperate through their local observations.

Original: The learning objective for ζ i should contain the information unknown to agent i , and not distract the agent’s attention.
Revised: The learning objective for ζ i should contain the information unknown to agent i , and not distract the agent’s attention.

Original: Therefore, we maximize:
Revised: Therefore, we maximize:

Original: [Equation]
Revised: [Equation]

Original: where s is the global state, and I ( · ; · ) means the mutual information.
Revised: where s is the global state, and I ( · ; · ) means the mutual information.

Original: f − i indicates the replenishment information for agent i .
Revised: f − i indicates the replenishment information for agent i .

Original: We leave the derivation of Eq.
Revised: We leave the derivation of Eq.

Original: (7) to Appendix B. Maximizing I ( ζ i ; f − i ) let ζ i be the summary of f − i .
Revised: (7) to Appendix B. Maximizing I ( ζ i ; f − i ) let ζ i be the summary of f − i .

Original: By feeding ζ i to agent i , the agent can sense the information beyond its sight range, and further alleviate the difficulties of cooperation caused by partial observability.
Revised: By feeding ζ i to agent i , the agent can sense the information beyond its sight range, and further alleviate the difficulties of cooperation caused by partial observability.

Original: Minimizing I ( ζ i ; s ) compresses the information ζ i has, which can be regarded as an information bottleneck constraint on ζ i (Wang et al., 2020b).
Revised: Minimizing I ( ζ i ; s ) compresses the information ζ i has, which can be regarded as an information bottleneck constraint on ζ i (Wang et al., 2020b).

Original: We use a hyper-parameter β ∈ [0 , 1] to control the compression degree of ζ i .
Revised: We use a hyper-parameter β ∈ [0 , 1] to control the compression degree of ζ i .

Original: Combining the two terms, we (1) discard the information in f i , which is already known to agent i , and (2) compress the sophisticated f − i into a brief message, which can hardly distract the agent while promotes coordination.
Revised: Combining the two terms, we (1) discard the information in f i , which is already known to agent i , and (2) compress the sophisticated f − i into a brief message, which can hardly distract the agent while promotes coordination.

Original: We then separately optimize the two mutual information term in the right side of Eq.
Revised: We then separately optimize the two mutual information term in the right side of Eq.

Original: Directly maximizing I ( ζ i ; s ) is difficult, but there exist some tools estimating its differentiable lower bound, e.g. infoNCE (Oord et al., 2018) and MINE (Belghazi et al., 2018).
Revised: Directly maximizing I ( ζ i ; s ) is difficult, but there exist some tools estimating its differentiable lower bound, e.g. infoNCE (Oord et al., 2018) and MINE (Belghazi et al., 2018).

Original: We choose the CatGen formulation (Fischer, 2020) of the former one and maximize the following lower bound of I ( ζ i ; s ) :
Revised: We choose the CatGen formulation (Fischer, 2020) of the former one and maximize the following lower bound of I ( ζ i ; s ) :

Original: [Equation]
Revised: [Equation]

Original: where K is the sample number of a mini-batch.
Revised: where K is the sample number of a mini-batch.

Original: Then we move on to minimizing I ( ζ i ; f i | f − i ) .
Revised: Then we move on to minimizing I ( ζ i ; f i | f − i ) .

Original: There are existing tools that minimize the upper bound of the mutual information between two random variables, such as CLUB (Cheng et al., 2020) and L1Out (Poole et al., 2019).
Revised: There are existing tools that minimize the upper bound of the mutual information between two random variables, such as CLUB (Cheng et al., 2020) and L1Out (Poole et al., 2019).

Original: But these methods can not be directly applied to the conditional mutual information paradigm.
Revised: But these methods can not be directly applied to the conditional mutual information paradigm.

Original: Therefore, we extend the CLUB estimator to the conditional form and present Conditional-CLUB (CC) estimator as a differentiable upper bound of I ( ζ i ; f i | f − i ) :
Revised: Therefore, we extend the CLUB estimator to the conditional form and present Conditional-CLUB (CC) estimator as a differentiable upper bound of I ( ζ i ; f i | f − i ) :

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For three random variables ζ i , f i and f − i ,
Revised: For three random variables ζ i , f i and f − i ,

Original: [Equation]
Revised: [Equation]

Original: The equality holds if and only if f i is independent of the joint distribution of ζ i , f − i .
Revised: The equality holds if and only if f i is independent of the joint distribution of ζ i , f − i .

Original: The proof of Thm.
Revised: The proof of Thm.

Original: 4.1 can be referred in Appendix C. Accordingly, we can minimize the conditional mutual information via minimizing I CC .
Revised: 4.1 can be referred in Appendix C. Accordingly, we can minimize the conditional mutual information via minimizing I CC .

Original: In practice, assuming we have the conditional distribution p ( ζ i | f i , f − i ) , we can sample pairs { ( ζ ik , f ik , f − i k ) }
Revised: In practice, assuming we have the conditional distribution p ( ζ i | f i , f − i ) , we can sample pairs { ( ζ ik , f ik , f − i k ) }

Original: Kk =1 and get an unbiased estimation of I CC :
Revised: Kk =1 and get an unbiased estimation of I CC :

Original: [Equation]
Revised: [Equation]

Original: Since the second part of Eq.
Revised: Since the second part of Eq.

Original: (11) requires O (cid:0) K 2 (cid:1) computational complexity, we use the following faster counterpart:
Revised: (11) requires O (cid:0) K 2 (cid:1) computational complexity, we use the following faster counterpart:

Original: [Equation]
Revised: [Equation]

Original: where µ ( · ) is a mapping from k ∈ { 1 , .., K } to its random permutation.
Revised: where µ ( · ) is a mapping from k ∈ { 1 , .., K } to its random permutation.

Original: Since ˆ I CC and ¯ I CC are both unbiased estimators, E [ˆ I CC ] =
Revised: Since ˆ I CC and ¯ I CC are both unbiased estimators, E [ˆ I CC ] =

Original: E [¯ I CC ] = I CC .
Revised: E [¯ I CC ] = I CC .

Original: In practice, we assume p ( ζ i | f i , f − i ) as a Gaussian distribution.
Revised: In practice, we assume p ( ζ i | f i , f − i ) as a Gaussian distribution.

Original: We use a neural network with input concat( f i , f − i ) to calculate its mean and variance and optimize it with the reparameterization trick (Kingma & Welling, 2013).
Revised: We use a neural network with input concat( f i , f − i ) to calculate its mean and variance and optimize it with the reparameterization trick (Kingma & Welling, 2013).

Original: Since concat( f i , f − i ) has involved all the information in the global state s , we can use the same distribution to represent p ( ζ i | s ) in ¯ I NCE and therefore optimizing L MI =
Revised: Since concat( f i , f − i ) has involved all the information in the global state s , we can use the same distribution to represent p ( ζ i | s ) in ¯ I NCE and therefore optimizing L MI =

Original: ¯ I CC − (1 − β )¯ I NCE .
Revised: ¯ I CC − (1 − β )¯ I NCE .

Original: If we need a coach with stronger capability by e.g., introducing the memory module, we will get the prior distribution p ( ζ i | ρ ) instead of p ( ζ i | f i , f − i ) , therefore we can not estimate ¯ I CC directly.
Revised: If we need a coach with stronger capability by e.g., introducing the memory module, we will get the prior distribution p ( ζ i | ρ ) instead of p ( ζ i | f i , f − i ) , therefore we can not estimate ¯ I CC directly.

Original: In Appendix D we introduce a variational distribution to estimate p ( ζ i | f i , f − i ) when it is unknown.E XPERIMENTS
Revised: In Appendix D we introduce a variational distribution to estimate p ( ζ i | f i , f − i ) when it is unknown.

Original: In this section we first show the cooperation problem/the attention distraction issue in low/large sight ranges in Sec. 5.1.

Original: Then we analyze each part of our method in Sec. 5.2.

Original: Finally we show our method’s performance in hard SC2 tasks in Sec.

Original: All experiments are repeated for 6 seeds.

Original: The results in figures and tables are presented by the form “mean±std”.

Original:   5. D ILEMMA IN S IGHT R ANGES
Revised:  E XPERIMENTS 5. D ILEMMA IN S IGHT R ANGES

Original: Coordination in Low Sight Ranges.
Revised: Coordination in Low Sight Ranges.

Original: We first show the defect of CTDE methods with a demo.
Revised: We first show the defect of CTDE methods with a demo.

Original: In a 11 × 11 grid-world, an apple is uniformly generated at one grid of the 3 × 3 grids in the map center, with obstacles of length 4 at its right&left or up&down, Fig. 4(b).
Revised: In a 11 × 11 grid-world, an apple is uniformly generated at one grid of the 3 × 3 grids in the map center, with obstacles of length 4 at its right&left or up&down, Fig. 4(b).

Original: 2 controllable agents are initialized at random places.
Revised: 2 controllable agents are initialized at random places.

Original: They can move towards 4 directions, one grid at a time.
Revised: They can move towards 4 directions, one grid at a time.

Original: The goal is that two agents should arrive the apple simultaneously, rewarded 10.
Revised: The goal is that two agents should arrive the apple simultaneously, rewarded 10.

Original: Any agent touching the apple or team reaching the episode time limit will end the game.
Revised: Any agent touching the apple or team reaching the episode time limit will end the game.

Original: The reward of one agent touching apple is 1 .
Revised: The reward of one agent touching apple is 1 .

Original: The reward for time penalty is − 0 .
Revised: The reward for time penalty is − 0 .

Original: The agents can only see the entities in the 3 × 3 grids, each centered on its position.
Revised: The agents can only see the entities in the 3 × 3 grids, each centered on its position.

Original: Agents can not overlap, which means the only way to achieve the team goal is the two agents moving towards the apple from the opposite sides of the obstacles, at which time they can not see each other.
Revised: Agents can not overlap, which means the only way to achieve the team goal is the two agents moving towards the apple from the opposite sides of the obstacles, at which time they can not see each other.

Revised: We train two CTDE

Original: We train two CTDE methods: Qatten (Yang et al., 2020) andREFIL (Iqbal et al., 2021), and two communication-based methods: our CAMA and MAIC (Yuan et al., 2022) 
Revised: : Qatten (Yang et al., ) and (Iqbal et ., 2021), and two communication-based methods: our CAMA and MAIC (Yuan et al., 2022) (with a global coach, see Sec.

Original: (with a global coach, see Sec.

Original: 5.2 for details).
Revised: 5.2 for details).

Original: The target achieving rate during training are shown in Fig. 4(c).
Revised: The target achieving rate during training are shown in Fig. 4(c).

Original: Only methods with communication can solve the task while CTDE methods fail completely.
Revised: Only methods with communication can solve the task while CTDE methods fail completely.

Original: To validate the effect of communication, we use t-SNE (Van der Maaten & Hinton, 2008) to visualize the messages ζ i from 160 testing episodes of our method in 3D-space.
Revised: To validate the effect of communication, we use t-SNE (Van der Maaten & Hinton, 2008) to visualize the messages ζ i from 160 testing episodes of our method in 3D-space.

Original: We show the results of the last time step of each episode in Fig. 4(d).
Revised: We show the results of the last time step of each episode in Fig. 4(d).

Original: Each point is a message and the color means the agent’s action after it receives the message.
Revised: Each point is a message and the color means the agent’s action after it receives the message.

Original: The results show that agents’ actions have obvious correlation with messages received, which shows the importance of communication in out-of-sight coordination.
Revised: The results show that agents’ actions have obvious correlation with messages received, which shows the importance of communication in out-of-sight coordination.

Original: [Table]
Revised: [Table]

Original: Attention Distraction in Large Sight Ranges.
Revised: Attention Distraction in Large Sight Ranges.

Original: We use the Resource Collection (RC) task (Fig. 4(a)) to show the attention concentration effect of CAMA.
Revised: We use the Resource Collection (RC) task (Fig. 4(a)) to show the attention concentration effect of CAMA.

Original: In this task, a group of 3-8 agents coordinate to collection resources from various places and bring them home.
Revised: In this task, a group of 3-8 agents coordinate to collection resources from various places and bring them home.

Original: It is slightly modified from Liu et al.
Revised: It is slightly modified from Liu et al.

Original: (2021) to make it suitable for both CTDE and communication-based settings.
Revised: (2021) to make it suitable for both CTDE and communication-based settings.

Original: Please refer to
Revised: Please refer to

Original: Appendix G.2 for a detailed description.
Revised: Appendix G.2 for a detailed description.

Original: We test the performance of the following methods on three sight ranges in Fig.
Revised: We test the performance of the following methods on three sight ranges in Fig.

Original: 5: (1) 2 CTDE methods: Qatten (Yang et al., 2020), REFIL (Iqbal et al., 2021).
Revised: 5: (1) 2 CTDE methods: Qatten (Yang et al., 2020), REFIL (Iqbal et al., 2021).

Original: (2) 1 graph communication method: EMP (Agarwal et al., 2019).
Revised: (2) 1 graph communication method: EMP (Agarwal et al., 2019).

Original: (3) 4 centralized communication methods: MAGIC (Niu et al., 2021), Gated_Qatten (Mao et al., 2020b), COPA (Liu et al., 2021), and our CAMA.
Revised: (3) 4 centralized communication methods: MAGIC (Niu et al., 2021), Gated_Qatten (Mao et al., 2020b), COPA (Liu et al., 2021), and our CAMA.

Original: MAGIC and Gated_Qatten are modified to make them suitable for the entity-wise input setting.
Revised: MAGIC and Gated_Qatten are modified to make them suitable for the entity-wise input setting.

Original: The results show that CAMA outperforms all baselines in all sight range settings.
Revised: The results show that CAMA outperforms all baselines in all sight range settings.

Original: And only CAMA performs better in SR = 1 .
Revised: And only CAMA performs better in SR = 1 .

Original: 5 than SR = 1 . 0 and SR
Revised: 5 than SR = 1 . 0 and SR

Original: = 0 .
Revised: = 0 .

Original: 5 , while others suffer from attention distraction and perform worse in larger sight range.
Revised: 5 , while others suffer from attention distraction and perform worse in larger sight range.

Original: 5. C OMPONENT A NALYSIS
Revised: 5. C OMPONENT A NALYSIS

Original: Entity Dividing Module.
Revised: Entity Dividing Module.

Original: In EDM, the parameter α plays theimportant role of balancing the observability between CEM and CRM.
Revised: In EDM, the parameter α plays the role of balancing the observability between AEM and ARM.

Original: We explore how to choose α in different sight ranges in Fig.
Revised: We explore how to choose α in different sight ranges in Fig.

Original: To make the error bars (std) more clear, points with the same α are slightly offset on the X axis. 
Revised: To make the error bars (std) more clear, points with the same α are slightly offset on the X axis. Note that α = 1 .means

Revised: no constraint on observability function, i.e., deleting EDM.

Original: We find that as the sight range increases, the agents can see more entities, and therefore a lower α can help attention concentration.
Revised: We find that as the sight range increases, the agents can see more entities, and therefore a lower α can help attention concentration.

Original: [Table]
Revised: [Table]

Original: Consciousness Enhancement.
Revised: Attention Enhancement.

Original: We analyze the contribution of each loss in Table 1.
Revised: We analyze the contribution of each loss in Table 1.

Original: L IM  is important when SR is large, which is reasonable since large SR brings attention distraction issue and requires agents to focus attention on execution-related entities teaser (SR=1). Method MAIC&NDQ
Revised: L IM (AEM) is important when SR is large, which is reasonable since large SR brings attention distraction issue and requires agents to focus attention on execution-related entities [Table]

Original: [Table]
Revised: [Table]

Original: Consciousness Replenishment.
Revised: Attention Replenishment.

Original: Table 1 shows that L MI  can bring obvious improvement under both SR conditions.
Revised: Table 1 shows that L MI (ARM) can bring obvious improvement under both SR conditions.

Original: To check whether the improvement comes from the communication mechanism or mutual information objective, we turn to a grid-world based traffic junction environment (TJ).
Revised: To check whether the improvement comes from the communication mechanism or mutual information objective, we turn to a grid-world based traffic junction environment (TJ).

Original: We use the map from Sukhbaatar et al.
Revised: We use the map from Sukhbaatar et al.

Original: (2016), which is a crossroads where cars are continuously generated from a Poisson distribution at one of the four entrances [Table] and aiming for one of the other three exits (Fig. 7).
Revised: (2016), which is a crossroads where cars are continuously generated from a Poisson distribution at one of the four entrances, and aiming for one of the other three exits (Fig. 7).

Original: Unlike the original simplified setting that the routes are fixed so that agents only choose to accelerate or brake, we use a more difficult setting that agents can move towards four directions freely [Table] which is harder and more realistic.
Revised: Unlike the original simplified setting that the routes are fixed so that agents only choose to accelerate or brake, we use a more difficult setting that agents can move towards four directions freely, which is harder and more realistic.

Original: Please refer to Appendix G.2 for environment details.
Revised: Please refer to Appendix G.2 for environment details.

Original: We compare four MI-based message generators under our entity-wise setting: MAIC&NDQ (Yuan et al., 2022; Wang et al., 2019), IMAC (Wang et al., 2020b), COPA (Liu et al., 2021) and our CAMA.
Revised: We compare four MI-based message generators under our entity-wise setting: MAIC&NDQ (Yuan et al., 2022; Wang et al., 2019), IMAC (Wang et al., 2020b), COPA (Liu et al., 2021) and our CAMA.

Original: To make a fair comparison on MI objective, we only implement the mutual information part of each method and keep the others same to ours, e.g., the structure of neural networks and communication mechanism.
Revised: To make a fair comparison on MI objective, we only implement the mutual information part of each method and keep the others same to ours, e.g., the structure of neural networks and communication mechanism.

Original: Since MAIC and NDQ use similar MI objectives under the centralized coach setting, we regard them as one method.
Revised: Since MAIC and NDQ use similar MI objectives under the centralized coach setting, we regard them as one method.

Original: The test return results are shown in Table 3, which exhibit the superior performance of our MI objective in limited sight range.
Revised: The test return results are shown in Table 3, which exhibit the superior performance of our MI objective in limited sight range.

Original: We check the effect of information compression degree β in L MI with different environment difficulties in Table 2, and find that in simple tasks such as 1-lane TJ, the coach can handle f − i without obvious information compression.
Revised: We check the effect of information compression degree β in L MI with different environment difficulties in Table 2, and find that in simple tasks such as 1-lane TJ, the coach can handle f − i without obvious information compression.

Original: While in hard tasks, e.g., RC with SR = 0 .
Revised: While in hard tasks, e.g., RC with SR = 0 .

Original: 5 , a large β can simplify communication messages, so that not to distract the agents and bring higher performance.
Revised: 5 , a large β can simplify communication messages, so that not to distract the agents and bring higher performance.

Original: 5. H ARD T ASK SMAC
Revised: 5. H ARD T ASK SMAC

Original: We test CAMA’s performance on the hard SMAC (Samvelyan et al., 2019) tasks with dynamic teams in this section.
Revised: We test CAMA’s performance on the hard SMAC (Samvelyan et al., 2019) tasks with dynamic teams in this section.

Original: We use the setting from Liu et al.
Revised: We use the setting from Liu et al.

Original: (2021) and Iqbal et al.
Revised: (2021) and Iqbal et al.

Original: (2021) that at the start of each episode a total of 3-8 agents are randomly divided into 2-4 groups and initialized at different places on the edge of a circle with the radius 9, and enemies are divided into 1-2 groups, Fig. 8(a).
Revised: (2021) that at the start of each episode a total of 3-8 agents are randomly divided into 2-4 groups and initialized at different places on the edge of a circle with the radius 9, and enemies are divided into 1-2 groups, Fig. 8(a).

Original: Agents have the sight range 9.
Revised: Agents have the sight range 9.

Original: Agents must learn to find teammates first before fighting against enemies with more quantities.
Revised: Agents must learn to find teammates first before fighting against enemies with more quantities.

Original: We show the results of test win rate (TWR) on 3 maps in Fig. 8(b).
Revised: We show the results of test win rate (TWR) on 3 maps in Fig. 8(b).

Original: Our method remarkably exceeds the current SOTA methods on all maps.
Revised: Our method remarkably exceeds the current SOTA methods on all maps.

Original: Sight Range Testing.
Revised: Sight Range Testing.

Original: In Fig.
Revised: In Fig.

Original: 9(a),we report performance comparisons with the varying SRs in { 3 , 6 , 9 , ∞} . Our CAMA achieves superior TWR over SOTA methods on all SR settings and exhibits impressive robustness.
Revised: 9(a), with the varying SR comparisons in { 3 , 6 , 9 , ∞} , our CAMA reports superior TWR over SOTA methods on all SR settings and exhibits impressive robustness The y is log( ω ω

Original: Dynamic Team Composition.
Revised: Dynamic Team Composition.

Original: In Fig. 9(b), we test the same model on different team sizes and plot the logarithm of the relative winning rate of the corresponding agent number against thewinning rate of total agents. Compared with other SOTA methods,
Revised: In Fig. 9(b), we test the same model on different team sizes and plot the logarithm of the relative winning rate of the corresponding agent number against the total agents, where CAMA shows out

Original: CAMA shows outstanding robustness to agent variation with a nearly stationary performance curve.
Revised: - standing robustness to agent variation with a nearly stationary performance curve.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we explore the dilemma of the partial observability in MARL.
Revised: In this paper, we explore the dilemma of the partial observability in MARL.

Original: With severe partial observability, agents usually can not sense the teammates and show poor team coordination.
Revised: With severe partial observability, agents usually can not sense the teammates and show poor team coordination.

Original: While with large sight range, agents are troubled by the attention distraction issue and exhibit apparent performance degradation.
Revised: While with large sight range, agents are troubled by the attention distraction issue and exhibit apparent performance degradation.

Original: To tackle such sight range dilemma, we propose a Consciousness-Aware Multi-Agent reinforcement learning (CAMA) approach.
Revised: To tackle such sight range dilemma, we propose a Concentrated Attention for Multi-Agent reinforcement learning (CAMA) approach.

Original: We first use an Entity Dividing Module  to dividethe raw entitiesinto two parts for the succeeding consciousness learning in large and low sight ranges, respectively.
Revised: First an Entity Dividing Module is used to divide raw entities for the local agents and the global coach separately. The suitable division percent varies due to the agents partial observability.

Original: For  attention distraction issue in large sight ranges, an inverse model is applied by the Consciousness Enhancement Module.
Revised: For local agents, an Attention Enhancement Module improves their attention on execution-related entities under the condition of large sight ranges.

Original: For sustainable coordination in low sight ranges, we use a Consciousness Replenishment Module to compress out-of-sight-range information with a conditional mutual information estimator.
Revised: For the global coach, the messages generated by novel conditional mutual information estimator replenish the information required for team coordination in all sight ranges.

Original: We evaluate our method on three commonly used MARL benchmarks: StarCraftII, MPE, and Traffic Junction.
Revised: We evaluate our method on three commonly used MARL benchmarks: StarCraftII, MPE, and Traffic Junction.

Original: With raised agents’ consciousness, the proposed CAMA reports significantly superior performance compared with SOTA methods and exhibits remarkable robustness to sight range variation and dynamic team composition.
Revised: With raised agents’ consciousness, the proposed CAMA reports significantly superior performance compared with SOTA methods and exhibits remarkable robustness to sight range variation and dynamic team composition.

Original: A P ROOF OF THE M ASK G ENERATOR
Revised: A P ROOF OF THE M ASK G ENERATOR

Original: We now prove that M s got by Eq.
Revised: We now prove that M s got by Eq.

Original: (5) is sufficient for Eq.
Revised: (5) is sufficient for Eq.

Original: We first show that M f in Eq.
Revised: We first show that M f in Eq.

Original: (5) satisfies ||M f || ∞ ≤ αn e .
Revised: (5) satisfies ||M f || ∞ ≤ αn e .

Original: Recall the definition of M f , it is a indicator matrix, that at each row, only the positions of top ⌊ αn e ⌋ values of the corresponding row in the attention weight matrix are 1, and others are 0.
Revised: Recall the definition of M f , it is a indicator matrix, that at each row, only the positions of top ⌊ αn e ⌋ values of the corresponding row in the attention weight matrix are 1, and others are 0.

Original: Therefore, at each row of M f , there are at most ⌊ αn e ⌋ 1s, which means the sum of the absolute value in each row is no larger than ⌊ αn e ⌋ .
Revised: Therefore, at each row of M f , there are at most ⌊ αn e ⌋ 1s, which means the sum of the absolute value in each row is no larger than ⌊ αn e ⌋ .

Original: So ||M f || ∞ ≤ ⌊ αn e ⌋ ≤ αn e .
Revised: So ||M f || ∞ ≤ ⌊ αn e ⌋ ≤ αn e .

Original: Then we show ||M s || ∞ ≤ αn e .
Revised: Then we show ||M s || ∞ ≤ αn e .

Original: Since the observability mask M is also a 0,1 mask, ||
Revised: Since the observability mask M is also a 0,1 mask, ||

Original: [Equation]
Revised: [Equation]

Original: Then we show that ¬M s ⊙
Revised: Then we show that ¬M s ⊙

Original: ¬M = ¬M , which equals to ¬ ( M f ⊙ M ) ⊙ ¬M
Revised: ¬M = ¬M , which equals to ¬ ( M f ⊙ M ) ⊙ ¬M

Original: = ¬M .
Revised: = ¬M .

Original: Recall that
Revised: Recall that

Original: M f and M are both 0,1 matrices with the same shape.
Revised: M f and M are both 0,1 matrices with the same shape.

Original: For any position in the matrices, we can use
Revised: For any position in the matrices, we can use

Original: Table 4 to conclude all the situations:
Revised: Table 4 to conclude all the situations:

Original: [Table]
Revised: [Table]

Original: We can find that at any situation ¬ ( M f ⊙ M ) ⊙ ¬M
Revised: We can find that at any situation ¬ ( M f ⊙ M ) ⊙ ¬M

Original: = ¬M , therefore ¬M s ⊙ ¬M
Revised: = ¬M , therefore ¬M s ⊙ ¬M

Original: = ¬M .
Revised: = ¬M .

Original: In summary, M s got by Eq.
Revised: In summary, M s got by Eq.

Original: (5) is sufficient for Eq.
Revised: (5) is sufficient for Eq.

Original: B D ERIVATION OF M UTUAL I NFORMATION O BJECTIVE
Revised: B D ERIVATION OF M UTUAL I NFORMATION O BJECTIVE

Original: We now give the derivation of Eq 7.
Revised: We now give the derivation of Eq 7.

Original: [Equation]
Revised: [Equation]

Original: //split
Revised: //split

Original: s into f i and f − i
Revised: s into f i and f − i

Original: [Equation]
Revised: [Equation]

Original: C P ROOF OF T HEOREM 4. 
Revised: C P ROOF OF T HEOREM 4.

Original: For three random variables ζ i , f i and f − i ,
Revised: For three random variables ζ i , f i and f − i ,

Original: I CC ( ζ i , f
Revised: I CC ( ζ i , f

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ≥ I (
Revised: ≥ I (

Original: [Equation]
Revised: [Equation]

Original: i , f i | f − i ) .
Revised: i , f i | f − i ) .

Original: The equality holds if and only if f i is independent of the joint distribution of ζ i , f − i .
Revised: The equality holds if and only if f i is independent of the joint distribution of ζ i , f − i .

Original: Proof.
Revised: Proof.

Original: Let ∆ be the gap between I CC ( ζ i , f i | f − i ) and I ( ζ i , f i | f − i ) :
Revised: Let ∆ be the gap between I CC ( ζ i , f i | f − i ) and I ( ζ i , f i | f − i ) :

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since log( · ) is a concave function, ∆ ≥ 0 due to Jensen’s Inequality.
Revised: Since log( · ) is a concave function, ∆ ≥ 0 due to Jensen’s Inequality.

Original: D G ENERATE M ESSAGES WITH M EMORY
Revised: D G ENERATE M ESSAGES WITH M EMORY

Original: In Sec. 4.4, we use concat( f i , f − i ) to generate the message ζ i .
Revised: In Sec. 4.4, we use concat( f i , f − i ) to generate the message ζ i .

Original: It means we have the true prior distribution p ( ζ i | f i , f − i ) , which can be expediently applied to estimate I CC .
Revised: It means we have the true prior distribution p ( ζ i | f i , f − i ) , which can be expediently applied to estimate I CC .

Original: However, if we wish to enhance the coach’s ability, e.g., introducing a memory structure (similar to the GRU cell in local agents) into the message generator, the prior distribution turns into p ( ζ i | ρ ) and therefore we can not obtain p ( ζ i | f i , f − i ) directly to estimate I CC .
Revised: However, if we wish to enhance the coach’s ability, e.g., introducing a memory structure (similar to the GRU cell in local agents) into the message generator, the prior distribution turns into p ( ζ i | ρ ) and therefore we can not obtain p ( ζ i | f i , f − i ) directly to estimate I CC .

Original: Similar to the idea in Cheng et al.
Revised: Similar to the idea in Cheng et al.

Original: (2020), we propose a variational term q
Revised: (2020), we propose a variational term q

Original: ( ζ i | f i , f − i ) to estimate p ( ζ i | f i , f − i ) :
Revised: ( ζ i | f i , f − i ) to estimate p ( ζ i | f i , f − i ) :

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 gives a sufficient condition to ensure I vCC ( ζ i , f i | f − i ) be an upper bound of
Revised: Theorem 1 gives a sufficient condition to ensure I vCC ( ζ i , f i | f − i ) be an upper bound of

Original: I ( ζ i , f i | f − i ) :
Revised: I ( ζ i , f i | f − i ) :

Original: Theorem 1 .
Revised: Theorem 1 .

Original: Denote q ( ζ i , f i , f − i ) =
Revised: Denote q ( ζ i , f i , f − i ) =

Original: q ( ζ i | f
Revised: q ( ζ i | f

Original: i , f − i ) p ( f i ) p ( f − i ) .
Revised: i , f − i ) p ( f i ) p ( f − i ) .

Original: If
Revised: If

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: then I vCC ( ζ i , f
Revised: then I vCC ( ζ i , f

Original: i | f − i ) ≥ I ( ζ i , f i | f − i ) .
Revised: i | f − i ) ≥ I ( ζ i , f i | f − i ) .

Original: The equality holds when f i and the join distribution ζ i , f − i are independent.
Revised: The equality holds when f i and the join distribution ζ i , f − i are independent.

Original: Proof.
Revised: Proof.

Original: Let ˆ∆ be the gap between I vCC ( ζ i , f i | f − i ) and I ( ζ i , f i | f − i ) .
Revised: Let ˆ∆ be the gap between I vCC ( ζ i , f i | f − i ) and I ( ζ i , f i | f − i ) .

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 reveals that I vCC is an MI upper bound if the variational joint distribution q ( ζ i , f i , f − i ) is more “closer” to p ( ζ i , f i , f − i ) than to p ( f i ) p ( ζ i , f − i ) .
Revised: Theorem 1 reveals that I vCC is an MI upper bound if the variational joint distribution q ( ζ i , f i , f − i ) is more “closer” to p ( ζ i , f i , f − i ) than to p ( f i ) p ( ζ i , f − i ) .

Original: Let q ϕ be the parameterization of q .
Revised: Let q ϕ be the parameterization of q .

Original: In addition to the Eq. 16 that optimizes I vCC , we should minimize the KL divergence between p ( ζ i , f i , f − i ) and q ( ζ i , f i , f − i ) :
Revised: In addition to the Eq. 16 that optimizes I vCC , we should minimize the KL divergence between p ( ζ i , f i , f − i ) and q ( ζ i , f i , f − i ) :

Original: [Equation]
Revised: [Equation]

Original: Therefore, with sample pairs { ( ζ ik , f ik , f − i k ) } Kk =1 , we can maximize the log-likelihood of q ϕ :max ϕ K K (cid:88) k = log
Revised: Therefore, with sample pairs { ( ζ ik , f ik , f − i k ) } Kk =1 , we can maximize the log-likelihood of q ϕ :

Original: [Equation]
Revised: [Equation]

Original: With enough optimization times of Eq. 20, I vCC is guaranteed to be an MI upper bound.
Revised: With enough optimization times of Eq. 20, I vCC is guaranteed to be an MI upper bound.

Original: We test the effect of coach with memory in the Resource Collection environment with SR = 0 .
Revised: We test the effect of coach with memory in the Resource Collection environment with SR = 0 .

Original: 5 and
Revised: 5 and

Original: SR = 1 . 0 , Table 5.
Revised: SR = 1 . 0 , Table 5.

Original: We find that although equip the coach with the memory module improves the average performance, it brings large variance that the method becomes unstable.
Revised: We find that although equip the coach with the memory module improves the average performance, it brings large variance that the method becomes unstable.

Original: Therefore, to keep a stable performance of our method, we still use the coach with MLP in the main paper.
Revised: Therefore, to keep a stable performance of our method, we still use the coach with MLP in the main paper.

Original: [Table]
Revised: [Table]

Original: E H EAT M APS OF A TTENTION W EIGHTS 
Revised: E H EAT M APS OF A TTENTION W EIGHTS E. A VERAGE H EAT M APS OF M ORE M ETHODS

Original: As mentioned in Sec.
Revised: As mentioned in Sec.

Original: 4.1, we visualize the attention weights of four methods in Fig.
Revised: 4.1, we visualize the attention weights of four methods in Fig.

Original: Only our
Revised: Only our

Original: CAMA pay more attention to the agents themselves when the sight range is large, and therefore keeps a high performance.
Revised: CAMA pay more attention to the agents themselves when the sight range is large, and therefore keeps a high performance

Revised: [Figure]

Revised: We also visualize the heat maps of our method and REFIL at single time steps.

Revised: We choose the first attention head of each saved model, and set the color bar’s range of attention weight to [0,0.35].

Original:  F H YPERPARAMETERS
Revised: We visualize in Fig. 11 both methods’ attention weights on all agents from t = 0 to t = 24 , at which time all the agents are usually alive. We find that the large sight range distracts the attention of REFIL, while our method CAMA keeps attention concentration. F H YPERPARAMETERS

Original: We summarize the hyperparameters in Table.[Table]
Revised: We summarize the hyperparameters in Table.

Original: G E NVIRONMENT D ETAILS G. R ESOURCE C OLLECTION
Revised: G E NVIRONMENT D ETAILS G. R ESOURCE C OLLECTION

Original: On a map of [- [Table] [Table] agents are initialized at random places with team size sampled from [3,8].
Revised: On a map of [-1,1], agents are initialized at random places with team size sampled from [3,8].

Original: They need to collect resources from 6 resource points and transport the goods home.
Revised: They need to collect resources from 6 resource points and transport the goods home.

Original: The locations of resource points and home are randomly sampled from the whole map at the start of each episode.
Revised: The locations of resource points and home are randomly sampled from the whole map at the start of each episode.

Revised: [Table]

Original: The radius of the home and the resource location & agent is 0.1 and 0.05, respectively.
Revised: The radius of the home and the resource location & agent is 0.1 and 0.05, respectively.

Original: There arekinds of resources, and each agent i has its own ability b e i uniformly sampled from {0.1, 0.5, 0.9} to collect each kind of resource e .
Revised: There arekinds of resources, and each agent i has its own ability b ei uniformly sampled from {0.1, 0.5, 0.9} to collect each kind of resource e .

Original: Each agent can accelerate towards 4 directions or apply no forces at each time-step.
Revised: Each agent can accelerate towards 4 directions or apply no forces at each time-step.

Original: Each agent has its maximal speed uniformly sampled from {0.3, 0.5, 0.7} and the acceleration is fixed to 3.0.
Revised: Each agent has its maximal speed uniformly sampled from {0.3, 0.5, 0.7} and the acceleration is fixed to 3.0.

Original: Every time the agent i collects resource e , the team will get a reward 10 ∗ b ei .
Revised: Every time the agent i collects resource e , the team will get a reward 10 ∗ b ei .

Original: When an agent brings the resource home, the team will get a reward 1.
Revised: When an agent brings the resource home, the team will get a reward 1.

Original: An agent can only carry one resource at a time, which means an agent needs to bring the collected resource home before it starts to collect the next one.
Revised: An agent can only carry one resource at a time, which means an agent needs to bring the collected resource home before it starts to collect the next one.

Original: The episode limit is 145.
Revised: The episode limit is 145.

Original: The number of the agents for training is uniformly sampled from {2,3,4,5}, while for testing it is sampled from {6,7,8}.
Revised: The number of the agents for training is uniformly sampled from {2,3,4,5}, while for testing it is sampled from {6,7,8}.

Original: Each agent has a sight range SR .
Revised: Each agent has a sight range SR .

Original: Entities including other agents and resource points that exceed agent i ’s SR are invisible to agent i .
Revised: Entities including other agents and resource points that exceed agent i ’s SR are invisible to agent i .

Original: G. T RAFFIC J UNCTION
Revised: G. T RAFFIC J UNCTION

Original: The simulated traffic junction environment from Sukhbaatar et al.
Revised: The simulated traffic junction environment from Sukhbaatar et al.

Original: (2016) has been a conventional and useful testbed for testing the performance of multi-agent communication algorithms Singh et al.
Revised: (2016) has been a conventional and useful testbed for testing the performance of multi-agent communication algorithms Singh et al.

Original: (2018); Das et al.
Revised: (2018); Das et al.

Original: Liu et al.
Revised: Liu et al.

Original: Despite its great success, cars in the original traffic junction environment can only move along pre-assigned routes on one or more road junctions and so the action space for each car only consists of two actions, i.e. gas and brake, which restricts its ability[Table] to simulate real-world environments and test communication algorithms.
Revised: Despite its great success, cars in the original traffic junction environment can only move along pre-assigned routes on one or more road junctions and so the action space for each car only consists of two actions, i.e. gas and brake, which restricts its ability to simulate real-world environments and test communication algorithms.

Original: Moreover, the original observation is not fit for the need of entity-wise input.
Revised: Moreover, the original observation is not fit for the need of entity-wise input.

Original: So, we modify the original traffic junction environment to a more flexible version.
Revised: So, we modify the original traffic junction environment to a more flexible version.

Original: In stead of preassigned routes, a random selected navigation target is assigned to each active car and the aim of each active car is navigating to its own target and avoiding collision which occurs when two cars are on same location.
Revised: In stead of preassigned routes, a random selected navigation target is assigned to each active car and the aim of each active car is navigating to its own target and avoiding collision which occurs when two cars are on same location.

Original: Along with the modified setting, the observation for each active car is altered to entity-wise form which contains the position of cars in a limited visibility (e.g. 3 x 3 for sight_range = 1) and the action space is more flexible with five actions i.e. forward, back, left, right and wait .
Revised: Along with the modified setting, the observation for each active car is altered to entity-wise form which contains the position of cars in a limited visibility (e.g. 3 x 3 for sight_range = 1) and the action space is more flexible with five actions i.e. forward, back, left, right and wait .

Original: Besides, the rewards consists of a linear time penalty − 0 .
Revised: Besides, the rewards consists of a linear time penalty − 0 .

Original: 01 τ , where τ is the number of active time steps for the car since the last resurrection, a collision penalty r collision = − 10 and the difference
Revised: 01 τ , where τ is the number of active time steps for the car since the last resurrection, a collision penalty r collision = − 10 and the difference

Original: Manhattan distance from the target between the previous and the current step .
Revised: Manhattan distance from the target between the previous and the current step .

Original: We choose two different difficulty levels following the settings in Singh et al.
Revised: We choose two different difficulty levels following the settings in Singh et al.

Original: (2018), illustrated in Fig.11.
Revised: (2018), illustrated in Fig.12.

Original: Moreover,  the total number of cars is fixed at N max and the cars will be turned inactive when reaching the target or colliding with others and new cars get added to the environment with probability p arrive at every time-step which varies during training consistent with the curriculum learning in Singh et al.
Revised: Moreover, [Table] the total number of cars is fixed at N max and the cars will be turned inactive when reaching the target or colliding with others and new cars get added to the environment with probability p arrive at every time-step which varies during training consistent with the curriculum learning in Singh et al.

Original: The detailed environment parameters settings are indicated in Table.7.
Revised: The detailed environment parameters settings are indicated in Table.7.

Original: [Figure]
Revised: [Figure]

Original: are two difficulty levels.

Original: H H ARDWARE
Revised: H H ARDWARE

Original: We ran experiments on 2 GPU servers, with each one having 8*RTX3090TI GPUS and 2*AMD
Revised: We ran experiments on 2 GPU servers, with each one having 8*RTX3090TI GPUS and 2*AMD

Original: EPYC 7H12 CPUs.
Revised: EPYC 7H12 CPUs.

Original: Each experiment (one seed) takes 12-24 hours on one GPU.
Revised: Each experiment (one seed) takes 12-24 hours on one GPU.

Revised: I G ENERALIZABILITY

Revised: We test the generalizability of our method in the environment Resource Collection.

Revised: We train each method for 10 7 time steps in the environment with 2-5 agents.

Revised: Every 5 × 10 4 time steps, the model is evaluated on the environment with 6-8 agents for 160 episodes, and we plot the test return curves in

Revised: Fig.

Revised: The results show that our method can generalize well to larger team sizes than training, which is probably due to the ARM module with the message generator that passes the global information.