Original: L EARNING N EARLY D ECOMPOSABLE V ALUE F UNCTIONS V IA C OMMUNICATION M INIMIZATION
Revised: L EARNING N EARLY D ECOMPOSABLE V ALUE F UNCTIONS V IA C OMMUNICATION M INIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity.
Revised: Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity.

Original: Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multi-agent systems.
Revised: Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multiagent systems.

Original: However, existing methods have been focusing on learning fully decentralized value function, which are not efﬁcient for tasks requiring communication.
Revised: However, existing methods have been focusing on learning fully decentralized value function, which are not efﬁcient for tasks requiring communication.

Original: To address this limitation, this paper presents a novel framework for learning nearly decomposable value functions with communication , with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination.
Revised: To address this limitation, this paper presents a novel framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination.

Original: This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers.
Revised: This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers.

Original: These regularizers are maximizing mutual information between decentralized Q functions and communication messages while minimizing the entropy of messages between agents.
Revised: These regularizers are maximizing mutual information between agents’ action selection and communication messages while minimizing the entropy of messages between agents.

Original: We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX.
Revised: We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX.

Original: Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework signiﬁcantly outperforms baseline methods and allows to cut off more than 80% communication without sacriﬁcing the performance.
Revised: Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework signiﬁcantly outperforms baseline methods and allows to cut off more than 80% communication without sacriﬁcing the performance.

Original: The video of our experiments is available at https://sites.google.com/view/ndvf .
Revised: The videos of our experiments are available at https://sites.google.com/view/ndvf .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Cooperative multi-agent reinforcement learning (MARL) are ﬁnding applications in many realworld domains, such as autonomous vehicle teams (Cao et al., 2012), intelligent warehouse systems (Now´e et al., 2012), and sensor networks (Zhang & Lesser, 2011).
Revised: Cooperative multi-agent reinforcement learning (MARL) are ﬁnding applications in many realworld domains, such as autonomous vehicle teams (Cao et al., 2012), intelligent warehouse systems (Now´e et al., 2012), and sensor networks (Zhang & Lesser, 2011).

Original: To help address these problems, recent years have made a great progress in MARL methods (Lowe et al., 2017; Foerster et al., 2018; Rashid et al., 2018;
Revised: To help address these problems, recent years have made a great progress in MARL methods (Lowe et al., 2017; Foerster et al., 2018; Rashid et al., 2018;

Original: Jaques et al., 2019).
Revised: Jaques et al., 2019).

Original: Among these successes, the paradigm of centralized training with decentralized execution has attracted much attention for its scalability andthe ability to deal with non-stationarity.
Revised: Among these successes, the paradigm of centralized training with decentralized execution has attracted much attention for its scalability and ability to deal with non-stationarity.

Original: Value function decomposition methods provide a promising way to exploit such paradigm.
Revised: Value function decomposition methods provide a promising way to exploit such paradigm.

Original: They learn a decentralized Q function for each agent and use a mixing network to combine these local Q values into a global action value.
Revised: They learn a decentralized Q function for each agent and use a mixing network to combine these local Q values into a global action value.

Original: In previous works, VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QTRAN (Son et al., 2019) have progressively enlarged the family of functions that can be represented by the mixing network.
Revised: In previous works, VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QTRAN (Son et al., 2019) have progressively enlarged the family of functions that can be represented by the mixing network.

Original: Despite their increasing ability in terms of value factorization representation, existing methods have been focusing on learning full decomposition, where each agent acts upon its local observations.
Revised: Despite their increasing ability in terms of value factorization representation, existing methods have been focusing on learning full decomposition, where each agent acts upon its local observations.

Original: However, many multi-agent tasks in the real world are not fully decomposable – agents sometimes require information from other agents in order to effectively coordinate their behaviors.
Revised: However, many multi-agent tasks in the real world are not fully decomposable – agents sometimes require information from other agents in order to effectively coordinate their behaviors.

Original: This is because partial observability and stochasticity in a multi-agent environment can exacerbate an agent’s uncertainty of other agents’ states and actions during decentralized execution, which may result in catastrophic miscoordination.
Revised: This is because partial observability and stochasticity in a multi-agent environment can exacerbate an agent’s uncertainty of other agents’ states and actions during decentralized execution, which may result in catastrophic miscoordination.

Original: To address this limitation, this paper presents a scalable multi-agent learning framework for learning nearly decomposable value functions with communication , with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination.
Revised: To address this limitation, this paper presents a scalable multi-agent learning framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination.

Original: This framework hybridizes value function factorization learning and communication learning by introducing an information-theoretic regularizer for maximizing mutual information between de- centralized Q functions and communication messages.
Revised: This framework hybridizes value function factorization learning and communication learning by introducing an information-theoretic regularizer for maximizing mutual information between agents’ action selection and communication messages.

Original: Messages are parameterized in a stochastic embedding space.
Revised: Messages are parameterized in a stochastic embedding space.

Original: To minimize communication, we introduce an additional information-theoretic regularizer for minimizing the entropy of messages between agents.
Revised: To optimize communication, we introduce an additional informationtheoretic regularizer to minimize the entropy of messages between agents.

Original: With these two regularizers, our framework implicitly learn when, what, and with whom to communicate and also ensure communication to be both expressive (i.e., effectively reducing the uncertainty of agents’ action-value functions) and succinct (i.e., only sending useful and necessary information).
Revised: With these two regularizers, our framework implicitly learn when, what, and with whom to communicate and also ensure communication to be both expressive (i.e., effectively reducing the uncertainty of agents’ action-value functions) and succinct (i.e., only sending useful and necessary information).

Original: To optimize these regularizers, we draw inspiration from variational inference and derive a tractable lower bound objective, which is easily integrated with existing value function factorization methods such as QMIX.
Revised: To optimize these regularizers, we draw inspiration from variational inference and derive a tractable lower bound objective, which is easily integrated with existing value function factorization methods such as QMIX.

Original: We demonstrate the effectiveness of our learning framework onthe StarCraft II 1 unit micromanagement benchmark used in Foerster et al.
Revised: We demonstrate the effectiveness of our learning framework on StarCraft II 1 unit micromanagement benchmark used in Foerster et al.

Original: (2017; 2018); Rashid et al.
Revised: (2017; 2018); Rashid et al.

Original: Samvelyan et al.
Revised: Samvelyan et al.

Original: (2019b). Empirical results show that it signiﬁcantly outperforms baseline methods and allows to cut off more than 80% communication without sacriﬁcing the performance.
Revised: Empirical results show that NDQ signiﬁcantly outperforms baseline methods and allows to cut off more than 80% communication without sacriﬁcing the performance.

Original: We also observe that agents can effectively learn to coordinate their actions at the cost of sending one or two bits of messages even in complex StarCraft II tasks.
Revised: We also observe that agents can effectively learn to coordinate their actions at the cost of sending one or two bits of messages even in complex StarCraft II tasks.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In our work, we consider a fully cooperative multi-agent task that can be modelled by a DecPOMDP (Oliehoek et al., 2016)
Revised: In our work, we consider a fully cooperative multi-agent task that can be modelled by a DecPOMDP (Oliehoek et al., 2016)

Original: G = (cid:104)
Revised: G = (cid:104)

Original: I, S, A, P, R, Ω , O, n, γ (cid:105) , where I ≡ { 1 , 2 , ..., n } is the ﬁnite set of agents.
Revised: I, S, A, P, R, Ω , O, n, γ (cid:105) , where I ≡ { 1 , 2 , ..., n } is the ﬁnite set of agents.

Original: s ∈ S is the true state of the environment from which each agent i draw an individual partial observation oi ∈ Ω
Revised: s ∈ S is the true state of the environment from which each agent i draws an individual partial observation o

Original:  according to the observation function O ( s, i ) .
Revised: i ∈ Ω according to the observation function O ( s, i ) .

Original: Each agent hasa an action-observation history τ i ∈ T ≡ (Ω × A ) ∗ .
Revised: Each agent has an action-observation history τ i ∈ T ≡ (Ω × A ) ∗ .

Original: At each timestep, each agent  selects an action a i ∈ A , forming a joint action a ∈
Revised: At each timestep, each agent i selects an action a i ∈ A , forming a joint action a ∈

Original: A n , resulting in a shared reward r = R
Revised: A n , resulting in a shared reward r = R

Original: ( s, a )
Revised: ( s, a )

Original: for each agent and the next state s (cid:48)
Revised: for each agent and the next state s (cid:48)

Original: according to the transition function P ( s (cid:48) | s, a ) .
Revised: according to the transition function P ( s (cid:48) | s, a ) .

Original: The joint policy π induces a joint action-value function: Q π tot ( τ, a ) =
Revised: The joint policy π induces a joint action-value function: Q π tot ( τ, a ) =

Original: E s 0: ∞ , a
Revised: E s 0: ∞ , a

Original: 0: ∞ [ (cid:80) ∞ t =0 γ t r t | s 0 =
Revised: 0: ∞ [ (cid:80) ∞ t =0 γ t r t | s 0 =

Original: s, a 0 = a , π ] , where τ is the joint action-observation history and γ ∈ [0 , 1) is the discount factor.
Revised: s, a 0 = a , π ] , where τ is the joint action-observation history and γ ∈ [0 , 1) is the discount factor.

Original: Learning the optimal action-value function encounters challenges in multi-agent settings.
Revised: Learning the optimal action-value function encounters challenges in multi-agent settings.

Original: On the one hand, to properly coordinate actions of agents, learning a centralized action value function Q tot seems a good choice.
Revised: On the one hand, to properly coordinate actions of agents, learning a centralized action-value function Q tot seems a good choice.

Original: However, such a function is difﬁcult to learn when the number of agents is large.
Revised: However, such a function is difﬁcult to learn when the number of agents is large.

Original: On the other hand, directly learning decentralized action-value function Q i for each agent alleviates the scalability problem (Tan, 1993; Tampuu et al., 2017).
Revised: On the other hand, directly learning decentralized action-value function Q i for each agent alleviates the scalability problem (Tan, 1993; Tampuu et al., 2017).

Original: Nevertheless, such independent learning method largely neglects interactions among agents, which often results in miscoordination and inferior performance.
Revised: Nevertheless, such independent learning method largely neglects interactions among agents, which often results in miscoordination and inferior performance.

Original: In between, value function factorization method provides a promising way to attenuate such dilemma by representing Q tot as a mixing of decentralized Q i conditioned on local information.
Revised: In between, value function factorization method provides a promising way to attenuate such dilemma by representing Q tot as a mixing of decentralized Q i conditioned on local information.

Original: Such methods has shown their effectiveness on complex task (Samvelyan et al., 2019b).
Revised: Such method has shown their effectiveness on complex task (Samvelyan et al., 2019).

Original: However, current value function factorization methods have  mainly focusing on full decomposition.
Revised: However, current value function factorization methods have been mainly focusing on full decomposition.

Original: Such decomposition reduces the complexity of learning Q tot by ﬁrst learning independent Q i and putting the burden of coordinating actions on the mixing networks whose input is all Q i ’s and output  Q tot .
Revised: Such decomposition reduces the complexity of learning Q tot by ﬁrst learning independent Q i and putting the burden of coordinating actions on the mixing networks whose input is all Q i ’s and output is Q tot .

Original: For many tasks with partial observability and stochastic dynamics, mixing networks are not sufﬁcient to learn coordinated actions, regardless of how powerful its representation ability is.
Revised: For many tasks with partial observability and stochastic dynamics, mixing networks are not sufﬁcient to learn coordinated actions, regardless of how powerful its representation ability is.

Original: The reason is that full decomposition cuts off all dependencies among decentralized action-value functions and agents will be uncertain about states and actions of other agents.
Revised: The reason is that full decomposition cuts off all dependencies among decentralized action-value functions and agents will be uncertain about states and actions of other agents.

Original: Such uncertainty will increase as time goes by and can result in severe miscordination and arbitrarily worse performance during decentralized execution.
Revised: Such uncertainty will increase as time goes by and can result in severe miscoordination and arbitrarily worse performance during decentralized execution.

Original:  M ETHODOLOTY
Revised:  M ETHODOLOGY

Original: In this section, we propose to learn nearly decomposable value functions with communication, a new framework to overcome the miscoordination issue of full factorization methods.
Revised: In this section, we propose to learn nearly decomposable Q-functions (NDQ) via communication minimization, a new framework to overcome the miscoordination issue of full factorization methods.

Original: Our framework adopts the centralized training and decentralized execution paradigm.

Original: During the centralized training, we assume the learning algorithm has access to all agents’ individual observation-action histories and global state s .

Original: During the decentralized execution, each agent acts based on its learned

Original: X message 𝑚 "# 𝑜 " 𝑜 # 𝑄 " (𝜏 " , 𝑎 " ,𝑚 ""+ ) 𝑄 # (𝜏 # , 𝑎 # ,𝑚 #"+ ) Mixing Network 𝑄 -.- 𝑚 ""+ 𝑚 #"+
Revised: X message 𝑚 "# 𝑜 " 𝑜 # 𝑄 " (𝜏 " , 𝑎 " ,𝑚 ""+ ) 𝑄 # (𝜏 # , 𝑎 # ,𝑚 #"+ ) Mixing Network 𝑄 -.- 𝑚 ""+ 𝑚 #"+

Original: X 𝑎 " 𝑎 # message 𝑚 #"
Revised: X 𝑎 " 𝑎 # message 𝑚 #"

Original: X 𝐺𝑅𝑈 " 𝐺𝑅𝑈 #
Revised: X 𝐺𝑅𝑈 " 𝐺𝑅𝑈 #

Original: Messages 𝑚 (2#)" from other agents Concatenate X
Revised: Messages 𝑚 (2#)" from other agents Concatenate X

Original: Agent 𝑖 MLP
Revised: Agent 𝑖 MLP

Original: MLP MLP
Revised: MLP MLP

Original: MLP
Revised: MLP

Original: Messages 𝑚 2" # from other agents
Revised: Messages 𝑚 2" # from other agents

Original: Agent
Revised: Agent

Original: 𝑗 𝑜 #
Revised: 𝑗 𝑜 #

Original: MLP
Revised: MLP

Original: GRU ℎ #- ℎ #- 𝜏 # 𝐼(𝐴 " ;M #" |Τ " ,𝑀 2# " )
Revised: GRU ℎ #- ℎ #- 𝜏 # 𝐼(𝐴 " ;M #" |Τ " ,𝑀 2# " )

Original: [Figure]
Revised: [Figure]

Original: 𝐼(𝐴 # ;M "# |Τ # , [Figure] action-value function Q i ( τ i , a i , m ini ) conditioning onits local action-observation history τ i and received messages m ini .
Revised: In our learning framework (Fig. 1), individual action-value functions condition on local actionobservation history and, at certain timesteps, messages received from a few other agents.

Original: Message communication is also centrally learned and executed in a decentralized manner.
Revised: Messages from agent i to agent j are drawn from a multivariate Gaussian distribution whose parameters are given by an encoder f m

Original: Our learning framework is shown in Fig.
Revised: ( τ i , j ; θ c ) , where τ i is the local observation-action history of agent i , and θ c are parameters of encoder f m . Formally, message m ij ∼ N ( f m ( τ i , j ; θ c ) , I ) , where I is an identity matrix.

Original: In our learning framework, individual action-value functions condition on local action-observation history and, at certain timesteps, messages from few other agents.
Revised: Here we use an identity covariance matrix and reasons will be discussed in the next section. m ( - i ) j is used to denote the message sent to j from agents other than i .

Original: We learn such near decomposable structure via learning minimized communication.
Revised: We learn a nearly decomposable structure via learning minimized communication.

Original: We thus expect the communication to have the following properties:
Revised: We thus expect the communication to have the following properties:

Original: i)
Revised: i)

Original: Expressiveness: The message passed to one agent should effectively reduce the uncertainty in its action-value function.
Revised: Expressiveness: The message passed to one agent should effectively reduce the uncertainty on its action-value function.

Original: ii)
Revised: ii)

Original: Succinctness: Agents are expected to send messages as short as possible to the agents who need it and  when necessary.
Revised: Succinctness: Agents are expected to send messages as short as possible to the agents who need it and only when necessary.

Original: To learn such a communicating strategy, we draw inspiration from variational inference for its proven ability in learning structure from data and endow a stochastic latent message space, which we also refer to as ”message embedding”.
Revised: To learn such a communicating strategy, we draw inspiration from variational inference for its proven ability in learning structure from data and endow a stochastic latent message space, which we also refer to as ”message embedding”.

Original: Messages are encoded as multi-variate Gaussian distributions with diagonal covariance in this latent space.

Original: Speciﬁcally, an encoder f m ( m i | τ i ; θ c ) conditions on individual observation-action history is learned and shared among agents.

Original: At each timestep, this encoder outputs a message distribution M i for each agent i , from which a message m ti is resampled.

Original: We adopt a point-to-point communication paradigm and m ti = (cid:104) m ti 1 , m ti 2 , ..., m tin (cid:105) where message m ij , sampled from the distribution M ij , is the message intended from agent i to agent j .

Original: We impose constraints, which will be discussed in detail in the next section, on the latent message embedding to enable  agent deciding locally which bits in a message should be sent (from 0 to all) according to their utility in terms of helping other agents make decision.
Revised: We impose constraints, which will be discussed in detail in the next section, on the latent message embedding to enable an agent to decide locally which bits in a message should be sent (from 0 to all) according to their utility in terms of helping other agents make decisions.

Original: Agent j will receive an input message m inj that has been  cut, on which it conditions the local action-value function Q j ( τ j , a j , m inj ) .
Revised: Agent j will receive an input message m inj that has been selectively cut, on which it conditions the local action-value function Q j ( τ j , a j , m inj ) .

Original: All the individual Q values are then feed into a mixing network such as in QMIX (Rashid et al., 2018).
Revised: All the individual Q values are then fed into a mixing network such as that used by QMIX (Rashid et al., 2018).

Original: Apart from the constraints on the message embedding, all the components (individual action-value function, message encoder, and mixing network) are trained  end-to-end  by  the downstream TD loss.
Revised: Apart from the constraints on the message embedding, all the components (individual action-value function, message encoder, and mixing network) are trained in an end-to-end manner by minimizing the downstream TD loss.

Original: Thus, our overall objective is to minimize
Revised: Thus, our overall objective is to minimize

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where L TD  =
Revised: where L TD ( θ ) =

Original: [ r + γ max a (cid:48)
Revised: [ r + γ max a (cid:48)

Original: Q tot ( s (cid:48) , a (cid:48) ; θ − ) − Q tot ( s, a ; θ )] 2 ( θ − are the parameters of a target network as in DQN) is the TD loss and θ is all parameters in the model.
Revised: Q tot ( s (cid:48) , a (cid:48) ; θ − ) − Q tot ( s, a ; θ )] 2 ( θ − are the parameters of a target network as in DQN) is the TD loss, θ are all parameters in the model, and λ is a weighting term.

Original: We now discuss how to get L c ( θ c ) to regularize the message embedding.
Revised: We will discuss how to deﬁne and optimize L c ( θ c ) to regularize the message embedding in the next section.

Original: 3. M INIMIZED C OMMUNICATION O BJECTIVE AND V ARIATIONAL B OUND
Revised: 3. M INIMIZED C OMMUNICATION O BJECTIVE AND V ARIATIONAL B OUND

Original: Introducing a latent variable facilitates the representation of  message, but it does not mean that the messages can reduce uncertainty in  action-value functions of other agents.
Revised: Introducing a latent variable facilitates the representation of the message, but it does not mean that the messages can reduce uncertainty in the action-value functions of other agents.

Original: To achieve this goal, we maximize the mutual information between messageM ij andoptimal action policy of agent j , A j ,
Revised: To make message expressive, we maximize the mutual information between message and agent j ’s action selection,

Original:  I θ c
Revised: A j . Formally, we maximize I θ c

Original: ( A j ; M ij | T j , M ( - i ) j )  .
Revised: ( A j ; M ij | T j , M ( - i ) j ) where T j is the random variable of the local action-observation history of agent j , M ij and M ( - i ) j are random variable of m ij and m ( - i ) j .

Original: However, if this is the only objective, the encoder can easily learn to cheat by giving messages under different histories a representation in different regions in the latent space, rendering cutting off useless message difﬁcult.
Revised: However, if this is the only objective, the encoder can easily learn to cheat by giving messages under different histories representations in different regions in the latent space, rendering cutting off useless messages difﬁcult.

Original: A natural constraint to avoid such representation is  the entropy of the messages.
Revised: A natural constraint to avoid such representations is to minimize the entropy of the messages.

Original: Therefore, our objective for message of agent i is to maximize:
Revised: Therefore, our objective for optimizing communication of agent i is to maximize:

Original: [Equation]
Revised: [Equation]

Original: where β is a scaling factor trading expressiveness and succinctness and M ij is the message distribution intended from i to j .
Revised: where β is a scaling factor trading expressiveness and succinctness.

Original: This objective is appealing because it agree exactly with the desiderata that we impose on the message embedding.
Revised: This objective is appealing because it agrees exactly with the desiderata that we impose on the message embedding.

Original: However, applying this objectiveto our message embedding needs extra efforts because  mutual information and entropy becomes intractable.
Revised: However, optimizing this objective needs extra efforts because computation involving mutual information is intractable.

Original: By introducing a variational approximator, a popular technique from variational toolkit (Alemi et al., 2017), we can construct a lower bound on the mutual information term in Eq. 2:
Revised: By introducing a variational approximator, a popular technique from variational toolkit (Alemi et al., 2017), we can derive a lower bound for the mutual information term in Eq. 2 (a detailed derivation can be found in Appendix A):

Original: [Equation]
Revised: [Equation]

Original:  where Tj is local action-observation history of agent j and T = (cid:104) T 1 , T 2 , . .
Revised: (3) where T = (cid:104) T 1 , T 2 , . .

Original: , T n (cid:105) is the joint local history sampled from the replay buffer D , q ξ ( A j | T j , M inj ) is the variational posterior estimator with parameters ξ . CE is the cross entropy.
Revised: , T n (cid:105) is the joint local history sampled from the replay buffer D , q ξ ( A j | T j , M inj ) is the variational posterior estimator with parameters ξ , and CE is the cross entropy operator.

Original: We share ξ among agents to accelerate learning.
Revised: We share ξ among agents to accelerate learning.

Original: Next we discuss how to minimize the term βH ( M ij ) .
Revised: Next we discuss how to minimize the term H θ c ( M ij ) .

Original: Directly minimize the entropy of Gaussian distribution can cause the variance to collapse to 0.
Revised: Directly minimizing the entropy of Gaussian distributions can cause the variance to collapse to 0.

Original: To deal with thispossible numeric issue, we set thecovariance to unit  matrix and  minimize H ( M ij ) − H ( M ij | T i ) instead, where T i is the random variable of local history of agent i .
Revised: To deal with this numeric issue, we use the unit covariance matrix and try to minimize H ( M ij ) − H ( M ij | T i ) instead.

Original: This is equivalent  because H ( M ij | T i ) is  a multivariate Gaussian random variable whose entropy is a constant ( log(det(2 πe Σ )) / 2 , where Σ is the unit matrix).
Revised: This is equivalent to minimizing H ( M ij ) because H ( M ij | T i ) is the entropy of a multivariate Gaussian random variable and thus is a constant ( log(det(2 πe Σ )) / 2 , where Σ is the unit matrix in our formulation).

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: We use the same technique as in the mutual information term – introducing an variational inference distribution r ( m ij ) to approximate p ( m ij ) , and we can get:
Revised: We use the same technique as for the mutual information term by introducing an variational inference distribution r ( m ij ) to approximate p ( m ij ) , and we can get:

Original: [Equation]
Revised: [Equation]

Original: This bound holds for any distribution r ( M ij ) .
Revised: This bound holds for any distribution r ( M ij ) .

Original: To facilitate cutting off messages, we use unit Gaussian distribution N (0 , I ) .
Revised: To facilitate cutting off messages, we use unit Gaussian distribution N (0 , I ) .

Original: Combining Eq.
Revised: Combining Eq.

Original: 3 and 5, we get a tractable variational lower bound of our objective in Eq. 2:
Revised: 3 and 5, we get a tractable variational lower bound of our objective in Eq. 2:

Original: [Equation]
Revised: [Equation]

Original: We optimize this bound to get a expressive and succinct message embedding.
Revised: We optimize this bound to generate an expressive and succinct message embedding.

Original: Speciﬁcally, we minimize:
Revised: Speciﬁcally, we minimize:

Original: [Equation]
Revised: [Equation]

Original: Intuitively [Equation] the ﬁrst term, which we callthe expressiveness loss, guarantees that messages sent can reduce the uncertainty in action-value functions of other agents.
Revised: Intuitively, the ﬁrst term, which we call expressiveness loss, ensures that communication aims to reduce the uncertainty on action-value functions of other agents.

Original: The second term, called succinctness loss, forces messages to get close to  unit Gaussian distribution.
Revised: The second term, called succinctness loss, forces messages to get close to the unit Gaussian distribution.

Original: Since we set  covariance of  message embedding to  unit matrix, this term actually pushes the mean of messages to the origin of the latent space.
Revised: Since we set the covariance of the latent message variable to the unit matrix, this term actually pushes the mean of message distributions to the origin of the latent space.

Original: Use this two losses leading to an embedding where useless messages distributes all near the origin, while messages that can contains information important  for decision processes of other agents occupy other spaces.
Revised: Using these two losses leads to an embedding where useless messages distribute near the origin, while messages that contain important information for the decision-making processes of other agents occupy other spaces.

Original: Note that loss in Eq.
Revised: Note that loss in Eq.

Original: 7 are used to update parameters in  message encoder.
Revised: 7 is used to update parameters in the message encoder.

Original: In the meantime, allthe components (individual action-value function, message encoder, and mixing network) are trainedend-to-end by the TD loss as in QMIX.
Revised: In the meantime, all components (individual action-value function, message encoder, and mixing network) are trained in an end-to-end manner.

Original: Thus, the message encoder f m ( M i | T i  ; θ c ) is updatedguided by two gradients: stochastic gradient given by Eq. 6 and the gradient associated with TD loss.
Revised: Thus, the message encoder f m ( τ i , j ; θ c ) is updated by two gradients: the gradient of L c ( θ c ) and the gradient associated with TD loss.

Original: 3. C UTTING O FF M ESSAGES
Revised: 3. C UTTING O FF M ESSAGES

Original: Our objective compresses messages which can not reduce the uncertainty in action-value functions of other agents  to the origin of the latent message space.
Revised: Our objective compresses messages which can not reduce the uncertainty in action-value functions of other agents close to the origin of the latent message space.

Original: This naturally gives us a hint  to dropout meaningless messages – we can order the message distributions according to their means and drop accordingly.
Revised: This naturally gives us a hint on how to drop meaningless messages – we can order the message distributions according to their means and drop accordingly.

Original: Note that since we adopt diagonal covariance  for our message embedding, bits in a message are independent.
Revised: Note that since we use a unit covariance matrix for the latent message variable, bits in a message are independent.

Original: We thus make decision bit by bit and send messages with various length.
Revised: Thus, we can make decisions in a bit-by-bit fashion and send messages with various lengths.

Original: In this way, such method  not onlyﬁnds out when  to communicate (agent keep silence when all bits are cut), but also what to send (how many bits are sent and their values).
Revised: In this way, our method learns not only when and who to communicate (agent keeps silence when all bits are cut), but also what to communicate (how many bits are sent and their values).

Original: Implementation details are discussed in Appendix B.
Revised: More details are discussed in Appendix B.

Revised: Our framework adopts the centralized training with decentralized execution paradigm.

Revised: During centralized training, we assume the learning algorithm has access to all agents’ individual observationaction histories and the global state s .

Revised: During execution, agents communicate and act in a decentralized fashion based on the learned message encoder f m ( τ i , j ; θ c ) and action-value functions

Revised: Q i ( τ i , a i , m ini ) .

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Deep multi-agent reinforcement learning has witnessed vigorous progress in recent years.
Revised: Deep multi-agent reinforcement learning has witnessed vigorous progress in recent years.

Original: COMA (Foerster et al., 2018), MADDPG (Lowe et al., 2017), and PR2 (Wen et al., 2019) explores multi-agent policy gradients and respectively address the problem of credit assignment, learning in mixed environments and recursive reasoning.
Revised: COMA (Foerster et al., 2018), MADDPG (Lowe et al., 2017), and PR2 (Wen et al., 2019) explores multi-agent policy gradients and respectively address the problem of credit assignment, learning in mixed environments and recursive reasoning.

Original: Another line of research focuses on value-based multi-agent RL, among which value-function factorization is the most popular method.
Revised: Another line of research focuses on value-based multi-agent RL, among which value-function factorization is the most popular method.

Original: Three representative examples: VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QTRAN (Son et al., 2019) gradually increase the representation ability of the mixing network.
Revised: Three representative examples: VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QTRAN (Son et al., 2019) gradually increase the representation ability of the mixing network.

Original: In particular,
Revised: In particular,

Original: QMIX (Rashid et al., 2018) stands out as a scalable and robust algorithm and achieves state-ofthe-art results on StarCraft unit micromanagement benchmark (Samvelyan et al., 2019b).
Revised: QMIX (Rashid et al., 2018) stands out as a scalable and robust algorithm and achieves state-ofthe-art results on StarCraft unit micromanagement benchmark (Samvelyan et al., 2019).

Original: Communication is a hot topic in multi-agent reinforcement learning.
Revised: Communication is a hot topic in multi-agent reinforcement learning.

Original: End-to-end learning with differentiable communication channel is a popular approach now.
Revised: End-to-end learning with differentiable communication channel is a popular approach now.

Original: Sukhbaatar et al.
Revised: Sukhbaatar et al.

Original: ; Hoshen (2017); Jiang & Lu (2018); Singh et al.
Revised: ; Hoshen (2017); Jiang & Lu (2018); Singh et al.

Original: Das et al.
Revised: Das et al.

Original: (2019) focus on learning decentralized communication protocol and address the problem of when and who to communicate.
Revised: (2019) focus on learning decentralized communication protocol and address the problem of when and who to communicate.

Original: Foerster et al.
Revised: Foerster et al.

Original: (2016); Das et al.
Revised: (2016); Das et al.

Original: ; Lazaridou et al.
Revised: ; Lazaridou et al.

Original: (2017); Mordatch & Abbeel (2018) study the emergence of natural language in the context of multi-agent learning.
Revised: (2017); Mordatch & Abbeel (2018) study the emergence of natural language in the context of multi-agent learning.

Original: IC3Net (Singh et al., 2019) learns gate to control the agents to only communicate with their teammates in mixed multi-agent environment.
Revised: IC3Net (Singh et al., 2019) learns gate to control the agents to only communicate with their teammates in mixed multi-agent environment.

Original: Zhang & Lesser (2013); Kim et al.
Revised: Zhang & Lesser (2013); Kim et al.

Original: (2019) study action coordination under limited communication channel and thus arealso related to our works.
Revised: (2019) study action coordination under limited communication channel and thus are related to our works.

Original: The difference lies in that they do not explicitly minimize communication.
Revised: The difference lies in that they do not explicitly minimize communication.

Original: Social inﬂuence (Jaques et al., 2019) and InfoBot (Goyal et al., 2019) penalize message that has no effect on policies of other agents.
Revised: Social inﬂuence (Jaques et al., 2019) and InfoBot (Goyal et al., 2019) penalize message that has no inﬂuence on policies of other agents.

Original: The most relatedwork to this paper is TarMAC (Das et al., 2019), where attention mechanism is used to differentiate the importance of incoming messages.
Revised: Work that is most related to this paper is TarMAC (Das et al., 2019), where attention mechanism is used to differentiate the importance of incoming messages.

Original: In comparison, we use variation inference to decide the content of messages and whether a message should be sent under the guidance of global reward signals.
Revised: In comparison, we use variation inference to decide the content of messages and whether a message should be sent under the guidance of global reward signals.

Original: We compare our method with TarMAC and a baseline combining TarMAC and QMIX in our methods.
Revised: We compare our method with TarMAC and a baseline combining TarMAC and QMIX in our experiments.

Original: StarCraft unit micromanagement has attracted lots of research interests for its high degree of control complexity and environmental stochasticity.
Revised: StarCraft unit micromanagement has attracted lots of research interests for its high degree of control complexity and environmental stochasticity.

Original: Usunier et al.
Revised: Usunier et al.

Original: (2017) and Peng et al.
Revised: (2017) and Peng et al.

Original: (2017) study this problem from a centralized perspective.
Revised: (2017) study this problem from a centralized perspective.

Original: In order to facilitate decentralized control, we use the setup introduced by Samvelyan et al.
Revised: In order to facilitate decentralized control, we use the setup introduced by Samvelyan et al.

Original: (2019b), which is the same in Foerster et al.
Revised: (2019), which is the same in Foerster et al.

Original: (2017; 2018) and Rashid et al.
Revised: (2017; 2018) and Rashid et al.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  strategy.
Revised: minimized communication strategy.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we show our experiments to answer the following questions: (i) Is miscoordination problem of full value function factorization methods widespread?
Revised: In this section, we show our experiments to answer the following questions: (i) Is miscoordination problem of full value function factorization methods widespread?

Original: (ii) Can our method learn the minimized message required by the task?
Revised: (ii) Can our method learn the minimized communication protocol required by a task?

Original: (iii) Can the learnt message distribution reduce uncertainty in value functions of other agents?
Revised: (iii) Can the learnt message distribution reduce uncertainty in value functions of other agents?

Original: (iv) How does our method differ from communication via attention mechanism?
Revised: (iv) How does our method differ from communication via attention mechanism?

Original: (v) How do β inﬂuence the communication protocol?
Revised: (v) How does β inﬂuence the communication protocol?

Original: We will ﬁrst show three simple examples to clarify our idea and then provide performance analysis on StarCraftII unit micromanagement task.
Revised: We will ﬁrst show three simple examples to clarify our idea and then provide performance analysis on StarCraftII unit micromanagement task.

Original: For evaluation, all experiments are carried out with 3 random seeds and results are shown with 95% conﬁdence interval.
Revised: For evaluation, all experiments are carried out withrandom seeds and results are shown with 95% conﬁdence interval.

Original: Videos of our experiments on StarCraft II are available online 2 .
Revised: Videos of our experiments on StarCraft II are available online 2 .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 5. B ASELINES AND A BLATIONS
Revised: 5. B ASELINES AND A BLATIONS

Original: We compare our method with various baselines and ablations (i) QMIX (Rashid et al., 2018).
Revised: We compare our method with various baselines and ablations (i) QMIX (Rashid et al., 2018).

Original: We use QMIX as representation for full factorization method; (ii) TarMAC
Revised: We use QMIX as representation for full factorization method; (ii) TarMAC

Original: (Das et al., 2019).
Revised: (Das et al., 2019).

Original: TarMAC is a state-of-the-art attentional communication method.
Revised: TarMAC is a state-of-the-art attentional communication method.

Original: Comparison with it can illustrate the difference between our method and attention mechanism.
Revised: Comparison with it can illustrate the difference between our method and attention mechanism.

Original: (iii) TarMAC+QMIX.
Revised: (iii) TarMAC+QMIX.

Original: Directly compare with TarMAC is perhaps unfair because it is not designed for tasks with shared rewards.
Revised: Directly compare with TarMAC is perhaps unfair because it is not designed for tasks with shared rewards.

Original: Therefore, we integrate the communication component of TarMAC into QMIX and compare with this strong baseline.
Revised: Therefore, we integrate the communication component of TarMAC into QMIX and compare with this strong baseline.

Original: 5. D IDACTIC E XAMPLES
Revised: 5. D IDACTIC E XAMPLES

Original: We ﬁrst demonstrate our idea on a set of didactic examples.
Revised: We ﬁrst demonstrate our idea on a set of didactic examples.

Original: Sensor network is a frequently used testbed in multi-agent learning ﬁeld (Kumar et al., 2011; Zhang & Lesser, 2011).
Revised: Sensor network is a frequently used testbed in multi-agent learning ﬁeld (Kumar et al., 2011; Zhang & Lesser, 2011).

Original: We use a 3-chain sensor conﬁguration in task sensor .
Revised: We use a 3-chain sensor conﬁguration in task sensor .

Original: Each sensor is controlled by one agent and they are rewarded for successfully locating targets, which requires two sensors to scan the same area simultaneously when the target appears.
Revised: Each sensor is controlled by one agent and they are rewarded for successfully locating targets, which requires two sensors to scan the same area simultaneously when the target appears.

Original: At each timestep, target 1 appears in areawith possibility 1 and locating it induces a team reward of 20; target 2 appears with probability 0.
Revised: At each timestep, target 1 appears in areawith possibility 1 and locating it induces a team reward of 20; target 2 appears with probability 0.

Original: in area 2 and agents are rewarded 30 for locating it.
Revised: in area 2 and agents are rewarded 30 for locating it.

Original: Agents can observe whether a target is present in nearby areas
Revised: Agents can observe whether a target is present in nearby areas

Original: and need to take one of ﬁve actions: scanning north , east , south , west , and noop .
Revised: and need to take one of ﬁve actions: scanning north , east , south , west , and noop .

Original: Every scan
Revised: Every scan

Original: induces
Revised: induces

Original: a cost of -
Revised: a cost of -

Original: In the optimal policy, when the target 2 appears, sensor 1 should turn itself off while sensor 2 andare expected to scan area 2 to get the reward.
Revised: In the optimal policy, when the target 2 appears, sensor 1 should turn itself off while sensor 2 andare expected to scan area 2 to get the reward.

Original: And when target 2 is absent, sensor 1 and 2 need to cooperatively scan area 1 while sensor 3 take noop .
Revised: And when target 2 is absent, sensor 1 and 2 need to cooperatively scan area 1 while sensor 3 take noop .

Original: Sensor is representative of a class of tasks where the uncertainty about the true state causes policies learned by full value function factorization method to be sub-optimal – sensor 1 has to know whether
Revised: Sensor is representative of a class of tasks where the uncertainty about the true state causes policies learned by full value function factorization method to be sub-optimal – sensor 1 has to know whether

Original: [Figure]
Revised: [Figure]

Original: the target is present in area 2 to make decision.
Revised: the target is present in area 2 to make decision.

Original: However, mixing network of QMIX cannot provide such information.
Revised: However, mixing network of QMIX cannot provide such information.

Original: As a result, QMIX learners converge to a sub-optimal policy which gets a team reward of 12.5 on average per step (see Fig. 2(b)).
Revised: As a result, QMIX learners converge to a sub-optimal policy which gets a team reward of 12.5 on average per step (see Fig. 2(b)).

Original: We are particularly interested in whether our method can learn the minimized communication strategy.
Revised: We are particularly interested in whether our method can learn the minimized communication strategy.

Original: Fig.
Revised: Fig.

Original: 3(a) - 3(f) shows the latent message space learned by our method.
Revised: 3(a) - 3(f) shows the latent message space learned by our method.

Original: Using our method and set β to 10 − 3 , agent 3 learns to send a bit to tell agent 1 whether target 2 appears.
Revised: Using NDQ and set β to 10 − 3 , agent 3 learns to send a bit to tell agent 1 whether target 2 appears.

Original: In the meantime, the latent message distribution between any other pair of agents is close unit Gaussian and thus is dropped.
Revised: In the meantime, the latent message distribution between any other pair of agents is close unit Gaussian and thus is dropped.

Original: This results indicates that our method has learnt the minimized conditional graph and can explain why our method can still perform optimally when 80% messages are cut off (Fig. 2(c)).
Revised: This results indicates that our method has learnt the minimized conditional graph and can explain why our method can still perform optimally when 80% messages are cut off (Fig. 2(c)).

Original: When β becomes too large (1.0), all the message bits are pushed below cutting threshold (Fig. 3(a) and 3(d)); and when β is too small ( 10 − 5 ), our method puts more attention on reducing uncertainty in Q-functions in stead of compressing messages, so both agent 3 and agent 2 pass message to agent 1 (Fig. 3(c) and 3(f)).
Revised: When β becomes too large (1.0), all the message bits are pushed below cutting threshold (Fig. 3(a) and 3(d)); and when β is too small ( 10 − 5 ), NDQ puts more attention on reducing uncertainty in Q-functions in stead of compressing messages, so both agent 3 and agent 2 pass message to agent 1 (Fig. 3(c) and 3(f)).

Original: The second example, called Hallway (Fig. 4(a)), Task hallway.
Revised: The second example, called hallway (Fig. 4(a)), Task hallway.

Original: A Dec-POMDP with two agents randomly starting at states a 1 to a m and b 1 to b n respectively.
Revised: A Dec-POMDP with two agents randomly starting at states a 1 to a m and b 1 to b n respectively.

Original: Agents can observe its own position and chooses to move left, move right, or keep still at each timestep.
Revised: Agents can observe its own position and chooses to move left, move right, or keep still at each timestep.

Original: Agents win if they arrive at state g simultaneously.
Revised: Agents win if they arrive at state g simultaneously.

Original: Otherwise, if any agent arrives at g earlier than the other, they receive no reward and the next episode begins.
Revised: Otherwise, if any agent arrives at g earlier than the other, they receive no reward and the next episode begins.

Original: Winning reward is set to 10 and the horizon is set to max ( m, n ) +to avoid inﬁnite loop.
Revised: Winning reward is set to 10 and the horizon is set to max ( m, n ) +to avoid inﬁnite loop.

Original: Hallway aims to show that the miscoordination problem of full factorization method can be severe in multi-step scenarios.
Revised: Hallway aims to show that the miscoordination problem of full factorization method can be severe in multi-step scenarios.

Original: We show results in Fig. 4(b) where m = n = 4 and our method outperforms all the baselines.
Revised: We show results in Fig. 4(b) where m = n = 4 and our method outperforms all the baselines.

Original: We are again particularly interested in the message embedding space learnt by our method.
Revised: We are again particularly interested in the message embedding space learnt by our method.

Original: We show an episode in Fig 5.
Revised: We show an episode in Fig 5.

Original: Two agents begin at a 4 and b 3 respectively.
Revised: Two agents begin at a 4 and b 3 respectively.

Original: They ﬁrst move left silently (t=1 and t=2) until agent B arrives b 1 .
Revised: They ﬁrst move left silently (t=1 and t=2) until agent B arrives b 1 .

Original: On arriving b 1 , it send a bit whose value is 5.24 to A. After sending this bit, B stays at b 1 until it receives a bit from A indicating that A has arrived at a 1 .
Revised: On arriving b 1 , it send a bit whose value is 5.24 to A. After sending this bit, B stays at b 1 until it receives a bit from A indicating that A has arrived at a 1 .

Original: After that, they move left together and win.
Revised: After that, they move left together and win.

Original: This is the minimized communication strategy and our method can therefore still win in 100% episodes when 80% communicating bits are dropped 4(c).
Revised: This is the minimized communication strategy and our method can therefore still win in 100% episodes when 80% communicating bits are dropped 4(c).

Original: The third task, independent search , aims to demonstrate that our method can learn not to communicate in scenarios where agents are independent.
Revised: The third task, independent search , aims to demonstrate that our method can learn not to communicate in scenarios where agents are independent.

Original: Task description and results analysis are deferred to Appendix C.1.
Revised: Task description and results analysis are deferred to Appendix C.1.

Original: 5. M AXIMUM V ALUE F UNCTION F ACTORIZATION IN S TAR C RAFT II
Revised: 5. M AXIMUM V ALUE F UNCTION F ACTORIZATION IN S TAR C RAFT II

Original: To demonstrate that the miscoordination problem is widespread in multi-agent scenarios, we apply our method and ablations to decentralized StarCraft II micromanagement benchmark.
Revised: To demonstrate that the miscoordination problem is widespread in multi-agent scenarios, we apply our method and ablations to decentralized StarCraft II micromanagement benchmark.

Original: We use the
Revised: We use the

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: setup introduced by Samvelyan et al.
Revised: setup introduced by Samvelyan et al.

Original: (2019b)

Original: and consider combat scenarios.
Revised: and consider combat scenarios.

Original: We describe the settings in detail in Appendix C.2.
Revised: We describe the settings in detail in Appendix C.2.

Original: We further increase the difﬁculty of action coordination in these tasks by i) reducing the sight range of the agents from 9 to 2; ii) introducing challenging maps that has complex terrain or with highly random spawning positions for units.
Revised: We further increase the difﬁculty of action coordination in these tasks by i) reducing the sight range of the agents from 9 to 2; ii) introducing challenging maps that has complex terrain or with highly random spawning positions for units.

Original: We compare and analysis our method on the six maps shown in
Revised: We compare and analysis our method on the six maps shown in

Original: Fig.
Revised: Fig.

Original: 6: 3b vs 1h1m, 3s vs 5z,1o2r vs 4r, 5z vs 1ul, MMM, and 1o10b vs1r.
Revised: 6: 3b vs 1h1m, 3s vs 5z, vs 4r, 5z vs 1ul, MMM, and 1o10b vs

Original: 3s vs 5z and MMM are the same as in Samvelyan et al.
Revised: 3s vs 5z and MMM are the same as in Samvelyan et al.

Original: (2019b)

Original: but with a narrow sight range.
Revised: but with a narrow sight range.

Original: Detailed descriptions of these scenarios are provided in Appendix C.2.
Revised: Detailed descriptions of these scenarios are provided in Appendix C.2.

Original: 5.3. P ERFORMANCE C OMPARISON
Revised: 5.3. P ERFORMANCE C OMPARISON

Original: Across all StarCraft II scenarios, we use the same set of hyper-parameters: the length of message m ij is set to 3 and β is set to 10 − 3 .
Revised: Across all StarCraft II scenarios, we use the same set of hyper-parameters: the length of message m ij is set to 3 and β is set to 10 − 3 .

Original: To evaluate the learned policy, we pause training every 100k environment steps and run 48 testing episode.
Revised: To evaluate the learned policy, we pause training every 100k environment steps and run 48 testing episode.

Original: We show the performance when no message is cut of our method and baselines in Fig.
Revised: We show the performance when no message is cut of our method and baselines in Fig.

Original: Superior performance of methods with communication over QMIX demonstrates that the miscoordination problem of full factorization method is widespread, especially in scenarios with high stochasticity, such as 1o2r vs 4r, 3b vs 1h1m, and 1o10b vs 1r.
Revised: Superior performance of methods with communication over QMIX demonstrates that the miscoordination problem of full factorization method is widespread, especially in scenarios with high stochasticity, such as 1o2r vs 4r, 3b vs 1h1m, and 1o10b vs 1r.

Original: Notably, our method outperforms attentional communication mechanism by a large margin.
Revised: Notably, our method outperforms attentional communication mechanism by a large margin.

Original: Since both of these two methods use the
Revised: Since both of these two methods use the

Original: TD-error to update parameters, these results highlights the role of constraints that we impose on our message embedding, especially the expressiveness requirement.
Revised: TD-error to update parameters, these results highlights the role of constraints that we impose on our message embedding, especially the expressiveness requirement.

Original: In all scenarios, performance of pure TarMAC struggles because it cannot deal with global rewards.
Revised: In all scenarios, performance of pure TarMAC struggles because it cannot deal with global rewards.

Original: 5.3. M ESSAGE C UT O FF
Revised: 5.3. M ESSAGE C UT O FF

Original: Results shown in Fig.
Revised: Results shown in Fig.

Original: 7 is the performance when the coordination graph is a complete graph.
Revised: 7 is the performance when the coordination graph is a complete graph.

Original: In this paper, we are interested in learning nearly decomposable structure – agents only need to coordinate their actions with few agents at a time.
Revised: In this paper, we are interested in learning nearly decomposable structure – agents only need to coordinate their actions with few agents at a time.

Original: Therefore, we cut off 80% messages according to the mean of distribution when testing and show the results in Fig.
Revised: Therefore, we cut off 80% messages according to the mean of distribution when testing and show the results in Fig.

Original: The results indicates that we can omit more than 80%  communication without signiﬁcantly affect performance.
Revised: The results indicate that we can omit more than 80% of communication without signiﬁcantly affect performance.

Original: We show the strategies learnt by our method in supplementary videos.
Revised: We show the strategies learnt by our method in supplementary videos.

Original: For comparison, we cut off messages in QMIX+TarMAC whose weights are 80% smallest and ﬁnd that the performance of QMIX+TarMAC drops signiﬁcantly (Fig.
Revised: For comparison, we cut off messages in QMIX+TarMAC whose weights are 80% smallest and ﬁnd that the performance of QMIX+TarMAC drops signiﬁcantly (Fig.

Original: These results indicate that our method is more powerful in terms of message cut off compared to  attentional communication method.
Revised: These results indicate that our method is more powerful in terms of message cut off compared to the attentional communication method.

Original: We further cut all the messages in models learnt by our methods and show the development of testing performance in Fig.
Revised: We further cut all the messages in models learnt by our methods and show the development of testing performance in Fig.

Original: The win rates decrease dramatically, proving that the superior performance of our method when 80% message is dropped comes from expressive and succinct communication protocols in stead of implicit coordination strategies are learnt.
Revised: The win rates decrease dramatically, proving that the superior performance of our method when 80% message is dropped comes from expressive and succinct communication protocols instead of implicit coordination strategies are learnt.

Original:  C LOSING R EMARKS
Revised:  C LOSING R EMARKS

Original: In this paper we presented a novel multi-agent learning framework within the paradigm of centralized training with decentralized execution.
Revised: In this paper we presented a novel multi-agent learning framework within the paradigm of centralized training with decentralized execution.

Original: This framework fuses value function factorization learning and communication learning, and efﬁciently learns nearly domcomposable value functions for agents to act independently most of the time and communicate when it is necessary for coordination.
Revised: This framework fuses value function factorization learning and communication learning, and efﬁciently learns nearly decomposable value functions for agents to act independently most of the time and communicate when it is necessary for coordination.

Original: We introduce two information-theoretical regularizers to minimize overall communication while maximizing the message information for coordination.
Revised: We introduce two information-theoretical regularizers to minimize overall communication while maximizing the message information for coordination.

Original: Empirical results in challenging StarCraft II tasks show that our method signiﬁcantly outperforms baseline methods and allows to reduce communication by more than 80% without sacriﬁcing the performance.
Revised: Empirical results in challenging StarCraft II tasks show that our method signiﬁcantly outperforms baseline methods and allows to reduce communication by more than 80% without sacriﬁcing the performance.

Original: We also observe that nearly minimal messages (e g., with one or two bits) are learned to communicate between agents in order to ensure effective coordination
Revised: We also observe that nearly minimal messages (e g., with one or two bits) are learned to communicate between agents in order to ensure effective coordination

Original: A PPENDIX A V ARIATIONAL B OUND ON M UTUAL I NFORMATION
Revised: A PPENDIX A V ARIATIONAL B OUND ON M UTUAL I NFORMATION

Original: In order to let messages effectively reduce the uncertainty in action-value functions of other agents, we propose to maximize mutual information between A j and M ij .
Revised: In order to let messages effectively reduce the uncertainty in action-value functions of other agents, we propose to maximize mutual information between A j and M ij .

Original: We borrow ideas from variational inference literature and derive a lower bound on this mutual information regularizer.
Revised: We borrow ideas from variational inference literature and derive a lower bound on this mutual information regularizer.

Original: Theorem 1.
Revised: Theorem 1.

Original: A lower bound of mutual information I θ c
Revised: A lower bound of mutual information I θ c

Original: ( A j ; M ij | T j , M ( - i ) j ) is
Revised: ( A j ; M ij | T j , M ( - i ) j ) is

Original: [Equation]
Revised: [Equation]

Original: where T j is local action-observation history of agent j and T = (cid:104) T 1 , T 2 , . .
Revised: where T j is local action-observation history of agent j and T = (cid:104) T 1 , T 2 , . .

Original: , T n (cid:105) is the joint local history sampled from the replay buffer D , q ξ ( A j | T j , M inj ) is the variational posterior estimator with parameters ξ .
Revised: , T n (cid:105) is the joint local history sampled from the replay buffer D , q ξ ( A j | T j , M inj ) is the variational posterior estimator with parameters ξ .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: where p ( a j | τ j , m inj ) is deﬁned by our encoder f m (m i | τ i  ; θ c ) and Markov Chain:
Revised: where p ( a j | τ j , m inj ) is determined by our encoder f m ( τ i , j ; θ c ) and Markov Chain:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We introduce q ξ ( a j | τ j , m inj ) as a variational approximation to p ( a j | τ j , m inj ) .
Revised: We introduce q ξ ( a j | τ j , m inj ) as a variational approximation to p ( a j | τ j , m inj ) .

Original: Since
Revised: Since

Original: DKL
Revised: D

Original: [Equation]
Revised: [Equation]

Original: p

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: 16) we have
Revised: 16) we have

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: for the mutual information term:
Revised: for the mutual information term:

Original: [Equation]
Revised: [Equation]

Original: [Equation] 
Revised: [Equation] [Equation] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Equation]  [Equation]
Revised: [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  Because H θ c ( A j | T j , M ( - i ) j ) ≥ 0 , we get the lower bound in Theorem 1.
Revised: c ( A j | T j , M ( - i ) j ) . Because H θ c ( A j | T j , M ( - i ) j ) ≥ 0 , we get the lower bound in Theorem 1.

Original: B I MPLEMENTATION D ETAILS B. D ETAILS OF M ESSAGE D ROPPING
Revised: B I MPLEMENTATION D ETAILS B. D ETAILS OF M ESSAGE D ROPPING

Original: In our methods, not only the number of messages, but also the length of messages are minimized.
Revised: In our methods, not only the number of messages, but also the length of messages are minimized.

Original: In other words, we send messages with varying length in the communication channel.
Revised: In other words, we send messages with varying lengths.

Original: However, messages at the recipient side are feed into  action-value function approximator, which requires inputs with ﬁxed length.
Revised: However, messages at the recipient side are feed into a action-value function approximator, which requires inputs to have the same length.

Original: To solve this problem, we send a mask indicating which bits are sent along with the messages.
Revised: To solve this problem, we send a mask indicating which bits are dropped along with the messages.

Original: To save channel width, masks are regraded as a binary number, so each of them only consumes a negligible log -scale space compared to the length of message.
Revised: To save channel bandwidth, masks are regarded as a binary number, so each of them only consumes a negligible log -scale space compared to the length of message.

Original: B. N ETWORK A RCHITECTURE , H YPERPARAMETERS , AND I NFRASTRUCTURE
Revised: B. N ETWORK A RCHITECTURE , H YPERPARAMETERS , AND I NFRASTRUCTURE

Original: We base our framework on PyMARL implementation of QMIX (Samvelyan et al., 2019a) and use its default parameters to carry out all the experiments.
Revised: We base our framework on PyMARL implementation of QMIX (Samvelyan et al., 2019) and use its default parameters to carry out all the experiments.

Original: We train our models on an NVIDIA RTX 2080TI GPU using experience sampled from 16 parallel environments.
Revised: We train our models on an NVIDIA RTX 2080TI GPU using experience sampled from 16 parallel environments.

Original: We have collected a total of 20 million time step data for each task.
Revised: We have collected a total of 20 million time step data for each task.

Original: Speciﬁc values of the number of agents n , the scaling weight β  and the communication bandwidth c len of each agent can be found in Table 1.
Revised: Speciﬁc values of the number of agents n , the scaling weight β , λ and the communication bandwidth l of each agent can be found in Table 1.

Original: C E XPERIMENTAL R ESULTS C. D IDACTIC EXAMPLE : INDEPENDENT SEARCH
Revised: C E XPERIMENTAL R ESULTS C. D IDACTIC EXAMPLE : INDEPENDENT SEARCH

Original: In independent search , two agents are ﬁnding landmarks in two independent 5 × 5 rooms fortime steps (see Fig. 9).
Revised: In independent search , two agents are ﬁnding landmarks in two independent 5 × 5 rooms fortime steps (see Fig. 9).

Original: Agent is rewarded 1 every time step they step on the landmark in its room.
Revised: Agent is rewarded 1 every time step they step on the landmark in its room.

Original: Independent search is an example where agents are totally independent.
Revised: Independent search is an example where agents are totally independent.

Original: This task aims to demonstrate that our method can learn not to communicate in independent scenarios.
Revised: This task aims to demonstrate that our method can learn not to communicate in independent scenarios.

Original: We show the demonstrative plot in Fig.
Revised: We show the demonstrative plot in Fig.

Original: 9 and team performance comparison in Table 2.
Revised: 9 and team performance comparison in Table 2.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: We ﬁrst describe the scenarios that we consider in details.
Revised: We ﬁrst describe the scenarios that we consider in details.

Original: We consider combat scenarios where the enemy units are controlled by StarCraft II built-in AI (difﬁculty level is set to medium ) and each of the ally unit is controlled by a learning agent.
Revised: We consider combat scenarios where the enemy units are controlled by StarCraft II built-in AI (difﬁculty level is set to medium ) and each of the ally unit is controlled by a learning agent.

Original: Units of two groups can be asymmetric and the initial placement is random.
Revised: Units of two groups can be asymmetric and the initial placement is random.

Original: At each timestep, each agent takes one action from the discrete action space consists of the following action: noop , move [ direction ] , attack [ enemy id ] , and stop .
Revised: At each timestep, each agent takes one action from the discrete action space consists of the following action: noop , move [ direction ] , attack [ enemy id ] , and stop .

Original: Under the control of these actions, agents move and attack in a continuous map.
Revised: Under the control of these actions, agents move and attack in a continuous map.

Original: A global reward that is equal to the total damage dealt on the enemy units is given at each timestep.
Revised: A global reward that is equal to the total damage dealt on the enemy units is given at each timestep.

Original: Killing each enemy unit and winning a combat induces an extra bonus of 10 and 200 respectively.
Revised: Killing each enemy unit and winning a combat induces an extra bonus of 10 and 200 respectively.

Original: 3b vs 1h1m : 3 Banelings try to kill a Hydralisk assisted by a Medivac.
Revised: 3b vs 1h1m : 3 Banelings try to kill a Hydralisk assisted by a Medivac.

Original: 3 Banelings together can just blow up the hydralisk.
Revised: 3 Banelings together can just blow up the hydralisk.

Original: Therefore, they should not give the hydralisk rest time during which the Medivec can restore its health.
Revised: Therefore, they should not give the hydralisk rest time during which the Medivec can restore its health.

Original: Banelings have to align their attacking time to get the winning reward.
Revised: Banelings have to align their attacking time to get the winning reward.

Original: This scenario is designed to test whether our method can learn communication protocol to coordinate actions.
Revised: This scenario is designed to test whether our method can learn communication protocol to coordinate actions.

Original: 3s vs 5z : 3 Stalkers encounter 5 Zealots on a map.
Revised: 3s vs 5z : 3 Stalkers encounter 5 Zealots on a map.

Original: Zealots can cause high damage but are much slower so that Stalkers have to take advantage of this to beat Zealots using a technique called kiting –
Revised: Zealots can cause high damage but are much slower so that Stalkers have to take advantage of this to beat Zealots using a technique called kiting –

Original: Stalkers should alternatively attack the Zealots and ﬂee for a distance.
Revised: Stalkers should alternatively attack the Zealots and ﬂee for a distance.

Original: Kiting requires the knowledge of exact positions of enemies.
Revised: Kiting requires the knowledge of exact positions of enemies.

Original: Since we narrow down the sight range of units, 3 Stalkers has to coordinate and learn to communicate necessary messages to
Revised: Since we narrow down the sight range of units, 3 Stalkers has to coordinate and learn to communicate necessary messages to

Original: win.
Revised: win.

Original: 1o2r vs 4r :
Revised: 1o2r vs 4r :

Original: An Overseer has found 4 Reapers.
Revised: An Overseer has found 4 Reapers.

Original: Its ally units, 2 Roaches, need to get to there and attack the Reapers.
Revised: Its ally units, 2 Roaches, need to get to there and attack the Reapers.

Original: At the beginning of an episode, the Overseer and Reapers spawn at a random point on the map while the Roaches are initialized at another random point.
Revised: At the beginning of an episode, the Overseer and Reapers spawn at a random point on the map while the Roaches are initialized at another random point.

Original: Given that only the Overseer knows the position of enemy, a learning algorithm has to learn to communicate this message to effectively win the combat.
Revised: Given that only the Overseer knows the position of enemy, a learning algorithm has to learn to communicate this message to effectively win the combat.

Original: 5z vs 1ul : 5 Zealots try to kill a powerful Ultralisk.
Revised: 5z vs 1ul : 5 Zealots try to kill a powerful Ultralisk.

Original: A sophisticated micro-trick demanding right positioning and attack timing has to be learnt to win.
Revised: A sophisticated micro-trick demanding right positioning and attack timing has to be learnt to win.

Original: MMM :
Revised: MMM :

Original: Symmetric teams consist of 7 Marines, 2 Marauders and 1 Medivac spawn at two ﬁxed points and the enemy team are tasked to attack the ally team.
Revised: Symmetric teams consist of 7 Marines, 2 Marauders and 1 Medivac spawn at two ﬁxed points and the enemy team are tasked to attack the ally team.

Original: To win the battle, agents have to learn to communicate their health to teammates.
Revised: To win the battle, agents have to learn to communicate their health to teammates.

Original: This task can demonstrate the scalability of our method.
Revised: This task can demonstrate the scalability of our method.

Original: 1o10b vs 1r :
Revised: 1o10b vs 1r :

Original: In a map full of cliffs, an Overseer detects a Roach, and its teammates, 10 banelings need to kill this Roach to get winning reward.
Revised: In a map full of cliffs, an Overseer detects a Roach, and its teammates, 10 banelings need to kill this Roach to get winning reward.

Original: The overseer and the Roach spawn at a random point while the Banelings spawn randomly all round the map.
Revised: The overseer and the Roach spawn at a random point while the Banelings spawn randomly all round the map.

Original: In the minimized communication strategy, Banelings can keep silence and the Overseer needs to encode its position and send it to Banelings.
Revised: In the minimized communication strategy, Banelings can keep silence and the Overseer needs to encode its position and send it to Banelings.

Original: We
Revised: We

Original: use
Revised: use

Original: this
Revised: this

Original: task to test the performance of our method in complex scenarios.
Revised: task to test the performance of our method in complex scenarios.

Original: [Figure]
Revised: [Figure]

Original: Performance of our methods when all messages are cut is shown in Fig.
Revised: Performance of our method when all messages are cut is shown in Fig.

Revised: As expected, performance of our method drops signiﬁcantly in most of the tasks.

Revised: Compared to the results when 80% messages are dropped (Fig.

Revised: 8), we can make the conclusion that the learnt communication strategies are only sending messages that are important for individual decision-making.