Original: M ULTI -A GENT S EQUENTIAL D ECISION -M AKING VIA C OMMUNICATION
Revised: M ULTI -A GENT S EQUENTIAL D ECISION -M AKING VIA C OMMUNICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Communication helps agents to obtain information about others so that better coordinated behavior can be learned.
Revised: Communication helps agents to obtain information about others so that better coordinated behavior can be learned.

Original: Some existing work communicates predicted future trajectory with others, hoping to get clues about what others would do for better coordination.
Revised: Some existing work communicates predicted future trajectory with others, hoping to get clues about what others would do for better coordination.

Original: However, circular dependencies sometimes can occur when agents are treated synchronously so it is hard to coordinate decision-making.
Revised: However, circular dependencies sometimes can occur when agents are treated synchronously so it is hard to coordinate decision-making.

Original: In this paper, we propose a novel communication scheme, Sequential Communication (SeqComm).
Revised: In this paper, we propose a novel communication scheme, Sequential Communication (SeqComm).

Original: SeqComm treats agents asynchronously (the upper-level agents make decisions before the lower-level ones) and has two communication phases.
Revised: SeqComm treats agents asynchronously (the upper-level agents make decisions before the lower-level ones) and has two communication phases.

Original: In negotiation phase, agents determine the priority of decision-making by communicating hidden states of observations and comparing the value of intention, which is obtained by modeling the environment dynamics.
Revised: In negotiation phase, agents determine the priority of decision-making by communicating hidden states of observations and comparing the value of intention, which is obtained by modeling the environment dynamics.

Original: In launching phase, the upper-level agents take the lead in making decisions and communicate their actions with the lower-level agents.
Revised: In launching phase, the upper-level agents take the lead in making decisions and communicate their actions with the lower-level agents.

Original: Theoretically, we prove the policies learned by SeqComm are guaranteed to improve monotonically and converge.
Revised: Theoretically, we prove the policies learned by SeqComm are guaranteed to improve monotonically and converge.

Original: Empirically, we show that
Revised: Empirically, we show that

Original: SeqComm outperforms existing methods in various multi-agent cooperative tasks.
Revised: SeqComm outperforms existing methods in various multi-agent cooperative tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The partial observability and stochasticity inherent to the nature of multi-agent systems can easily impede the cooperation among agents and lead to catastrophic miscoordination (Ding et al., 2020).
Revised: The partial observability and stochasticity inherent to the nature of multi-agent systems can easily impede the cooperation among agents and lead to catastrophic miscoordination (Ding et al., 2020).

Original: Communication has been exploited to help agents obtain extra information during both training and execution to mitigate such problems (Foerster et al., 2016;
Revised: Communication has been exploited to help agents obtain extra information during both training and execution to mitigate such problems (Foerster et al., 2016;

Original: Sukhbaatar et al., 2016; Peng et al., 2017).
Revised: Sukhbaatar et al., 2016; Peng et al., 2017).

Original: Speciﬁcally, agents can share their information with others via a trainable communication channel.
Revised: Speciﬁcally, agents can share their information with others via a trainable communication channel.

Original: Centralized training with decentralized execution (CTDE) is a popular learning paradigm in cooperative multi-agent reinforcement learning (MARL).
Revised: Centralized training with decentralized execution (CTDE) is a popular learning paradigm in cooperative multi-agent reinforcement learning (MARL).

Original: Although the centralized value function can be learned to evaluate the joint policy of agents, the decentralized policies of agents are essentially independent.
Revised: Although the centralized value function can be learned to evaluate the joint policy of agents, the decentralized policies of agents are essentially independent.

Original: Therefore, a coordination problem arises.
Revised: Therefore, a coordination problem arises.

Original: That is, agents may make sub-optimal actions by mistakenly assuming others’ actions when there exist multiple optimal joint actions (Busoniu et al., 2008).
Revised: That is, agents may make sub-optimal actions by mistakenly assuming others’ actions when there exist multiple optimal joint actions (Busoniu et al., 2008).

Original: Communication allows agents to obtain information about others to avoid miscoordination.
Revised: Communication allows agents to obtain information about others to avoid miscoordination.

Original: However, most existing work only focuses on communicating messages, e.g., the information of agents’ current observation or historical trajectory (Jiang & Lu, 2018;
Revised: However, most existing work only focuses on communicating messages, e.g., the information of agents’ current observation or historical trajectory (Jiang & Lu, 2018;

Original: Singh et al., 2019; Das et al., 2019; Ding et al., 2020).
Revised: Singh et al., 2019; Das et al., 2019; Ding et al., 2020).

Original: It is impossible for an agent to acquire other’s actions before making decisions since the game model is usually synchronous, i.e. , agents make decisions and execute actions simultaneously.
Revised: It is impossible for an agent to acquire other’s actions before making decisions since the game model is usually synchronous, i.e. , agents make decisions and execute actions simultaneously.

Original: Recently, intention or imagination, depicted by a combination of predicted actions and observations of many future steps, has been proposed as part of messages (Kim et al., 2021; Pretorius et al., 2021).
Revised: Recently, intention or imagination, depicted by a combination of predicted actions and observations of many future steps, has been proposed as part of messages (Kim et al., 2021; Pretorius et al., 2021).

Original: However, circular dependencies can still occur, so it may be hard to coordinate decision-making under synchronous settings.
Revised: However, circular dependencies can still occur, so it may be hard to coordinate decision-making under synchronous settings.

Original: A general approach to solving the coordination problem is to make sure that ties between equally good actions are broken by all agents.
Revised: A general approach to solving the coordination problem is to make sure that ties between equally good actions are broken by all agents.

Original: One simple mechanism for doing so is to know exactly what others will do and adjust the behavior accordingly under a unique ordering of agents and actions (Busoniu et al., 2008).
Revised: One simple mechanism for doing so is to know exactly what others will do and adjust the behavior accordingly under a unique ordering of agents and actions (Busoniu et al., 2008).

Original: Inspired by this, we reconsider the cooperative game from an asynchronous perspective.
Revised: Inspired by this, we reconsider the cooperative game from an asynchronous perspective.

Original: In other words, each agent is assigned a priority ( i.e., order) of decision-making each step in both training and execution, thus the Stackelberg equilibrium (SE) (Von Stackelberg, 2010) is naturally set up as the learning objective.
Revised: In other words, each agent is assigned a priority ( i.e., order) of decision-making each step in both training and execution, thus the Stackelberg equilibrium (SE) (Von Stackelberg, 2010) is naturally set up as the learning objective.

Original: Speciﬁcally, the upper-level agents make decisions before the lower-level agents.
Revised: Speciﬁcally, the upper-level agents make decisions before the lower-level agents.

Original: Therefore, the lower-level agents can acquire the actual actions of the upper-level agents by communication and make their decisions conditioned on what the upper-level agents would do.
Revised: Therefore, the lower-level agents can acquire the actual actions of the upper-level agents by communication and make their decisions conditioned on what the upper-level agents would do.

Original: Under this setting, the SE is likely to be Pareto superior to the average Nash equilibrium (NE) in games that require a high cooperation level (Zhang et al., 2020).
Revised: Under this setting, the SE is likely to be Pareto superior to the average Nash equilibrium (NE) in games that require a high cooperation level (Zhang et al., 2020).

Original: However, is it necessary to decide a speciﬁc priority of decision-making for each agent?
Revised: However, is it necessary to decide a speciﬁc priority of decision-making for each agent?

Original: Ideally, the optimal joint policy can be decomposed by any orders (Wen et al., 2019), e.g., π ∗ ( a 1 , a 2 | s ) =
Revised: Ideally, the optimal joint policy can be decomposed by any orders (Wen et al., 2019), e.g., π ∗ ( a 1 , a 2 | s ) =

Original: π ∗ ( a 1 | s ) π ∗ ( a 2 | s, a 1 ) =
Revised: π ∗ ( a 1 | s ) π ∗ ( a 2 | s, a 1 ) =

Original: π ∗ ( a 2 | s ) π ∗ ( a 1 | s, a 2 ) .
Revised: π ∗ ( a 2 | s ) π ∗ ( a 1 | s, a 2 ) .

Original: But during the learning process, it is unlikely for agents to use the optimal actions of other agents for gradient calculation, making it still vulnerable to the relative overgeneralization problem (Wei et al., 2018).
Revised: But during the learning process, it is unlikely for agents to use the optimal actions of other agents for gradient calculation, making it still vulnerable to the relative overgeneralization problem (Wei et al., 2018).

Original: Overall, there is no guarantee that the above equation will hold in the learning process, thus ordering should be carefully concerned.
Revised: Overall, there is no guarantee that the above equation will hold in the learning process, thus ordering should be carefully concerned.

Original: In this paper, we propose a novel model-based multi-round communication scheme for cooperative
Revised: In this paper, we propose a novel model-based multi-round communication scheme for cooperative

Original: MARL, Sequential Communication (SeqComm), to enable agents to explicitly coordinate with each other.
Revised: MARL, Sequential Communication (SeqComm), to enable agents to explicitly coordinate with each other.

Original: Speciﬁcally, SeqComm has two-phase communication, negotiation phase and launching phase.
Revised: Speciﬁcally, SeqComm has two-phase communication, negotiation phase and launching phase.

Original: In the negotiation phase, agents communicate their hidden states of observations with others simultaneously.
Revised: In the negotiation phase, agents communicate their hidden states of observations with others simultaneously.

Original: Then they are able to generate multiple predicted trajectories, called intention , by modeling the environmental dynamics and other agents’ actions.
Revised: Then they are able to generate multiple predicted trajectories, called intention , by modeling the environmental dynamics and other agents’ actions.

Original: In addition, the priority of decision-making is determined by communicating and comparing the corresponding values of agents’ intentions.
Revised: In addition, the priority of decision-making is determined by communicating and comparing the corresponding values of agents’ intentions.

Original: The value of each intention represents the rewards obtained by letting that agent take the upper-level position of the order sequence.
Revised: The value of each intention represents the rewards obtained by letting that agent take the upper-level position of the order sequence.

Original: The sequence of others follows the same procedure as aforementioned with the upper-level agents ﬁxed.
Revised: The sequence of others follows the same procedure as aforementioned with the upper-level agents ﬁxed.

Original: In the launching phase, the upper-level agents take the lead in decision-making and communicate their actual actions with the lower-level agents.
Revised: In the launching phase, the upper-level agents take the lead in decision-making and communicate their actual actions with the lower-level agents.

Original: Note that the actual actions will be executed simultaneously in the environment without any changes.
Revised: Note that the actual actions will be executed simultaneously in the environment without any changes.

Original: SeqCommadopts the CTDE paradigm (Lowe et al., 2017) and is currently built on MAPPO (Yu et al., 2021).
Revised: SeqComm is currently built on MAPPO (Yu et al., 2021).

Original: Theoretically, we prove the policies learned by SeqComm are guaranteed to improve monotonically and converge.
Revised: Theoretically, we prove the policies learned by SeqComm are guaranteed to improve monotonically and converge.

Original: Empirically, we evaluate SeqComm on a set of tasks in multi-agent particle environment (MPE) (Lowe et al., 2017) and StarCraft multi-agent challenge (SMAC) (Samvelyan et al., 2019).
Revised: Empirically, we evaluate SeqComm on a set of tasks in multi-agent particle environment (MPE) (Lowe et al., 2017) and StarCraft multi-agent challenge (SMAC) (Samvelyan et al., 2019).

Original: In all these tasks, we demonstrate that SeqComm outperforms prior communication-free and communication-based methods.
Revised: In all these tasks, we demonstrate that SeqComm outperforms prior communication-free and communication-based methods.

Original: By ablation studies, we conﬁrm that treating agents asynchronously is a more effective way to promote coordination and SeqComm can provide the proper priority of decision-making for agents to develop better coordination.
Revised: By ablation studies, we conﬁrm that treating agents asynchronously is a more effective way to promote coordination and SeqComm can provide the proper priority of decision-making for agents to develop better coordination.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Communication.
Revised: Communication.

Original: Existing studies (Jiang & Lu, 2018;
Revised: Existing studies (Jiang & Lu, 2018;

Original: Kim et al., 2019; Singh et al., 2019; Das et al., 2019; Zhang et al., 2019; Jiang et al., 2020; Ding et al., 2020; Konan et al., 2022) in this realm mainly focus on how to extract valuable messages.
Revised: Kim et al., 2019; Singh et al., 2019; Das et al., 2019; Zhang et al., 2019; Jiang et al., 2020; Ding et al., 2020; Konan et al., 2022) in this realm mainly focus on how to extract valuable messages.

Original: ATOC (Jiang & Lu, 2018) and IC3Net (Singh et al., 2019) utilize gate mechanisms to decide when to communicate with other agents.
Revised: ATOC (Jiang & Lu, 2018) and IC3Net (Singh et al., 2019) utilize gate mechanisms to decide when to communicate with other agents.

Original: Many works (Das et al., 2019; Konan et al., 2022) employ multi-round communication to fully reason the intentions of others and establish complex collaboration strategies.
Revised: Many works (Das et al., 2019; Konan et al., 2022) employ multi-round communication to fully reason the intentions of others and establish complex collaboration strategies.

Original: Social inﬂuence (Jaques et al., 2019) uses communication to inﬂuence the behaviors of others.
Revised: Social inﬂuence (Jaques et al., 2019) uses communication to inﬂuence the behaviors of others.

Original: I2C (Ding et al., 2020) only communicates with agents that are relevant and inﬂuential which are determined by causal inference.
Revised: I2C (Ding et al., 2020) only communicates with agents that are relevant and inﬂuential which are determined by causal inference.

Original: However, all these methods focus on how to exploit valuable information from current or past partial observations effectively and properly.
Revised: However, all these methods focus on how to exploit valuable information from current or past partial observations effectively and properly.

Original: More recently, some studies (Kim et al., 2021; Du et al., 2021;
Revised: More recently, some studies (Kim et al., 2021; Du et al., 2021;

Original: Pretorius et al., 2021) begin to answer the question: can we favor cooperation beyond sharing partial observation?
Revised: Pretorius et al., 2021) begin to answer the question: can we favor cooperation beyond sharing partial observation?

Original: They allow agents to imagine their future states with a world model and communicate those with others.
Revised: They allow agents to imagine their future states with a world model and communicate those with others.

Original: IS (Pretorius et al., 2021), as the representation of this line of research, enables each agent to share its intention with other agents in the form of the encoded imagined trajectory and use the attention module to ﬁgure out the importance of the received intention.
Revised: IS (Pretorius et al., 2021), as the representation of this line of research, enables each agent to share its intention with other agents in the form of the encoded imagined trajectory and use the attention module to ﬁgure out the importance of the received intention.

Original: However, two concerns arise.
Revised: However, two concerns arise.

Original: On one hand, circular dependencies can lead to inaccurate predicted future trajectories as long as the multi-agent system treats agents synchronously.
Revised: On one hand, circular dependencies can lead to inaccurate predicted future trajectories as long as the multi-agent system treats agents synchronously.

Original: On the other hand, MARL struggles in extracting useful information from numerous messages, not to mention more complex and dubious messages, i.e., predicted future trajectories.
Revised: On the other hand, MARL struggles in extracting useful information from numerous messages, not to mention more complex and dubious messages, i.e., predicted future trajectories.

Original: Unlike these works, we treat the agents from an asynchronously perspective therefore circular dependencies can be naturally resolved.
Revised: Unlike these works, we treat the agents from an asynchronously perspective therefore circular dependencies can be naturally resolved.

Original: Furthermore, agents only send actions to lower-level agents besides partial observations to make sure the messages are compact as well as informative.
Revised: Furthermore, agents only send actions to lower-level agents besides partial observations to make sure the messages are compact as well as informative.

Original: Coordination.
Revised: Coordination.

Original: The agents are essentially independent decision makers in execution and may break ties between equally good actions randomly.
Revised: The agents are essentially independent decision makers in execution and may break ties between equally good actions randomly.

Original: Thus, in the absence of additional mechanisms, different agents may break ties in different ways, and the resulting joint actions may be suboptimal.
Revised: Thus, in the absence of additional mechanisms, different agents may break ties in different ways, and the resulting joint actions may be suboptimal.

Original: Coordination graphs (Guestrin et al., 2002; B¨ohmer et al., 2020; Wang et al., 2021b) simplify the coordination when the global Q-function can be additively decomposed into local Q-functions that only depend on the actions of a subset of agents.
Revised: Coordination graphs (Guestrin et al., 2002; B¨ohmer et al., 2020; Wang et al., 2021b) simplify the coordination when the global Q-function can be additively decomposed into local Q-functions that only depend on the actions of a subset of agents.

Original: Typically, a coordination graph expresses a higher-order value decomposition among agents.
Revised: Typically, a coordination graph expresses a higher-order value decomposition among agents.

Original: This improves the representational capacity to distinguish other agents’ effects on local utility functions, which addresses the miscoordination problems caused by partial observability.
Revised: This improves the representational capacity to distinguish other agents’ effects on local utility functions, which addresses the miscoordination problems caused by partial observability.

Original: Another general approach to solving the coordination problem is to make sure that ties are broken by all agents in the same way, requiring that random action choices are somehow coordinated or negotiated.
Revised: Another general approach to solving the coordination problem is to make sure that ties are broken by all agents in the same way, requiring that random action choices are somehow coordinated or negotiated.

Original: Social conventions (Boutilier, 1996) or role assignments (Prasad et al., 1998) encode prior preferences towards certain joint actions and help break ties during action selection.
Revised: Social conventions (Boutilier, 1996) or role assignments (Prasad et al., 1998) encode prior preferences towards certain joint actions and help break ties during action selection.

Original: Communication (Fischer et al., 2004; Vlassis, 2007) can be used to negotiate action choices, either alone or in combination with the aforementioned techniques.
Revised: Communication (Fischer et al., 2004; Vlassis, 2007) can be used to negotiate action choices, either alone or in combination with the aforementioned techniques.

Original: Our method follows this line of research by utilizing the ordering of agents and actions to break the ties, other than the enhanced representational capacity of the local value function.
Revised: Our method follows this line of research by utilizing the ordering of agents and actions to break the ties, other than the enhanced representational capacity of the local value function.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: Cost-Free Communication.
Revised: Cost-Free Communication.

Original: The decentralized partially observable Markov decision process (DecPOMDP) can be extended to explicitly incorporate communicated messages and observations.
Revised: The decentralized partially observable Markov decision process (DecPOMDP) can be extended to explicitly incorporate broadcasting observations.

Original: The resulting model is called Dec-POMDP-Com (Goldman & Zilberstein, 2003; 2004).
Revised: The resulting model is called multi-agent POMDP (Oliehoek et al., 2016).

Original: Pynadath & Tambe (2002) showed that under cost-free communication, a joint communication policy that shares local observations at each stage is optimal.
Revised: Pynadath & Tambe (2002) showed that under cost-free communication, a joint communication policy that shares local observations at each stage is optimal.

Original: Many studies have also investigated sharing local observations in models that are similar to Dec-POMDP-Com (Pynadath & Tambe, 2002; Ooi & 
Revised: Many studies have also investigated sharing local observations in models that are similar to multi-agent POMDP (Pynadath & Tambe, 2002; Ooi & Wornell, 1996; Nair et al., 2004;

Original: Wornell, 1996; Nair et al., 2004; Roth et al., 2005a;b; Spaan et al., 2006; Oliehoek et al., 2007; Becker et al., 2004).
Revised: Roth et al., 2005a;b; Spaan et al., 2006; Oliehoek et al., 2007; Becker et al., 2004).

Original: These works focus on issues other than communication cost.
Revised: These works focus on issues other than communication cost and we foucs on the coordination problem.

Original: Although real world rarely exhibits such ideal conditions.
Revised: Note that even under multi-agent POMDP where agents can get joint observations, coordination problem can still arise (Busoniu et al., 2008).

Original: They intend to model some domains as having approximately free communication to a sufﬁcient degree and believe analyzing such cases gives us some insight to the beneﬁt of communication.
Revised: Suppose the centralized critic has learnt actions pairs [ a 1 , a 2 ] and [ b 1 , b 2 ] are equally optimal. Without any prior information, the individual policies π 1 and π 2 learnt from the centralized critic can break the ties randomly and may choose a 1 and b 2 , respectively.

Original: Multi-Agent Sequential Decision-Making.
Revised: Multi-Agent Sequential Decision-Making.

Original: We consider fully cooperative multi-agent tasks that are modeled as Dec-POMDP-Com, where n agents interact with the environment according to the following procedure, which we refer to as multi-agent sequential decision-making .
Revised: We consider fully cooperative multi-agent tasks that are modeled as multi-agent POMDP, where n agents interact with the environment according to the following procedure, which we refer to as multi-agent sequential decision-making .

Original: At each timestep t , assume the priority ( i.e., order) of decision-making for all agents is given and each priority level has only one agent ( i.e. , agents make decisions one by one).
Revised: At each timestep t , assume the priority ( i.e., order) of decision-making for all agents is given and each priority level has only one agent ( i.e. , agents make decisions one by one).

Original: Note that the smaller the level index, the higher priority of decision-making is.
Revised: Note that the smaller the level index, the higher priority of decision-making is.

Original: The agent at each level k gets its own observation o kt drawn from the state s t , and receives messages m − k t from all other agents, where m − k t (cid:44) {{ o 1 t , a 1 t } , . .
Revised: The agent at each level k gets its own observation o kt drawn from the state s t , and receives messages m − k t from all other agents, where m − k t (cid:44) {{ o 1 t , a 1 t } , . .

Original: , { o k − 1 t , a k − 1 t } , o k +1 t , . . .
Revised: , { o k − 1 t , a k − 1 t } , o k +1 t , . . .

Original: , o nt } .
Revised: , o nt } .

Original: Equivalently, m − k t can be written as { o − k t , a 1: k − 1 t } , where o − k t denotes the joint observations of all agents except k , and a 1: k − 1 t denotes the joint actions of agents 1 to k − 1 .
Revised: Equivalently, m − k t can be written as { o − k t , a 1: k − 1 t } , where o − k t denotes the joint observations of all agents except k , and a 1: k − 1 t denotes the joint actions of agents 1 to k − 1 .

Original: For the agent at the ﬁrst level ( i.e. , k = 1 ), a 1: k − 1 t = ∅ .
Revised: For the agent at the ﬁrst level ( i.e. , k = 1 ), a 1: k − 1 t = ∅ .

Original: Then, the agent determines its action a kt sampled from its policy π k ( ·| o kt , m − k t ) or equivalently π k ( ·| o t , a 1: k − 1 t ) and sends it to the lower-level agents.
Revised: Then, the agent determines its action a kt sampled from its policy π k ( ·| o kt , m − k t ) or equivalently π k ( ·| o t , a 1: k − 1 t ) and sends it to the lower-level agents.

Original: After all agents have determined their actions, they perform the joint actions a t , which can be seen as sampled from the joint policy π ( ·| s t ) factorized as (cid:81)
Revised: After all agents have determined their actions, they perform the joint actions a t , which can be seen as sampled from the joint policy π ( ·| s t ) factorized as (cid:81)

Original: nk =
Revised: nk =

Original: 1 π k ( ·| o t , a 1: k − 1 t ) , in the environment and get a shared reward r ( s t , a t ) and the state transitions to next state s (cid:48) according to the transition probability p ( s (cid:48) | s t , a t ) .
Revised: 1 π k ( ·| o t , a 1: k − 1 t ) , in the environment and get a shared reward r ( s t , a t ) and the state transitions to next state s (cid:48) according to the transition probability p ( s (cid:48) | s t , a t ) .

Original: All agents aim to maximize the expected return (cid:80) ∞ t =

Revised: All agents aim to maximize the expected return (cid:80) ∞ t =

Original: γ t r t , where γ is the discount factor.
Revised: γ t r t , where γ is the discount factor.

Original: The state-value function and action-value function of the level- k agent are deﬁned as follows:
Revised: The state-value function and action-value function of the level- k agent are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: For the setting of multi-agent sequential decision-making discussed above, we have the following proposition.
Revised: For the setting of multi-agent sequential decision-making discussed above, we have the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: If all the agents update its policy with individual TRPO (Schulman et al., 2015) sequentially in multi-agent sequential decision-making, then the joint policy of all agents is guaranteed to improve monotonically and converge.
Revised: If all the agents update its policy with individual TRPO (Schulman et al., 2015) sequentially in multi-agent sequential decision-making, then the joint policy of all agents is guaranteed to improve monotonically and converge.

Original: Proof.
Revised: Proof.

Original: The proof is given in Appendix A.
Revised: The proof is given in Appendix A.

Original: Proposition 1 indicates that SeqComm has the performance guarantee regardless of the priority of decision-making in multi-agent sequential decision-making.
Revised: Proposition 1 indicates that SeqComm has the performance guarantee regardless of the priority of decision-making in multi-agent sequential decision-making.

Original: However, the priority of decision-making indeed affects the optimality of the converged joint policy, and we have the following claim.
Revised: However, the priority of decision-making indeed affects the optimality of the converged joint policy, and we have the following claim.

Original: Claim 1.
Revised: Claim 1.

Original: The different priorities of decision-making affect the optimality of the convergence of the learning algorithm due to the relative overgeneralization problem.
Revised: The different priorities of decision-making affect the optimality of the convergence of the learning algorithm due to the relative overgeneralization problem.

Original: [Figure]
Revised: [Figure]

Original: We use a one-step matrix game as an example, as illustrated in Figure 1(a), to demonstrate the inﬂuence of the priority of decision-making on the learning process.
Revised: We use a one-step matrix game as an example, as illustrated in Figure 1(a), to demonstrate the inﬂuence of the priority of decision-making on the learning process.

Original: Due to relative overgeneralization, agent B tends to choose bor b 3 even under the CTDE paradigm (Wei et al., 2018).
Revised: Due to relative overgeneralization (Wei et al., 2018), agent B tends to choose b 2 or b 3 .

Original: Speciﬁcally, b 2 or b 3 in the suboptimal equilibrium is a better choice than b 1 in the optimal equilibrium when matched with arbitrary actions from agent A .
Revised: Speciﬁcally, b 2 or b 3 in the suboptimal equilibrium is a better choice than b 1 in the optimal equilibrium when matched with arbitrary actions from agent A .

Original: Therefore, as shown in Figure 1(b), B → A ( i.e. , agent B makes decisions before
Revised: Therefore, as shown in Figure 1(b), B → A ( i.e. , agent B makes decisions before

Original: A , and A ’s policy conditions on the action of B ) and Simultaneous ( i.e. , two agents make decisions simultaneously and independently) are easily trapped into local optima.
Revised: A , and A ’s policy conditions on the action of B ) and Simultaneous ( i.e. , two agents make decisions simultaneously and independently) are easily trapped into local optima.

Original: However, things can be different if agent A goes ﬁrst, as A → B achieves the optimum.
Revised: However, things can be different if agent A goes ﬁrst, as A → B achieves the optimum.

Original: As long as agent A does not suffer from relative overgeneralization, it can help agent B get rid of local optima by narrowing down the search space of B .
Revised: As long as agent A does not suffer from relative overgeneralization, it can help agent B get rid of local optima by narrowing down the search space of B .

Original: Besides, a policy that determines the priority of decision-making can be learned under the guidance of the state-value function, denoted as Learned .
Revised: Besides, a policy that determines the priority of decision-making can be learned under the guidance of the state-value function, denoted as Learned .

Original: It obtains better performance than B → A and Simultaneous , which indicates that dynamically determining the order during policy learning can be beneﬁcial as we do not know the optimal priority in advance.
Revised: It obtains better performance than B → A and Simultaneous , which indicates that dynamically determining the order during policy learning can be beneﬁcial as we do not know the optimal priority in advance.

Original: Remark 1.
Revised: Remark 1.

Original: The priority ( i.e., order) of decision-making affects the optimality of the converged joint policy in multi-agent sequential decision-making, thus it is critical to determine the order.
Revised: The priority ( i.e., order) of decision-making affects the optimality of the converged joint policy in multi-agent sequential decision-making, thus it is critical to determine the order.

Original: However, learning the order directly requires an additional centralized policy in execution, whichdoes not ﬁt the CTDE paradigm and is not generalizable in the scenario where the number of agents varies.
Revised: However, learning the order directly requires an additional centralized policy in execution, which is not generalizable in the scenario where the number of agents varies.

Original: Moreover, its learning complexity exponentially increases with the number of agents, making it infeasible in many cases.
Revised: Moreover, its learning complexity exponentially increases with the number of agents, making it infeasible in many cases.

Original:  S EQUENTIAL C OMMUNICATION
Revised:  S EQUENTIAL C OMMUNICATION

Original: In this paper, we cast our eyes in another direction and resort to the world model.
Revised: In this paper, we cast our eyes in another direction and resort to the world model.

Original: Ideally, we can randomly sample candidate order sequences, evaluate them under the world model (see Section 4.1), and choose the order sequence that is deemed the most promising under the true dynamic.
Revised: Ideally, we can randomly sample candidate order sequences, evaluate them under the world model (see Section 4.1), and choose the order sequence that is deemed the most promising under the true dynamic.

Original: SeqComm is designed based on this principle to determine the priority of decision-making via communication.
Revised: SeqComm is designed based on this principle to determine the priority of decision-making via communication.

Original: SeqComm adopts a multi-round communication mechanism, i.e. , agents are allowed to communicate with others in multiple rounds.
Revised: SeqComm adopts a multi-round communication mechanism, i.e. , agents are allowed to communicate with others in multiple rounds.

Original: Importantly, communication is separated into phases serving different purposes.
Revised: Importantly, communication is separated into phases serving different purposes.

Original: One is the negotiation phase for agents to determine the priority of decision-making.
Revised: One is the negotiation phase for agents to determine the priority of decision-making.

Original: Another is the launching phase for agents to act conditioning on actual actions upper-level agents will take to implement explicit coordination via communication .
Revised: Another is the launching phase for agents to act conditioning on actual actions upper-level agents will take to implement explicit coordination via communication .

Original: The overview of SeqComm is illustrated in Figure 2.
Revised: The overview of SeqComm is illustrated in Figure 2.

Original: Each SeqComm agent consists of a policy, a critic, and a world model, as illustrated in
Revised: Each SeqComm agent consists of a policy, a critic, and a world model, as illustrated in

Original: Figure 3, and the parameters of all networks are shared across agents (Gupta et al., 2017).
Revised: Figure 3, and the parameters of all networks are shared across agents (Gupta et al., 2017).

Original: agent 1 a g e n t 1 ’ s o b s agent 2 agentagent 4 r e q u e s t
Revised: agent 1 a g e n t 1 ’ s o b s agent 2 agentagent 4 r e q u e s t

Original: r
Revised: r

Original: e p l y agent 1 a g e n t 1 ’ s o b s agent 2 agentagent
Revised: e p l y agent 1 a g e n t 1 ’ s o b s agent 2 agentagent

Original: r
Revised: r

Original: e p l y r e q u e s t r e
Revised: e p l y r e q u e s t r e

Original: q u e s t
Revised: q u e s t

Original: r
Revised: r

Original: e q u e s t r e p l y r e p l y Agent 1 chooses to send request to agent 2 and ignore agent 32 1 24 Agent 1 chooses to send request to agent 2, 3, 4 t t+
Revised: e q u e s t r e p l y r e p l y Agent 1 chooses to send request to agent 2 and ignore agent 32 1 24 Agent 1 chooses to send request to agent 2, 3, 4 t t+

Original: B order set
Revised: B order set

Original: A s A C s C B s B
Revised: A s A C s C B s B

Original: B C 1 B Cr 1 r 2 intention reward
Revised: B C 1 B Cr 1 r 2 intention reward

Original: [Figure]
Revised: [Figure]

Original: A C B 1 a C
Revised: A C B 1 a C

Original: [Figure]
Revised: [Figure]

Original: A C B a A A C B a B a A a C
Revised: A C B a A A C B a B a A a C

Original: A
Revised: A

Original: C
Revised: C

Original: B
Revised: B

Original: [Figure]
Revised: [Figure]

Original: agents who hold the upper-level positions will make decisions prior to the lower-level agents.
Revised: agents who hold the upper-level positions will make decisions prior to the lower-level agents.

Original: Besides, their actions will be shared with anyone that has not yet made decisions.
Revised: Besides, their actions will be shared with anyone that has not yet made decisions.

Original: 4. N EGOTIATION P HASE
Revised: 4. N EGOTIATION P HASE

Original: In the negotiation phase, the observation encoder ﬁrst takes o t as input and outputs a hidden state h t , which is used to communicate with others.
Revised: In the negotiation phase, the observation encoder ﬁrst takes o t as input and outputs a hidden state h t , which is used to communicate with others.

Original: Agents then determine the priority of decision-making by intention which is established and evaluated based on the world model.
Revised: Agents then determine the priority of decision-making by intention which is established and evaluated based on the world model.

Original: World Model.
Revised: World Model.

Original: The world model is needed to predict and evaluate future trajectories.
Revised: The world model is needed to predict and evaluate future trajectories.

Original: SeqComm, unlike previous works (Kim et al., 2021; Du et al., 2021; Pretorius et al., 2021), can utilize received hidden states of other agents in the ﬁrst round of communication to model more precise environment dynamics for the explicit coordination in the next round of communication.
Revised: SeqComm, unlike previous works (Kim et al., 2021; Du et al., 2021; Pretorius et al., 2021), can utilize received hidden states of other agents in the ﬁrst round of communication to model more precise environment dynamics for the explicit coordination in the next round of communication.

Original: Once an agent can access other agents’ hidden states, it shall have adequate information to estimate their actions since all agents are homogeneous and parameter-sharing.
Revised: Once an agent can access other agents’ hidden states, it shall have adequate information to estimate their actions since all agents are homogeneous and parameter-sharing.

Original: Therefore, the world model M ( · ) takes as input the joint hidden states h t = { h 1 t , . .
Revised: Therefore, the world model M ( · ) takes as input the joint hidden states h t = { h 1 t , . .

Original: , h nt } and actions a t , and predicts the next joint observations and reward,
Revised: , h nt } and actions a t , and predicts the next joint observations and reward,

Original: [Equation]
Revised: [Equation]

Original: where AM w is the attention module.
Revised: where AM w is the attention module.

Original: The reason that we adopt the attention module is to entitle the world model to be generalizable in the scenarios where additional agents are introduced or existing agents are removed.
Revised: The reason that we adopt the attention module is to entitle the world model to be generalizable in the scenarios where additional agents are introduced or existing agents are removed.

Original: AM a
Revised: AM a

Original: Policy o
Revised: Policy o

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: upper t
Revised: upper t

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: Cri/c
Revised: Cri/c

Original: AM a
Revised: AM a

Original: Policy
Revised: Policy

Original: [Equation]
Revised: [Equation]

Original: t a
Revised: t a

Original: [Equation]
Revised: [Equation]

Original: t a upper t
Revised: t a upper t

Original: [Equation]
Revised: [Equation]

Original: { a 1 t }
Revised: { a 1 t }

Original: Cri/c
Revised: Cri/c

Original: AM a
Revised: AM a

Original: Policy
Revised: Policy

Original: [Equation]
Revised: [Equation]

Original: 1 t
Revised: 1 t

Original: [Equation]
Revised: [Equation]

Original: 1 t a upper t
Revised: 1 t a upper t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: /c
Revised: /c

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: ⋯
Revised: ⋯

Original: Priority of Decision-Making.
Revised: Priority of Decision-Making.

Original: The intention is the key element to determine the priority of decision-making.
Revised: The intention is the key element to determine the priority of decision-making.

Original: The notion of intention is described as an agent’s future behavior in previous works (Rabinowitz et al., 2018; Raileanu et al., 2018;
Revised: The notion of intention is described as an agent’s future behavior in previous works (Rabinowitz et al., 2018; Raileanu et al., 2018;

Original: Kim et al., 2021).
Revised: Kim et al., 2021).

Original: However, we deﬁne the intention as an agent’s future behavior without considering others .
Revised: However, we deﬁne the intention as an agent’s future behavior without considering others .

Original: As mentioned before, an agent’s intention considering others can lead to circular dependencies and cause miscoordination.
Revised: As mentioned before, an agent’s intention considering others can lead to circular dependencies and cause miscoordination.

Original: By our deﬁnition, the intention of an agent should be depicted as all future trajectories considering that agent as the ﬁrst-mover and ignoring the others.
Revised: By our deﬁnition, the intention of an agent should be depicted as all future trajectories considering that agent as the ﬁrst-mover and ignoring the others.

Original: However, there are many possible future trajectories as the priority of the rest agents is unﬁxed .
Revised: However, there are many possible future trajectories as the priority of the rest agents is unﬁxed .

Original: In practice, we use the Monte Carlo method to evaluate intention.
Revised: In practice, we use the Monte Carlo method to evaluate intention.

Original: Taking agent i at timestep t to illustrate, it ﬁrstly considers itself as the ﬁrst-mover and produces its action only based on the joint hidden states, ˆ a it ∼ π i ( ·| AM a ( h t )) , where we again use an attention module AM a to handle the input.
Revised: Taking agent i at timestep t to illustrate, it ﬁrstly considers itself as the ﬁrst-mover and produces its action only based on the joint hidden states, ˆ a it ∼ π i ( ·| AM a ( h t )) , where we again use an attention module AM a to handle the input.

Original: For the order sequence of lower-level agents, we randomly sample a set of order sequences from unﬁxed agents.
Revised: For the order sequence of lower-level agents, we randomly sample a set of order sequences from unﬁxed agents.

Original: Assume agent j is the second-mover, agent i models j ’s action by considering the upper-level action following its own policy ˆ a jt ∼ π i ( ·| AM a ( h t , ˆ a it )) .
Revised: Assume agent j is the second-mover, agent i models j ’s action by considering the upper-level action following its own policy ˆ a jt ∼ π i ( ·| AM a ( h t , ˆ a it )) .

Original: The same procedure is applied to predict the actions of all other agents following the sampled order sequence.
Revised: The same procedure is applied to predict the actions of all other agents following the sampled order sequence.

Original: Based on the joint hidden states and predicted actions, the next joint observations ˆ o t +1 and corresponding reward ˆ r t +1 can be predicted by the world model.
Revised: Based on the joint hidden states and predicted actions, the next joint observations ˆ o t +1 and corresponding reward ˆ r t +1 can be predicted by the world model.

Original: The length of the predicted future trajectory is H and it can then be written as τ t
Revised: The length of the predicted future trajectory is H and it can then be written as τ t

Original: = { ˆ o t
Revised: = { ˆ o t

Original: +

Revised: +

Original: , ˆ a t +1 , . . .
Revised: , ˆ a t +1 , . . .

Original: , ˆ o t + H , ˆ a t
Revised: , ˆ o t + H , ˆ a t

Original: + H } by repeating the procedure aforementioned and the value of one trajectory is deﬁned as the return of that trajectory v τ t = (cid:80) t
Revised: + H } by repeating the procedure aforementioned and the value of one trajectory is deﬁned as the return of that trajectory v τ t = (cid:80) t

Original: + H t (cid:48)
Revised: + H t (cid:48)

Original: =
Revised: =

Original: t +1 γ t (cid:48) − t − 1 ˆ r t (cid:48) /H .
Revised: t +1 γ t (cid:48) − t − 1 ˆ r t (cid:48) /H .

Original: In addition, the intention value is deﬁned as the average value of F future trajectories with different sampled order sequences.
Revised: In addition, the intention value is deﬁned as the average value of F future trajectories with different sampled order sequences.

Original: The choice of F is a tradeoff between the computation overhead and the accuracy of the estimation.
Revised: The choice of F is a tradeoff between the computation overhead and the accuracy of the estimation.

Original: After all the agents have computed their own intention and the corresponding value, they again communicate their intention values to others.
Revised: After all the agents have computed their own intention and the corresponding value, they again communicate their intention values to others.

Original: Then agents would compare and choose the agent with the highest intention value to be the ﬁrst-mover.
Revised: Then agents would compare and choose the agent with the highest intention value to be the ﬁrst-mover.

Original: The priority of lower-level decision-making follows the same procedure with the upper-level agents ﬁxed.
Revised: The priority of lower-level decision-making follows the same procedure with the upper-level agents ﬁxed.

Original: Note that some agents are required to communicate intention values with others multiple times until the priority of decision-making is ﬁnally determined.
Revised: Note that some agents are required to communicate intention values with others multiple times until the priority of decision-making is ﬁnally determined.

Original: 4. L AUNCHING P HASE
Revised: 4. L AUNCHING P HASE

Original: As for the launching phase, agents communicate for obtaining additional information to make decisions.
Revised: As for the launching phase, agents communicate for obtaining additional information to make decisions.

Original: Apart from the received hidden states from the last phase, we allow agents to get what actual actions the upper-level agents will take in execution, while other studies can only infer others’ actions by opponent modeling (Rabinowitz et al., 2018; Raileanu et al., 2018) or communicating intentions (Kim et al., 2021).
Revised: Apart from the received hidden states from the last phase, we allow agents to get what actual actions the upper-level agents will take in execution, while other studies can only infer others’ actions by opponent modeling (Rabinowitz et al., 2018; Raileanu et al., 2018) or communicating intentions (Kim et al., 2021).

Original: Therefore, miscoordination can be naturally avoided and a better cooperation strategy is possible since lower-level agents can adjust their behaviors accordingly.
Revised: Therefore, miscoordination can be naturally avoided and a better cooperation strategy is possible since lower-level agents can adjust their behaviors accordingly.

Original: A lower-level agent i make a decision following the policy
Revised: A lower-level agent i make a decision following the policy

Original: π i ( ·| AM a ( h t , a uppert )) , where a uppert means received actual actions from all upper-level agents.
Revised: π i ( ·| AM a ( h t , a uppert )) , where a uppert means received actual actions from all upper-level agents.

Original: As long as the agent has decided its action, it will send its action to all other lower-level agents by the communication channel.
Revised: As long as the agent has decided its action, it will send its action to all other lower-level agents by the communication channel.

Original: Note that the actions are executed simultaneously and distributedly in execution, though agents make decisions sequentially.
Revised: Note that the actions are executed simultaneously and distributedly in execution, though agents make decisions sequentially.

Original: Communication Overhead.
Revised: Communication Overhead.

Original: Two communication phases alternate until all agents determine their levels and get upper-level actions.
Revised: Two communication phases alternate until all agents determine their levels and get upper-level actions.

Original: Note that many previous works also adopt the multi-round communication scheme (Das et al., 2019; Singh et al., 2019).
Revised: Note that many previous works also adopt the multi-round communication scheme (Das et al., 2019; Singh et al., 2019).

Original: As for implementation in practice, compared with communicating high-dimensional hidden states/observations by multiple rounds (Das et al., 2019; Singh et al., 2019), or transferring multi-step trajectory (Kim et al., 2021), SeqComm needs more rounds, but it only transmits hidden states for one time.
Revised: As for implementation in practice, compared with communicating high-dimensional hidden states/observations by multiple rounds (Das et al., 2019; Singh et al., 2019), or transferring multi-step trajectory (Kim et al., 2021), SeqComm needs more rounds, but it only transmits hidden states for one time.

Original: For the rest n − 1 round communication with total ( n − 1) / 2 broadcasts per agent, only a single intention value and an action will be exchanged.
Revised: For the rest n − 1 round communication with total ( n − 1) / 2 broadcasts per agent, only a single intention value and an action will be exchanged.

Original: Considering there are n !
Revised: Considering there are n !

Original: permutations of different order choices for n agents, our method has greatly reduced computation overhead since each agent needs to calculate up to n times to search for a satisfying order.
Revised: permutations of different order choices for n agents, our method has greatly reduced computation overhead since each agent needs to calculate up to n times to search for a satisfying order.

Original: Although SeqComm is more suitable for latency-tolerate MARL tasks, e.g., power dispatch (minutes) (Wang et al., 2021a), inventory management (hours) (Feng et al., 2021), maritime transportation (days) (Li et al., 2019), it is possible for SeqComm to have a wider range of applications given the rapid development of the communication technology, e.g., 5G.
Revised: Although SeqComm is more suitable for latency-tolerate MARL tasks, e.g., power dispatch (minutes) (Wang et al., 2021a), inventory management (hours) (Feng et al., 2021), maritime transportation (days) (Li et al., 2019), it is possible for SeqComm to have a wider range of applications given the rapid development of the communication technology, e.g., 5G.

Original: 4. T HEORETICAL A NALYSIS
Revised: 4. T HEORETICAL A NALYSIS

Original: As the priority of decision-making is determined by intention values, SeqComm is likely to choose different orders at different timesteps during training.
Revised: As the priority of decision-making is determined by intention values, SeqComm is likely to choose different orders at different timesteps during training.

Original: However, we have the following proposition that theoretically guarantees the performance of the learned joint policy under SeqComm.
Revised: However, we have the following proposition that theoretically guarantees the performance of the learned joint policy under SeqComm.

Original: Proposition 2.
Revised: Proposition 2.

Original: The monotonic improvement and convergence of the joint policy in SeqComm are independent of the priority of decision-making of agents at each timestep.
Revised: The monotonic improvement and convergence of the joint policy in SeqComm are independent of the priority of decision-making of agents at each timestep.

Original: Proof.
Revised: Proof.

Original: The proof is given in Appendix A.
Revised: The proof is given in Appendix A.

Original: The priority of decision-making is chosen under the world model, thus the compounding errors in the world model can result in discrepancies between the predicted returns of the same order under the
Revised: The priority of decision-making is chosen under the world model, thus the compounding errors in the world model can result in discrepancies between the predicted returns of the same order under the

Original: 0.0 0.2 0.4 0.6 0.8 1.0 step 1e7 2.
Revised: 0.0 0.2 0.4 0.6 0.8 1.0 step 1e7 2.

Original: a v e r a g e r e w a r d
Revised: a v e r a g e r e w a r d

Original: SeqCommTarMACmappoISI2C (a) PP 0.0 0.2 0.4 0.

Revised: SeqCommTarMACmappoISI2C (a) PP 0.0 0.2 0.4 0.

Original: 0.8 1.0 step 1e7 2.
Revised: 0.8 1.0 step 1e7 2.

Original: a v e r a g e r e w a r d
Revised: a v e r a g e r e w a r d

Original: SeqCommTarMACmappoISI2C (b) CN 0.0 0.2 0.4 0.

Revised: SeqCommTarMACmappoISI2C (b) CN 0.0 0.2 0.4 0.

Original: 0.8 1.0 step 1e7 1.
Revised: 0.8 1.0 step 1e7 1.

Original: a v e r a g e r e w a r d
Revised: a v e r a g e r e w a r d

Original: SeqCommTarMACmappoISI2C (c) KA
Revised: SeqCommTarMACmappoISI2C (c) KA

Original: Theorem 1.
Revised: Theorem 1.

Original: Let the expected total variation between two transition distributions be bounded at each timestep as max t E s ∼ π β,t
Revised: Let the expected total variation between two transition distributions be bounded at each timestep as max t E s ∼ π β,t

Original: [ D TV ( p ( s (cid:48) | s, a )
Revised: [ D TV ( p ( s (cid:48) | s, a )

Original: ||
Revised: ||

Original: ˆ p ( s (cid:48) | s, a ))] ≤ (cid:15) m , and the policy divergences at level
Revised: ˆ p ( s (cid:48) | s, a ))] ≤ (cid:15) m , and the policy divergences at level

Original: k be bounded as max s, a 1: k − 1 D TV ( π β,k ( a k | s, a 1: k − 1 )
Revised: k be bounded as max s, a 1: k − 1 D TV ( π β,k ( a k | s, a 1: k − 1 )

Original: ||
Revised: ||

Original: π k ( a k | s, a 1: k − 1 )) ≤ (cid:15) π k , where π β is the data collecting policy for the model and ˆ p ( s (cid:48) | s, a ) is the transition distribution under the model.
Revised: π k ( a k | s, a 1: k − 1 )) ≤ (cid:15) π k , where π β is the data collecting policy for the model and ˆ p ( s (cid:48) | s, a ) is the transition distribution under the model.

Original: Then the model return ˆ η and true return η of the policy π are bounded as:
Revised: Then the model return ˆ η and true return η of the policy π are bounded as:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is given in Appendix B.
Revised: The proof is given in Appendix B.

Original: Remark 2.
Revised: Remark 2.

Original: Theorem 1 provides a useful relationship between the compounding errors and the policy update.
Revised: Theorem 1 provides a useful relationship between the compounding errors and the policy update.

Original: As long as we improve the return under the true dynamic by more than the gap, C ( (cid:15) m , (cid:15) π 1: n ) , we can guarantee the policy improvement under the world model.
Revised: As long as we improve the return under the true dynamic by more than the gap, C ( (cid:15) m , (cid:15) π 1: n ) , we can guarantee the policy improvement under the world model.

Original: If no such policy exists to overcome the gap, it implies the model error is too high, that is, there is a large discrepancy between the world model and true dynamics.
Revised: If no such policy exists to overcome the gap, it implies the model error is too high, that is, there is a large discrepancy between the world model and true dynamics.

Original: Thus the order sequence obtained under the world model is not reliable.
Revised: Thus the order sequence obtained under the world model is not reliable.

Original: Such an order sequence is almost the same as a random one.
Revised: Such an order sequence is almost the same as a random one.

Original: Though a random order sequence also has the theoretical guarantee of Proposition 2, we will show in Section 5.2 that a random order sequence leads to a poor local optimum empirically.
Revised: Though a random order sequence also has the theoretical guarantee of Proposition 2, we will show in Section 5.2 that a random order sequence leads to a poor local optimum empirically.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Sequential communication (SeqComm) is currently instantiated based on MAPPO (Yu et al., 2021).
Revised: Sequential communication (SeqComm) is currently instantiated based on MAPPO (Yu et al., 2021).

Original: We evaluate SeqComm on three tasks in multi-agent particle environment (MPE) (Lowe et al., 2017) and four maps in StarCraft multi-agent challenge (SMAC) (Samvelyan et al., 2019).
Revised: We evaluate SeqComm on three tasks in multi-agent particle environment (MPE) (Lowe et al., 2017) and four maps in StarCraft multi-agent challenge (SMAC) (Samvelyan et al., 2019).

Original: For these experiments, we compare SeqComm against the following communication-free and communication-basedCTDE baselines: MAPPO (Yu et al., 2021), QMIX (Rashid et al., 2018), IS (Kim et al., 2021), TarMAC (Das et al., 2019), and I2C (Ding et al., 2020).
Revised: For these experiments, we compare SeqComm against the following communication-free and communication-based baselines: MAPPO (Yu et al., 2021), QMIX (Rashid et al., 2018), IS (Kim et al., 2021), TarMAC (Das et al., 2019), and I2C (Ding et al., 2020).

Original: In more detail, IS communicates predicted future trajectories (observations and actions), and predictions are made by the environment model.
Revised: In more detail, IS communicates predicted future trajectories (observations and actions), and predictions are made by the environment model.

Original: TarMAC uses the attention model to focus more on important incoming messages (the hidden states of observations).
Revised: TarMAC uses the attention model to focus more on important incoming messages (the hidden states of observations).

Original: TarMAC is reproduced based on MAPPO instead of A2C in the original paper for better performance.
Revised: TarMAC is reproduced based on MAPPO instead of A2C in the original paper for better performance.

Original: I2C infers one-to-one communication to reduce the redundancy of messages (also conditioning on observations).
Revised: I2C infers one-to-one communication to reduce the redundancy of messages (also conditioning on observations).

Original: In the experiments, all the methods are parameter-sharing for fast convergence.
Revised: In the experiments, all the methods are parameter-sharing for fast convergence.

Original: We have ﬁne-tuned the baselines for a fair comparison.
Revised: We have ﬁne-tuned the baselines for a fair comparison.

Original: Please refer to Appendix E for experimental settings and Appendix F for implementation details.
Revised: Please refer to Appendix E for experimental settings and Appendix F for implementation details.

Original: All results are presented in terms of the mean and standard deviation of ﬁve runs with different random seeds.
Revised: All results are presented in terms of the mean and standard deviation of ﬁve runs with different random seeds.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: MPE.
Revised: MPE.

Original: We experiment on predator-prey (PP), cooperative navigation (CN), and keep-away (KA) in
Revised: We experiment on predator-prey (PP), cooperative navigation (CN), and keep-away (KA) in

Original: MPE.
Revised: MPE.

Original: In PP, ﬁve predators (agents) try to capture three prey.
Revised: In PP, ﬁve predators (agents) try to capture three prey.

Original: In CN, ﬁve agents try to occupy ﬁve landmarks.
Revised: In CN, ﬁve agents try to occupy ﬁve landmarks.

Original: In KA, three attackers (agents) try to occupy three landmarks, however, there are three
Revised: In KA, three attackers (agents) try to occupy three landmarks, however, there are three

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: agents is set to be larger than the original settings so that collisions occur more easily, following the settings in (Kim et al., 2021).
Revised: agents is set to be larger than the original settings so that collisions occur more easily, following the settings in (Kim et al., 2021).

Original: In addition, agents cannot observe any other agents, and this makes the task more difﬁcult and communication more important.
Revised: In addition, agents cannot observe any other agents, and this makes the task more difﬁcult and communication more important.

Original: We can observe similar modiﬁcations in previous works (Foerster et al., 2016;
Revised: We can observe similar modiﬁcations in previous works (Foerster et al., 2016;

Original: Ding et al., 2020).
Revised: Ding et al., 2020).

Original: After all, we want to demonstrate the superior over communication-based baselines, and communication-based methods are more suitable for scenarios with limited vision.
Revised: After all, we want to demonstrate the superior over communication-based baselines, and communication-based methods are more suitable for scenarios with limited vision.

Original: More details about experimental settings are available in Appendix E.
Revised: More details about experimental settings are available in Appendix E.

Original: Figure 4 shows the learning curves of all the methods in terms of the mean reward averaged over timesteps in PP, CN, and KA.
Revised: Figure 4 shows the learning curves of all the methods in terms of the mean reward averaged over timesteps in PP, CN, and KA.

Original: We can see that SeqComm converges to the highest mean reward compared with all the baselines.
Revised: We can see that SeqComm converges to the highest mean reward compared with all the baselines.

Original: The results demonstrate the superiority of SeqComm.
Revised: The results demonstrate the superiority of SeqComm.

Original: In more detail, all communication-based methods outperform MAPPO, indicating the necessity of communication in these difﬁcult tasks.
Revised: In more detail, all communication-based methods outperform MAPPO, indicating the necessity of communication in these difﬁcult tasks.

Original: Apart from MAPPO, IS performs the worst since it may access inaccurate predicted information due to the circular dependencies.
Revised: Apart from MAPPO, IS performs the worst since it may access inaccurate predicted information due to the circular dependencies.

Original: The substantial improvement SeqComm over
Revised: The substantial improvement SeqComm over

Original: I2C and TarMAC is attributed to that SeqComm allows agents to get more valuable action information for explicit coordination.
Revised: I2C and TarMAC is attributed to that SeqComm allows agents to get more valuable action information for explicit coordination.

Original: The agents learned by SeqComm show sophisticated coordination strategies induced by the priority of decision-making, which can be witnessed by the visualization of agent behaviors.
Revised: The agents learned by SeqComm show sophisticated coordination strategies induced by the priority of decision-making, which can be witnessed by the visualization of agent behaviors.

Original: More details are given in Appendix C. Note that QMIX is omitted in the comparison for clear presentation since Yu et al.
Revised: More details are given in Appendix C. Note that QMIX is omitted in the comparison for clear presentation since Yu et al.

Original: (2021) have shown QMIX and MAPPO exhibit similar performance in various MPE tasks.
Revised: (2021) have shown QMIX and MAPPO exhibit similar performance in various MPE tasks.

Original: SMAC.
Revised: SMAC.

Original: We also evaluate SeqComm against the baselines on four customized maps in SMAC: 6h vs 8z, MMM2, 10m vs 11m, and 8m vs 9m, where we have made some minor changes to the observation part of agents to make it more difﬁcult.
Revised: We also evaluate SeqComm against the baselines on four customized maps in SMAC: 6h vs 8z, MMM2, 10m vs 11m, and 8m vs 9m, where we have made some minor changes to the observation part of agents to make it more difﬁcult.

Original: Speciﬁcally, the sight range of agents is reduced from 9 to 2 , and agents cannot perceive any information about their allies even if they are within the sight range.
Revised: Speciﬁcally, the sight range of agents is reduced from 9 to 2 , and agents cannot perceive any information about their allies even if they are within the sight range.

Original: NDQ (Wang et al., 2020) adopts a similar change to increase the difﬁculty of action coordination and demonstrates that the miscoordination problem is widespread in multi-agent learning.
Revised: NDQ (Wang et al., 2020) adopts a similar change to increase the difﬁculty of action coordination and demonstrates that the miscoordination problem is widespread in multi-agent learning.

Original: The rest settings remain the same as the default.
Revised: The rest settings remain the same as the default.

Original: The learning curves of SeqComm and the baselines in terms of the win rate are illustrated in
Revised: The learning curves of SeqComm and the baselines in terms of the win rate are illustrated in

Original: Figure 5.
Revised: Figure 5.

Original: IS and I2C fail in this task and get a zero win rate because these two methods are built on
Revised: IS and I2C fail in this task and get a zero win rate because these two methods are built on

Original: MADDPG.
Revised: MADDPG.

Original: However, MADDPG cannot work well in SMAC, especially when we reduce the sight range of agents, which is also supported by other studies (Papoudakis et al., 2021).
Revised: However, MADDPG cannot work well in SMAC, especially when we reduce the sight range of agents, which is also supported by other studies (Papoudakis et al., 2021).

Original: SeqComm and
Revised: SeqComm and

Original: TarMAC converge to better performances than MAPPO and QMIX, which demonstrates the beneﬁt of communication.
Revised: TarMAC converge to better performances than MAPPO and QMIX, which demonstrates the beneﬁt of communication.

Original: Moreover, SeqComm outperforms TarMAC, which again veriﬁes the gain of explicit action coordination.
Revised: Moreover, SeqComm outperforms TarMAC, which again veriﬁes the gain of explicit action coordination.

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: Priority of Decision-Making.
Revised: Priority of Decision-Making.

Original: We compare SeqComm with two ablation baselines with only a difference in the priority of decision-making: the priority of decision-making is ﬁxed throughout one episode, denoted as Fix-C, and the priority of decision-making is determined randomly at each timestep, denoted as Random-C. TarMAC is also compared as a reference without explicit action coordination.
Revised: We compare SeqComm with two ablation baselines with only a difference in the priority of decision-making: the priority of decision-making is ﬁxed throughout one episode, denoted as Fix-C, and the priority of decision-making is determined randomly at each timestep, denoted as Random-C. TarMAC is also compared as a reference without explicit action coordination.

Original: As depicted in Figure 6, SeqComm achieves a higher mean reward or win rate than Fix-C, Random-C, and TarMAC in all the tasks.
Revised: As depicted in Figure 6, SeqComm achieves a higher mean reward or win rate than Fix-C, Random-C, and TarMAC in all the tasks.

Original: These results verify the importance of the priority of decision-making and the necessity to continuously adjust it during one episode.
Revised: These results verify the importance of the priority of decision-making and the necessity to continuously adjust it during one episode.

Original: It is also demonstrated that SeqComm can provide a proper priority of decision-making.
Revised: It is also demonstrated that SeqComm can provide a proper priority of decision-making.

Original: As discussed in Section 4.3, although Fix-C and Random-C also have the theoretical guarantee, they converge to poor local optima in practice.
Revised: As discussed in Section 4.3, although Fix-C and Random-C also have the theoretical guarantee, they converge to poor local optima in practice.

Original: Moreover, Fix-C and Random-C show better performance than TarMAC in most tasks.
Revised: Moreover, Fix-C and Random-C show better performance than TarMAC in most tasks.

Original: This result accords with the hypothesis that the SE is likely to be Pareto superior to the average NE in games with a high cooperation level.
Revised: This result accords with the hypothesis that the SE is likely to be Pareto superior to the average NE in games with a high cooperation level.

Original: Additionally, the learned policy of SeqComm can generalize well to the same task with a different number of agents in MPE, which is detailed in Appendix C.
Revised: Additionally, the learned policy of SeqComm can generalize well to the same task with a different number of agents in MPE, which is detailed in Appendix C.

Original: Communication Range.
Revised: Communication Range.

Original: We also carry out ablation studies on communication range in MPE tasks.
Revised: We also carry out ablation studies on communication range in MPE tasks.

Original: Note that communication range means how many nearest neighbors each agent is allowed to communicate with, following the setting in Ding et al.
Revised: Note that communication range means how many nearest neighbors each agent is allowed to communicate with, following the setting in Ding et al.

Original: We reduce the communication range of SeqComm from 4 to 2 and 0 .
Revised: We reduce the communication range of SeqComm from 4 to 2 and 0 .

Original: As there are only three agents in KA, it is omitted in this study.
Revised: As there are only three agents in KA, it is omitted in this study.

Original: The results are shown in Figure 7.
Revised: The results are shown in Figure 7.

Original: Communication-based agents perform better than communication-free agents, which accords with the results of many previous studies.
Revised: Communication-based agents perform better than communication-free agents, which accords with the results of many previous studies.

Original: More importantly, the superiority of SeqComm with communication range 2 over the corresponding TarMAC again demonstrates the effectiveness of sequential communication even in reduced communication ranges.
Revised: More importantly, the superiority of SeqComm with communication range 2 over the corresponding TarMAC again demonstrates the effectiveness of sequential communication even in reduced communication ranges.

Original: [Figure]
Revised: [Figure]

Original: However, as the communication range decreases from 4 to 2 , there is no performance reduction in these two MPE tasks.
Revised: However, as the communication range decreases from 4 to 2 , there is no performance reduction in these two MPE tasks.

Original: On the contrary, the agents with communication range 2 perform the best.
Revised: On the contrary, the agents with communication range 2 perform the best.

Original: It accords with the results in I2C (Ding et al., 2020) and ATOC (Jiang & Lu, 2018) that redundant information can impair the learning process sometimes.
Revised: It accords with the results in I2C (Ding et al., 2020) and ATOC (Jiang & Lu, 2018) that redundant information can impair the learning process sometimes.

Original: In other settings, this conclusion might not be true.
Revised: In other settings, this conclusion might not be true.

Original: Moreover, since under our communication scheme agents can obtain more information, i.e., the actual actions of others, it is more reasonable that SeqComm can still outperform other methods in reduced communication ranges.
Revised: Moreover, since under our communication scheme agents can obtain more information, i.e., the actual actions of others, it is more reasonable that SeqComm can still outperform other methods in reduced communication ranges.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have proposed SeqComm, which enables agents explicitly coordinate with each other.
Revised: We have proposed SeqComm, which enables agents explicitly coordinate with each other.

Original: SeqComm from an asynchronous perspective allows agents to make decisions sequentially.
Revised: SeqComm from an asynchronous perspective allows agents to make decisions sequentially.

Original: A two-phase communication scheme has been adopted for determining the priority of decision-making and communicating messages accordingly.
Revised: A two-phase communication scheme has been adopted for determining the priority of decision-making and communicating messages accordingly.

Original: Theoretically, we prove the policies learned by SeqComm are guaranteed to improve monotonically and converge.
Revised: Theoretically, we prove the policies learned by SeqComm are guaranteed to improve monotonically and converge.

Original: Empirically, it is demonstrated that SeqComm outperforms baselines in a variety of cooperative multi-agent tasks and SeqComm can provide a proper priority of decision-making.
Revised: Empirically, it is demonstrated that SeqComm outperforms baselines in a variety of cooperative multi-agent tasks and SeqComm can provide a proper priority of decision-making.

Original: A P ROOFS OF P ROPOSITION 1 AND P ROPOSITION
Revised: A P ROOFS OF P ROPOSITION 1 AND P ROPOSITION

Original: Lemma 1 (Agent-by-Agent PPO) .
Revised: Lemma 1 (Agent-by-Agent PPO) .

Original: If we update the policy of each agent i with TRPO Schulman et al.
Revised: If we update the policy of each agent i with TRPO Schulman et al.

Original: (2015) (or approximately PPO) when ﬁxing all the other agent’s policies, then the joint policy will improve monotonically.
Revised: (2015) (or approximately PPO) when ﬁxing all the other agent’s policies, then the joint policy will improve monotonically.

Original: Proof.
Revised: Proof.

Original: We consider the joint surrogate objective in TRPO L π old ( π new ) where π old is the joint policy before updating and π new is the joint policy after updating.
Revised: We consider the joint surrogate objective in TRPO L π old ( π new ) where π old is the joint policy before updating and π new is the joint policy after updating.

Original: Given that π − i new = π − i old
Revised: Given that π − i new = π − i old

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: where A i π old ( s, a i ) =
Revised: where A i π old ( s, a i ) =

Original: E a − i ∼ π − i old [ A π old ( s, a i , a − i )] is the individual advantage of agent
Revised: E a − i ∼ π − i old [ A π old ( s, a i , a − i )] is the individual advantage of agent

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: and the third equation is from the condition π − i new =
Revised: and the third equation is from the condition π − i new =

Original: π − i old .
Revised: π − i old .

Original: With the result of TRPO
Revised: With the result of TRPO

Original: [Equation]
Revised: [Equation]

Original: we have the following conclusion:
Revised: we have the following conclusion:

Original: [Equation]
Revised: [Equation]

Original: This means the individual objective is the same as the joint objective so the monotonic improvement is guaranteed.
Revised: This means the individual objective is the same as the joint objective so the monotonic improvement is guaranteed.

Original: Then we can show the proof of Proposition 1.
Revised: Then we can show the proof of Proposition 1.

Original: Proof.
Revised: Proof.

Original: We will build a new MDP ˜ M based on the original MDP.
Revised: We will build a new MDP ˜ M based on the original MDP.

Original: We keep the action space ˜
Revised: We keep the action space ˜

Original: A =
Revised: A =

Original: A = × ni =

Revised: A = × ni =

Original: A i , where A i is the original action space of agent i .
Revised: A i , where A i is the original action space of agent i .

Original: The new state space contains multiple layers.
Revised: The new state space contains multiple layers.

Original: We deﬁne ˜ S
Revised: We deﬁne ˜ S

Original: k = S × ( × ki =1 A i ) for k = 1 , 2 , · · · , n − 1 and ˜ S 

Revised: k = S × ( × ki =1 A i ) for k = 1 , 2 , · · · , n − 1 and ˜ S

Original: = S , where S is the original state space.
Revised: = S , where S is the original state space.

Original: Then a new state ˜ s k ∈ ˜
Revised: Then a new state ˜ s k ∈ ˜

Original: S k means that ˜ s k
Revised: S k means that ˜ s k

Original: =
Revised: =

Original: ( s, a 1 , a 2 , · · · , a k ) .
Revised: ( s, a 1 , a 2 , · · · , a k ) .

Original: The total new state space is deﬁned as ˜ S
Revised: The total new state space is deﬁned as ˜ S

Original: = ∪ n − 1 i =

Revised: = ∪ n − 1 i =

Original: ˜ S i .
Revised: ˜ S i .

Original: Next we deﬁne the transition probability ˜ P as following:
Revised: Next we deﬁne the transition probability ˜ P as following:

Original: [Equation]
Revised: [Equation]

Original: This means that the state in the layer k can only transition to the state in the layer k + 1 with the corresponding action, and the state in the layer n − 1 will transition to the layer 0 with the probability
Revised: This means that the state in the layer k can only transition to the state in the layer k + 1 with the corresponding action, and the state in the layer n − 1 will transition to the layer 0 with the probability

Original: P in the original MDP.
Revised: P in the original MDP.

Original: The reward function ˜ r is deﬁned as following:
Revised: The reward function ˜ r is deﬁned as following:

Original: [Equation]
Revised: [Equation]

Original: This means the reward is only obtained when the state in layer 

Revised: This means the reward is only obtained when the state in layer

Original: and the value is the same as the original reward function.
Revised: and the value is the same as the original reward function.

Original: Now we obtain the total deﬁnition of the new MDP ˜ M = { ˜ S, ˜
Revised: Now we obtain the total deﬁnition of the new MDP ˜ M = { ˜ S, ˜

Original: [Equation]
Revised: [Equation]

Original: , ˜ P, ˜ r, γ } .
Revised: , ˜ P, ˜ r, γ } .

Original: Then we claim that if all agents learn in multi-agent sequential decision-making by PPO, they are actually taking agent-by-agent PPO in the new MDP ˜ M .
Revised: Then we claim that if all agents learn in multi-agent sequential decision-making by PPO, they are actually taking agent-by-agent PPO in the new MDP ˜ M .

Original: To be precise, one update of multi-agent sequential decision-making in the original MDP M equals to a round of update from agent 1 to agent n by agent-by-agent PPO in the new MDP ˜ M .
Revised: To be precise, one update of multi-agent sequential decision-making in the original MDP M equals to a round of update from agent 1 to agent n by agent-by-agent PPO in the new MDP ˜ M .

Original: Moreover, the total reward of a round in the new
Revised: Moreover, the total reward of a round in the new

Original: MDP ˜ M is the same as the reward in one timestep in the original MDP M .
Revised: MDP ˜ M is the same as the reward in one timestep in the original MDP M .

Original: With this conclusion and
Revised: With this conclusion and

Original: Lemma 1, we complete the proof.
Revised: Lemma 1, we complete the proof.

Original: The proof of Proposition 2 can be seen as a corollary of the proof of Proposition 1.
Revised: The proof of Proposition 2 can be seen as a corollary of the proof of Proposition 1.

Original: Proof.
Revised: Proof.

Original: From Lemma 1 we know that the monotonic improvement of the joint policy in the new MDP ˜
Revised: From Lemma 1 we know that the monotonic improvement of the joint policy in the new MDP ˜

Original: M is guaranteed for each update of one single agent’s policy.
Revised: M is guaranteed for each update of one single agent’s policy.

Original: So even if the different round of updates in the new MDP ˜ M is with different order of the decision-making, the monotonic improvement of the joint policy is still guaranteed.
Revised: So even if the different round of updates in the new MDP ˜ M is with different order of the decision-making, the monotonic improvement of the joint policy is still guaranteed.

Original: Finally, from the proof of Proposition 1, we know that the monotonic improvement in the new MDP ˜ M equals to the monotonic improvement in the original MDP M .
Revised: Finally, from the proof of Proposition 1, we know that the monotonic improvement in the new MDP ˜ M equals to the monotonic improvement in the original MDP M .

Original: These complete the proof.
Revised: These complete the proof.

Original: B P ROOFS OF T HEOREM
Revised: B P ROOFS OF T HEOREM

Original: Lemma 2 (TVD of the joint distributions) .
Revised: Lemma 2 (TVD of the joint distributions) .

Original: Suppose we have two distribution p 1 ( x, y )
Revised: Suppose we have two distribution p 1 ( x, y )

Original: =
Revised: =

Original: p 1 ( x ) p 1 ( x | y ) and p 2 ( x, y ) =
Revised: p 1 ( x ) p 1 ( x | y ) and p 2 ( x, y ) =

Original: p 2 ( x ) p 2 ( x | y ) .
Revised: p 2 ( x ) p 2 ( x | y ) .

Original: We can bound the total variation distance of the joint as:
Revised: We can bound the total variation distance of the joint as:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See (Janner et al., 2019)
Revised: See (Janner et al., 2019)

Original: (Lemma B.1).
Revised: (Lemma B.1).

Original: Lemma 3 (Markov chain TVD bound, time-varing) .
Revised: Lemma 3 (Markov chain TVD bound, time-varing) .

Original: Suppose the expected KL-divergence between two transition is bounded as max t E s ∼ p 1 ,t ( s ) D KL ( p 1 ( s (cid:48) | s )
Revised: Suppose the expected KL-divergence between two transition is bounded as max t E s ∼ p 1 ,t ( s ) D KL ( p 1 ( s (cid:48) | s )

Original: || p 2 ( s (cid:48) | s ))
Revised: || p 2 ( s (cid:48) | s ))

Original: ≤ δ , and the initial state distributions are the same p 1 ,t =0 ( s ) =
Revised: ≤ δ , and the initial state distributions are the same p 1 ,t =0 ( s ) =

Original: p 2 ,t =0 ( s ) .
Revised: p 2 ,t =0 ( s ) .

Original: Then the distance in the state marginal is bounded as:
Revised: Then the distance in the state marginal is bounded as:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See (Janner et al., 2019) (Lemma B.2).
Revised: See (Janner et al., 2019) (Lemma B.2).

Original: Lemma 4 (Branched Returns Bound) .
Revised: Lemma 4 (Branched Returns Bound) .

Original: Suppose the expected KL-divergence between two dynamics distributions is bounded as max
Revised: Suppose the expected KL-divergence between two dynamics distributions is bounded as max

Original: [Equation]
Revised: [Equation]

Original: TV
Revised: TV

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: | s, a ))] , and the policy divergences at level k are bounded as
Revised: | s, a ))] , and the policy divergences at level k are bounded as

Original: [Equation]
Revised: [Equation]

Original: Then the returns are bounded as:
Revised: Then the returns are bounded as:

Original: [Equation]
Revised: [Equation]

Original: where r max is the upper bound of the reward function.
Revised: where r max is the upper bound of the reward function.

Original: Proof.
Revised: Proof.

Original: Here, η 1 denotes the returns of π 1 under dynamics p 1 ( s (cid:48) | s, a ) , and η 2 denotes the returns of π 2 under dynamics p 2 ( s (cid:48) | s, a ) .
Revised: Here, η 1 denotes the returns of π 1 under dynamics p 1 ( s (cid:48) | s, a ) , and η 2 denotes the returns of π 2 under dynamics p 2 ( s (cid:48) | s, a ) .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: t (cid:88) s, a γ t | p 1 ,t ( s, a ) − p 2 ,t ( s, a ) | .
Revised: t (cid:88) s, a γ t | p 1 ,t ( s, a ) − p 2 ,t ( s, a ) | .

Original: By Lemma 2, we get
Revised: By Lemma 2, we get

Original: [Equation]
Revised: [Equation]

Original: We then apply Lemma 3, using δ = (cid:15) m
Revised: We then apply Lemma 3, using δ = (cid:15) m

Original: + (cid:80) nk =1 (cid:15) π k (via Lemma 3 and 2) to get
Revised: + (cid:80) nk =1 (cid:15) π k (via Lemma 3 and 2) to get

Original: [Equation]
Revised: [Equation]

Original: 2 ,t (
Revised: 2 ,t (

Original: [Equation]
Revised: [Equation]

Original: ≤ t max t
Revised: ≤ t max t

Original: [Equation]
Revised: [Equation]

Original: ∼
Revised: ∼

Original: [Equation]
Revised: [Equation]

Original: π k )
Revised: π k )

Original: And we also get D TV ( p 1 ,t ( s, a ) || p 2 ,t
Revised: And we also get D TV ( p 1 ,t ( s, a ) || p 2 ,t

Original: ( s, a ))
Revised: ( s, a ))

Original: ≤ t
Revised: ≤ t

Original: [Equation]
Revised: [Equation]

Original: m +
Revised: m +

Original: [Equation]
Revised: [Equation]

Original: nk =
Revised: nk =

Original: [Equation]
Revised: [Equation]

Original: π k )
Revised: π k )

Original: + (cid:80) nk =1 (cid:15)
Revised: + (cid:80) nk =1 (cid:15)

Original: π k by Lemma 2.
Revised: π k by Lemma 2.

Original: Thus, by plugging this back, we get:
Revised: Thus, by plugging this back, we get:

Original: [Equation]
Revised: [Equation]

Original: Then we can show the proof of Theorem 1.
Revised: Then we can show the proof of Theorem 1.

Original: Proof.
Revised: Proof.

Original: Let π β denote the data collecting policy.
Revised: Let π β denote the data collecting policy.

Original: We use Lemma 4 to bound the returns, but it will require bounded model error under the new policy π .
Revised: We use Lemma 4 to bound the returns, but it will require bounded model error under the new policy π .

Original: Thus, we need to introduce π β by adding and subtracting η
Revised: Thus, we need to introduce π β by adding and subtracting η

Original: [ π β ] , to get:
Revised: [ π β ] , to get:

Original: [Equation]
Revised: [Equation]

Original: we can bound L 1 and L 2 both using Lemma 4 by using δ = (cid:80) nk =1 (cid:15) π k and δ = (cid:15) m + (cid:80) nk =1 (cid:15)
Revised: we can bound L 1 and L 2 both using Lemma 4 by using δ = (cid:80) nk =1 (cid:15) π k and δ = (cid:15) m + (cid:80) nk =1 (cid:15)

Original: π k respectively, and obtain:
Revised: π k respectively, and obtain:

Original: [Equation]
Revised: [Equation]

Original: Adding these two bounds together yields the conclusion.
Revised: Adding these two bounds together yields the conclusion.

Original: pplevelagentagent 3 agentlevellevellevellevellevellevellevellevellevellevellevellevellevellevel 5 a b c d e levellevellevel 1 agentagent 5 agentlevellevellevellevellevellevellevellevellevellevellevellevellevellevelcn a b c d e
Revised: pplevelagentagent 3 agentlevellevellevellevellevellevellevellevellevellevellevellevellevellevel 5 a b c d e levellevellevel 1 agentagent 5 agentlevellevellevellevellevellevellevellevellevellevellevellevellevellevelcn a b c d e

Original: [Table]
Revised: [Table]

Original: C A DDITIONAL E XPERIMENTS C. I LLUSTRATION OF L EARNED P RIORITY OF D ECISION -M AKING
Revised: C A DDITIONAL E XPERIMENTS C. I LLUSTRATION OF L EARNED P RIORITY OF D ECISION -M AKING

Original: Figure 8 (upper panel from a to e ) shows the priority order of decision-making determined by
Revised: Figure 8 (upper panel from a to e ) shows the priority order of decision-making determined by

Original: SeqComm in PP.
Revised: SeqComm in PP.

Original: Agent 2 that is far away from other preys and predators is chosen to be the ﬁrstmover.
Revised: Agent 2 that is far away from other preys and predators is chosen to be the ﬁrstmover.

Original: If agents want to encircle and capture the preys, the agents ( e.g. , agent 2 and 5) that are on the periphery of the encircling circle should hold upper-level positions since they are able to decide how to narrow the encirclement.
Revised: If agents want to encircle and capture the preys, the agents ( e.g. , agent 2 and 5) that are on the periphery of the encircling circle should hold upper-level positions since they are able to decide how to narrow the encirclement.

Original: In addition, agent 3 makes decisions prior to agent 5 so that collision can be avoided after agent 5 obtains the intention of agent 3 .
Revised: In addition, agent 3 makes decisions prior to agent 5 so that collision can be avoided after agent 5 obtains the intention of agent 3 .

Original: For CN, as illustrated in Figure 8 (lower panel from a to e ), agent 2 is far away from all the landmarks and all other agents are in a better position to occupy landmarks.
Revised: For CN, as illustrated in Figure 8 (lower panel from a to e ), agent 2 is far away from all the landmarks and all other agents are in a better position to occupy landmarks.

Original: Therefore, agents 2 is chosen to be the ﬁrst-mover, which is similar to the phenomenon observed in PP.
Revised: Therefore, agents 2 is chosen to be the ﬁrst-mover, which is similar to the phenomenon observed in PP.

Original: Once it has determined the target to occupy, other agents (agent 5 and 3 ) can adjust their actions accordingly and avoid conﬂict of goals.
Revised: Once it has determined the target to occupy, other agents (agent 5 and 3 ) can adjust their actions accordingly and avoid conﬂict of goals.

Original: Otherwise, if agent 5 makes a decision ﬁrst and chooses to occupy the closest landmark, then agent 2 has to approach to a further landmark which would take more steps.
Revised: Otherwise, if agent 5 makes a decision ﬁrst and chooses to occupy the closest landmark, then agent 2 has to approach to a further landmark which would take more steps.

Original: C. G ENERALIZATION
Revised: C. G ENERALIZATION

Original: Generalization to different numbers of agents has always been a key problem in MARL.
Revised: Generalization to different numbers of agents has always been a key problem in MARL.

Original: For most algorithms in communication, once the model is trained in one scenario, it is unlikely for agents to maintain relatively competitive performance in other scenarios with different numbers of agents.
Revised: For most algorithms in communication, once the model is trained in one scenario, it is unlikely for agents to maintain relatively competitive performance in other scenarios with different numbers of agents.

Original: However, as we employ attention modules to process communicated messages so that agents can handle messages of different lengths.
Revised: However, as we employ attention modules to process communicated messages so that agents can handle messages of different lengths.

Original: In addition, the module used to determine the priority of decision-making is also not restricted by the number of agents.
Revised: In addition, the module used to determine the priority of decision-making is also not restricted by the number of agents.

Original: Thus, we investigate whether
Revised: Thus, we investigate whether

Original: SeqComm generalizes well to different numbers of agents in CN and PP.
Revised: SeqComm generalizes well to different numbers of agents in CN and PP.

Original: For both tasks, SeqComm is trained on 5-agent settings.
Revised: For both tasks, SeqComm is trained on 5-agent settings.

Original: Then, we test SeqComm in 3-agent and 7-agent settings of CN and 7-agent setting of PP.
Revised: Then, we test SeqComm in 3-agent and 7-agent settings of CN and 7-agent setting of PP.

Original: We use Fix-C trained directly on these test tasks to illustrate the performance of SeqComm.
Revised: We use Fix-C trained directly on these test tasks to illustrate the performance of SeqComm.

Original: Note that the quantity of both landmarks and preys is  adjusted according to the number of agents in CN and PP.
Revised: Note that the quantity of both landmarks and preys is [Figure] adjusted according to the number of agents in CN and PP.

Original: The test results are shown in Table 1.
Revised: The test results are shown in Table 1.

Original: SeqComm exhibits the superiority in CN and PP, demonstrating that SeqComm may have a good generalization to the number of agents.
Revised: SeqComm exhibits the superiority in CN and PP, demonstrating that SeqComm may have a good generalization to the number of agents.

Original: A thorough study of the generalization of SeqComm is left to future work.
Revised: A thorough study of the generalization of SeqComm is left to future work.

Revised: C. M ORE SMAC M APS

Revised: We have evaluated our method on two additional maps, i.e., 3s vs 4z and corridor.

Revised: As illustrated in

Revised: Figure 9, we can ﬁnd out the similar conclusions as section 5.1.

Original: D A DDITIONAL R ELATED W ORK
Revised: D A DDITIONAL R ELATED W ORK

Original: Multi-Agent Path Finding (MAPF).
Revised: Multi-Agent Path Finding (MAPF).

Original: MAPF aims to plan collision-free paths for multiple agents on a given graph from their given start vertices to target vertices.
Revised: MAPF aims to plan collision-free paths for multiple agents on a given graph from their given start vertices to target vertices.

Original: In MAPF, prioritized planning is deeply coupled with collision avoidance (Van Den Berg & Overmars, 2005; Ma et al., 2019), where collision is used to design constraints or heuristics for planning.
Revised: In MAPF, prioritized planning is deeply coupled with collision avoidance (Van Den Berg & Overmars, 2005; Ma et al., 2019), where collision is used to design constraints or heuristics for planning.

Original: Unlike MAPF, our method couples the priority of decision-making with the learning objective and thus is more general.
Revised: Unlike MAPF, our method couples the priority of decision-making with the learning objective and thus is more general.

Original: In addition, the different motivations and problem settings may lead to the incompatibility of the methods in the two ﬁelds.
Revised: In addition, the different motivations and problem settings may lead to the incompatibility of the methods in the two ﬁelds.

Original: Reinforcement Learning in Stackelberg Game.
Revised: Reinforcement Learning in Stackelberg Game.

Original: Many studies (K¨on¨onen, 2004; Sodomka et al., 2013; Greenwald et al., 2003;
Revised: Many studies (K ¨ on ¨ onen, 2004; Sodomka et al., 2013; Greenwald et al., 2003;

Original: Zhang et al., 2020) have investigated reinforcement learning in ﬁnding the Stackelberg equilibrium.
Revised: Zhang et al., 2020) have investigated reinforcement learning in ﬁnding the Stackelberg equilibrium.

Original: Bi-AC (Zhang et al., 2020) is a bi-level actor-critic method that allows agents to have different knowledge bases so that the Stackelberg equilibrium (SE) is possible to ﬁnd.
Revised: Bi-AC (Zhang et al., 2020) is a bi-level actor-critic method that allows agents to have different knowledge bases so that the Stackelberg equilibrium (SE) is possible to ﬁnd.

Original: The actions still can be executed simultaneously and distributedly.
Revised: The actions still can be executed simultaneously and distributedly.

Original: It empirically studies the relationship between the cooperation level and the superiority of the SE over the Nash equilibrium.
Revised: It empirically studies the relationship between the cooperation level and the superiority of the SE over the Nash equilibrium.

Original: AQL (K ¨ on ¨ onen, 2004) updates the Q-value by solving the SE in each iteration and can be regarded as the value-based version of Bi-AC.
Revised: AQL (K¨on¨onen, 2004) updates the Q-value by solving the SE in each iteration and can be regarded as the value-based version of Bi-AC.

Original: Existing work mainly focuses on two-agent settings and their order is ﬁxed in advance.
Revised: Existing work mainly focuses on two-agent settings and their order is ﬁxed in advance.

Original: However, the ﬁxed order can hardly be an optimal solution as we will show in the next section.
Revised: However, the ﬁxed order can hardly be an optimal solution as we will show in the next section.

Original: To address this issue, we exploit agents’ intentions to dynamically determine the priority of decision-making along the way of interacting with each other.
Revised: To address this issue, we exploit agents’ intentions to dynamically determine the priority of decision-making along the way of interacting with each other.

Original: E E XPERIMENTAL S ETTINGS
Revised: E E XPERIMENTAL S ETTINGS

Original: In cooperative navigation, there are 5 agents and the size of each is 0.15.
Revised: In cooperative navigation, there are 5 agents and the size of each is 0.15.

Original: They need to occupylandmarks with the size of 0.05.
Revised: They need to occupylandmarks with the size of 0.05.

Original: The acceleration of agents is 7.
Revised: The acceleration of agents is 7.

Original: In predator-prey, the number of predators (agents) and prey is set to 5 and 3, respectively, and their sizes are 0.15 and 0.05.
Revised: In predator-prey, the number of predators (agents) and prey is set to 5 and 3, respectively, and their sizes are 0.15 and 0.05.

Original: The acceleration is 5 for predators and 7 for prey.
Revised: The acceleration is 5 for predators and 7 for prey.

Original: In keep away, the number of attackers (agents) and defenders is set to 3, and their sizes are respectively 0.15 and 0.05.
Revised: In keep away, the number of attackers (agents) and defenders is set to 3, and their sizes are respectively 0.15 and 0.05.

Original: Besides, the acceleration isfor attackers and 4 for defenders.
Revised: Besides, the acceleration is [Table] for attackers and 4 for defenders.

Original: The three landmarks are located at (0 . 00 , 0 . 30) , (0 . 25 , − 0 . 15) , and ( − 0 .
Revised: The three landmarks are located at (0 . 00 , 0 . 30) , (0 . 25 , − 0 . 15) , and ( − 0 .

Original: 25 , − 0 . 15) .
Revised: 25 , − 0 . 15) .

Original: Note that each agent is allowed to communicate with all other agents in all three tasks.
Revised: Note that each agent is allowed to communicate with all other agents in all three tasks.

Original: The team reward is similar across tasks.
Revised: The team reward is similar across tasks.

Original: At a timestep t , it can be written as r t team = − (cid:80) ni =1 d ti + C t r collision , where d ti is the distance of landmark/prey i to its nearest agent/predator,
Revised: At a timestep t , it can be written as r t team = − (cid:80) n i =1 d t i + C t r collision , where d t i is the distance of landmark/prey i to its nearest agent/predator,

Original: C t is the number of collisions (when the distance between two agents is less than the sum of their sizes) occurred at timestep t , and r collision = − 1 .
Revised: C t is the number of collisions (when the distance between two agents is less than the sum of their sizes) occurred at timestep t , and r collision = − 1 .

Original: In addition, agents act discretely and have 5 actions (stay and move up, down, left, right).
Revised: In addition, agents act discretely and have 5 actions (stay and move up, down, left, right).

Original: The length of each episode is 20, 30, and 20 in cooperative navigation, predator-prey, and keep-away, respectively.
Revised: The length of each episode is 20, 30, and 20 in cooperative navigation, predator-prey, and keep-away, respectively.

Original: F I MPLEMENTATION D ETAILS F. A RCHITECTURE AND H YPERPARAMETERS
Revised: F I MPLEMENTATION D ETAILS F. A RCHITECTURE AND H YPERPARAMETERS

Original: Our models, including SeqComm, Fix-C, and Random-C are trained based on MAPPO.
Revised: Our models, including SeqComm, Fix-C, and Random-C are trained based on MAPPO.

Original: The critic and policy network are realized by two fully connected layers.
Revised: The critic and policy network are realized by two fully connected layers.

Original: As for the attention module, key, query, and value have one fully connected layer each.
Revised: As for the attention module, key, query, and value have one fully connected layer each.

Original: The size of hidden layers is 100.
Revised: The size of hidden layers is 100.

Original: Tanh functions are used as nonlinearity.
Revised: Tanh functions are used as nonlinearity.

Original: For I2C, we use their ofﬁcial code with default settings of basic hyperparameters and networks.
Revised: For I2C, we use their ofﬁcial code with default settings of basic hyperparameters and networks.

Original: As there is no released code of IS and TarMAC, we implement IS and TarMAC by ourselves, following the instructions mentioned in the original papers (Kim et al., 2021; Das et al., 2019).
Revised: As there is no released code of IS and TarMAC, we implement IS and TarMAC by ourselves, following the instructions mentioned in the original papers (Kim et al., 2021; Das et al., 2019).

Original: For the world model, observations and actions are ﬁrstly encoded by a fully connected layer.
Revised: For the world model, observations and actions are ﬁrstly encoded by a fully connected layer.

Original: The output size for the observation encoder is 48, and the output size for the action encoder is 16.
Revised: The output size for the observation encoder is 48, and the output size for the action encoder is 16.

Original: Then the outputs of the encoder will be passed into the attention module with the same structure[Table] aforementioned.
Revised: Then the outputs of the encoder will be passed into the attention module with the same structure aforementioned.

Original: Finally, we use a fully connected layer to decode.
Revised: Finally, we use a fully connected layer to decode.

Original: In these layers, Tanh is used as the nonlinearity.
Revised: In these layers, Tanh is used as the nonlinearity.

Original: Table 2 summarize the hyperparameters used by SeqComm and the baselines in the experiments.
Revised: Table 2 summarize the hyperparameters used by SeqComm and the baselines in the MPE.

Revised: For SMAC, SeqComm, Random-C, Fix-C are based on the same architecture, the hyperparameters stay the same.

Original: F. A TTENTION M ODULE
Revised: For MMM2, 6z vs 8z, and 8m vs 9m, the learning rate is 5e − 5 , while for 10m vs 11m, corridor, and 3s vs 4z, learning rate is 7e − 5 . The ppo epoch is set to 10 for 6 h vs 8 z , and is 5 for rest maps. H and F is set to 5 and 1, respectively. However, 20 and 2 is a better value of H and F if computing resources is sufﬁcient.

Revised: For TarMAC, the learning rate is 7e − 5 for all maps.

Revised: The ppo epoch is set to 10 for 6 h vs 8 z , and is 5 for rest maps.

Revised: For MAPPO, the learning rate is 5e − 5 for MMM2 and 6z vs 8z, and 7e − 5 for 8m vs 9m and 10m vs 11m.

Revised: For these four methods, the mini batch is set to 1.

Revised: As for other hyperparameters, we follow the default settings of the ofﬁcial code (Yu et al., 2021).

Revised: For QMIX, the learning rate is 5e − 5 .

Revised: The (cid:15) is 1 and the batch size is 32.

Revised: The buffer size is 5e 3 .

Original:  Attention module (AM) is applied to process messages in the world model, critic network, and policy network.
Revised: For others, we F. A TTENTION M ODULE Attention module (AM) is applied to process messages in the world model, critic network, and policy network.

Original: AM consists of three components: query, key, and values.
Revised: AM consists of three components: query, key, and values.

Original: The output of AM is the weighted sum of values, where the weight of value is determined by the dot product of the query and the corresponding key.
Revised: The output of AM is the weighted sum of values, where the weight of value is determined by the dot product of the query and the corresponding key.

Original: For AM in the world model denoted as AM w , agent i gets messages m − i t
Revised: For AM in the world model denoted as AM w , agent i gets messages m − i t

Original: = h − i t from all other agents at timestep t in negotiation phase, and predicts a query vector q it following AM i w ,q ( h it ) .
Revised: = h − i t from all other agents at timestep t in negotiation phase, and predicts a query vector q it following AM i w ,q ( h it ) .

Original: The query is used to compute a dot product with keys k t =
Revised: The query is used to compute a dot product with keys k t =

Original: [ k 1 t , · · · , k nt ] .
Revised: [ k 1 t , · · · , k nt ] .

Original: Note that k jt is obtained by the message from agent j following AM i a ,k ( h jt ) for j (cid:54)
Revised: Note that k jt is obtained by the message from agent j following AM i a ,k ( h jt ) for j (cid:54)

Original: =
Revised: =

Original: i , and k it is from AM i neg ,k
Revised: i , and k it is from AM i neg ,k

Original: ( h it ) .
Revised: ( h it ) .

Original: Besides, it is scaled by 1 / √ d k followed by a softmax to obtain attention weights α for each value vector:
Revised: Besides, it is scaled by 1 / √ d k followed by a softmax to obtain attention weights α for each value vector:

Original: [Equation]
Revised: [Equation]

Original:  The output of attention module is deﬁned as: c it = (cid:80) nj =1 α ij v jt , where v jt is obtained from messages or its own hidden state of observation following AM i w ,v ( · ) .
Revised: [Equation] The output of attention module is deﬁned as: c it = (cid:80) nj =1 α ij v jt , where v jt is obtained from messages or its own hidden state of observation following AM i w ,v ( · ) .

Original: As for AM in the policy and critic network denoted as AM a , agent i gets additional messages from upper-level agent in the launching phase.
Revised: As for AM in the policy and critic network denoted as AM a , agent i gets additional messages from upper-level agent in the launching phase.

Original: The message from upper-level and lower-level agent can be expanded as m uppert =
Revised: The message from upper-level and lower-level agent can be expanded as m uppert =

Original: [ h uppert , a uppert ] and m lowert
Revised: [ h uppert , a uppert ] and m lowert

Original: =
Revised: =

Original: [ h lowert , 0] , respectively.
Revised: [ h lowert , 0] , respectively.

Original: In addition, the query depends on agent’s own hidden state of observation h it , but keys and values are only from messages of other agents.
Revised: In addition, the query depends on agent’s own hidden state of observation h it , but keys and values are only from messages of other agents.

Original: F. T RAINING
Revised: F. T RAINING

Original: The training of SeqComm is an extension of MAPPO.
Revised: The training of SeqComm is an extension of MAPPO.

Original: The observation encoder e , the critic V , and the policy π are respectively parameterized by θ e , θ v , θ π .
Revised: The observation encoder e , the critic V , and the policy π are respectively parameterized by θ e , θ v , θ π .

Original: Besides, the attention module AM a is parameterized by θ a and takes as input the agent’s hidden state, the messages (hidden states of other agents) in the negotiation phase, and the messages (the actions of upper-level agents) in launching phase.
Revised: Besides, the attention module AM a is parameterized by θ a and takes as input the agent’s hidden state, the messages (hidden states of other agents) in the negotiation phase, and the messages (the actions of upper-level agents) in launching phase.

Original: Let D =
Revised: Let D =

Original: { τ k } K k =1 be a set of trajectories by running policy in the environment.
Revised: { τ k } Kk =1 be a set of trajectories by running policy in the environment.

Original: Note that we drop time t in the following notations for simplicity.
Revised: Note that we drop time t in the following notations for simplicity.

Original: The value function is ﬁtted by regression on mean-squared error:
Revised: The value function is ﬁtted by regression on mean-squared error:

Original: [Equation]
Revised: [Equation]

Original: where ˆ R is the discount rewards-to-go.
Revised: where ˆ R is the discount rewards-to-go.

Original: We update the policy by maximizing the PPO-Clip objective:
Revised: We update the policy by maximizing the PPO-Clip objective:

Original: [Equation]
Revised: [Equation]

Original: where g ( (cid:15), A ) = 
Revised: where g ( (cid:15), A ) = (cid:26) (1 + (cid:15) ) A A ≥(1 −

Original: [Equation]
Revised: [Equation]

Original: (1 + (cid:15) )

Original: A A ≥(1 − (cid:15) )

Original: A A ≤ 0 , and A π old ( o , a upper , a ) is computed using the GAE method.
Revised: A A ≤ 0 , and A π old ( o , a upper , a ) is computed using the GAE method.

Original: The world model M is parameterized by θ w is trained as a regression model using the training data set S .
Revised: The world model M is parameterized by θ w is trained as a regression model using the training data set S .

Original: It is updated with the loss:
Revised: It is updated with the loss:

Original: [Equation]
Revised: [Equation]

Original: We trained our model on one GeForce GTX 

Revised: We trained our model on one GeForce GTX

Original: Ti and Intel(R) Core(TM) [Equation] -9900K CPU @
Revised: Ti and Intel(R) Core(TM) i9-9900K CPU @

Original: [Equation]
Revised: 60GHz.