Original: CM

Original: : C OOPERATIVE

Original: M

Original:  ULTI - GOAL M ULTI  STAGE M ULTI - AGENT R EINFORCEMENT 
Revised: CM3: C OOPERATIVE M ULTI - GOAL M ULTI - STAGE M ULTI - AGENT R EINFORCEMENT L EARNING

Original: L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success.
Revised: A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success.

Original: This multi-goal multiagent setting poses difﬁculties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efﬁcient exploration for learning both individual goal attainment and cooperation for others’ success, and credit-assignment for interactions between actions and goals of different agents.
Revised: This multi-goal multiagent setting poses difﬁculties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efﬁcient exploration for learning both individual goal attainment and cooperation for others’ success, and credit-assignment for interactions between actions and goals of different agents.

Original: To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment.
Revised: To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment.

Original: We use a function augmentation scheme to bridge value and policy functions across the curriculum.
Revised: We use a function augmentation scheme to bridge value and policy functions across the curriculum.

Original: The complete architecture, called CM3, learns signiﬁcantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difﬁcult formations, negotiating multi-vehicle lane changes in the
Revised: The complete architecture, called CM3, learns signiﬁcantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difﬁcult formations, negotiating multi-vehicle lane changes in the

Original: SUMO trafﬁc simulator, and strategic cooperation in a Checkers environment.
Revised: SUMO trafﬁc simulator, and strategic cooperation in a Checkers environment.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many real-world scenarios that require cooperation among multiple autonomous agents are multi-goal multi-agent control problems: each agent needs to achieve its own individual goal, but the global optimum where all agents succeed is only attained when agents cooperate to allow the success of other agents.
Revised: Many real-world scenarios that require cooperation among multiple autonomous agents are multi-goal multi-agent control problems: each agent needs to achieve its own individual goal, but the global optimum where all agents succeed is only attained when agents cooperate to allow the success of other agents.

Original: In autonomous driving, multiple vehicles must execute cooperative maneuvers when their individual goal locations and nominal trajectories are in conﬂict (e.g., double lane merges)
Revised: In autonomous driving, multiple vehicles must execute cooperative maneuvers when their individual goal locations and nominal trajectories are in conﬂict (e.g., double lane merges)

Original: (Cao et al. , 2013).
Revised: (Cao et al. , 2013).

Original: In social dilemmas, mutual cooperation has higher global payoff but agents’ individual goals may lead to defection out of fear or greed (Van Lange et al. , 2013).
Revised: In social dilemmas, mutual cooperation has higher global payoff but agents’ individual goals may lead to defection out of fear or greed (Van Lange et al. , 2013).

Original: Even settings with a global objective that seem unfactorizable can be formulated as multi-goal problems: in Starcraft
Revised: Even settings with a global objective that seem unfactorizable can be formulated as multi-goal problems: in Starcraft

Original: II micromanagement, a unit that gathers resources must not accidentally jeopardize a teammate’s attempt to scout the opponent base (Blizzard Entertainment, 2019); in trafﬁc ﬂow optimization, different intersection controllers may have local throughput goals but must cooperate for high global performance (Zhang et al. , 2019).
Revised: II micromanagement, a unit that gathers resources must not accidentally jeopardize a teammate’s attempt to scout the opponent base (Blizzard Entertainment, 2019); in trafﬁc ﬂow optimization, different intersection controllers may have local throughput goals but must cooperate for high global performance (Zhang et al. , 2019).

Original: While the framework of multi-agent reinforcement learning (MARL) (Littman, 1994; Stone and Veloso, 2000; Shoham et al. , 2003) has been equipped with methods in deep reinforcement learning (RL) (Mnih et al. , 2015; Lillicrap et al. , 2016) and shown promise on high-dimensional problems with complex agent interactions (Lowe et al. , 2017; Mordatch and Abbeel, 2018; Foerster et al. , 2018; Lin et al. , 2018; Srinivasan et al. , 2018), learning multi-agent cooperation in the multi-goal scenario involves signiﬁcant open challenges.
Revised: While the framework of multi-agent reinforcement learning (MARL) (Littman, 1994; Stone and Veloso, 2000; Shoham et al. , 2003) has been equipped with methods in deep reinforcement learning (RL) (Mnih et al. , 2015; Lillicrap et al. , 2016) and shown promise on high-dimensional problems with complex agent interactions (Lowe et al. , 2017; Mordatch and Abbeel, 2018; Foerster et al. , 2018; Lin et al. , 2018; Srinivasan et al. , 2018), learning multi-agent cooperation in the multi-goal scenario involves signiﬁcant open challenges.

Original: First, given that exploration is crucial for RL (Thrun, 1992) and even more so in MARL with larger state and joint action spaces, how should agents explore to learn both individual goal attainment and cooperation for others’ success?
Revised: First, given that exploration is crucial for RL (Thrun, 1992) and even more so in MARL with larger state and joint action spaces, how should agents explore to learn both individual goal attainment and cooperation for others’ success?

Original: Uniform random exploration is common in deep MARL (Hernandez-Leal et al. , 2018) but can be highly inefﬁcient as the value of cooperative actions may be discoverable only in small regions of state space where cooperation is needed.
Revised: Uniform random exploration is common in deep MARL (Hernandez-Leal et al. , 2018) but can be highly inefﬁcient as the value of cooperative actions may be discoverable only in small regions of state space where cooperation is needed.

Original: Furthermore, the conceptual difference between attaining one’s own goal and cooperating for others’ success calls for more modularized and targeted approaches.
Revised: Furthermore, the conceptual difference between attaining one’s own goal and cooperating for others’ success calls for more modularized and targeted approaches.

Original: Second, while there are methods for multi-agent credit assignment when all agents share a single goal (i.e., a global reward)
Revised: Second, while there are methods for multi-agent credit assignment when all agents share a single goal (i.e., a global reward)

Original: (Chang et al. , 2004; Foerster et al. , 2018; Nguyen et al. , 2018), and while one could treat the cooperative multi-goal scenario as a problem with a single joint goal, this coarse approach makes it extremely difﬁcult to evaluate the impact of an agent’s action on another agent’s success.
Revised: (Chang et al. , 2004; Foerster et al. , 2018; Nguyen et al. , 2018), and while one could treat the cooperative multi-goal scenario as a problem with a single joint goal, this coarse approach makes it extremely difﬁcult to evaluate the impact of an agent’s action on another agent’s success.

Original: Instead, the multi-goal scenario can beneﬁt from ﬁne-grained credit assignment that leverages available structure in action-goal interactions, such as local interactions where only few agents affect another agent’s goal attainment at any time.
Revised: Instead, the multi-goal scenario can beneﬁt from ﬁne-grained credit assignment that leverages available structure in action-goal interactions, such as local interactions where only few agents affect another agent’s goal attainment at any time.

Original: Given these open challenges, our paper focuses on the cooperative multi-goal multi-agent setting where each agent is assigned a goal 1 and must learn to cooperate with other agents with possibly different goals.
Revised: Given these open challenges, our paper focuses on the cooperative multi-goal multi-agent setting where each agent is assigned a goal 1 and must learn to cooperate with other agents with possibly different goals.

Original: To tackle the problems of efﬁcient exploration and credit assignment in this complex problem setting, we develop CM3, a novel general framework involving three synergistic components:We
Revised: To tackle the problems of efﬁcient exploration and credit assignment in this complex problem setting, we develop CM3, a novel general framework involving three synergistic components:

Original:  approach the difﬁculty of multi-agent exploration from a novel curriculum learning perspective, by ﬁrst training an actor-critic pair to achieve different goals in an induced single-agent setting (Stage 1), then using them to initialize all agents in the multi-agent environment (Stage 2).
Revised: We approach the difﬁculty of multi-agent exploration from a novel curriculum learning perspective, by ﬁrst training an actor-critic pair to achieve different goals in an induced single-agent setting (Stage 1), then using them to initialize all agents in the multi-agent environment (Stage 2).

Original: The key insight is that agents who can already act toward individual objectives are better prepared for discovery of cooperative solutions with additional exploration once other agents are introduced.
Revised: The key insight is that agents who can already act toward individual objectives are better prepared for discovery of cooperative solutions with additional exploration once other agents are introduced.

Original: In contrast to hierarchical learning where sub-goals are selected sequentially in time Sutton et al. ,  all agents act toward their goals simultaneously in Stage 2 of our curriculum.
Revised: In contrast to hierarchical learning where sub-goals are selected sequentially in time (Sutton et al. , 1999), all agents act toward their goals simultaneously in Stage 2 of our curriculum.

Original: Observing that a wide array of complex MARL problems permit a decomposition of agents’ observations and state vectors into components of self, others, and non-agent speciﬁc environment information (Hernandez-Leal et al. , 2018), we employ function augmentation to bridge Stages 1-2: we reduce the number of trainable parameters of the actor-critic in Stage 1 by limiting their input space to the part that is sufﬁcient for single-agent training, then augment the architecture in Stage 2 with additional inputs and trainable parameters for learning in the multi-agent environment.
Revised: Observing that a wide array of complex MARL problems permit a decomposition of agents’ observations and state vectors into components of self, others, and non-agent speciﬁc environment information (Hernandez-Leal et al. , 2018), we employ function augmentation to bridge Stages 1-2: we reduce the number of trainable parameters of the actor-critic in Stage 1 by limiting their input space to the part that is sufﬁcient for single-agent training, then augment the architecture in Stage 2 with additional inputs and trainable parameters for learning in the multi-agent environment.

Original: We propose a credit function , which is an action-value function that speciﬁcally evaluates actiongoal pairs, for localized credit assignment in multi-goal MARL.
Revised: We propose a credit function , which is an action-value function that speciﬁcally evaluates actiongoal pairs, for localized credit assignment in multi-goal MARL.

Original: We use it to derive a multi-goal multi-agent policy gradient for Stage 2.
Revised: We use it to derive a multi-goal multi-agent policy gradient for Stage 2.

Original: In synergy with the curriculum, the credit function is constructed via function augmentation from the critic in Stage 1.
Revised: In synergy with the curriculum, the credit function is constructed via function augmentation from the critic in Stage 1.

Original: We evaluate our method on challenging multi-goal multi-agent environments with high-dimensional state spaces: cooperative navigation with difﬁcult formations, double lane merges in the SUMO simulator (Lopez et al. , 2018), and strategic teamwork in a Checkers game.
Revised: We evaluate our method on challenging multi-goal multi-agent environments with high-dimensional state spaces: cooperative navigation with difﬁcult formations, double lane merges in the SUMO simulator (Lopez et al. , 2018), and strategic teamwork in a Checkers game.

Original: CM3 solved all domains signiﬁcantly faster than IAC and COMA (Tan, 1993; Foerster et al. , 2018), and solved four out of ﬁve environments signiﬁcantly faster than QMIX (Rashid et al. , 2018).
Revised: CM3 solved all domains signiﬁcantly faster than IAC and COMA (Tan, 1993; Foerster et al. , 2018), and solved four out of ﬁve environments signiﬁcantly faster than QMIX (Rashid et al. , 2018).

Original: Exhaustive ablation experiments show that the combination of all three components is crucial for CM3’s overall high performance.
Revised: Exhaustive ablation experiments show that the combination of all three components is crucial for CM3’s overall high performance.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: While early theoretical work analyzed Markov games in discrete state and action spaces (Tan, 1993;
Revised: While early theoretical work analyzed Markov games in discrete state and action spaces (Tan, 1993;

Original: Littman, 1994; Hu and Wellman, 2003), recent literature have leveraged techniques from deep RL to develop general algorithms for high dimensional environments with complex agent interactions (Tampuu et al. , 2017; Mordatch and Abbeel, 2018; Lowe et al. , 2017), which pose difﬁculty for traditional methods that do not generalize by learning interactions (Bhattacharya et al. , 2010).
Revised: Littman, 1994; Hu and Wellman, 2003), recent literature have leveraged techniques from deep RL to develop general algorithms for high dimensional environments with complex agent interactions (Tampuu et al. , 2017; Mordatch and Abbeel, 2018; Lowe et al. , 2017), which pose difﬁculty for traditional methods that do not generalize by learning interactions (Bhattacharya et al. , 2010).

Original: Cooperative multi-agent learning is important since many real-world problems can be formulated as distributed systems in which decentralized agents must coordinate to achieve shared objectives (Panait and Luke, 2005).
Revised: Cooperative multi-agent learning is important since many real-world problems can be formulated as distributed systems in which decentralized agents must coordinate to achieve shared objectives (Panait and Luke, 2005).

Original: The multi-agent credit assignment problem arises when agents share a global reward (Chang et al. , 2004).
Revised: The multi-agent credit assignment problem arises when agents share a global reward (Chang et al. , 2004).

Original: While credit assignment be resolved when independent individual rewards are available (Singh et al. , 2019), this may not be suitable for the fully cooperative setting:
Revised: While credit assignment be resolved when independent individual rewards are available (Singh et al. , 2019), this may not be suitable for the fully cooperative setting:

Original: Austerweil et al.
Revised: Austerweil et al.

Original: (2016) showed that agents whose rewards depend on the success of other agents can cooperate better than agents who optimize for their own success.
Revised: (2016) showed that agents whose rewards depend on the success of other agents can cooperate better than agents who optimize for their own success.

Original: In the special case when all agents have a single goal and share a global reward, COMA (Foerster et al. , 2018) uses a counterfactual baseline, while Nguyen et al.
Revised: In the special case when all agents have a single goal and share a global reward, COMA (Foerster et al. , 2018) uses a counterfactual baseline, while Nguyen et al.

Original: employs count-based variance reduction limited to discrete-state environments.
Revised: employs count-based variance reduction limited to discrete-state environments.

Original: However, their centralized critic does not evaluate the speciﬁc impact of an agent’s action on another’s success in the general multi-goal setting.
Revised: However, their centralized critic does not evaluate the speciﬁc impact of an agent’s action on another’s success in the general multi-goal setting.

Original: When a global objective is the sum of agents’ individual objectives, value-decomposition methods optimize a centralized Q-function while preserving scalable decentralized execution (Sunehag et al. , 2018; Rashid et al. , 2018; Son et al. , 2019), but do not address credit assignment.
Revised: When a global objective is the sum of agents’ individual objectives, value-decomposition methods optimize a centralized Q-function while preserving scalable decentralized execution (Sunehag et al. , 2018; Rashid et al. , 2018; Son et al. , 2019), but do not address credit assignment.

Original: While MADDPG (Lowe et al. , 2017) and M3DDPG (Li et al. , 2019) apply to agents with different rewards, they do not address multi-goal cooperation as they do not distinguish between cooperation and competition, despite the fundamental difference.
Revised: While MADDPG (Lowe et al. , 2017) and M3DDPG (Li et al. , 2019) apply to agents with different rewards, they do not address multi-goal cooperation as they do not distinguish between cooperation and competition, despite the fundamental difference.

Original: Multi-goal MARL was considered in Zhang et al.
Revised: Multi-goal MARL was considered in Zhang et al.

Original: (2018), who analyzed convergence in a special networked setting restricted to fully-decentralized training, while we conduct centralized training with decentralized execution (Oliehoek et al. , 2008).
Revised: (2018), who analyzed convergence in a special networked setting restricted to fully-decentralized training, while we conduct centralized training with decentralized execution (Oliehoek et al. , 2008).

Original: In contrast to multi- task MARL, which aims for generalization among non-simultaneous tasks (Omidshaﬁei et al. , 2017), and in contrast to hierarchical methods that sequentially select subtasks (Vezhnevets et al. , 2017; Shu and Tian, 2019), our decentralized agents must cooperate concurrently to attain all goals.
Revised: In contrast to multi- task MARL, which aims for generalization among non-simultaneous tasks (Omidshaﬁei et al. , 2017), and in contrast to hierarchical methods that sequentially select subtasks (Vezhnevets et al. , 2017; Shu and Tian, 2019), our decentralized agents must cooperate concurrently to attain all goals.

Original: Methods for optimizing high-level agent-task assignment policies in a hierarchical framework (Carion et al. , 2019) are complementary to our work, as we focus on learning low-level cooperation after goals are assigned.
Revised: Methods for optimizing high-level agent-task assignment policies in a hierarchical framework (Carion et al. , 2019) are complementary to our work, as we focus on learning low-level cooperation after goals are assigned.

Original: Prior application of curriculum learning (Bengio et al. , 2009) to MARL include a single cooperative task deﬁned by the number of agents (Gupta et al. , 2017) and the probability of agent appearance (Sukhbaatar et al. , 2016), without explicit individual goals.
Revised: Prior application of curriculum learning (Bengio et al. , 2009) to MARL include a single cooperative task deﬁned by the number of agents (Gupta et al. , 2017) and the probability of agent appearance (Sukhbaatar et al. , 2016), without explicit individual goals.

Original: Rusu et al.
Revised: Rusu et al.

Original: (2016) instantiate new neural network columns for task transfer in single-agent RL.
Revised: (2016) instantiate new neural network columns for task transfer in single-agent RL.

Original: Techniques in transfer learning (Pan and Yang, 2010) are complementary to our novel curriculum approach to MARL.
Revised: Techniques in transfer learning (Pan and Yang, 2010) are complementary to our novel curriculum approach to MARL.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In multi-goal MARL, each agent should achieve a goal drawn from a ﬁnite set, cooperate with other agents for collective success, and act independently with limited local observations.
Revised: In multi-goal MARL, each agent should achieve a goal drawn from a ﬁnite set, cooperate with other agents for collective success, and act independently with limited local observations.

Original: We formalize the problem as an episodic multi-goal Markov game, review an actor-critic approach to centralized training of decentralized policies, and summarize counterfactual-based multi-agent credit assignment.
Revised: We formalize the problem as an episodic multi-goal Markov game, review an actor-critic approach to centralized training of decentralized policies, and summarize counterfactual-based multi-agent credit assignment

Original: Multi-goal Markov games.
Revised: Multi-goal Markov games.

Original: A multi-goal Markov game is a tuple (cid:104)S , {O n } , {A n } , P, R, G , N, γ (cid:105) with N agents labeled by n ∈ [ N ] .
Revised: A multi-goal Markov game is a tuple (cid:104)S , {O n } , {A n } , P, R, G , N, γ (cid:105) with N agents labeled by n ∈ [ N ] .

Original: In each episode, each agent n has one ﬁxed goal g n ∈ G that is known only to itself.
Revised: In each episode, each agent n has one ﬁxed goal g n ∈ G that is known only to itself.

Original: At time t and global state s t ∈ S , each agent n receives an observation o nt :
Revised: At time t and global state s t ∈ S , each agent n receives an observation o nt :

Original: =
Revised: =

Original: o n ( s t ) ∈ O n and chooses an action a nt ∈ A n .
Revised: o n ( s t ) ∈ O n and chooses an action a nt ∈ A n .

Original: The environment moves to s t +1 due to joint action a t := { a 1 t , . .
Revised: The environment moves to s t +1 due to joint action a t := { a 1 t , . .

Original: . , a Nt } , according to transition probability P ( s t +1 | s t , a t ) .
Revised: . , a Nt } , according to transition probability P ( s t +1 | s t , a t ) .

Original: Each agent receives a reward R nt :
Revised: Each agent receives a reward R nt :

Original: = R ( s t , a t , g n ) , and the learning task is to ﬁnd stochastic decentralized policies π n : O n × G × A n → [0 , 1] , conditioned only on local observations and goals, to maximize
Revised: = R ( s t , a t , g n ) , and the learning task is to ﬁnd stochastic decentralized policies π n : O n × G × A n → [0 , 1] , conditioned only on local observations and goals, to maximize

Original: J ( π ) :
Revised: J ( π ) :

Original: = E π (cid:104)(cid:80) ∞ t =0 γ t (cid:80) Nn =1 R ( s t , a t , g n ) (cid:105) , where γ ∈ (0 , 1) and joint policy π factorizes as π ( a | s, g ) :
Revised: = E π (cid:104)(cid:80) ∞ t =0 γ t (cid:80) Nn =1 R ( s t , a t , g n ) (cid:105) , where γ ∈ (0 , 1) and joint policy π factorizes as π ( a | s, g ) :

Original: = (cid:81)
Revised: = (cid:81)

Original: Nn =1 π n ( a n | o n , g n ) due to decentralization.
Revised: Nn =1 π n ( a n | o n , g n ) due to decentralization.

Original: Let a − n and g − n denote all agents’ actions and goals, respectively, except that of agent n .
Revised: Let a − n and g − n denote all agents’ actions and goals, respectively, except that of agent n .

Original: Let boldface a
Revised: Let boldface a

Original: and g denote the joint action and joint goals, respectively.
Revised: and g denote the joint action and joint goals, respectively.

Original: For brevity, let π ( a n ) :
Revised: For brevity, let π ( a n ) :

Original: = π n ( a n | o n , g n ) .
Revised: = π n ( a n | o n , g n ) .

Original: This model covers a diverse set of cooperation problems in the literature (Hernandez-Leal et al. , 2018), without constraining how the attainability of a goal depends on other agents: at a trafﬁc intersection, each vehicle can easily reach its target location if not for the presence of other vehicles; in contrast, agents in a strategic game may not be able to maximize their rewards in the absence of cooperators (Sunehag et al. , 2018).
Revised: This model covers a diverse set of cooperation problems in the literature (Hernandez-Leal et al. , 2018), without constraining how the attainability of a goal depends on other agents: at a trafﬁc intersection, each vehicle can easily reach its target location if not for the presence of other vehicles; in contrast, agents in a strategic game may not be able to maximize their rewards in the absence of cooperators (Sunehag et al. , 2018).

Original: Centralized learning of decentralized policies.
Revised: Centralized learning of decentralized policies.

Original: A centralized critic that receives full state-action information can speed up training of decentralized actors that receive only local information (Lowe et al. , 2017; Foerster et al. , 2018).
Revised: A centralized critic that receives full state-action information can speed up training of decentralized actors that receive only local information (Lowe et al. , 2017; Foerster et al. , 2018).

Original: Directly extending the single-goal case, for each n ∈ [1 ..N ] in a multigoal
Revised: Directly extending the single-goal case, for each n ∈ [1 ..N ] in a multigoal

Original: Markov game, critics are represented by the value function V π n ( s ) :
Revised: Markov game, critics are represented by the value function V π n ( s ) :

Original: =
Revised: =

Original: E π (cid:2)(cid:80) ∞ t =

Revised: E π (cid:2)(cid:80) ∞ t =

Original: γ t R nt (cid:12)(cid:12) s 

Revised: γ t R nt (cid:12)(cid:12) s

Original: = s (cid:3) and the action-value function Q π n ( s, a ) :
Revised: = s (cid:3) and the action-value function Q π n ( s, a ) :

Original: =
Revised: =

Original: E π (cid:2)(cid:80) ∞ t =

Revised: E π (cid:2)(cid:80) ∞ t =

Original: γ t R nt (cid:12)(cid:12) s 

Revised: γ t R nt (cid:12)(cid:12) s

Original: = s, a 0 = a (cid:3) , which evaluate the joint policy π against the reward R n for each goal g n .
Revised: = s, a 0 = a (cid:3) , which evaluate the joint policy π against the reward R n for each goal g n .

Original: Multi-agent credit assignment.
Revised: Multi-agent credit assignment.

Original: In MARL with a single team objective, COMA addresses credit assignment by using a counterfactual baseline in an advantage function A n ( s, a ) :
Revised: In MARL with a single team objective, COMA addresses credit assignment by using a counterfactual baseline in an advantage function A n ( s, a ) :

Original: = Q π ( s, a ) − (cid:80) ˆ a n π n (ˆ a n | o n ) Q π ( s, (ˆ a n , a − n ))
Revised: = Q π ( s, a ) − (cid:80) ˆ a n π n (ˆ a n | o n ) Q π ( s, (ˆ a n , a − n ))

Original: (Foerster et al. , 2018, Lemma 1) , which evaluates the contribution of a chosen action a n versus the average of all possible counterfactuals ˆ a n , keeping a − n ﬁxed.
Revised: (Foerster et al. , 2018, Lemma 1) , which evaluates the contribution of a chosen action a n versus the average of all possible counterfactuals ˆ a n , keeping a − n ﬁxed.

Original: The analysis in Wu et al.
Revised: The analysis in Wu et al.

Original: (2018) for a formally equivalent action-dependent baseline in RL suggests that
Revised: (2018) for a formally equivalent action-dependent baseline in RL suggests that

Original: COMA is a low-variance estimator for single-goal MARL.
Revised: COMA is a low-variance estimator for single-goal MARL.

Original: We derive its variance in Appendix C.1.
Revised: We derive its variance in Appendix C.1.

Original: However, COMA is unsuitable for credit assignment in multi-goal MARL, as it would treat the collection of goals g as a global goal and only learn from total reward, making it extremely difﬁcult to disentangle each agent’s impact on other agents’ goal attainment.
Revised: However, COMA is unsuitable for credit assignment in multi-goal MARL, as it would treat the collection of goals g as a global goal and only learn from total reward, making it extremely difﬁcult to disentangle each agent’s impact on other agents’ goal attainment.

Original: Furthermore, a global Q-function does not explicitly capture structure in agents’ interactions, such as local interactions involving a limited number of agents.
Revised: Furthermore, a global Q-function does not explicitly capture structure in agents’ interactions, such as local interactions involving a limited number of agents.

Original: We substantiate these arguments by experimental results in Section 6.
Revised: We substantiate these arguments by experimental results in Section 6.

Original:  M ETHODS
Revised:  M ETHODS

Original: We describe the complete CM3 learning framework as follows.
Revised: We describe the complete CM3 learning framework as follows.

Original: First we deﬁne a credit function as a mechanism for credit assignment in multi-goal MARL, then derive a new cooperative multi-goal policy gradient with localized credit assignment.
Revised: First we deﬁne a credit function as a mechanism for credit assignment in multi-goal MARL, then derive a new cooperative multi-goal policy gradient with localized credit assignment.

Original: Next we motivate the possibility of signiﬁcant training speedup via a curriculum for multi-goal MARL.
Revised: Next we motivate the possibility of signiﬁcant training speedup via a curriculum for multi-goal MARL.

Original: We describe function augmentation as a mechanism for efﬁciently bridging policy and value functions across the curriculum stages, and ﬁnally synthesize all three components into a synergistic learning framework.
Revised: We describe function augmentation as a mechanism for efﬁciently bridging policy and value functions across the curriculum stages, and ﬁnally synthesize all three components into a synergistic learning framework.

Original: 4. C REDIT ASSIGNMENT IN MULTI - GOAL MARL
Revised: 4. C REDIT ASSIGNMENT IN MULTI - GOAL MARL

Original: If all agents take greedy goal-directed actions that are individually optimal in the absence of other agents, the joint action can be sub-optimal (e.g. straight-line trajectory towards target in trafﬁc).
Revised: If all agents take greedy goal-directed actions that are individually optimal in the absence of other agents, the joint action can be sub-optimal (e.g. straight-line trajectory towards target in trafﬁc).

Original: Instead rewarding agents for both individual and collective success can avoid such bad local optima.
Revised: Instead rewarding agents for both individual and collective success can avoid such bad local optima.

Original: A naïve approach based on previous works (Foerster et al. , 2018; Lowe et al. , 2017) would evaluate the joint action a via a global Q-function Q π n ( s, a ) for each agent’s goal g n , but this does not precisely capture each agent’s contribution to another agent’s attainment of its goal.
Revised: A naïve approach based on previous works (Foerster et al. , 2018; Lowe et al. , 2017) would evaluate the joint action a via a global Q-function Q π n ( s, a ) for each agent’s goal g n , but this does not precisely capture each agent’s contribution to another agent’s attainment of its goal.

Original: Instead, we propose an explicit mechanism for credit assignment by learning an additional function Q π n ( s, a m ) that evaluates pairs of action a m and goal g n , for use in a multi-goal actor-critic algorithm.
Revised: Instead, we propose an explicit mechanism for credit assignment by learning an additional function Q π n ( s, a m ) that evaluates pairs of action a m and goal g n , for use in a multi-goal actor-critic algorithm.

Original: We deﬁne this function and show that it satisﬁes the classical relation needed for sample-based model-free learning.
Revised: We deﬁne this function and show that it satisﬁes the classical relation needed for sample-based model-free learning.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: For n, m ∈ [ N ] , s ∈ S , the credit function for goal g n and a m ∈
Revised: For n, m ∈ [ N ] , s ∈ S , the credit function for goal g n and a m ∈

Original: A m by agent m is:
Revised: A m by agent m is:

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: For all m, n ∈ [ N ] , the credit function (1) satisﬁes the following relations:
Revised: For all m, n ∈ [ N ] , the credit function (1) satisﬁes the following relations:

Original: [Equation]
Revised: [Equation]

Original: Derivations are given in Appendix B.1, including the relation between Q π n ( s, a m ) and Q π n ( s, a ) .
Revised: Derivations are given in Appendix B.1, including the relation between Q π n ( s, a m ) and Q π n ( s, a ) .

Original: Equation (2) takes the form of the Bellman expectation equation, which justiﬁes learning the credit function, parameterized by θ Q c , by optimizing the standard loss function in deep RL:
Revised: Equation (2) takes the form of the Bellman expectation equation, which justiﬁes learning the credit function, parameterized by θ Q c , by optimizing the standard loss function in deep RL:

Original: [Equation]
Revised: [Equation]

Original: While centralized training means the input space scales linearly with agent count, many practical environments involving only local interactions between agents allows centralized training with few agents while retaining decentralized performance when deployed at scale (evidenced in Appendix E).
Revised: While centralized training means the input space scales linearly with agent count, many practical environments involving only local interactions between agents allows centralized training with few agents while retaining decentralized performance when deployed at scale (evidenced in Appendix E).

Original: 4. C OOPERATIVE MULTI - GOAL MULTI - AGENT POLICY GRADIENT
Revised: 4. C OOPERATIVE MULTI - GOAL MULTI - AGENT POLICY GRADIENT

Original: We use the credit function as a critic within a policy gradient for multi-goal MARL.
Revised: We use the credit function as a critic within a policy gradient for multi-goal MARL.

Original: Letting θ parameterize π , the overall objective J ( π ) is maximized by ascending the following gradient:
Revised: Letting θ parameterize π , the overall objective J ( π ) is maximized by ascending the following gradient:

Original: Proposition 2.
Revised: Proposition 2.

Original: The cooperative multi-goal credit function based MARL policy gradient is
Revised: The cooperative multi-goal credit function based MARL policy gradient is

Original: [Equation]
Revised: [Equation]

Original: This is derived in Appendix B.2.
Revised: This is derived in Appendix B.2.

Original: For a ﬁxed agent m , the inner summation over n considers all agents’ goals g n and updates m ’s policy based on the advantage of a m over all counterfactual actions ˆ a m , as measured by the credit function for g n .
Revised: For a ﬁxed agent m , the inner summation over n considers all agents’ goals g n and updates m ’s policy based on the advantage of a m over all counterfactual actions ˆ a m , as measured by the credit function for g n .

Original: The strength of interaction between action-goal pairs is captured by the extent to which Q π n ( s, ˆ a m ) varies with ˆ a m , which directly impacts the magnitude of the gradient on agent m ’s policy.
Revised: The strength of interaction between action-goal pairs is captured by the extent to which Q π n ( s, ˆ a m ) varies with ˆ a m , which directly impacts the magnitude of the gradient on agent m ’s policy.

Original: For example, strong interaction results in non-constant Q π n ( s, · ) , which implies larger magnitude of A π n,m and larger weight on ∇ θ log π ( a m ) .
Revised: For example, strong interaction results in non-constant Q π n ( s, · ) , which implies larger magnitude of A π n,m and larger weight on ∇ θ log π ( a m ) .

Original: The double summation accounts for ﬁrst-order interaction between all action-goal pairs, but complexity can be reduced by omitting terms when interactions are known to be sparse, and our empirical runtimes are on par with other methods due to efﬁcient batch computation (Appendix F).
Revised: The double summation accounts for ﬁrst-order interaction between all action-goal pairs, but complexity can be reduced by omitting terms when interactions are known to be sparse, and our empirical runtimes are on par with other methods due to efﬁcient batch computation (Appendix F).

Original: As the second term in A π n,m is a baseline, the reduction of variance can be analyzed similarly to that for COMA, given in Appendix C.2.
Revised: As the second term in A π n,m is a baseline, the reduction of variance can be analyzed similarly to that for COMA, given in Appendix C.2.

Original: While A π n,m = Q π n ( s, a ) − V π n ( s ) (due to (3)), ablation results show stability improvement due to the credit function (Section 6).
Revised: While A π n,m = Q π n ( s, a ) − V π n ( s ) (due to (3)), ablation results show stability improvement due to the credit function (Section 6).

Original: As the credit function takes in a single agent’s action, it synergizes with both CM3’s curriculum and function augmentation as described in Section 4.5.
Revised: As the credit function takes in a single agent’s action, it synergizes with both CM3’s curriculum and function augmentation as described in Section 4.5.

Original: 4. C URRICULUM FOR MULTI - GOAL MARL
Revised: 4. C URRICULUM FOR MULTI - GOAL MARL

Original: Multi-goal MARL poses a signiﬁcant challenge for exploration.
Revised: Multi-goal MARL poses a signiﬁcant challenge for exploration.

Original: Random exploration can be highly inefﬁcient for concurrently learning both individual task completion and cooperative behavior.
Revised: Random exploration can be highly inefﬁcient for concurrently learning both individual task completion and cooperative behavior.

Original: Agents who cannot make progress toward individual goals may rarely encounter the region of state space where cooperation is needed, rendering any exploration useless for learning cooperative behavior.
Revised: Agents who cannot make progress toward individual goals may rarely encounter the region of state space where cooperation is needed, rendering any exploration useless for learning cooperative behavior.

Original: On the other extreme, exploratory actions taken in situations that require precise coordination can easily lead to penalties that cause agents to avoid the coordination problem and fail to achieve individual goals.
Revised: On the other extreme, exploratory actions taken in situations that require precise coordination can easily lead to penalties that cause agents to avoid the coordination problem and fail to achieve individual goals.

Original: Instead, we hypothesize and conﬁrm in experiments that agents who can achieve individual goals in the absence of other agents can more reliably produce state conﬁgurations where cooperative solutions are easily discovered with additional exploration in the multi-agent environment 2 .
Revised: Instead, we hypothesize and conﬁrm in experiments that agents who can achieve individual goals in the absence of other agents can more reliably produce state conﬁgurations where cooperative solutions are easily discovered with additional exploration in the multi-agent environment 2 .

Original: We propose a MARL curriculum that ﬁrst solves a single-agent Markov decision process (MDP), as preparation for subsequent exploration speedup.
Revised: We propose a MARL curriculum that ﬁrst solves a single-agent Markov decision process (MDP), as preparation for subsequent exploration speedup.

Original: Given a cooperative multi-goal Markov game
Revised: Given a cooperative multi-goal Markov game

Original: MG , we induce an MDP M to be the tuple (cid:104)S n , O n , A n , P n , R, γ (cid:105) , where an agent n is selected to be the single agent in M .
Revised: MG , we induce an MDP M to be the tuple (cid:104)S n , O n , A n , P n , R, γ (cid:105) , where an agent n is selected to be the single agent in M .

Original: Entities S n , P n , and R are deﬁned by removing all dependencies on agent interactions, so that only components depending on agent n remain.
Revised: Entities S n , P n , and R are deﬁned by removing all dependencies on agent interactions, so that only components depending on agent n remain.

Original: This reduction to M is possible in almost all fully cooperative multi-agent environments used in a large body of work 3 (Hernandez-Leal et al. , 2018), precisely because they support a variable number of agents, including N = 1 .
Revised: This reduction to M is possible in almost all fully cooperative multi-agent environments used in a large body of work 3 (Hernandez-Leal et al. , 2018), precisely because they support a variable number of agents, including N = 1 .

Original: Important real-world settings that allow this reduction include autonomous driving, multi trafﬁc light control, and warehouse commissioning (removing all but one car/controller/robot, respectively, from the environment).
Revised: Important real-world settings that allow this reduction include autonomous driving, multi trafﬁc light control, and warehouse commissioning (removing all but one car/controller/robot, respectively, from the environment).

Original: Given a full Markov game implementation, the reduction involves only deletion of components associated with all other agents from state vectors (since an agent is uniquely deﬁned by its attributes), deletion of if-else conditions from the reward function corresponding to agent interactions, and likewise from the transition function if a simulation is used.
Revised: Given a full Markov game implementation, the reduction involves only deletion of components associated with all other agents from state vectors (since an agent is uniquely deﬁned by its attributes), deletion of if-else conditions from the reward function corresponding to agent interactions, and likewise from the transition function if a simulation is used.

Original: Appendix G provides practical guidelines for the reduction.
Revised: Appendix G provides practical guidelines for the reduction.

Original: Based on M , we deﬁne a greedy policy for MG .
Revised: Based on M , we deﬁne a greedy policy for MG .

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: A greedy policy π n by agent n for cooperative multi-goal MG is deﬁned as the optimal policy π ∗ for the induced MDP M where only agent n is present.
Revised: A greedy policy π n by agent n for cooperative multi-goal MG is deﬁned as the optimal policy π ∗ for the induced MDP M where only agent n is present.

Original: This naturally leads to our proposed curriculum: Stage 1 trains a single agent in M to achieve a greedy policy, which is then used for initialization in MG in Stage 2.
Revised: This naturally leads to our proposed curriculum: Stage 1 trains a single agent in M to achieve a greedy policy, which is then used for initialization in MG in Stage 2.

Original: Next we explain in detail how to leverage the structure of decentralized MARL to bridge the two curriculum stages.
Revised: Next we explain in detail how to leverage the structure of decentralized MARL to bridge the two curriculum stages.

Original: 4. F UNCTION AUGMENTATION FOR MULTI - GOAL CURRICULUM
Revised: 4. F UNCTION AUGMENTATION FOR MULTI - GOAL CURRICULUM

Original: In Markov games with decentralized execution, an agent’s observation space decomposes into
Revised: In Markov games with decentralized execution, an agent’s observation space decomposes into

Original: O n = O n self ∪ O n others , where o n self ∈ O n self captures the agent’s own properties, which must be observable by the agent for closed-loop control, while o n others ∈ O n others is the agent’s egocentric observation of other agents, which in our work does not include their private information (Pynadath and Tambe, 2002).
Revised: O n = O n self ∪O n others , where o n self ∈ O n self captures the agent’s own properties, which must be observable by the agent for closed-loop control, while o n others ∈ O n others is the agent’s egocentric observation of other agents. In our work, egocentric observations are private and not accessible by other agents (Pynadath and Tambe, 2002).

Original: Similarly, global state s decomposes into s :=
Revised: Similarly, global state s decomposes into s :=

Original: ( s env , s n , s − n ) , wheres env is environment information not speciﬁc to any agent (e.g., position of a landmark), and s n captures agent n ’s information. While this decomposition is implicitly available in a wide range of complex multi-agent environments (Bansal et al. , 2018; Foerster et al. , 2018; Lowe et al. , 2017; Rashid et al. , 2018; Liu et al. , 2019; Jaderberg et al. , 2019), we explicitly use it to implement our curriculum. In Stage 1, as the ability to process o n others and s − n is unnecessary, we reduce the input space of
Revised: ( s env , s n , s − n ) , where

Original: Environment
Revised: Environment

Original: Environment h i
Revised: Environment h i

Original: *h
Revised: *h

Original: i * − 1⋮ ⋮
Revised: i * − 1⋮ ⋮

Original: W i *h K⋮ ( o selfn , g n )
Revised: W i *h K⋮ ( o selfn , g n )

Original: o othersn input input
Revised: o othersn input input

Original: Stage
Revised: Stage

Original: Stage∇ J
Revised: Stage∇ J

Original: Qπ( o self ,g )
Revised: Qπ( o self ,g )

Original: R ( s,a,g ) a π 2 W 1: ( o N ,g N ) ( o 1, g 1 ) a( s, g , { R n } n )
Revised: R ( s,a,g ) a π 2 W 1: ( o N ,g N ) ( o 1, g 1 ) a( s, g , { R n } n )

Original: Q 1 ( s, a ) ππ N
Revised: Q 1 ( s, a ) ππ N

Original: Q 1 ( s,a m )
Revised: Q 1 ( s,a m )

Original: a N ∇ J ⋯ ⋯ ⋯ aa N
Revised: a N ∇ J ⋯ ⋯ ⋯ aa N

Original: Q N ( s,a m )
Revised: Q N ( s,a m )

Original: Q N ( s, a )
Revised: Q N ( s, a )

Original: Function augmentation π π  policy and value functions, thereby reducing the number of trainable parameters and lowering the computation cost.
Revised: Function augmentation π π s env is environment information not speciﬁc to any agent (e.g., position of a landmark), and s n captures agent n ’s information. While this decomposition is implicitly available in a wide range of complex multi-agent environments (Bansal et al. , 2018; Foerster et al. , 2018; Lowe et al. , 2017; Rashid et al. , 2018; Liu et al. , 2019; Jaderberg et al. , 2019), we explicitly use it to implement our curriculum. In Stage 1, as the ability to process o n others and s − n is unnecessary, we reduce the input space of policy and value functions, thereby reducing the number of trainable parameters and lowering the computation cost.

Original: In Stage 2, we restore Stage 1 parameters and activate new modules to process additional inputs o n others and s − n .
Revised: In Stage 2, we restore Stage 1 parameters and activate new modules to process additional inputs o n others and s − n .

Original: This augmentation is especially suitable for efﬁciently learning the credit function (1) and global Q-function, since Q ( s, a ) can be augmented into both Q π n ( s, a ) and
Revised: This augmentation is especially suitable for efﬁciently learning the credit function (1) and global Q-function, since Q ( s, a ) can be augmented into both Q π n ( s, a ) and

Original: Q π n ( s, a m ) , as explained below.
Revised: Q π n ( s, a m ) , as explained below.

Original: 4. A COMPLETE INSTANTIATION OF CM 
Revised: 4. A COMPLETE INSTANTIATION OF CM

Original: We combine the preceding components to create CM3, using deep neural networks for function approximation (Figure 1 and Algorithm 1).
Revised: We combine the preceding components to create CM3, using deep neural networks for function approximation (Figure 1 and Algorithm 1).

Original: Without loss of generality, we assume parameter-sharing (Foerster et al. , 2018) among homogeneous agents with goals as input (Schaul et al. , 2015).
Revised: Without loss of generality, we assume parameter-sharing (Foerster et al. , 2018) among homogeneous agents with goals as input (Schaul et al. , 2015).

Original: The inhomogeneous case can be addressed by N actor-critics.
Revised: The inhomogeneous case can be addressed by N actor-critics.

Original: Drawing from multi-task learning (Taylor and Stone, 2009), we sample goal(s) in each episode for the agent(s), to train one model for all goals.
Revised: Drawing from multi-task learning (Taylor and Stone, 2009), we sample goal(s) in each episode for the agent(s), to train one model for all goals.

Original: Stage 1.
Revised: Stage 1.

Original: We train an actor π 1 ( a | o, g ) and critic Q 1 ( s 1 , a, g ) to convergence according to (4) and (5) in the induced MDP (with N = 1 , see Appendix J).
Revised: We train an actor π 1 ( a | o, g ) and critic Q 1 ( s 1 , a, g ) to convergence according to (4) and (5) in the induced MDP with N = 1 and random goal sampling (see Appendix J).

Original: This uses orders of magnitude fewer samples than for the full multi-agent environment—compare Figure 6 with Figure 5 below.
Revised: This uses orders of magnitude fewer samples than for the full multi-agent environment—compare Figure 6 with Figure 5.

Original: Stage 2.
Revised: Stage 2.

Original: The Markov game is instantiated with all N agents.
Revised: The Markov game is instantiated with all N agents.

Original: We retain the trained π 1 parameters, instantiate a second neural network π 2 for agents to process o n others , and connect the output of πto a selected hidden layer of π 1 .
Revised: We restore the trained π 1 parameters, instantiate a second neural network π 2 for agents to process o n others , and connect the output of π 2 to a selected hidden layer of π 1 .

Original: Concretely, let
Revised: Concretely, let

Original: h 1 i ∈
Revised: h 1 i ∈

Original: R m i denote hidden layer i ≤ L with m i units in an L -layer network π 1 , connected to layer i − 1 via h 1 i = f
Revised: R m i denote hidden layer i ≤ L with m i units in an Llayer network π 1 , connected to layer i − 1 via h 1 i = f

Original: ( W 1 i h 

Revised: ( W 1 i h

Original: i − 1 ) with W 1 i ∈
Revised: i − 1 ) with W 1 i ∈

Original: R m i × m i − 1 and nonlinear activation f .
Revised: R m i × m i − 1 and nonlinear activation f .

Original: Stage 2 introduces a K -layer network π 2 ( o n others ) with outputs
Revised: Stage 2 introduces a K -layer network π 2 ( o n others ) with outputs

Original: h 2 K ∈ R m K , chooses aparticular layer 4 i ∗ of π 1 , and augmentsthe hidden activations h 1 i ∗ to be h 1 i ∗ =
Revised: h 2 K ∈ R m K , chooses a layer 4 i ∗ of π 1 , and augments h 1 i ∗ to be h 1 i ∗ =

Original: f ( W 1 i ∗ h 1 i ∗ − 1 + W 1:2 h 2 K ) with W 1:2 ∈ R m i ∗ × m K .
Revised: f ( W 1 i ∗ h 1 i ∗ − 1 + W 1:2 h 2 K ) with W 1:2 ∈ R m i ∗ × m K .

Original: Hence, hidden layers i < i ∗ begin with the ability to process ( o n self , g n ) , while the new module speciﬁcally learns the effect of surrounding agents.
Revised: Being restored from Stage 1, not re-initialized, hidden layers i < i ∗ begin with the ability to process ( o n self , g n ) , while the new weights in π 2 and W 1:2 speciﬁcally learn the effect of surrounding agents.

Original: Higher layers i ≥ i ∗ that already take greedy actions to achieve goals in Stage 1 must now do so while cooperating to allow other agents’ success.
Revised: Higher layers i ≥ i ∗ that already take greedy actions to achieve goals in Stage 1 must now do so while cooperating to allow other agents’ success.

Original: This augmentation scheme is simplest for deep policy and value networks using fully-connected or convolutional layers.
Revised: This augmentation scheme is simplest for deep policy and value networks using fully-connected or convolutional layers.

Original: The middle panel of Figure 1 depicts the construction of π from π 1 and π 2 .
Revised: The middle panel of Figure 1 depicts the construction of π from π 1 and π 2 .

Original: The global Q π ( s, a , g n ) is constructed from Q 1 similarly: when the input to Q 1 is ( s env , s n , a n , g n ) , a new module takes input ( s − n , a − n ) and connects to a chosen hidden layer of Q 1 .
Revised: The global Q π ( s, a , g n ) is constructed from Q 1 similarly: when the input to Q 1 is ( s env , s n , a n , g n ) , a new module takes input ( s − n , a − n ) and connects to a chosen hidden layer of Q 1 .

Original: Credit function Q π ( s, a m , g n ) isaugmented from a copy of Q 1 , such that when Q 1 inputs are ( s env , s n , a m , g n ) , the new module’s inputs are ( s m , s − n ) . We train the policy using (5), train the credit function with loss (4), and train the global Q-function with the joint-action analogue of (4).
Revised: Credit function Q π ( s, a m , g n ) is

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Policy generalization was tested on such conditions
Revised: augmented from a copy of Q 1 , such that when Q 1 inputs are ( s env , s n , a m , g n ) , the new module’s inputs are ( s m , s − n ) . We train the policy using (5), train the credit function with loss (4), and train the global Q-function with the joint-action analogue of (4).

Original:  E XPERIMENTAL SETUP
Revised:  E XPERIMENTAL SETUP

Original: We investigated the performance and robustness of CM3 versus existing methods on diverse and challenging multi-goal MARL environments: cooperative navigation in difﬁcult formations, double lane merge in autonomous driving, and strategic cooperation in a Checkers game.
Revised: We investigated the performance and robustness of CM3 versus existing methods on diverse and challenging multi-goal MARL environments: cooperative navigation in difﬁcult formations, double lane merge in autonomous driving, and strategic cooperation in a Checkers game.

Original: We evaluated ablations of CM3 on all domains.
Revised: We evaluated ablations of CM3 on all domains.

Original: We describe key setup here, with full details in Appendices G to J.
Revised: We describe key setup here, with full details in Appendices G to J.

Original: Cooperative navigation: We created three variants of the cooperative navigation scenario in Lowe et al.
Revised: Cooperative navigation: We created three variants of the cooperative navigation scenario in Lowe et al.

Original: (2017), where N agents cooperate to reach a set of targets.
Revised: (2017), where N agents cooperate to reach a set of targets.

Original: We increased the difﬁculty by giving each agent only an individual reward based on distance to its designated target, not a global team reward, but initial and target positions require complex cooperative maneuvers to avoid collision penalties (Figure 3).
Revised: We increased the difﬁculty by giving each agent only an individual reward based on distance to its designated target, not a global team reward, but initial and target positions require complex cooperative maneuvers to avoid collision penalties (Figure 3).

Original: Agents observe relative positions and velocities (details in Appendix G.1).
Revised: Agents observe relative positions and velocities (details in Appendix G.1).

Original: SUMO:
Revised: SUMO:

Original: Previous work modeled autonomous driving tasks as MDPs in which all other vehicles do not learn to respond to a single learning agent (Isele et al. , 2018; Kueﬂer et al. , 2017).
Revised: Previous work modeled autonomous driving tasks as MDPs in which all other vehicles do not learn to respond to a single learning agent (Isele et al. , 2018; Kueﬂer et al. , 2017).

Original: However, real-world driving requires cooperation among different drivers’ with personal goals.
Revised: However, real-world driving requires cooperation among different drivers’ with personal goals.

Original: Built in the
Revised: Built in the

Original: SUMO trafﬁc simulator with sublane resolution (Lopez et al. , 2018), this experiment requires agent vehicles to learn double-merge maneuvers to reach goal lane assignments (Figure 4).
Revised: SUMO trafﬁc simulator with sublane resolution (Lopez et al. , 2018), this experiment requires agent vehicles to learn double-merge maneuvers to reach goal lane assignments (Figure 4).

Original: Agents have limited ﬁeld of view and receive sparse rewards (Appendix G.2).
Revised: Agents have limited ﬁeld of view and receive sparse rewards (Appendix G.2).

Original: Checkers:
Revised: Checkers:

Original: We implemented a challenging strategic game (Appendix G.3, an extension of Sunehag et al.
Revised: We implemented a challenging strategic game (Appendix G.3, an extension of Sunehag et al.

Original: (2018)), to investigate whether CM3 is beneﬁcial even when an agent cannot maximize its reward in the absence of another agent.
Revised: (2018)), to investigate whether CM3 is beneﬁcial even when an agent cannot maximize its reward in the absence of another agent.

Original: In a gridworld with red and yellow squares that disappear when collected (Figure 2), Agent A receives +1 for red and -0.5 for yellow; Agent B receives -0.5 for red  yellow.
Revised: In a gridworld with red and yellow squares that disappear when collected (Figure 2), Agent A receives +1 for red and -0.5 for yellow; Agent B receives -0.5 for red and +1 for yellow.

Original:  have a  global 
Revised: Both have a limited 5x5 ﬁeld of view. The global optimum requires each agent to clear the path for the other.

Original: Algorithm implementations.
Revised: Algorithm implementations.

Original: We describe key points here, leaving complete architecture details and hyperparameter tables to Appendices H and I. CM3: Stage 1 is deﬁned for each environment as follows (Appendix G): in cooperative navigation, a single particle learns to reach any speciﬁed landmark; in SUMO, a car learns to reach any speciﬁed goal lane; in Checkers, we alternate between training one agent as A and B. Appendix H describes function augmentation in Stage 2 of CM3.
Revised: We describe key points here, leaving complete architecture details and hyperparameter tables to Appendices H and I. CM3: Stage 1 is deﬁned for each environment as follows (Appendix G): in cooperative navigation, a single particle learns to reach any speciﬁed landmark; in SUMO, a car learns to reach any speciﬁed goal lane; in Checkers, we alternate between training one agent as A and B. Appendix H describes function augmentation in Stage 2 of CM3.

Original: COMA (Foerster et al. , 2018): the joint goal g and total reward (cid:80) n R n can be used to train COMA’s global Q function, which receives input ( s, o n , g n , n, a − n , g − n ) .
Revised: COMA (Foerster et al. , 2018): the joint goal g and total reward (cid:80) n R n can be used to train COMA’s global Q function, which receives input ( s, o n , g n , n, a − n , g − n ) .

Original: Each output node i represents
Revised: Each output node i represents

Original: Q ( s, a n = i, a − n , g ) .
Revised: Q ( s, a n = i, a − n , g ) .

Original: IAC (Tan, 1993; Foerster et al. , 2018): IAC trains each agent’s actor and critic independently, using the agent’s own observation.
Revised: IAC (Tan, 1993; Foerster et al. , 2018): IAC trains each agent’s actor and critic independently, using the agent’s own observation.

Original: The TD error of value function V ( o n , g n ) is used in a standard policy gradient (Sutton et al. , 2000).
Revised: The TD error of value function V ( o n , g n ) is used in a standard policy gradient (Sutton et al. , 2000).

Original: QMIX (Rashid et al. , 2018): we used the original hypernetwork, giving all goals to the mixer and individual goals to each agent network.
Revised: QMIX (Rashid et al. , 2018): we used the original hypernetwork, giving all goals to the mixer and individual goals to each agent network.

Original: We used a manual coordinate descent on exploration and learning rate hyperparameters, including values reported in the original works.
Revised: We used a manual coordinate descent on exploration and learning rate hyperparameters, including values reported in the original works.

Original: We ensured the number of trainable parameters are similar among all methods, up to method-speciﬁc architecture requirements for COMA and QMIX.
Revised: We ensured the number of trainable parameters are similar among all methods, up to method-speciﬁc architecture requirements for COMA and QMIX.

Original: Ablations.
Revised: Ablations.

Original: We conducted ablation experiments in all domains.
Revised: We conducted ablation experiments in all domains.

Original: To discover the speedup from the curriculum with function augmentation, we trained the full Stage 2 architecture of CM3 (labeled as
Revised: To discover the speedup from the curriculum with function augmentation, we trained the full Stage 2 architecture of CM3 (labeled as

Original: Direct ) without ﬁrst training components π 1 and Q 1 in an induced MDP. To investigate the beneﬁt of the new credit function and multi-goal policy gradient, we trained an ablation (labeled QV ) with advantage function A π n ( s, a ) : = Q π n ( s, a ) − V π n ( s ) where credit assignment between action-goal pairs is lost
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: ) 1e4 10080 7050 

Revised: ) 1e4 10080 7050

Original: S u m o f i n d i v i d u a l r e w a r d s
Revised: S u m o f i n d i v i d u a l r e w a r d s

Original: CM3QVDirect (h) Merge     5 Episode (1e4) 1e48 1014 

Revised: CM3QVDirect (h) Merge     5 Episode (1e4) 1e48 1014

Original: S u m o f i n d i v i d u a l r e w a r d s
Revised: S u m o f i n d i v i d u a l r e w a r d s

Original: CM3QVDirect (i) SUMO     5 Episode (1e4)
Revised: CM3QVDirect (i) SUMO     5 Episode (1e4)

Original: 1e

Revised: 1e

Original: S u m o f i n d i v i d u a l r e w a r d s
Revised: S u m o f i n d i v i d u a l r e w a r d s

Original: CM3QVDirect (j) Checkers
Revised: CM3QVDirect (j) Checkers

Revised: Direct ) without ﬁrst training components π 1 and Q 1 in an induced MDP.

Revised: To investigate the beneﬁt of the new credit function and multi-goal policy gradient, we trained an ablation (labeled QV ) with advantage function A π n ( s, a ) :

Revised: = Q π n ( s, a ) − V π n ( s ) , where credit assignment between action-goal pairs is lost.

Revised: QV uses the same π 1 , Q 1 , and function augmentation as CM3.

Original:  R ESULTS AND D ISCUSSIONS
Revised:  R ESULTS AND D ISCUSSIONS

Original: CM3 ﬁnds optimal or near-optimal policies signiﬁcantly faster than IAC and COMA on all domains, and performs signiﬁcantly higher than QMIX in four out of ﬁve.
Revised: CM3 ﬁnds optimal or near-optimal policies signiﬁcantly faster than IAC and COMA on all domains, and performs signiﬁcantly higher than QMIX in four out of ﬁve.

Original: We report absolute runtime in
Revised: We report absolute runtime in

Original: Appendix F and account for CM3’s Stage 1 episodes (Appendix J) when comparing sample efﬁciency.
Revised: Appendix F and account for CM3’s Stage 1 episodes (Appendix J) when comparing sample efﬁciency.

Original: Main comparison.
Revised: Main comparison.

Original: Over all cooperative navigation scenarios (Figures 5a to 5c), CM3 (with 1k episodes in Stage 1) converged more than 15k episodes faster than IAC.
Revised: Over all cooperative navigation scenarios (Figures 5a to 5c), CM3 (with 1k episodes in Stage 1) converged more than 15k episodes faster than IAC.

Original: IAC reached the same ﬁnal performance as CM3 because dense individual rewards simpliﬁes the learning problem for
Revised: IAC reached the same ﬁnal performance as CM3 because dense individual rewards simpliﬁes the learning problem for

Original: IAC’s fully decentralized approach, but CM3 beneﬁted signiﬁcantly from curriculum learning, as evidenced by comparison to “Direct” in Figure 5f.
Revised: IAC’s fully decentralized approach, but CM3 beneﬁted signiﬁcantly from curriculum learning, as evidenced by comparison to “Direct” in Figure 5f.

Original: QMIX and COMA settled at suboptimal behavior.
Revised: QMIX and COMA settled at suboptimal behavior.

Original: Both learn global critics that use all goals as input, in contrast to CM3 and IAC that process each goal separately.
Revised: Both learn global critics that use all goals as input, in contrast to CM3 and IAC that process each goal separately.

Original: This indicates the difﬁculty of training agents for individual goals under a purely global approach.
Revised: This indicates the difﬁculty of training agents for individual goals under a purely global approach.

Original: While COMA was shown to outperform IAC in SC2 micromanagement where
Revised: While COMA was shown to outperform IAC in SC2 micromanagement where

Original: IAC must learn from a single team reward (Foerster et al. , 2018), our IAC agents have access to individual rewards that resolve the credit assignment issue and improve performance (Singh et al. , 2019).
Revised: IAC must learn from a single team reward (Foerster et al. , 2018), our IAC agents have access to individual rewards that resolve the credit assignment issue and improve performance (Singh et al. , 2019).

Original: In SUMO (Figure 5d), CM3 and QMIX found cooperative solutions with performances within the margin of error, while COMA and IAC could not break out of local optima where vehicles move straight but do not perform merge maneuvers.
Revised: In SUMO (Figure 5d), CM3 and QMIX found cooperative solutions with performances within the margin of error, while COMA and IAC could not break out of local optima where vehicles move straight but do not perform merge maneuvers.

Original: Since initial states force agents into the region of state space requiring cooperation, credit assignment rather than exploration is the dominant challenge, which CM3 addressed via the credit function, as evidenced in Figure 5i.
Revised: Since initial states force agents into the region of state space requiring cooperation, credit assignment rather than exploration is the dominant challenge, which CM3 addressed via the credit function, as evidenced in Figure 5i.

Original: IAC underperformed because SUMO requires a longer sequence of cooperative actions and gave much sparser rewards than the “Merge” scenario in cooperative navigation.
Revised: IAC underperformed because SUMO requires a longer sequence of cooperative actions and gave much sparser rewards than the “Merge” scenario in cooperative navigation.

Original: We also show that centralized training of merely two decentralized agents allows them to generalize to settings with much heavier trafﬁc (Appendix E).
Revised: We also show that centralized training of merely two decentralized agents allows them to generalize to settings with much heavier trafﬁc (Appendix E).

Original: In Checkers (Figure 5e), CM3 (with 5k episodes in Stage 1) converged 10k episodes faster than COMA and QMIX to the global optimum with score 24.
Revised: In Checkers (Figure 5e), CM3 (with 5k episodes in Stage 1) converged 10k episodes faster than COMA and QMIX to the global optimum with score 24.

Original: Both exploration of the combinatorially large joint trajectory space and credit assignment for path clearing are challenges that CM3 successfully addressed.
Revised: Both exploration of the combinatorially large joint trajectory space and credit assignment for path clearing are challenges that CM3 successfully addressed.

Original: COMA only solved Checkers among all domains, possibly because the small bounded environment alleviates COMA’s difﬁculty with individual goals in large state spaces.
Revised: COMA only solved Checkers among all domains, possibly because the small bounded environment alleviates COMA’s difﬁculty with individual goals in large state spaces.

Original: IAC underperformed all centralized learning methods because cooperative actions that give no instantaneous reward are hard for selﬁsh agents to discover in Checkers.
Revised: IAC underperformed all centralized learning methods because cooperative actions that give no instantaneous reward are hard for selﬁsh agents to discover in Checkers.

Original: These results demonstrate
Revised: These results demonstrate

Original: CM3’s ability to attain individual goals and ﬁnd cooperative solutions in diverse multi-agent systems.
Revised: CM3’s ability to attain individual goals and ﬁnd cooperative solutions in diverse multi-agent systems.

Original: Ablations.
Revised: Ablations.

Original: The signiﬁcantly better performance of CM3 versus “Direct” (Figures 5f to 5j) shows that learning individual goal attainment prior to learning multi-agent cooperation, and initializing
Revised: The signiﬁcantly better performance of CM3 versus “Direct” (Figures 5f to 5j) shows that learning individual goal attainment prior to learning multi-agent cooperation, and initializing

Original: Stage 2 with Stage 1 parameters, are crucial for improving learning speed and stability.
Revised: Stage 2 with Stage 1 parameters, are crucial for improving learning speed and stability.

Original: It gives evidence that while global action-value and credit functions may be difﬁcult to train from scratch, function augmentation signiﬁcantly eases the learning problem.
Revised: It gives evidence that while global action-value and credit functions may be difﬁcult to train from scratch, function augmentation signiﬁcantly eases the learning problem.

Original: While “QV” initially learns quickly to attain individual goals, it does so at the cost of frequent collisions, higher variance, and inability to maintain a cooperative solution, giving clear evidence for the necessity of the credit function.
Revised: While “QV” initially learns quickly to attain individual goals, it does so at the cost of frequent collisions, higher variance, and inability to maintain a cooperative solution, giving clear evidence for the necessity of the credit function.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented CM3, a general framework for cooperative multi-goal MARL.
Revised: We presented CM3, a general framework for cooperative multi-goal MARL.

Original: CM3 addresses the need for efﬁcient exploration to learn both individual goal attainment and cooperation, via a two-stage curriculum bridged by function augmentation.
Revised: CM3 addresses the need for efﬁcient exploration to learn both individual goal attainment and cooperation, via a two-stage curriculum bridged by function augmentation.

Original: It achieves local credit assignment between action and goals using a credit function in a multi-goal policy gradient.
Revised: It achieves local credit assignment between action and goals using a credit function in a multi-goal policy gradient.

Original: In diverse experimental domains, CM attains signiﬁcantly higher performance, faster learning, and overall robustness than existing MARL methods, displaying strengths of both independent learning and centralized credit assignment while avoiding shortcomings of existing methods.
Revised: In diverse experimental domains, CM attains signiﬁcantly higher performance, faster learning, and overall robustness than existing MARL methods, displaying strengths of both independent learning and centralized credit assignment while avoiding shortcomings of existing methods.

Original: Ablations demonstrate each component is crucial to the whole framework.
Revised: Ablations demonstrate each component is crucial to the whole framework.

Original: Our results motivate future work on analyzing CM3’s theoretical properties and generalizing to inhomogeneous systems or settings without known goal assignments
Revised: Our results motivate future work on analyzing CM3’s theoretical properties and generalizing to inhomogeneous systems or settings without known goal assignments. A CKNOWLEDGMENTS

Revised: JY thanks Rakshit Trivedi (Georgia Institute of Technology), Ahmad Beirami (Facebook AI), and

Revised: Peter Sunehag (DeepMind) for detailed and helpful feedback on this work.

Original: B D ERIVATIONS B. P ROPOSITION
Revised: B D ERIVATIONS B. P ROPOSITION

Original: By stationarity and relabeling t
Revised: By stationarity and relabeling t

Original: [Equation]
Revised: [Equation]

Original: the credit function can be written:
Revised: the credit function can be written:

Original: [Equation]
Revised: [Equation]

Original: Using the law of iterated expectation
Revised: Using the law of iterated expectation

Original: [Equation]
Revised: [Equation]

Original: the credit function satisﬁes the Bellman expectation equation (2):
Revised: the credit function satisﬁes the Bellman expectation equation (2):

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: The goal-speciﬁc joint value function is the marginal of the credit function:
Revised: The goal-speciﬁc joint value function is the marginal of the credit function:

Original: [Equation]
Revised: [Equation]

Original: The credit function can be expressed in terms of the goal-speciﬁc action-value function:
Revised: The credit function can be expressed in terms of the goal-speciﬁc action-value function:

Original: [Equation]
Revised: [Equation]

Original: B. P ROPOSITION
Revised: B. P ROPOSITION

Original: First we state some elementary relations between global functions V πn ( s ) and Q πn ( s, a ) .
Revised: First we state some elementary relations between global functions V πn ( s ) and Q πn ( s, a ) .

Original: These carry over directly from the case of an MDP
Revised: These carry over directly from the case of an MDP

Original: [Equation]
Revised: [Equation]

Original: by treating the joint policy π as as an effective “single-agent” policy and restricting attention to a single goal g n (standard derivations are included at the end of this section)
Revised: by treating the joint policy π as as an effective “single-agent” policy and restricting attention to a single goal g n (standard derivations are included at the end of this section).

Original: [Equation]
Revised: [Equation]

Original: We follow the proof of the policy gradient theorem (Sutton et al. , 2000):
Revised: We follow the proof of the policy gradient theorem (Sutton et al. , 2000):

Original: [Equation]
Revised: [Equation]

Original: recursively unrolling) 
Revised: recursively unrolling) ∇

Original: [Equation]
Revised: [Equation]

Original: We can replace Q π n ( s, a ) by the advantage function A
Revised: We can replace Q π n ( s, a ) by the advantage function A

Original: [Equation]
Revised: [Equation]

Original: − V π n ( s ) , which does not change the expectation in Equation (9) because:
Revised: V π n ( s ) , which does not change the expectation in Equation (9) because:

Original: [Equation]
Revised: [Equation]

Original: So the gradient (9) can be written
Revised: So the gradient (9) can be written

Original: [Equation]
Revised: [Equation]

Original: Recall that from (3), for any choice of agent label k ∈ [1 ..N ] :
Revised: Recall that from (3), for any choice of agent label k ∈ [1 ..N ] :

Original: [Equation]
Revised: [Equation]

Original: Then substituting (3) into (10):
Revised: Then substituting (3) into (10):

Original: [Equation]
Revised: [Equation]

Original: Now notice that the choice of k in (13) is completely arbitrary, since (3) holds for any k ∈ [1 ..N ] .
Revised: Now notice that the choice of k in (13) is completely arbitrary, since (3) holds for any k ∈ [1 ..N ] .

Original: Therefore, it is valid to distribute A π n,k ( s, a ) into the summation in (12) using the summation index m instead of k .
Revised: Therefore, it is valid to distribute A π n,k ( s, a ) into the summation in (12) using the summation index m instead of k .

Original: Further summing (12) over all n , we arrive at the result of Proposition 2:
Revised: Further summing (12) over all n , we arrive at the result of Proposition 2:

Original: [Equation]
Revised: [Equation]

Original: The relation between V πn ( s ) and Q πn ( s, a ) in (7) and (8) are derived as follows:
Revised: The relation between V πn ( s ) and Q πn ( s, a ) in (7) and (8) are derived as follows:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: C V ARIANCE C. V ARIANCE OF COMA GRADIENT .
Revised: C V ARIANCE C. V ARIANCE OF COMA GRADIENT .

Original: For convenience, let Q :=
Revised: Let Q :=

Original: Q π ( s, a , g ) denote the centralized Q function, let π ( a n ) :
Revised: Q π ( s, a , g ) denote the centralized Q function, let π ( a n ) :

Original: = π ( a n | o n , g n ) denote a single agent’s policy, and let π ( a − n ) :
Revised: = π ( a n | o n , g n ) denote a single agent’s policy, and let π ( a − n ) :

Original: = π ( a − n | o − n , g − n ) denote the other agents’ joint policy.
Revised: = π ( a − n | o − n , g − n ) denote the other agents’ joint policy.

Original: Inthe cooperative multi-goal MARL context, the direct application of COMA has the following gradient.
Revised: In cooperative multi-goal MARL, the direct application of COMA has the following gradient.

Original: [Equation]
Revised: [Equation]

Original: Deﬁne the following:
Revised: Deﬁne the following:

Original: [Equation]
Revised: [Equation]

Original: Deﬁne M nm :=
Revised: Deﬁne M nm :=

Original: E π [ f n ] T E π [ f m ] and let M := (cid:80) n,m M nm .
Revised: E π [ f n ] T E π [ f m ] and let M := (cid:80) n,m M nm .

Original: Then we have M nm =
Revised: Then we have M nm =

Original: E π [ z n Q ] T E π
Revised: E π [ z n Q ] T E π

Original: [ z m Q ] since
Revised: [ z m Q ] since

Original: [Equation]
Revised: [Equation]

Original: Since the COMA gradient is E π
Revised: Since the COMA gradient is E π

Original: [ (cid:80) Nn =1 f n ] .
Revised: [ (cid:80) Nn =1 f n ] .

Original: its variance can be derived to be (Wu et al. , 2018):
Revised: its variance can be derived to be (Wu et al. , 2018):

Original: [Equation]
Revised: [Equation]

Original: C. V ARIANCE OF THE CM3 GRADIENT
Revised: C. V ARIANCE OF THE CM3 GRADIENT

Original: For convenience, let Q n :=
Revised: For convenience, let Q n :=

Original: Q π n ( s, a ) =
Revised: Q π n ( s, a ) =

Original: Q π ( s, a , g n ) denote the global Q function for goal g n , and let π ( a m ) :
Revised: Q π ( s, a , g n ) denote the global Q function for goal g n [Equation] and let π ( a m ) :

Original: = π ( a m | o m , g m ) .
Revised: = π ( a m | o m , g m ) .

Original: The CM3 gradient can be rewritten as
Revised: The CM3 gradient can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: As before, z m := ∇ θ log π ( a m ) .
Revised: As before, z m := ∇ θ log π ( a m ) .

Original: Deﬁne h nm :=
Revised: Deﬁne h nm :=

Original: z m ( Q n − b nm ( s )) and let h n := (cid:80) m h nm .
Revised: z m ( Q n − b nm ( s )) and let h n := (cid:80) m h nm .

Original: Then the variance is
Revised: Then the variance is

Original: Var

Original: [Equation]
Revised: [Equation]

Original: D E XAMPLE OF GREEDY INITIALIZATION FOR MARL EXPLORATION
Revised: D E XAMPLE OF GREEDY INITIALIZATION FOR MARL EXPLORATION

Original: A greedy initialization can provide signiﬁcant improvement in multi-agent exploration versus naïve random exploration, as shown by a simple thought experiment.
Revised: A greedy initialization can provide signiﬁcant improvement in multi-agent exploration versus naïve random exploration, as shown by a simple thought experiment.

Original: Consider a two-player MG deﬁned by a 4 × 3 gridworld with unit actions (up, down, left, right).
Revised: Consider a two-player MG deﬁned by a 4 × 3 gridworld with unit actions (up, down, left, right).

Original: Agent A starts at (1,2) with goal (4,2), while agent B starts at (4,2) with goal (1,2).
Revised: Agent A starts at (1,2) with goal (4,2), while agent B starts at (4,2) with goal (1,2).

Original: The greedy policy for each agent in MG is to move horizontally toward its target, since this is optimal in the induced M (when the other agent is absent).
Revised: The greedy policy for each agent in MG is to move horizontally toward its target, since this is optimal in the induced M (when the other agent is absent).

Original: Case 1: Suppose that for (cid:15) ∈ (0 , 1) , A and B follow greedy policies with probability 1 − (cid:15) , and take random actions ( p ( a ) = 1 / 4 ) with probability (cid:15) .
Revised: Case 1: Suppose that for (cid:15) ∈ (0 , 1) , A and B follow greedy policies with probability 1 − (cid:15) , and take random actions ( p ( a ) = 1 / 4 ) with probability (cid:15) .

Original: Then the probability of a symmetric optimal trajectory is P ( cooperate ) = 2 (cid:15) 2 ((1 − (cid:15) )
Revised: Then the probability of a symmetric optimal trajectory is P ( cooperate ) = 2 (cid:15) 2 ((1 − (cid:15) )

Original: + (cid:15)/ 4) 8 .
Revised: + (cid:15)/ 4) 8 .

Original: For (cid:15) = 0 .
Revised: For (cid:15) = 0 .

Original: 5 , P ( cooperate )
Revised: 5 , P ( cooperate )

Original: ≈ 0 .
Revised: ≈ 0 .

Original: Case 2: If agents execute uniform random exploration, then P ( cooperate ) = 3 . 05 e-5 (cid:28) 0 .
Revised: Case 2: If agents execute uniform random exploration, then P ( cooperate ) = 3 . 05 e-5 (cid:28) 0 .

Original: E G ENERALIZATION
Revised: E G ENERALIZATION

Original: [Table]
Revised: [Table]

Original: We investigated whether policies trained with few agent vehicles ( N = 2 ) on an empty road can generalize to situations with heavy SUMO-controlled trafﬁc.
Revised: We investigated whether policies trained with few agent vehicles ( N = 2 ) on an empty road can generalize to situations with heavy SUMO-controlled trafﬁc.

Original: We also tested on initial and goal lane conﬁgurations (C3 and C4) which occur with low probability when training with conﬁgurations
Revised: We also tested on initial and goal lane conﬁgurations (C3 and C4) which occur with low probability when training with conﬁgurations

Original: C1 and C2.
Revised: C1 and C2.

Original: Table 1 shows the sum of agents’ reward, averaged over 100 test episodes, on these conﬁgurations that require cooperation with each other and with minimally-interactive SUMOcontrolled vehicles for success.
Revised: Table 1 shows the sum of agents’ reward, averaged over 100 test episodes, on these conﬁgurations that require cooperation with each other and with minimally-interactive SUMOcontrolled vehicles for success.

Original: CM3’s higher performance than IAC and COMA in training is reﬂected by better generalization performance on these test conﬁgurations.
Revised: CM3’s higher performance than IAC and COMA in training is reﬂected by better generalization performance on these test conﬁgurations.

Original: There is almost negligible decrase in performance from train Figure 5d to test, giving evidence to our hypothesis that centralized training with few agents is feasible even for deployment in situations with many agents, for certain applications where local interactions are dominant
Revised: There is almost negligible decrase in performance from train Figure 5d to test, giving evidence to our hypothesis that centralized training with few agents is feasible even for deployment in situations with many agents, for certain applications where local interactions are dominant

Original: [Table]
Revised: [Table]

Original: F A BSOLUTE RUNTIME
Revised: F A BSOLUTE RUNTIME

Original: CM3’s higher sample efﬁciency does not come at greater computational cost, as all methods’ runtimes are within an order of magnitude of one another.
Revised: CM3’s higher sample efﬁciency does not come at greater computational cost, as all methods’ runtimes are within an order of magnitude of one another.

Original: Test times have no signiﬁcant difference as all neural networks were similar.
Revised: Test times have no signiﬁcant difference as all neural networks were similar.

Original: [Table]
Revised: [Table]

Original: G E NVIRONMENT DETAILS
Revised: G E NVIRONMENT DETAILS

Original: The full Markov game for each experimental domain, along with the single-agent MDP induced from the Markov game, are deﬁned in this section.
Revised: The full Markov game for each experimental domain, along with the single-agent MDP induced from the Markov game, are deﬁned in this section.

Original: In all domains, each agent’s observation in the
Revised: In all domains, each agent’s observation in the

Original: Markov game consists of two components, o self and o others .
Revised: Markov game consists of two components, o self and o others .

Original: CM3 leverages this decomposition 
Revised: CM3 leverages this decomposition for

Original: G. C OOPERATIVE NAVIGATION
Revised: G. C OOPERATIVE NAVIGATION

Original: This domain is adapted from the multi-agent particle environment in Lowe et al.
Revised: This domain is adapted from the multi-agent particle environment in Lowe et al.

Original: Movable agents and static landmarks are represented as circular objects located in a 2D unbounded world with real-valued position and velocity.
Revised: Movable agents and static landmarks are represented as circular objects located in a 2D unbounded world with real-valued position and velocity.

Original: Agents experience contact forces during collisions.
Revised: Agents experience contact forces during collisions.

Original: A simple model of inertia and friction is involved.
Revised: A simple model of inertia and friction is involved.

Original: State.
Revised: State.

Original: The global state vector is the concatenation of all agents’ absolute position ( x, y ) ∈
Revised: The global state vector is the concatenation of all agents’ absolute position ( x, y ) ∈

Original: R 2 and velocity ( v x , v y
Revised: R 2 and velocity ( v x , v y

Original: ) ∈ R 2 .
Revised: ) ∈ R 2 .

Original: Observation.
Revised: Observation.

Original: Each agent’s observation of itself, o self , is its own absolute position and velocity.
Revised: Each agent’s observation of itself, o self , is its own absolute position and velocity.

Original: Each agent’s observation of others, o others , is the concatenation of the relative positions and velocities of all other agents with respect to itself.
Revised: Each agent’s observation of others, o others , is the concatenation of the relative positions and velocities of all other agents with respect to itself.

Original: Actions.
Revised: Actions.

Original: Agents take actions from the discrete set do nothing, up, down, left, right, where the movement actions produce an instantaneous velocity (with inertia effects).
Revised: Agents take actions from the discrete set do nothing, up, down, left, right, where the movement actions produce an instantaneous velocity (with inertia effects).

Original: Goals and initial state assignment.
Revised: Goals and initial state assignment.

Original: With probability 0.2, landmarks are given uniform random locations in the set ( − 1 , 1) 2 , and agents are assigned initial positions uniformly at random within the set ( − 1 , 1) 2 .
Revised: With probability 0.2, landmarks are given uniform random locations in the set ( − 1 , 1) 2 , and agents are assigned initial positions uniformly at random within the set ( − 1 , 1) 2 .

Original: With probability 0.8, they are predeﬁned as follows (see Figure 3).
Revised: With probability 0.8, they are predeﬁned as follows (see Figure 3).

Original: In “Antipodal”, landmarks for agents 1 to 4 have ( x, y ) coordinates [(0.9,0.9), (-0.9,-0.9), (0.9,-0.9), (-0.9,0.9)], while agents 1 to 4 are placed at [(-0.9,-0.9), (0.9,0.9), (-0.9,0.9), (0.9,-0.9)].
Revised: In “Antipodal”, landmarks for agents 1 to 4 have ( x, y ) coordinates [(0.9,0.9), (-0.9,-0.9), (0.9,-0.9), (-0.9,0.9)], while agents 1 to 4 are placed at [(-0.9,-0.9), (0.9,0.9), (-0.9,0.9), (0.9,-0.9)].

Original: In “Intersection”, landmark coordinates are [(0.9,-0.15), (-0.9,0.15), (0.15,0.9), (-0.15,-0.9)], while agents are placed at [(-0.9,- 0.15), (0.9,0.15), (0.15,-0.9), (-0.15,0.9)].
Revised: In “Intersection”, landmark coordinates are [(0.9,-0.15), (-0.9,0.15), (0.15,0.9), (-0.15,-0.9)], while agents are placed at [(-0.9,- 0.15), (0.9,0.15), (0.15,-0.9), (-0.15,0.9)].

Original: In “Merge”, landmark coordinates are [(0.9,-0.2), (0.9,0.2)], while agents are [(-0.9,0.2), (-0.9,-0.2)].
Revised: In “Merge”, landmark coordinates are [(0.9,-0.2), (0.9,0.2)], while agents are [(-0.9,0.2), (-0.9,-0.2)].

Original: Each agent’s goal is the assigned landmark position vector.
Revised: Each agent’s goal is the assigned landmark position vector.

Original: Reward.
Revised: Reward.

Original: At each time step, each agent’s individual reward is the negative distance between its position and the position of its assigned landmark.
Revised: At each time step, each agent’s individual reward is the negative distance between its position and the position of its assigned landmark.

Original: If a collision occurs between any pair of agents, both agents receive an additional -1 penalty.
Revised: If a collision occurs between any pair of agents, both agents receive an additional -1 penalty.

Original: A collision occurs when two agents’ distance is less than the sum of their radius.
Revised: A collision occurs when two agents’ distance is less than the sum of their radius.

Original: Termination.
Revised: Termination.

Original: Episode terminates when all agents are less than 0.05 distance from assigned landmarks.
Revised: Episode terminates when all agents are less than 0.05 distance from assigned landmarks.

Original: Induced MDP.
Revised: Induced MDP.

Original: This is the N = 1 case of the Markov game, used by Stage 1 of CM3.
Revised: This is the N = 1 case of the Markov game, used by Stage 1 of CM3.

Original: The single agent only receives o self .
Revised: The single agent only receives o self .

Original: In each episode, its initial position and the assigned landmark’s initial position are both uniform randomly chosen from ( − 1 , 1) 2 .
Revised: In each episode, its initial position and the assigned landmark’s initial position are both uniform randomly chosen from ( − 1 , 1) 2 .

Original: G. SUMO
Revised: G. SUMO

Original: We constructed a straight road of total length 200m and width 12.8m, consisting of four lanes.
Revised: We constructed a straight road of total length 200m and width 12.8m, consisting of four lanes.

Original: All lanes have width 3.2m, and vehicles can be aligned along any of four sub-lanes within a lane, with lateral spacing 0 .
Revised: All lanes have width 3.2m, and vehicles can be aligned along any of four sub-lanes within a lane, with lateral spacing 0 .

Original: 8 m .
Revised: 8 m .

Original: Vehicles are emitted at average speed 30m/s with small deviation.
Revised: Vehicles are emitted at average speed 30m/s with small deviation.

Original: Simulation time resolution was 0 .
Revised: Simulation time resolution was 0 .

Original: 2 s per step.
Revised: 2 s per step.

Original: Supplementary ﬁle merge_stage3_dense.rou.xml contains all vehicle parameters, and merge.net.xml deﬁnes the complete road architecture.
Revised: SUMO ﬁle merge_stage3_dense.rou.xml contains all vehicle parameters, and merge.net.xml deﬁnes the complete road architecture.

Original: State.
Revised: State.

Original: The global state vector s is the concatenation of all agents’ absolute position ( x, y ) , normalized respectively by the total length and width of the road, and horizontal speed v normalized by 29m/s.
Revised: The global state vector s is the concatenation of all agents’ absolute position ( x, y ) , normalized respectively by the total length and width of the road, and horizontal speed v normalized by 29m/s.

Original: Observation.
Revised: Observation.

Original: Each agent observation of itself o n self is a vector consisting of: agent speed normalized by 29m/s, normalized number of sub-lanes between agent’s current sub-lane and center sub-lane of goal lane, and normalized longitudinal distance to goal position.
Revised: Each agent observation of itself o n self is a vector consisting of: agent speed normalized by 29m/s, normalized number of sub-lanes between agent’s current sub-lane and center sub-lane of goal lane, and normalized longitudinal distance to goal position.

Original: Each agent’s observation of others o n others is a discretized observation tensor of shape [13, Actions.
Revised: Each agent’s observation of others o n others is a discretized observation tensor of shape [13,9,2] centered have ﬁve

Original: All agents have the same discrete action space, consisting of ﬁve options: no-op (maintain current speed and lane), accelerate ( decelerate ( − shift one sub lane to the left shift one sub-lane to the right

Original: Each agent’s action a n is represented as a one-hot vector length 5.

Original: Goals and initial state assignment.
Revised: Goals and initial state assignment.

Original: Each goal vector g n is a one-hot vector of length 4, indicating the goal lane at which agent n should arrive once it crosses position x =190m. With probability 0.2, agents are assigned goals uniformly at random, and agents are assigned initial lanes uniformly at random at position x =0.
Revised: Each goal vector g n is a one-hot vector of length 4, indicating the goal lane at which agent n should arrive once it crosses position x =190m. With probability 0.2, agents are assigned goals uniformly at random, and agents are assigned initial lanes uniformly at random at position x =0.

Original: With probability 0.8, agent 1’s goal is lane 2 and agent 2’s goal is lane 1, while agent 1 is initialized at lane 1 and agent 2 is initialized at lane 2 (see Figure 4).
Revised: With probability 0.8, agent 1’s goal is lane 2 and agent 2’s goal is lane 1, while agent 1 is initialized at lane 1 and agent 2 is initialized at lane 2 (see Figure 4).

Original: Departure times were drawn from a normal distribution with mean 0s and standard deviation 0.5s for each agent.
Revised: Departure times were drawn from a normal distribution with mean 0s and standard deviation 0.5s for each agent.

Original: Reward.
Revised: Reward.

Original: The reward R ( s t , a t , g n ) for agent n with goal g n is given according to the conditions: -1 for a collision; -10 for time-out (exceed 33 simulation steps during an episode); 10(1 − ∆) for reaching the end of the road and having a normalized sub-lane difference of ∆ from the center of the goal lane; and -0.1 if current speed exceeds 35.7m/s.
Revised: The reward R ( s t , a t , g n ) for agent n with goal g n is given according to the conditions: -1 for a collision; -10 for time-out (exceed 33 simulation steps during an episode); 10(1 − ∆) for reaching the end of the road and having a normalized sub-lane difference of ∆ from the center of the goal lane; and -0.1 if current speed exceeds 35.7m/s.

Original: Termination.
Revised: Termination.

Original: Episode terminates when 33 simulation steps have elapsed or all agents have x > 190m.
Revised: Episode terminates when 33 simulation steps have elapsed or all agents have x > 190m.

Original: Induced MDP.
Revised: Induced MDP.

Original: This is the N = 1 case of the Markov game deﬁned above, used by Stage 1 of
Revised: This is the N = 1 case of the Markov game deﬁned above, used by Stage 1 of

Original: CM3.
Revised: CM3.

Original: The single agent receives only o self .
Revised: The single agent receives only o self .

Original: For each episode, agent initial and goal lanes are assigned uniformly at random from the available lanes.
Revised: For each episode, agent initial and goal lanes are assigned uniformly at random from the available lanes.

Original: G. C HECKERS
Revised: G. C HECKERS

Original: This domain is adapted from the Checkers environment in Sunehag et al.
Revised: This domain is adapted from the Checkers environment in Sunehag et al.

Original: It is a gridworld with 5 rows and 13 columns (Figure 2).
Revised: It is a gridworld with 5 rows and 13 columns (Figure 2).

Original: Agents cannot move to the two highest and lowest rows and the two highest and lowest columns, which are placed for agents’ ﬁnite observation grid to be well-deﬁned.
Revised: Agents cannot move to the two highest and lowest rows and the two highest and lowest columns, which are placed for agents’ ﬁnite observation grid to be well-deﬁned.

Original: Agents cannot be in the same grid location.
Revised: Agents cannot be in the same grid location.

Original: Red and yellow collectible reward are placed in a checkered pattern in the middle 3x8 region, and they disappear when any agent moves to their location.
Revised: Red and yellow collectible reward are placed in a checkered pattern in the middle 3x8 region, and they disappear when any agent moves to their location.

Original: State.
Revised: State.

Original: The global state s consists of two components.
Revised: The global state s consists of two components.

Original: The ﬁrst is s T , a tensor of shape [3,9,2], where the two “channels” in the last dimension represents the presence/absence of red and yellow rewards as 1-hot matrices.
Revised: The ﬁrst is s T , a tensor of shape [3,9,2], where the two “channels” in the last dimension represents the presence/absence of red and yellow rewards as 1-hot matrices.

Original: The second is s V , the concatenation of all agents’ ( x, y ) location (integer-valued) and the number of red and yellow each agent has collected so far.
Revised: The second is s V , the concatenation of all agents’ ( x, y ) location (integer-valued) and the number of red and yellow each agent has collected so far.

Original: Observation.
Revised: Observation.

Original: Each agent’s obsevation of others, o n others , is the concatenation of all other agents’ normalized coordinates (normalized by total size of grid).
Revised: Each agent’s obsevation of others, o n others , is the concatenation of all other agents’ normalized coordinates (normalized by total size of grid).

Original: An agent’s observation of itself, o n self , consists of two components.
Revised: An agent’s observation of itself, o n self , consists of two components.

Original: First, o n self ,V is a vector concatenation of agent n ’s normalized coordinate and the number of red and yellow it has collected so far.
Revised: First, o n self ,V is a vector concatenation of agent n ’s normalized coordinate and the number of red and yellow it has collected so far.

Original: Second, o n self ,T is a tensor of shape [5,5,3], centered on its current location in the grid.
Revised: Second, o n self ,T is a tensor of shape [5,5,3], centered on its current location in the grid.

Original: The tensor has three “channels”, where the ﬁrst two represent presence/absence of red and yellow rewards as 1-hot matrices, and the last channel indicates the invalid locations as a 1-hot matrix.
Revised: The tensor has three “channels”, where the ﬁrst two represent presence/absence of red and yellow rewards as 1-hot matrices, and the last channel indicates the invalid locations as a 1-hot matrix.

Original: The agent’s own grid location is a valid location, while other agents’ locations are invalid.
Revised: The agent’s own grid location is a valid location, while other agents’ locations are invalid.

Original: Actions.
Revised: Actions.

Original: Agents choose from a discrete set of actions do-nothing, up, down, left, right.
Revised: Agents choose from a discrete set of actions do-nothing, up, down, left, right.

Original: Movement actions transport the agent one grid cell in the chosen direction.
Revised: Movement actions transport the agent one grid cell in the chosen direction.

Original: Goals.
Revised: Goals.

Original: Agent A’s goal is to collect all red rewards without touching yellow.
Revised: Agent A’s goal is to collect all red rewards without touching yellow.

Original: Agent B’s goal is to collect all yellow without touching red.
Revised: Agent B’s goal is to collect all yellow without touching red.

Original: The goal is represented as a 1-hot vector of length 2.
Revised: The goal is represented as a 1-hot vector of length 2.

Original: Reward.
Revised: Reward.

Original: Agent A gets +1 for red, -0.5 for yellow.
Revised: Agent A gets +1 for red, -0.5 for yellow.

Original: Agent B gets -0.5 for red, +1 for yellow.
Revised: Agent B gets -0.5 for red, +1 for yellow.

Original: Initial state distribution.
Revised: Initial state distribution.

Original: Agent A is initialized at (2,8), Agent B is initialized at (4,8).
Revised: Agent A is initialized at (2,8), Agent B is initialized at (4,8).

Original: (0,0) is the top-left cell (Figure 2).
Revised: (0,0) is the top-left cell (Figure 2).

Original: Termination.
Revised: Termination.

Original: Each episode ﬁnishes when either 75 time steps have elapsed, or when all rewards have been collected.
Revised: Each episode ﬁnishes when either 75 time steps have elapsed, or when all rewards have been collected.

Original: Induced MDP.
Revised: Induced MDP.

Original: For Stage 1 of CM3, the single agent is randomly assigned the role of either Agent A or Agent B in each episode.
Revised: For Stage 1 of CM3, the single agent is randomly assigned the role of either Agent A or Agent B in each episode.

Original: Everything else is deﬁned as above.
Revised: Everything else is deﬁned as above.

Original: H A RCHITECTURE
Revised: H A RCHITECTURE

Original: For all experiment domains, ReLU nonlinearity was used for all neural network layers unless otherwise speciﬁed.
Revised: For all experiment domains, ReLU nonlinearity was used for all neural network layers unless otherwise speciﬁed.

Original: All layers are fully-connected feedforward layers, unless otherwise speciﬁed.
Revised: All layers are fully-connected feedforward layers, unless otherwise speciﬁed.

Original: All experiment domains have a discrete action space (with |A| = 5 actions), and action probabilities were computed by lower-bounding softmax outputs of all policy networks by
Revised: All experiment domains have a discrete action space (with |A| = 5 actions), and action probabilities were computed by lower-bounding softmax outputs of all policy networks by

Original: P ( a n = i ) =
Revised: P ( a n = i ) =

Original: (1 − (cid:15) ) softmax ( i ) + (cid:15)/ |A| , where (cid:15) is a decaying exploration parameter.
Revised: (1 − (cid:15) ) softmax ( i ) + (cid:15)/ |A| , where (cid:15) is a decaying exploration parameter.

Original: To keep neural network architectures as similar as possible among all algorithms, our neural networks for
Revised: To keep neural network architectures as similar as possible among all algorithms, our neural networks for

Original: COMA differ from those of Foerster et al.
Revised: COMA differ from those of Foerster et al.

Original: (2018) in that we do not use recurrent networks, and we do not feed previous actions into the Q function.
Revised: (2018) in that we do not use recurrent networks, and we do not feed previous actions into the Q function.

Original: For the Q network in all implementations of
Revised: For the Q network in all implementations of

Original: COMA, the value of each output node i is interpreted as the action-value Q ( s, a − n , a n =
Revised: COMA, the value of each output node i is interpreted as the action-value Q ( s, a − n , a n =

Original: i, g ) for agent n taking action i and all other agents taking action a − n .
Revised: i, g ) for agent n taking action i and all other agents taking action a − n .

Original: Also for COMA, agent n ’s label vector (one-hot indicator vector) and observation o self were used as input to COMA’s global Q function, to differentiate between evaluations of the Q-function for different agents.
Revised: Also for COMA, agent n ’s label vector (one-hot indicator vector) and observation o self were used as input to COMA’s global Q function, to differentiate between evaluations of the Q-function for different agents.

Original: These were choices in
Revised: These were choices in

Original: Foerster et al.
Revised: Foerster et al.

Original: (2018) that we retain.
Revised: (2018) that we retain.

Original: H. C OOPERATIVE NAVIGATION
Revised: H. C OOPERATIVE NAVIGATION

Original: CM3.
Revised: CM3.

Original: The policy network π 1 in Stage 1 feeds the concatenation of o self and goal g to one layer with 64 units, which is connected to the special layer h 1 ∗ with 64 units, then connected to the softmax output layer with 5 units, each corresponding to one discrete action.
Revised: The policy network π 1 in Stage 1 feeds the concatenation of o self and goal g to one layer with 64 units, which is connected to the special layer h 1 ∗ with 64 units, then connected to the softmax output layer with 5 units, each corresponding to one discrete action.

Original: In Stage 2, o others is connected to a new layer with 128 units, then connected to h 1 ∗ .
Revised: In Stage 2, o others is connected to a new layer with 128 units, then connected to h 1 ∗ .

Original: The Q 1 function in Stage 1 feeds the concatenation of state s , goal g , and 1-hot action a to one layer with 64 units, which is connected to the special layer h 1 ∗ with 64 units, then to a single linear output unit.
Revised: The Q 1 function in Stage 1 feeds the concatenation of state s , goal g , and 1-hot action a to one layer with 64 units, which is connected to the special layer h 1 ∗ with 64 units, then to a single linear output unit.

Original: In Stage 2, Q 1 is augmented into both Q π n ( s, a ) and Q π n ( s, a m ) as separate networks.
Revised: In Stage 2, Q 1 is augmented into both Q π n ( s, a ) and Q π n ( s, a m ) as separate networks.

Original: For
Revised: For

Original: Q π n ( s, a ) , s − n (part of state s excluding agent n ) and a − n are concatenated and connected to a layer with 128 units, then connected to h 1 ∗ .
Revised: Q π n ( s, a ) , s − n (part of state s excluding agent n ) and a − n are concatenated and connected to a layer with 128 units, then connected to h 1 ∗ .

Original: For Q π n ( s, a m ) , s m (agent m portion of state s ) and s − n are concatenated and connected to a layer with 128 units, then connected to h 1 ∗ .
Revised: For Q π n ( s, a m ) , s m (agent m portion of state s ) and s − n are concatenated and connected to a layer with 128 units, then connected to h 1 ∗ .

Original: IAC.
Revised: IAC.

Original: IAC uses the same policy network as Stage 2 of CM3.
Revised: IAC uses the same policy network as Stage 2 of CM3.

Original: The value function of IAC concatenates o n self and goal g n , connects to a layer with 64 units, which connects to a second layer h 2 with 64 units, then to a single linear output unit.
Revised: The value function of IAC concatenates o n self and goal g n , connects to a layer with 64 units, which connects to a second layer h 2 with 64 units, then to a single linear output unit.

Original: o n others is connected to a layer with 128 units, then connected to h 2 .
Revised: o n others is connected to a layer with 128 units, then connected to h 2 .

Original: COMA.
Revised: COMA.

Original: COMA uses the same policy network as Stage 2 of CM3.
Revised: COMA uses the same policy network as Stage 2 of CM3.

Original: The global Q function of COMA computes Q ( s, ( a n , a − n )) for each agent n as follows.
Revised: The global Q function of COMA computes Q ( s, ( a n , a − n )) for each agent n as follows.

Original: Input is the concatenation of state s , all other agents’ 1-hot actions a − n , agent n’s goal g n , all other agent goals g − n , agent label n , and agent n ’s observation o n self .
Revised: Input is the concatenation of state s , all other agents’ 1-hot actions a − n , agent n’s goal g n , all other agent goals g − n , agent label n , and agent n ’s observation o n self .

Original: This is passed through two layers of 128 units each, then connected to a linear output layer with 5 units.
Revised: This is passed through two layers of 128 units each, then connected to a linear output layer with 5 units.

Original: QMIX.
Revised: QMIX.

Original: Individual value functions take input ( o n self , o n others , g n ) and connects to one hidden layer with 64 units, which connects to the output layer.
Revised: Individual value functions take input ( o n self , o n others , g n ) and connects to one hidden layer with 64 units, which connects to the output layer.

Original: The mixing network follows the exact architecture of
Revised: The mixing network follows the exact architecture of

Original: Rashid et al.
Revised: Rashid et al.

Original: (2018) with embedding dimension 64.
Revised: (2018) with embedding dimension 64.

Original: H. SUMO
Revised: H. SUMO

Original: CM3.
Revised: CM3.

Original: The policy network π 1 during Stage 1 feeds each of the inputs o self and goal g n to a layer with 32 units.
Revised: The policy network π 1 during Stage 1 feeds each of the inputs o self and goal g n to a layer with 32 units.

Original: The concatenation is then connected to the layer h 1 ∗ with 64 units, and connected to a softmax output layer with 5 units, each corresponding to one discrete action.
Revised: The concatenation is then connected to the layer h 1 ∗ with 64 units, and connected to a softmax output layer with 5 units, each corresponding to one discrete action.

Original: In Stage 2, the input observation grid o n others is processed by a convolutional layer with 4 ﬁlters of size 5x3 and stride 1x1, ﬂattened and connected to a layer with 64 units, then connected to the layer h 1 ∗ of π 1 .
Revised: In Stage 2, the input observation grid o n others is processed by a convolutional layer with 4 ﬁlters of size 5x3 and stride 1x1, ﬂattened and connected to a layer with 64 units, then connected to the layer h 1 ∗ of π 1 .

Original: The Q 1 function in Stage 1 feeds the concatenation of state s , goal g , and 1-hot action a to one layer with 256 units, which is connected to the special layer h 1 ∗ with 256 units, then to a single linear output unit.
Revised: The Q 1 function in Stage 1 feeds the concatenation of state s , goal g , and 1-hot action a to one layer with 256 units, which is connected to the special layer h 1 ∗ with 256 units, then to a single linear output unit.

Original: In Stage 2, Q 1 is augmented into both Q π n ( s, a ) and Q π n ( s, a m ) as separate networks.
Revised: In Stage 2, Q 1 is augmented into both Q π n ( s, a ) and Q π n ( s, a m ) as separate networks.

Original: For Q π n ( s, a ) , s − n (part of state s excluding agent n ), a − n , and g − n are concatenated and connected to a layer with 128 units, then connected to h 1 ∗ .
Revised: For Q π n ( s, a ) , s − n (part of state s excluding agent n ), a − n , and g − n are concatenated and connected to a layer with 128 units, then connected to h 1 ∗ .

Original: For Q π n ( s, a m ) , s m (agent m portion of state s ), s − n , and g − n are concatenated and connected to a layer with 128 units, then connected to h 1 ∗ .
Revised: For Q π n ( s, a m ) , s m (agent m portion of state s ), s − n , and g − n are concatenated and connected to a layer with 128 units, then connected to h 1 ∗ .

Original: IAC.
Revised: IAC.

Original: IAC uses the same policy network as Stage 2 of CM3.
Revised: IAC uses the same policy network as Stage 2 of CM3.

Original: The value function of IAC concatenates o n self and g n , feeds it into a layer with 64 units, which connects to a layer h 2 with 64 units, which connects to one linear output unit.
Revised: The value function of IAC concatenates o n self and g n , feeds it into a layer with 64 units, which connects to a layer h 2 with 64 units, which connects to one linear output unit.

Original: o n others is processed by a convolutional layer with 4 ﬁlters of size 5x3 and stride 1x1, ﬂattened and connected to a layer with 128 units, then connected to h 2 .
Revised: o n others is processed by a convolutional layer with 4 ﬁlters of size 5x3 and stride 1x1, ﬂattened and connected to a layer with 128 units, then connected to h 2 .

Original: COMA.
Revised: COMA.

Original: COMA uses the same policy network as Stage 2 of CM3.
Revised: COMA uses the same policy network as Stage 2 of CM3.

Original: The Q function of COMA is exactly the same as the one in COMA for cooperative navigation deﬁned above.
Revised: The Q function of COMA is exactly the same as the one in COMA for cooperative navigation deﬁned above.

Original: QMIX.
Revised: QMIX.

Original: Individual value functions take input ( o n self , g n ) and connects to one hidden layer withunits, which connects to layer h 2 with 64 units.
Revised: Individual value functions take input ( o n self , g n ) and connects to one hidden layer withunits, which connects to layer h 2 with 64 units.

Original: o n others is passed through the same convolutional layer as above and connected to h 2 .
Revised: o n others is passed through the same convolutional layer as above and connected to h 2 .

Original: h 2 is fully-connected to an output layer.
Revised: h 2 is fully-connected to an output layer.

Original: The mixing network follows the exact architecture of Rashid et al.
Revised: The mixing network follows the exact architecture of Rashid et al.

Original: (2018) with embedding dimension 64.
Revised: (2018) with embedding dimension 64.

Original: H. C HECKERS
Revised: H. C HECKERS

Original: CM3.
Revised: CM3.

Original: The policy network π 1 during Stage 1 feeds o n self ,T to a convolution layer with 6 ﬁlters of size 3x3 and stride 1x1, which is ﬂattened and connected to a layer with 32 units, which is concatenated with o n self ,V , previous action, and its goal vector.
Revised: The policy network π 1 during Stage 1 feeds o n self ,T to a convolution layer with 6 ﬁlters of size 3x3 and stride 1x1, which is ﬂattened and connected to a layer with 32 units, which is concatenated with o n self ,V , previous action, and its goal vector.

Original: The concatenation is connected to a layer withunits, then to the special layer h 1 ∗ with 256 units, ﬁnally to a softmax output layer with 5 units.
Revised: The concatenation is connected to a layer withunits, then to the special layer h 1 ∗ with 256 units, ﬁnally to a softmax output layer with 5 units.

Original: In
Revised: In

Original: Stage 2, o n others is connected to a layer with 256 units, then to the layer h 1 ∗ of π 1 .
Revised: Stage 2, o n others is connected to a layer with 256 units, then to the layer h 1 ∗ of π 1 .

Original: The Q 1 function in Stage 1 is deﬁned as: state tensor s T is fed to a convolutional layer with 4 ﬁlters of size 3x5 and stride 1x1 and ﬂattened.
Revised: The Q 1 function in Stage 1 is deﬁned as: state tensor s T is fed to a convolutional layer with 4 ﬁlters of size 3x5 and stride 1x1 and ﬂattened.

Original: o n self ,T is given to a convolution layer with 6 ﬁlters of size 3x3 and stride 1x1 and ﬂattened.
Revised: o n self ,T is given to a convolution layer with 6 ﬁlters of size 3x3 and stride 1x1 and ﬂattened.

Original: Both are concatenated with s n (agent n part of the s V vector), goal g n , action a n and o n self ,V .
Revised: Both are concatenated with s n (agent n part of the s V vector), goal g n , action a n and o n self ,V .

Original: The concatenation is fed to a layer with 256 units, then to the special layer h 1 ∗ with 256 units, then to a single linear output unit.
Revised: The concatenation is fed to a layer with 256 units, then to the special layer h 1 ∗ with 256 units, then to a single linear output unit.

Original: In Stage 2, Q 1 is augmented into both
Revised: In Stage 2, Q 1 is augmented into both

Original: Q π n ( s, a ) and Q π n ( s, a m ) as separate networks.
Revised: Q π n ( s, a ) and Q π n ( s, a m ) as separate networks.

Original: For Q π n ( s, a ) , s − n (part of state vector s V excluding agent n ) and a − n are concatenated and connected to a layer with 32 units, then connected to h 1 ∗ .
Revised: For Q π n ( s, a ) , s − n (part of state vector s V excluding agent n ) and a − n are concatenated and connected to a layer with 32 units, then connected to h 1 ∗ .

Original: For
Revised: For

Original: Q π n ( s, a m ) , s m (agent m portion of state s V ) and s − n are concatenated and connected to a layer with 32 units, then connected to h 1 ∗ .
Revised: Q π n ( s, a m ) , s m (agent m portion of state s V ) and s − n are concatenated and connected to a layer with 32 units, then connected to h 1 ∗ .

Original: IAC.
Revised: IAC.

Original: IAC uses the same policy network as Stage 2 of CM3.
Revised: IAC uses the same policy network as Stage 2 of CM3.

Original: The value function of IAC feeds o n self ,T to a convolutional layer with 6 ﬁlters of size 3x3 and stride 1x1, which is ﬂattened and concatenated with o n self ,V and goal g n .
Revised: The value function of IAC feeds o n self ,T to a convolutional layer with 6 ﬁlters of size 3x3 and stride 1x1, which is ﬂattened and concatenated with o n self ,V and goal g n .

Original: The concatenation is connected to a layer with 256 units, then to a layer hwith 256 units, then to a single linear output unit.
Revised: The concatenation is connected to a layer with 256 units, then to a layer hwith 256 units, then to a single linear output unit.

Original: o n others is connected to a layer with 32 units, then to the layer
Revised: o n others is connected to a layer with 32 units, then to the layer

Original: h 2 .
Revised: h 2 .

Original: COMA.
Revised: COMA.

Original: COMA uses the same policy network as Stage 2 of CM3.
Revised: COMA uses the same policy network as Stage 2 of CM3.

Original: The global Q ( s, ( a n , a − n )) function of COMA is deﬁned as follows for each agent n .
Revised: The global Q ( s, ( a n , a − n )) function of COMA is deﬁned as follows for each agent n .

Original: Tensor part of global state s T is given to a convolutional layer with 4 ﬁlters of size 3x5 and stride 1x1.
Revised: Tensor part of global state s T is given to a convolutional layer with 4 ﬁlters of size 3x5 and stride 1x1.

Original: Tensor part of agent n ’s observation o n self ,T is given to a convolutional layer with 6 ﬁlters of size 3x3 and stride 1x1.
Revised: Tensor part of agent n ’s observation o n self ,T is given to a convolutional layer with 6 ﬁlters of size 3x3 and stride 1x1.

Original: Outputs of both convolutional layers are ﬂattened, then concatenated with s V , all other agents’ actions a − n , agent n ’s goal g n , other agents’ goals g − n , agent n ’s label vector, and agent n ’s vector observation o n self ,V .
Revised: Outputs of both convolutional layers are ﬂattened, then concatenated with s V , all other agents’ actions a − n , agent n ’s goal g n , other agents’ goals g − n , agent n ’s label vector, and agent n ’s vector observation o n self ,V .

Original: The concatenation is passed through two layers with 256 units each, then to a linear output layer withunits.
Revised: The concatenation is passed through two layers with 256 units each, then to a linear output layer withunits.

Original: QMIX.
Revised: QMIX.

Original: Individual value functions are deﬁned as: o n self ,T is passed through the same convolutional layer as above, connected to hidden layer with 32 units, then concatenated with o n self ,V , a nt − 1 , and g n .
Revised: Individual value functions are deﬁned as: o n self ,T is passed through the same convolutional layer as above, connected to hidden layer with 32 units, then concatenated with o n self ,V , a nt − 1 , and g n .

Original: This is connected to layer h 2 with 64 units.
Revised: This is connected to layer h 2 with 64 units.

Original: o n others is connected to a layer with 64 units then connectd to h 2 .
Revised: o n others is connected to a layer with 64 units then connectd to h 2 .

Original: h 2 is fully-connected to an output layer.
Revised: h 2 is fully-connected to an output layer.

Original: The mixing network feeds s T into the same convolutional network as above and follows the exact architecture of Rashid et al.
Revised: The mixing network feeds s T into the same convolutional network as above and follows the exact architecture of Rashid et al.

Original: (2018) with embedding dimension 128.
Revised: (2018) with embedding dimension 128.

Original: I P ARAMETERS
Revised: I P ARAMETERS

Original: We used the Adam optimizer in Tensorﬂow with hyperparameters in Tables 3 to 5.
Revised: We used the Adam optimizer in Tensorﬂow with hyperparameters in Tables 3 to 5.

Original: (cid:15) div is used to compute the exploration decrement (cid:15)
Revised: (cid:15) div is used to compute the exploration decrement (cid:15)

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] start 0. (cid:15) end 0.

Original: [Table]
Revised: (cid:15) step 2e 1e 4e 4e 1e 1e 4e Replay buffer 1e 2e 2e 2e 2e 2e 2e Minibatch size Steps

Original: [Table]
Revised: [Table]

Original: N
Revised: train N/A N/A Episodes

Original: [Table]
Revised: [Table]

Revised: train

Original: N/A
Revised: N/A

Revised: N/A

Revised: N/A

Revised: N/A

Original: [Table]
Revised: N/A Learning rate π 1e1e- 1e1e- 1e1e- N/A Learning rate Q 1e1e- 1e1e-

Original: N/A 1e
Revised: N/A 1e1e-

Original: [Table]
Revised: Learning rate V N/A N/A 1eN/A 1eN/A N/A

Revised: Epochs

Revised: N/A

Revised: N/A

Revised: N/A

Original:  [Table]
Revised: N/A N/A [Table] [Table]

Revised: COMA

Original: [Table] 
Revised: [Table] [Table]

Revised: (cid:15)

Original: [Table]
Revised: [Table]

Original:  1e 1e 1e 2e 1e 1e
Revised: 5e 1e 1e 1e 2e 1e 1e

Original: Replay buffer 1e 1e 1e 1e 1e 1e 1e
Revised: Replay buffer 1e 1e 1e 1e 1e 1e 1e

Original: Minibatch size
Revised: Minibatch size

Original: Steps per train
Revised: Steps per train

Original: N/A
Revised: N/A

Original: N/A N/A
Revised: N/A N/A

Original: Episodes per train
Revised: Episodes per train

Original: N/A
Revised: N/A

Original: N/A N/A
Revised: N/A N/A

Original: N/A
Revised: N/A

Original: Learning rate π 1e1e- 1e1e- 1e1e-
Revised: Learning rate π 1e1e- 1e1e- 1e1e-

Original: N/A
Revised: N/A

Original: Learning rate Q 1e1e- 1e1e-
Revised: Learning rate Q 1e1e- 1e1e-

Original: N/A 1e1e-
Revised: N/A 1e1e-

Original: Learning rate V
Revised: Learning rate V

Original: N/A
Revised: N/A

Original: N/A 1eN/A 1eN/A
Revised: N/A 1eN/A 1eN/A

Original: N/A
Revised: N/A

Original: Epochs
Revised: Epochs

Original: N/A N/A
Revised: N/A N/A

Original: N/A N/A
Revised: N/A N/A

Original: Max env steps
Revised: Max env steps

Original: J S TAGE
Revised: J S TAGE

Original: The Stage 1 functions Q 1 and π 1 for a single agent are trained with the N = 1 equivalents of (4) and (5): 
Revised: The Stage 1 functions Q 1 and π 1 for a single agent are trained with the N = 1 equivalents of (4) and (5): [Equation]

Original: [Equation]
Revised: [Equation]

Original: Stage 1 training curves for all three experimental domains are shown in Figure
Revised: Stage 1 training curves for all three experimental domains are shown in Figure

Original: [Figure]
Revised: [Figure]