Original: S ELF -O RGANIZED P OLYNOMIAL -T IME C OORDINATION G RAPHS
Revised: S ELF -O RGANIZED P OLYNOMIAL -T IME C OORDINATION G RAPHS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Coordination graph is a promising approach to model agent collaboration in multiagent reinforcement learning.
Revised: Coordination graph is a promising approach to model agent collaboration in multiagent reinforcement learning.

Original: It factorizes a large multi-agent system into a suite of overlapping groups that represent the underlying coordination dependencies.
Revised: It factorizes a large multi-agent system into a suite of overlapping groups that represent the underlying coordination dependencies.

Original: One critical challenge in this paradigm is the complexity of computing maximumvalue actions for a graph-based value factorization.
Revised: One critical challenge in this paradigm is the complexity of computing maximumvalue actions for a graph-based value factorization.

Original: It refers to the decentralized constraint optimization problem (DCOP), which and whose constant-ratio approximation are NP-hard problems.
Revised: It refers to the decentralized constraint optimization problem (DCOP), which and whose constant-ratio approximation are NP-hard problems.

Original: To bypass this fundamental hardness, this paper proposes a novel method, named Self-Organized Polynomial-time Coordination
Revised: To bypass this fundamental hardness, this paper proposes a novel method, named Self-Organized Polynomial-time Coordination

Original: Graphs (SOP-CG), which uses structured graph classes to guarantee the optimality of the induced DCOPs with sufﬁcient function expressiveness.
Revised: Graphs (SOP-CG), which uses structured graph classes to guarantee the optimality of the induced DCOPs with sufficient function expressiveness.

Original: We extend the graph topology to be state-dependent, formulate the graph selection as an imaginary agent, and ﬁnally derive an end-to-end learning paradigm from the uniﬁed
Revised: We extend the graph topology to be state-dependent, formulate the graph selection as an imaginary agent, and finally derive an end-to-end learning paradigm from the unified

Original: Bellman optimality equation.
Revised: Bellman optimality equation.

Original: In experiments, we show that our approach learns interpretable graph topologies, induces effective coordination, and improves performance across a variety of cooperative multi-agent tasks.
Revised: In experiments, we show that our approach learns interpretable graph topologies, induces effective coordination, and improves performance across a variety of cooperative multi-agent tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Cooperative multi-agent reinforcement learning (MARL) is a promising approach to a variety of real-world applications, such as sensor networks (Zhang & Lesser, 2011; Ye et al., 2015), trafﬁc light control (Van der Pol & Oliehoek, 2016), and multi-robot formation (Alonso-Mora et al., 2017).
Revised: Cooperative multi-agent reinforcement learning (MARL) is a promising approach to a variety of real-world applications, such as sensor networks (Zhang & Lesser, 2011; Ye et al., 2015), traffic light control (Van der Pol & Oliehoek, 2016), and multi-robot formation (Alonso-Mora et al., 2017).

Original: One of the long-lasting challenges of cooperative MARL is how to organize coordination for a large multi-agent system.
Revised: One of the long-lasting challenges of cooperative MARL is how to organize coordination for a large multi-agent system.

Original: Coordination graph (Guestrin et al., 2001) is a classical approach to represent coordination relations in the reinforcement learning framework.
Revised: Coordination graph (Guestrin et al., 2001) is a classical approach to represent coordination relations in the reinforcement learning framework.

Original: It decomposes a multi-agent system into a suite of overlapping factors.
Revised: It decomposes a multi-agent system into a suite of overlapping factors.

Original: Each factor is a hyper-edge covering a subset of agents that may involve coordinated behaviors.
Revised: Each factor is a hyper-edge covering a subset of agents that may involve coordinated behaviors.

Original: More speciﬁcally, the joint value function of the multi-agent system is factorized to the summation of local value functions based on factors or edges.
Revised: More specifically, the joint value function of the multi-agent system is factorized to the summation of local value functions based on factors or edges.

Original: Such a graph structure is a particular type of inductive bias that encodes task-speciﬁc coordination dependencies.
Revised: Such a graph structure is a particular type of inductive bias that encodes task-specific coordination dependencies.

Original: It is known that conducting appropriate inductive bias can both reduce the model complexity and improve the generalizability (Battaglia et al., 2018).
Revised: It is known that conducting appropriate inductive bias can both reduce the model complexity and improve the generalizability (Battaglia et al., 2018).

Original: [Figure]
Revised: [Figure]

Original: One fundamental problem for coordination graphs is the trade-off between the representational capacity of value functions and the computational complexity of policy execution.
Revised: One fundamental problem for coordination graphs is the trade-off between the representational capacity of value functions and the computational complexity of policy execution.

Original: To obtain value functions with high expressiveness, an advanced approach, deep coordination graphs (DCG) (B¨ohmer et al., 2020), considers a static complete graph connecting all pairs of agents.
Revised: To obtain value functions with high expressiveness, an advanced approach, deep coordination graphs (DCG) (B¨ohmer et al., 2020), considers a static complete graph connecting all pairs of agents.

Original: This graph structure has high representational capacity in terms of function expressiveness but raises a challenge for computation in the execution phase.
Revised: This graph structure has high representational capacity in terms of function expressiveness but raises a challenge for computation in the execution phase.

Original: The greedy action selection over a coordination graph can be formalized to a decentralized constraint optimization problem (DCOP), ﬁnding the maximum-value joint actions (Guestrin et al., 2001; Zhang & Lesser, 2013;
Revised: The greedy action selection over a coordination graph can be formalized to a decentralized constraint optimization problem (DCOP), finding the maximum-value joint actions (Guestrin et al., 2001; Zhang & Lesser, 2013;

Original: B¨ohmer et al., 2020).
Revised: B¨ohmer et al., 2020).

Original: Note that the DCOP and its any constant-ratio approximation are NP-hard problems, especially for the complete graph used in DCG (Dagum & Luby,
Revised: Note that the DCOP and its any constant-ratio approximation are NP-hard problems, especially for the complete graph used in DCG (Dagum & Luby,

Original: 1993; Park & Darwiche, 2004).
Revised: 1993; Park & Darwiche, 2004).

Original: From the perspective of theoretical computer science, any scalable heuristic algorithm (e.g., max-sum algorithm (Pearl, 1988)) for DCOP problem may have an unpredictable gap with the optimal solution.
Revised: From the perspective of theoretical computer science, any scalable heuristic algorithm (e.g., max-sum algorithm (Pearl, 1988)) for DCOP problem may have an unpredictable gap with the optimal solution.

Original: To visualize this phenomenon, we conduct a motivating example for generating a suite of complete-graph DCOPs with random edge-values.
Revised: To visualize this phenomenon, we conduct a motivating example for generating a suite of complete-graph DCOPs with random edge-values.

Original: More detailed experiment setting are deferred to to Appendix A. We investigate the accuracy and the relative joint
Revised: More detailed experiment setting are deferred to to Appendix A. We investigate the accuracy and the relative joint

Original: Q error of the max-sum algorithm in computing optimal joint actions, which is the default implementation of MARL methods based on coordination graphs (Stranders et al., 2009; Zhang & Lesser, 2013; B¨ohmer et al., 2020).
Revised: Q error of the max-sum algorithm in computing optimal joint actions, which is the default implementation of MARL methods based on coordination graphs (Stranders et al., 2009; Zhang & Lesser, 2013; B¨ohmer et al., 2020).

Original: The results are shown in Figure 1.
Revised: The results are shown in Figure 1.

Original: As the number of agents increases, the accuracy of greedy action selection consistently decreases, and the relative joint Q error between the selected and optimal joint actions increases accordingly.
Revised: As the number of agents increases, the accuracy of greedy action selection consistently decreases, and the relative joint Q error between the selected and optimal joint actions increases accordingly.

Original: How to incorporating polynomial-time complexity guarantee into coordination graphs with sufﬁcient function expressiveness remains an open problem in MARL.
Revised: How to incorporating polynomial-time complexity guarantee into coordination graphs with sufficient function expressiveness remains an open problem in MARL.

Original: In this paper, we study a dilemma in coordination graphs, i.e., expressive graph structures lead to computationally intractable DCOPs but concise graph structures are in lack of function expressiveness.
Revised: In this paper, we aim to overcome a dilemma in coordination graphs, i.e., expressive graph structures lead to computationally intractable DCOPs but concise graph structures are in lack of function expressiveness.

Original: To tackle this problem, we propose a novel graph-based MARL method for multi-agent coordination, named Self-Organized Polynomial-time Coordination Graphs (SOP-CG), that  utilizes structured graph topologies with polynomial-time guarantees for DCOPs and  extends their representation expressiveness through a dynamic graph organization mechanism. 
Revised: To tackle this problem, we propose a novel graph-based MARL method for multiagent coordination, named Self-Organized Polynomial-time Coordination Graphs (SOP-CG), that (1) utilizes structured graph topologies with polynomial-time guarantees for DCOPs and (2) extends their representation expressiveness through a dynamic graph organization mechanism. These two characteristics open up a new family of MARL algorithms.

Original: More speciﬁcally, the topology of coordination graph G can depend on the environment state.
Revised: More specifically, we first construct graph classes guaranteeing polynomial-time DCOPs and then conduct a state-dependent graph selection mechanism.

Original: Such a dynamic factorization structure can be self-organized, which is a major characteristic of our proposed approach.
Revised: Such a dynamic factorization structure can be self-organized.

Original: We introduce an imaginary coordinator agent, whose action is to select a graph topology in a given class at each time step.
Revised: We introduce an imaginary coordinator agent, whose action is to select a graph topology in a given class at each time step.

Original: SOP-CG incorporates this imaginary agent into the Bellman optimality equation supporting end-to-end learning.
Revised: SOP-CG incorporates this imaginary agent into the Bellman optimality equation supporting end-to-end learning.

Original: Figure 2 illustrates the basic idea of SOP-CG.
Revised: Figure 2 illustrates the basic idea of SOP-CG.

Original: We integrate the search of graph structures into the MARL paradigm.
Revised: We integrate the search of graph structures into the MARL paradigm.

Original: The utilities of different graph topologies are learned along with the RL objective.
Revised: The utilities of different graph topologies are learned along with the RL objective.

Original: As the graph utilities are updated through agent-environment interactions, the coordination graph will evolve to the topology characterizing the coordination dependencies among agents.
Revised: As the graph utilities are updated through agent-environment interactions, the coordination graph will evolve to the topology characterizing the coordination dependencies.

Original: We evaluate the performance of SOP-CG in both grid-world and simulated physical environments.
Revised: We evaluate the performance of SOP-CG in both grid-world and simulated physical environments.

Original: Empirical results on challenging tasks demonstrate that SOP-CG outperforms state-of-the-art baselines.
Revised: Empirical results on challenging tasks demonstrate that SOP-CG outperforms state-of-the-art baselines.

Original: By extensive ablation studies, we verify that harnessing speciﬁc structures with optimality guarantee of the induced DCOPs improves the sample quality and the accuracy of the one-step TD target.
Revised: By extensive ablation studies, we verify that harnessing specific structures with optimality guarantee of the induced DCOPs improves the sample quality and the accuracy of the one-step TD target.

Original: Furthermore, we show that SOP-CG can learn interpretable and context-dependent coordination graphs, which induces effective and dynamic coordination among agents.
Revised: Furthermore, we show that SOP-CG can learn interpretable and context-dependent coordination graphs, which induces effective and dynamic coordination among agents.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-agent reinforcement learning (OroojlooyJadid & Hajinezhad, 2019) is challenged by the size of joint action space, which grows exponentially with the number of agents.
Revised: Multi-agent reinforcement learning (OroojlooyJadid & Hajinezhad, 2019) is challenged by the size of joint action space, which grows exponentially with the number of agents.

Original: Independent Q-learning
Revised: Independent Q-learning

Original: (Tan, 1993; Foerster et al., 2017) models agents as independent learners, which makes the environment non-stationary in the perspective of each agent.
Revised: (Tan, 1993; Foerster et al., 2017) models agents as independent learners, which makes the environment non-stationary in the perspective of each agent.

Original: An alternative paradigm called centralized training and decentralized execution (CTDE; Kraemer & Banerjee, 2016) is widely used in both policy-based and value-based methods.
Revised: An alternative paradigm called centralized training and decentralized execution (CTDE; Kraemer & Banerjee, 2016) is widely used in both policy-based and value-based methods.

Original: Policy-based multi-agent reinforcement learning methods use a centralized critic to compute gradient for the local actors (Lowe et al., 2017; Foerster et al., 2018; Wen et al., 2019; Wang et al., 2020d).
Revised: Policy-based multi-agent reinforcement learning methods use a centralized critic to compute gradient for the local actors (Lowe et al., 2017; Foerster et al., 2018; Wen et al., 2019; Wang et al., 2020d).

Original: Value-based methods usually decompose the joint value function into individual value functions under the IGM (individual-global-max) principle, which guarantees the consistency between local action selection and joint action optimization (Sunehag et al., 2018;
Revised: Value-based methods usually decompose the joint value function into individual value functions under the IGM (individual-global-max) principle, which guarantees the consistency between local action selection and joint action optimization (Sunehag et al., 2018;

Original: Rashid et al., 2020b; Son et al., 2019; Wang et al., 2021a; Rashid et al., 2020a).
Revised: Rashid et al., 2020b; Son et al., 2019; Wang et al., 2021a; Rashid et al., 2020a).

Original: Other work also studies this problem from the perspective of agent roles and individuality (Wang et al., 2020a;b; Jiang & Lu, 2021) or communication learning (Singh et al., 2018; Das et al., 2019; Wang et al., 2020c).
Revised: Other work also studies this problem from the perspective of agent roles and individuality (Wang et al., 2020a;b; Jiang & Lu, 2021) or communication learning (Singh et al., 2018; Das et al., 2019; Wang et al., 2020c).

Original: Compared to these methods, our work is built upon graph-based value decomposition, which explicitly models the interaction among agents.
Revised: Compared to these methods, our work is built upon graph-based value decomposition, which explicitly models the interaction among agents.

Original: Coordination graphs are classical technique for planning in multi-agent systems (Guestrin et al., 2001; 2002b).
Revised: Coordination graphs are classical technique for planning in multi-agent systems (Guestrin et al., 2001; 2002b).

Original: They are combined with multi-agent deep reinforcement learning by recent work (Castellini et al., 2019; B¨ohmer et al., 2020; Li et al., 2020; Wang et al., 2021b).
Revised: They are combined with multi-agent deep reinforcement learning by recent work (Castellini et al., 2019; B¨ohmer et al., 2020; Li et al., 2020; Wang et al., 2021b).

Original: Joint action selection on coordination graphs can be modeled as a decentralized constraint optimization problem (DCOP), and previous methods compute approximate solutions by message passing among agents (Pearl, 1988).
Revised: Joint action selection on coordination graphs can be modeled as a decentralized constraint optimization problem (DCOP), and previous methods compute approximate solutions by message passing among agents (Pearl, 1988).

Original: As the work which is most closed to our method, Zhang & Lesser (2013) presents an algorithm based on coordination graph searching.
Revised: As the work which is most closed to our method, Zhang & Lesser (2013) presents an algorithm based on coordination graph searching.

Original: They deﬁne a measurement to quantify the potential loss of the lack of coordination between agents and search for a coordination structure to minimize the communication cost within restricted loss of utilities.
Revised: They define a measurement to quantify the potential loss of the lack of coordination between agents and search for a coordination structure to minimize the communication cost within restricted loss of utilities.

Original: However, their induced DCOPs still remain NP-hard.
Revised: However, their induced DCOPs still remain NP-hard.

Original: In contrast, minimizing communication is not our core motivation, and we aim to use a structured graph class to maintain sufﬁcient function expressiveness when bypassing the computational hardness of large-scale DCOPs (Dagum & Luby, 1993; Park & Darwiche, 2004).
Revised: In contrast, minimizing communication is not our core motivation, and we aim to use a structured graph class to maintain sufficient function expressiveness when bypassing the computational hardness of large-scale DCOPs (Dagum & Luby, 1993; Park & Darwiche, 2004).

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this paper, we consider about fully cooperative multi-agent tasks that can be modelled as a DecPOMDP (Oliehoek et al., 2016) deﬁned as M = (cid:104) D, S, { A i } ni =1 , T, { O i } ni =1 , { σ i } ni =1 , R, h, b 0 , γ (cid:105) , where D = { 1 , . . . , n } is the set of n agents, S is a set of states, h is the horizon of the environment, γ ∈ [0 , 1) is the discount factor, and b 0 ∈ ∆( S ) denotes the initial state distribution.
Revised: In this paper, we consider about fully cooperative multi-agent tasks that can be modelled as a DecPOMDP (Oliehoek et al., 2016) defined as M = ⟨ D, S, { A i } ni =1 , T, { O i } ni =1 , { σ i } ni =1 , R, h, b 0 , γ ⟩ , where D = { 1 , . . . , n } is the set of n agents, S is a set of states, h is the horizon of the environment, γ ∈ [0 , 1) is the discount factor, and b 0 ∈ ∆( S ) denotes the initial state distribution.

Original: At each stage t , each agent i takes an action a i ∈
Revised: At each stage t , each agent i takes an action a i ∈

Original: A i and forms the joint action
Revised: A i and forms the joint action

Original: a = ( a 1 . . .
Revised: a = ( a 1 . . .

Original: , a n ) , which leads to a next state s (cid:48) according to the transition function T ( s (cid:48) | s, a ) and an immediate reward
Revised: , a n ) , which leads to a next state s ′ according to the transition function T ( s ′ | s, a ) and an immediate reward

Original: R ( s, a ) shared by all agents.
Revised: R ( s, a ) shared by all agents.

Original: Each agent i observes the state only partially by drawing observations
Revised: Each agent i observes the state only partially by drawing observations

Original: o i ∈
Revised: o i ∈

Original: O i , according to σ i .
Revised: O i , according to σ i .

Original: The joint history of agent i ’s
Revised: The joint history of agent i ’s

Original: observations o i,t and actions a i,t is denoted as τ i,t =
Revised: observations o i,t and actions a i,t is denoted as τ i,t =

Original: ( o i, 0 , a i, 0 , . . .
Revised: ( o i, 0 , a i, 0 , . . .

Original: , o i,t − 1 , a i,t − 1 , o i,t ) ∈ (cid:0)
Revised: , o i,t − 1 , a i,t − 1 , o i,t ) ∈ (cid:0)

Original: O i × A i (cid:1) t
Revised: O i × A i (cid:1) t

Original: × O i .
Revised: × O i .

Original: Deep Q-Learning.
Revised: Deep Q-Learning.

Original: Q-learning algorithms is a well-known algorithm to ﬁnd the optimal joint action-value function Q ∗ ( s, a ) =
Revised: Q-learning algorithms is a well-known algorithm to find the optimal joint action-value function Q ∗ ( s, a ) =

Original: r ( s, a )+ γ E s (cid:48)
Revised: r ( s, a )+ γ E s ′

Original: [max a (cid:48) Q ∗ ( s (cid:48) , a (cid:48) )] .
Revised: [max a ′ Q ∗ ( s ′ , a ′ )] .

Original: Deep Q-learning approximates the action-value function with a deep neural network with parameters θ .
Revised: Deep Q-learning approximates the action-value function with a deep neural network with parameters θ .

Original: In Multi-agent Q-learning algorithms (Sunehag et al., 2018; Rashid et al., 2020b; Son et al., 2019; Wang et al., 2021a), a replay memory D is used to store the transition tuple ( τ , a , r, τ (cid:48) ) , where r is the immediate reward when taking action a at joint action-observation history τ with a transition to τ (cid:48) .
Revised: In Multi-agent Q-learning algorithms (Sunehag et al., 2018; Rashid et al., 2020b; Son et al., 2019; Wang et al., 2021a), a replay memory D is used to store the transition tuple ( τ , a , r, τ ′ ) , where r is the immediate reward when taking action a at joint action-observation history τ with a transition to τ ′ .

Original: Q ( τ , a ; θ ) is used in place of Q ( s, a ; θ ) , because of partial observability.
Revised: Q ( τ , a ; θ ) is used in place of Q ( s, a ; θ ) , because of partial observability.

Original: Hence, parameters θ are learnt by minimizing the following expect TD error:
Revised: Hence, parameters θ are learnt by minimizing the following expect TD error:

Original: [Equation]
Revised: [Equation]

Original: where V ( τ (cid:48) ; θ − )
Revised: where V ( τ ′ ; θ − )

Original: = max a (cid:48) Q ( τ (cid:48) , a (cid:48) ; θ − ) is the one-step expected future return of the TD target and θ − are the parameters of the target network, which will be periodically updated with θ .
Revised: = max a ′ Q ( τ ′ , a ′ ; θ − ) is the one-step expected future return of the TD target and θ − are the parameters of the target network, which will be periodically updated with θ .

Original: Coordination graphs.
Revised: Coordination graphs.

Original: An undirect coordination graph (CG, Guestrin et al.
Revised: An undirect coordination graph (CG, Guestrin et al.

Original: G = (cid:104)V , E(cid:105) contains vertex v i ∈ V for each agent 1 ≤ i ≤ n and a set of (hyper-)edges in E ⊆ 2 V which represents coordination dependencies among agents.
Revised: G = ⟨V , E⟩ contains vertex v i ∈ V for each agent 1 ≤ i ≤ n and a set of (hyper-)edges in E ⊆ 2 V which represents coordination dependencies among agents.

Original: Prior work considers higher order coordination where the edges depend on actions of several agents (Guestrin et al., 2002a;
Revised: Prior work considers higher order coordination where the edges depend on actions of several agents (Guestrin et al., 2002a;

Original: Kok & Vlassis, 2006;
Revised: Kok & Vlassis, 2006;

Original: Guestrin et al., 2002b).
Revised: Guestrin et al., 2002b).

Original: Such a coordination graph induces a factorization of global Q function:
Revised: Such a coordination graph induces a factorization of global Q function:

Original: [Equation]
Revised: [Equation]

Original:  The special case that E is an empty set yields VDN, but each additional edge enables the value representation of the joint actions of a pair of agents and can thus help to avoid relativeovergeneralization (B¨ohmer et al., 2020).
Revised: , where f i represents the individual utility of agent i and f e specifies the payoff contribution for the actions of the agents connected by the (hyper-)edge e , so that the global optimal solution can be found through maximizing this joint value. The special case that E is an empty set yields VDN, but each additional edge enables the value representation of the joint actions of a pair of agents and can thus help to avoid relative-overgeneralization (B¨ohmer et al., 2020).

Original: In many coordination graph learning works (Zhang & Lesser, 2013; B¨ohmer et al., 2020; Wang et al., 2021b), the hyper-edges are simpliﬁed into pairwise edges.
Revised: In many coordination graph learning works (Zhang & Lesser, 2013; B¨ohmer et al., 2020; Wang et al., 2021b), the hyper-edges are simplified into pairwise edges.

Original: The graph is usually considered to be speciﬁed before training.
Revised: The graph is usually considered to be specified before training.

Original: Guestrin et al.
Revised: Guestrin et al.

Original: (2002b)
Revised: (2002b)

Original: and Zhang & Lesser (2013) suggest that the graph could also depend on states, which means each state can have its own unique CG.
Revised: and Zhang & Lesser (2013) suggest that the graph could also depend on states, which means each state can have its own unique CG.

Original:  S ELF -O RGANIZED P OLYNOMIAL -T IME C OORDINATION G RAPHS
Revised:  S ELF -O RGANIZED P OLYNOMIAL -T IME C OORDINATION G RAPHS

Original: A common method for multi-agent reinforcement learning is to decompose the joint value function into a linear combination of local value functions, each of which conditions on actions of a subset of agents (Guestrin et al., 2001; B ¨ ohmer et al., 2020).
Revised: A common method for multi-agent reinforcement learning is to decompose the joint value function into a linear combination of local value functions, each of which conditions on actions of a subset of agents (Guestrin et al., 2001; B ¨ ohmer et al., 2020).

Original: With this paradigm, computing joint action with maximum value can be modeled as a distributed constraint optimization problem (DCOP).
Revised: With this paradigm, computing joint action with maximum value can be modeled as a distributed constraint optimization problem (DCOP).

Original: General
Revised: General

Original: DCOP and any constant-ratio approximation have been proved to be NP-hard (Dagum & Luby, 1993; Park & Darwiche, 2004).
Revised: DCOP and any constant-ratio approximation have been proved to be NP-hard (Dagum & Luby, 1993; Park & Darwiche, 2004).

Original: Previous work adopts a variety of heuristic algorithms for action selection (Kok & Vlassis, 2006; Wang et al., 2021b).
Revised: Previous work adopts a variety of heuristic algorithms for action selection (Kok & Vlassis, 2006; Wang et al., 2021b).

Original: The imperfection of heuristics may lead to two side effects: (a) collecting bad samples and (b) errors in constructing one-step TD target, which hurt the learning performance.
Revised: The imperfection of heuristics may lead to two side effects: (a) collecting bad samples and (b) errors in constructing one-step TD target, which hurt the learning performance.

Original: To address these issues, we investigate polynomial-time coordination graphs , in which the induced
Revised: To address these issues, we investigate polynomial-time coordination graphs , in which the induced

Original: DCOPs can be solved in polynomial time.
Revised: DCOPs can be solved in polynomial time.

Original: We present a novel algorithm called Self-Organized
Revised: We present a novel algorithm called Self-Organized

Original: Polynomial-time Coordination Graphs (SOP-CG), that utilizes a class of polynomial-time coordination graphs to construct a dynamic and state-dependent topology.
Revised: Polynomial-time Coordination Graphs (SOP-CG), that utilizes a class of polynomial-time coordination graphs to construct a dynamic and state-dependent topology.

Original: In Section 4.1, we introduce our value factorization upon coordination graphs, which can hold the optimality guarantee of the induced DCOPs without signiﬁcantly sacriﬁcing the representational capacity by using a bottom-up design.
Revised: In Section 4.1, we introduce our value factorization upon coordination graphs, which can hold the optimality guarantee of the induced DCOPs without significantly sacrificing the representational capacity by using a bottom-up design.

Original: To enable efﬁcient topology self-organization, in Section 4.2, we unify the graph selection into the Q-learning framework by formulating the graphs as actions of an imaginary coordinator agent, and depict the whole framework of our algorithm.
Revised: To enable efficient topology self-organization, in Section 4.2, we unify the graph selection into the Q-learning framework by formulating the graphs as actions of an imaginary coordinator agent, and depict the whole framework of our algorithm.

Original: In Section 4.3, we further propose two polynomial-time graph class instances and discuss their theoretical advantages in SOP-CG.
Revised: In Section 4.3, we further propose two polynomial-time graph class instances and discuss their theoretical advantages in SOP-CG.

Original: 4. V ALUE F ACTORIZATION ON P OLYNOMIAL -T IME C OORDINATION G RAPHS
Revised: 4. V ALUE F ACTORIZATION ON P OLYNOMIAL -T IME C OORDINATION G RAPHS

Original: To establish an algorithm with optimality guarantee in action selection, our approach ﬁrst models coordination relations upon the graph-based value factorization speciﬁed by deep coordination graphs (DCG; B ¨ ohmer et al., 2020), in which the joint value function of the multi-agent system is factorized to the summation of individual utility functions
Revised: To establish an algorithm with optimality guarantee in action selection, our approach first models coordination relations upon the graph-based value factorization specified by deep coordination graphs (DCG; B¨ohmer et al., 2020), in which the joint value function of the multi-agent system is factorized to the summation of individual utility functions

Original: q i and pairwise payoff functions q ij as follows:
Revised: q i and pairwise payoff functions q ij as follows:

Original: [Equation]
Revised: [Equation]

Original: where the coordination graph G is represented by a set of undirected edges.
Revised: where the coordination graph G is represented by a set of undirected edges.

Original: The induced DCOP can be solved in polynomial time if G lies in G uac , the set of undirected acyclic graphs (Fioretto et al., 2018).
Revised: The induced DCOP can be solved in polynomial time if G lies in G uac , the set of undirected acyclic graphs (Fioretto et al., 2018).

Original: However, an undirected acyclic graph can contain at most n − 1 edges in an environment with n agents.
Revised: However, an undirected acyclic graph can contain at most n − 1 edges in an environment with n agents.

Original: For long-horizon complex multi-agent tasks, such a static sparse graph usually cannot represent all potential coordination dependencies.
Revised: For long-horizon complex multi-agent tasks, such a static sparse graph usually cannot represent all potential coordination dependencies.

Original: To alleviate this problem, our approach allows the graph topology to dynamically evolve through the transitions of environment status.
Revised: To alleviate this problem, our approach allows the graph topology to dynamically evolve through the transitions of environment status.

Original: Given different environmental states, the joint values can be factorized with different coordination graphs chosen from a predeﬁned graph class G ⊆ G uac .
Revised: Given different environmental states, the joint values can be factorized with different coordination graphs chosen from a predefined graph class G ⊆ G uac .

Original: This design is based on an assumption that, although a long-horizon task cannot be characterized by a static sparse coordination graph, the coordination relations at each single-time step are sparse and manageable.
Revised: This design is based on an assumption that, although a long-horizon task cannot be characterized by a static sparse coordination graph, the coordination relations at each single-time step are sparse and manageable.

Original: The employment of this graph class ﬁlls the lack of representational capacity as well as maintains a polynomial complexity of the induced DCOPs.
Revised: The employment of this graph class fills the lack of representational capacity as well as maintains a polynomial complexity of the induced DCOPs.

Original: 4. L EARNING S ELF -O RGANIZED T OPOLOGY WITH A N E XPLICIT C OORDINATOR
Revised: 4. L EARNING S ELF -O RGANIZED T OPOLOGY WITH A N E XPLICIT C OORDINATOR

Original: Now we present a novel framework to render the self-organized coordination graph.
Revised: Now we present a novel framework to render the self-organized coordination graph.

Original: We introduce an imaginary coordinator agent whose action space refers to the selection of graph topologies, aiming to select a proper graph for minimizing the loss within restricted coordination.
Revised: We introduce an imaginary coordinator agent whose action space refers to the selection of graph topologies, aiming to select a proper graph for minimizing the loss within restricted coordination.

Original: When using the graph-based value factorization stated in Eq.
Revised: When using the graph-based value factorization stated in Eq.

Original: (3), the graph topology G can be regarded as an input of joint value function Q ( τ ( t ) , a ; G ) .
Revised: (3), the graph topology G can be regarded as an input of joint value function Q ( τ ( t ) , a ; G ) .

Original: The objective of the coordinator agent is to maximize the joint value function over the speciﬁc graph class, which is a dual problem for ﬁnding a graph to minimize the loss within the different restricted coordination structures (Zhang & Lesser, 2013).
Revised: The objective of the coordinator agent is to maximize the joint value function over the specific graph class, which is a dual problem for finding a graph to minimize the loss within the different restricted coordination structures (Zhang & Lesser, 2013).

Original: Under this interpretation, we can integrate the selection of graph topologies into the trial-and-error loop of reinforcement learning.
Revised: Under this interpretation, we can integrate the selection of graph topologies into the trial-and-error loop of reinforcement learning.

Original: We handle the imaginary coordinator as a usual agent in multi-agent
Revised: We handle the imaginary coordinator as a usual agent in multi-agent

Original: Q-learning framework and rewrite the joint action as a cg
Revised: Q-learning framework and rewrite the joint action as a cg

Original: = ( a 1 , · · · , a n , G ) .
Revised: = ( a 1 , · · · , a n , G ) .

Original: Execution.
Revised: Execution.

Original: Formally, at time step t , greedy action selection indicates the following joint action:
Revised: Formally, at time step t , greedy action selection indicates the following joint action:

Original: [Equation]
Revised: [Equation]

Original: Hence the action of coordinator agent G ( t ) is naturally the corresponding component in Eq.
Revised: Hence the action of coordinator agent G ( t ) is naturally the corresponding component in Eq.

Original: [Equation]
Revised: [Equation]

Original: After determining the graph topology G ( t ) , the agents can choose their individual actions to jointly maximize the value function Q ( τ ( t ) , a ; G ( t ) ) upon the selected topology.
Revised: After determining the graph topology G ( t ) , the agents can choose their individual actions to jointly maximize the value function Q ( τ ( t ) , a ; G ( t ) ) upon the selected topology.

Original: Training.
Revised: Training.

Original: With the imaginary coordinator, we can re-formulate the Bellman optimality equation and maximizing the future value over the coordinator agent’s action:
Revised: With the imaginary coordinator, we can re-formulate the Bellman optimality equation and maximizing the future value over the coordinator agent’s action:

Original: [Equation]
Revised: [Equation]

Original: Since the graph selection is a part of agent action, the associative value Q ( τ , a ; G ) of graph G can be updated through temporal difference learning.
Revised: Since the graph selection is a part of agent action, the associative value Q ( τ , a ; G ) of graph G can be updated through temporal difference learning.

Original: The network parameters θ can be updated by minimizing the standard Q-learning TD loss:
Revised: The network parameters θ can be updated by minimizing the standard Q-learning TD loss:

Original: [Equation]
Revised: [Equation]

Original: where y cg = max ( a (cid:48) ,G (cid:48) )
Revised: where y cg = max ( a ′ ,G ′ )

Original: Q ( τ (cid:48) , a (cid:48) ; G (cid:48) ; θ − ) is the one-step TD target and θ − are the parameters of target network.
Revised: Q ( τ ′ , a ′ ; G ′ ; θ − ) is the one-step TD target and θ − are the parameters of target network.

Original: G ( t ) is regarded as part of the transition and is stored in the replay buffer together with the realistic agent actions.
Revised: G ( t ) is regarded as part of the transition and is stored in the replay buffer together with the realistic agent actions.

Original: We include more implementation details of temporal difference learning in Appendix C.
Revised: We include more implementation details of temporal difference learning in Appendix C.

Original: The overall algorithm of our approach is summarized in Algorithm 1.
Revised: The overall algorithm of our approach is summarized in Algorithm 1.

Original: The graph selection step at line 6 is the main characteristic of our method, which differs from the static graph representation used by prior work (Guestrin et al., 2002a; Kok & Vlassis, 2006; B¨ohmer et al., 2020).
Revised: The graph selection step at line 6 is the main characteristic of our method, which differs from the static graph representation used by prior work (Guestrin et al., 2002a; Kok & Vlassis, 2006; B¨ohmer et al., 2020).

Original: To support the self-organization of coordination graphs, the graph selection step raises a new computation demand, i.e., we need to ﬁnd the best candidate from the graph class G , which will be another non-trivial combinatorial optimization problem.
Revised: To support the self-organization of coordination graphs, the graph selection step raises a new computation demand, i.e., we need to find the best candidate from the graph class G , which will be another non-trivial combinatorial optimization problem.

Original: We will show that, by designing a proper graph class G , we can maintain both computational efﬁciency and function representational capacity in the next subsection.
Revised: We will show that, by designing a proper graph class G , we can maintain both computational efficiency and function representational capacity in the next subsection.

Original: 4. H ARNESSING S TRUCTURED G RAPH C LASSES FOR E FFICIENT C OORDINATION
Revised: 4. H ARNESSING S TRUCTURED G RAPH C LASSES FOR E FFICIENT C OORDINATION

Original: To achieve the efﬁcient graph search over the polynomial-time class instances, we investigate the following structures for the graph class G , which are subsets of G uac :
Revised: To achieve the efficient graph search over the polynomial-time class instances, we investigate the following structures for the graph class G , which are subsets of G uac :

Original: • Pairwise grouping G P .
Revised: • Pairwise grouping G P .

Original: This class is introduced by Castellini et al.
Revised: This class is introduced by Castellini et al.

Original: n agents are partitioned to (cid:98) n/ 2 (cid:99) non-overlapping pairwise groups.
Revised: n agents are partitioned to ⌊ n/ 2 ⌋ non-overlapping pairwise groups.

Original: Only pairwise interactions are considered and two agents are connected with an edge if they lie in the same group.
Revised: Only pairwise interactions are considered and two agents are connected with an edge if they lie in the same group.

Original: Require:
Revised: Require:

Original: Predeﬁned graph class G 1: Initialize replay buffer D 2: Initialize network with random weights 3: for episode = 1 , · · · , M do 4:
Revised: Predefined graph class G 1: Initialize replay buffer D 2: Initialize network with random weights 3: for episode = 1 , · · · , M do 4:

Original: Receive initial observation τ (0) 5: for t = 0 , · · · , T do 6:
Revised: Receive initial observation τ (0) 5: for t = 0 , · · · , T do 6:

Original: Choose coordination graph G ( t )
Revised: Choose coordination graph G ( t )

Original: ∈ G according to Eq.
Revised: ∈ G according to Eq.

Original: (5) (cid:46) Section 4.
Revised: (5) ▷ Section 4.

Original: Select actions a ( t ) w.r.t G ( t ) and an (cid:15) -greedy exploration 8:
Revised: Select actions a ( t ) w.r.t G ( t ) and an ϵ -greedy exploration 8:

Original: Execute actions a ( t ) and receive reward r ( t ) and observation o ( t +1) 9:
Revised: Execute actions a ( t ) and receive reward r ( t ) and observation o ( t +1) 9:

Original: Store transition ( τ ( t ) , a ( t ) , r ( t ) , τ ( t +1) ) in D 10: end for 11:
Revised: Store transition ( τ ( t ) , a ( t ) , r ( t ) , τ ( t +1) ) in D 10: end for 11:

Original: Sample random minibatch of transitions from D 12:
Revised: Sample random minibatch of transitions from D 12:

Original: Perform a gradient descent step on loss deﬁned in Eq.
Revised: Perform a gradient descent step on loss defined in Eq.

Original: (7) 13: end for
Revised: (7) 13: end for

Original: • Hierarchical organization G H .
Revised: • Hierarchical organization G H .

Original: Overlapping edges are allowed in this class.
Revised: Overlapping edges are allowed in this class.

Original: In a system with n agents, the graph is organized as a tree with n − 1 edges so that all agents form a connected component.
Revised: In a system with n agents, the graph is organized as a tree with n − 1 edges so that all agents form a connected component.

Original: Even if two agents are not directly connected by an edge, their actions are implicitly coordinated through the path on the tree.
Revised: Even if two agents are not directly connected by an edge, their actions are implicitly coordinated through the path on the tree.

Original: This provides the potential to approximate complex joint value functions.
Revised: This provides the potential to approximate complex joint value functions.

Original: Graph Class
Revised: Graph Class

Original: Select graph G ( t )
Revised: Select graph G ( t )

Original: Select actions on a given graph
Revised: Select actions on a given graph

Original: [Table] 
Revised: [Table] √

Revised: Complete graph

Original: N/A −
Revised: N/A −

Original: G  is actually the set of maximal undirected acyclic graphs.
Revised: G H is actually the set of maximal undirected acyclic graphs.

Original: It is noteworthy that we do not further incorporate the smaller graphs into both classes, since their representational capacity are dominated by existing graphs in the classes.
Revised: It is noteworthy that we do not further incorporate the smaller graphs into both classes, since their representational capacity are dominated by existing graphs in the classes.

Original: We summarize the hardness of computing these problems upon different graph class in Table 1.
Revised: We summarize the hardness of computing these problems upon different graph class in Table 1.

Original: Although the representational capacity of G P is relatively weaker, selecting G ( t ) from this class can be rigorously solved by polynomial-time algorithms (Edmonds, 1965), which highlights the key beneﬁt of G P .
Revised: Although the representational capacity of G P is relatively weaker, selecting G ( t ) from this class can be rigorously solved by polynomial-time algorithms (Edmonds, 1965), which highlights the key benefit of G P .

Original: In comparison, selecting G ( t ) from G H is non-trivial, so we design a greedy algorithm to compute approximated solutions.
Revised: In comparison, selecting G ( t ) from G H is non-trivial, so we design a greedy algorithm to compute approximated solutions.

Original: Note that, although the graph selection of G H may be imprecise, its represented values are formally optimized by temporal difference learning with accurate action selection.
Revised: Note that, although the graph selection of G H may be imprecise, its represented values are formally optimized by temporal difference learning with accurate action selection.

Original: Thus, as we will show in the experiment section, G H can still work as a promising graph class in absence of rigorous graph selection.
Revised: Thus, as we will show in the experiment section, G H can still work as a promising graph class in absence of rigorous graph selection.

Original: Please refer to Appendix B for the implementation details of the induced combinatorial optimization for graph selection.
Revised: Please refer to Appendix B for the implementation details of the induced combinatorial optimization for graph selection.

Original: Remark.
Revised: Remark.

Original: In this paper, SOP-CG trades off the representational capacity of value function with the computational complexity within the polynomial-time boundary.
Revised: In this paper, SOP-CG trades off the representational capacity of value function with the computational complexity within the polynomial-time boundary.

Original: Compared to fully connected graph, our sparse graphs restrict the value function class to a smaller space.
Revised: Compared to fully connected graph, our sparse graphs restrict the value function class to a smaller space.

Original: This polynomial-time graph class enables SOP-CG to achieve accurate greedy action selection using limited message passing along the edges and achieve effective learning in the centralized training process.
Revised: This polynomial-time graph class enables SOP-CG to achieve accurate greedy action selection using limited message passing along the edges and achieve effective learning in the centralized training process.

Original: Moreover, if these graph classes are enough to express the coordination dependencies in a multi-agent task, using such topologies will be more sample-efﬁcient by virtue of the concise function classes (Battaglia et al., 2018), which becomes another potential advantage of our algorithm.
Revised: Moreover, if these graph classes are enough to express the coordination dependencies in a multi-agent task, using such topologies will be more sample-efficient by virtue of the concise function classes (Battaglia et al., 2018), which becomes another potential advantage of our algorithm.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct experiments to answer the following questions: (1) Does the class of polynomial-time coordination graphs improve our performance (see Section 5.1)?
Revised: In this section, we conduct experiments to answer the following questions: (1) Does the class of polynomial-time coordination graphs improve our performance (see Section 5.1)?

Original: (2) How well
Revised: (2) How well

Original: [Figure]
Revised: [Figure]

Original: does SOP-CG perform on complex cooperative multi-agent tasks (see Fig. 5)?
Revised: does SOP-CG perform on complex cooperative multi-agent tasks (see Fig. 5)?

Original: (3) Can SOP-CG extract interpretable dynamic coordination structures (see Fig. 6)?
Revised: (3) Can SOP-CG extract interpretable dynamic coordination structures (see Fig. 6)?

Original: D IDACTIC EXAMPLE
Revised: D IDACTIC EXAMPLE

Original: We ﬁrst study a didactic example to demonstrate two main contributions of this paper: (a) harnessing polynomial-time coordination graphs to obtain optimality guarantee of the induced DCOPs; (b) maintaining function expressiveness by incorporating a class of structured graphs into the Q-learning framework.
Revised: We first study a didactic example to demonstrate two main contributions of this paper: (a) harnessing polynomial-time coordination graphs to obtain optimality guarantee of the induced DCOPs; (b) maintaining function expressiveness by incorporating a class of structured graphs into the Q-learning framework.

Original: We conduct ablation studies on Sensor-network , which is a classic task for testing multi-agent coordination (Lesser et al., 2003; Zhang & Lesser, 2011).
Revised: We conduct ablation studies on Sensor-network , which is a classic task for testing multi-agent coordination (Lesser et al., 2003; Zhang & Lesser, 2011).

Original: We adopt the same task conﬁguration as Wang et al.
Revised: We adopt the same task configuration as Wang et al.

Original: (2021b).
Revised: (2021b).

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 4, 15 sensor agents are arranged as a 3 × 5 matrix while three targets randomly walk in the same grid space.
Revised: 4, 15 sensor agents are arranged as a 3 × 5 matrix while three targets randomly walk in the same grid space.

Original: At each timestep, an agent can choose to scan one of the eight nearby positions (inducing a cost) or perform no operation.
Revised: At each timestep, an agent can choose to scan one of the eight nearby positions (inducing a cost) or perform no operation.

Original: If two or more agents scan the target simultaneously, the target is captured and produces a positive joint reward.
Revised: If two or more agents scan the target simultaneously, the target is captured and produces a positive joint reward.

Original: We compare our algorithm with DCG (B ¨ ohmer et al., 2020) on this game.
Revised: We compare our algorithm with DCG (B ¨ ohmer et al., 2020) on this game.

Original: The default implementation of DCG uses a fully connected coordination graph.
Revised: The default implementation of DCG uses a fully connected coordination graph.

Original: In addition, DCG provides alternative implementations on two static graphs Line (agent i is connected with agent i − 

Revised: In addition, DCG provides alternative implementations on two static graphs Line (agent i is connected with agent i −

Original: and i +1 )
Revised: and i +1 )

Original: and Star (agent 1 is linked with others), which are two special instances of G H .
Revised: and Star (agent 1 is linked with others), which are two special instances of G H .

Original: SOP-CG ( G H ) has absolutely higher function expressiveness than DCG (line) and DCG (star).
Revised: SOP-CG ( G H ) has absolutely higher function expressiveness than DCG (line) and DCG (star).

Original: For the ﬁrst contribution, we verify that the optimality guarantee of the induced DCOP in our algorithm improve the performance.
Revised: For the first contribution, we verify that the optimality guarantee of the induced DCOP in our algorithm improve the performance.

Original: We test these algorithms in both online (Fig. 3a) and ofﬂine learning (Fig.
Revised: We test these algorithms in both online (Fig. 3a) and offline learning (Fig.

Original: 3b) settings.
Revised: 3b) settings.

Original: For fair evaluation, we train DCG (line) and collect 15K experienced episodes throughout the training process to set up the diverse ofﬂine dataset.
Revised: For fair evaluation, we train DCG (line) and collect 15K experienced episodes throughout the training process to set up the diverse offline dataset.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3(ab), SOP-CG outperforms DCG in both settings.
Revised: 3(ab), SOP-CG outperforms DCG in both settings.

Original: In the ofﬂine setting, as the dataset is ﬁxed for both algorithms, the main difference between SOP-CG and DCG lies in the construction of one-step
Revised: In the offline setting, as the dataset is fixed for both algorithms, the main difference between SOP-CG and DCG lies in the construction of one-step

Original: TD target.
Revised: TD target.

Original: Our algorithm is guaranteed to ﬁnd the optimal solution of the induced DCOPs, which must play an important role in reducing the errors in computing TD target.
Revised: Our algorithm is guaranteed to find the optimal solution of the induced DCOPs, which must play an important role in reducing the errors in computing TD target.

Original: We also observe that
Revised: We also observe that

Original: DCG sticks in a bad local optimum with online data collection, but its performance is improved in the ofﬂine setting.
Revised: DCG sticks in a bad local optimum with online data collection, but its performance is improved in the offline setting.

Original: This result indicates that the hardness of greedy action selection causes DCG to collect low-quality samples in the online setting, which may hurt the ﬁnal performance.
Revised: This result indicates that the hardness of greedy action selection causes DCG to collect low-quality samples in the online setting, which may hurt the final performance.

Original: In contrast, our method converges to the same superior value in both settings, highlighting the effectiveness of polynomial-time coordination graphs.
Revised: In contrast, our method converges to the same superior value in both settings, highlighting the effectiveness of polynomial-time coordination graphs.

Original: For the second contribution, we show the importance of learning a state-dependent structure.
Revised: For the second contribution, we show the importance of learning a state-dependent structure.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 3c, the performance of SOP-CG is much better than both DCG (line) and DCG (star) with online data collection.
Revised: 3c, the performance of SOP-CG is much better than both DCG (line) and DCG (star) with online data collection.

Original: A static sparse topology usually cannot express the coordination dependencies in all steps, so these methods could not work well even if they are built upon polynomial-time coordination graphs.
Revised: A static sparse topology usually cannot express the coordination dependencies in all steps, so these methods could not work well even if they are built upon polynomial-time coordination graphs.

Original: 5. P ERFORMANCE C OMPARISON
Revised: 5. P ERFORMANCE C OMPARISON

Original: To further demonstrate the effectiveness of our algorithm and discuss the difference among the graph classes, we evaluate SOP-CG on three large-scale complex tasks with highly dynamic coordination requirements: Pursuit, Chase and Postman.
Revised: To further demonstrate the effectiveness of our algorithm and discuss the difference among the graph classes, we evaluate SOP-CG on three large-scale complex tasks with highly dynamic coordination requirements and partial observation: Pursuit, Chase and Postman.

Original: 3.0 T (mil)
Revised: 3.0 T (mil)

Original: T e s t R e t u r n
Revised: T e s t R e t u r n

Original: Pursuit     5 T (mil) 2.
Revised: Pursuit     5 T (mil) 2.

Original: T e s t R e t u r n
Revised: T e s t R e t u r n

Original: Chase 0.
Revised: Chase 0.

Original: 3.0 T (mil)
Revised: 3.0 T (mil)

Original: T e s t R e t u r n
Revised: T e s t R e t u r n

Original: Postman SOP-CG ( P )
Revised: Postman SOP-CG ( P )

Original: SOP-CG ( H )
Revised: SOP-CG ( H )

Original: DCG
Revised: DCG

Original: CASEC
Revised: CASEC

Original: VDN
Revised: VDN

Original: QMIX
Revised: QMIX

Revised: NDQ

Original: Optimal Return
Revised: Optimal Return

Original: • Pursuit .
Revised: • Pursuit .

Original: This environment is also called predator-and-prey, which is widely used in recent work (Son et al., 2019; B¨ohmer et al., 2020; Wang et al., 2021b).
Revised: This environment is also called predator-and-prey, which is widely used in recent work (Son et al., 2019; B¨ohmer et al., 2020; Wang et al., 2021b).

Original: 20 agents and 10 random walking prey interact on a 10 × 10 grid-world.
Revised: 20 agents and 10 random walking prey interact on a 10 × 10 grid-world.

Original: Based on a limited sight range, an agent can move on the map or catch adjacent prey.
Revised: Based on a limited sight range, an agent can move on the map or catch adjacent prey.

Original: One prey is captured only if at least two adjacent agents catch it simultaneously, inducing a joint reward of 1.
Revised: One prey is captured only if at least two adjacent agents catch it simultaneously, inducing a joint reward of 1.

Original: After that, both agents and the captured prey are removed from the game.
Revised: After that, both agents and the captured prey are removed from the game.

Original: If only one agent tries to catch the prey, it will fail and receive a punishment of -1.
Revised: If only one agent tries to catch the prey, it will fail and receive a punishment of -1.

Original: • Chase .
Revised: • Chase .

Original: This is a challenging task on the particle world with trained adversaries instead of built-in AIs created by Lowe et al.
Revised: This is a challenging task on the particle world with trained adversaries instead of built-in AIs created by Lowe et al.

Original: On the map with 3 randomly generated obstacles, 10 slower agents chase 3 faster adversaries.
Revised: On the map with 3 randomly generated obstacles, 10 slower agents chase 3 faster adversaries.

Original: For each collision between an agent and an adversary, the agents receive a reward while the adversaries are punished.
Revised: For each collision between an agent and an adversary, the agents receive a reward while the adversaries are punished.

Original: An agent or adversary can observe the units in a restricted circular area that is centered at its position.
Revised: An agent or adversary can observe the units in a restricted circular area that is centered at its position.

Original: We train the agents by a speciﬁc algorithm, and the adversaries are concurrently trained by VDN (Sunehag et al., 2018).
Revised: We train the agents by a specific algorithm, and the adversaries are concurrently trained by VDN (Sunehag et al., 2018).

Original: • Postman .
Revised: • Postman .

Original: 12 postmen work in a vast community to deliver letters from residents to the post ofﬁce.
Revised: 12 postmen work in a vast community to deliver letters from residents to the post office.

Original: With the post ofﬁce located in the northwest of the community, the whole community is separated into three parts according to the distance to the post ofﬁce, and each postman will only work in one region.
Revised: With the post office located in the northwest of the community, the whole community is separated into three parts according to the distance to the post office, and each postman will only work in one region.

Original: Postmen need to pick randomly generated letters in the community.
Revised: Postmen need to pick randomly generated letters in the community.

Original: While two postmen meet each other, any one of them can offer its letters to another one, so they can collaborate in order to deliver the letters to the post ofﬁce.
Revised: While two postmen meet each other, any one of them can offer its letters to another one, so they can collaborate in order to deliver the letters to the post office.

Original: A postman can observe the information of other postmen and the letters in a sight range.
Revised: A postman can observe the information of other postmen and the letters in a sight range.

Original: We compare our method with the state-of-the-art  value-basedMARL methods (VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2020b)) and coordination graph learning methods (DCG (B¨ohmer et al., 2020), CASEC (Wang et al., 2021b)).
Revised: We compare our method with the state-of-the-art coordination graph learning methods (DCG (B¨ohmer et al., 2020), CASEC (Wang et al., 2021b)), fully decomposed value-based methods (VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2020b)) and communication-based method (NDQ (Wang et al., 2020c)).

Original: The implementation details and hyperparameter settings are included in appendix D. As presented in Fig.
Revised: The implementation details and hyperparameter settings are included in Appendix D. As presented in Fig.

Original: 5, SOP-CG outperforms the baselines in all three environments.
Revised: 5, SOP-CG outperforms the baselines in all three environments.

Original: Using a smaller graph class will be more efﬁcient if such a concise graph class is sufﬁcient to express the coordination dependency in the task of interest.
Revised: Using a smaller graph class will be more efficient if such a concise graph class is sufficient to express the coordination dependency in the task of interest.

Original: Taking the task Chase as an example, DCG  performs as poorly as the fully-decomposed methods, as their graph classes are too large and complex to learn.
Revised: Taking the task Chase as an example, DCG and CASEC performs as poorly as the fully-decomposed methods, as their graph classes are too large and complex to learn.

Original: However, our method signiﬁcantly outperformsVDN, QMIX, and coordination graph learning algorithms with a smaller graph class, which strongly conﬁrms our statement.
Revised: However, our method significantly outperforms coordination graph learning algorithms with a smaller graph class, which strongly confirms our statement.

Original: Furthermore, SOP-CG ( G H ) has better performance than SOP-CG ( G P ) on this task, which may be due to the lack of representational capacity.
Revised: Furthermore, SOPCG ( G H ) has better performance than SOP-CG ( G P ) on this task, which may be due to the lack of representational capacity of G P .

Original: On the contrary, Pursuit mainly involves pairwise cooperation, so SOP-CG ( G P ) achieves better sample efﬁciency than SOP-CG ( G H ).
Revised: On the contrary, Pursuit mainly involves pairwise cooperation, so SOP-CG ( G P ) achieves better sample efficiency than SOP-CG ( G H ).

Original: We will show that reducing the error of DCOP can also improve performance.
Revised: We will show that reducing the error of DCOP can also improve performance.

Original: On task Pursuit with 20 agents, our method achieves almost the optimal of this task while DCG has over two times larger gap with the optimal value than ours.
Revised: On task Pursuit with 20 agents, our method achieves almost the optimal of this task while DCG has over two times larger gap with the optimal value than ours.

Original: The reason is that Pursuit requires precise collaborations.
Revised: The reason is that Pursuit requires precise collaborations.

Original: If only one agent tries to catch the prey, it will fail and induce a large punishment.
Revised: If only one agent tries to catch the prey, it will fail and induce a large punishment.

Original: When facing growing scales, DCG and CASEC have a poor accuracy on DCOP, as we showed before, which may lead to inaccurate greedy action selection, impeding DCG to better performance.
Revised: When facing growing scales, DCG and CASEC have a poor accuracy on DCOP, as we showed before, which may lead to inaccurate greedy action selection, impeding DCG to better performance.

Original: In the super hard task
Revised: In the super hard task

Original: Postman, which requires agents to collaborate during long horizon, our method also outperforms other methods.
Revised: Postman, which requires agents to collaborate during long horizon, our method also outperforms other methods.

Original: 6  32        7 2 Sight range of agent(a) (b) (c)  46  7agent and its id adversary and its moving direction edge in the chosen graph
Revised: 6  32        7 2 Sight range of agent(a) (b) (c)  46  7agent and its id adversary and its moving direction edge in the chosen graph

Original: [Figure]
Revised: [Figure]

Original: 5. V ISUALIZATION OF S ELF -O RGANIZED C OORDINATION
Revised: 5. V ISUALIZATION OF S ELF -O RGANIZED C OORDINATION

Original: A major strength of our method is that SOP-CG can organize different coordination graphs to adapt to different situations.
Revised: A major strength of our method is that SOP-CG can organize different coordination graphs to adapt to different situations.

Original: Such an adaptive organization mechanism can improve the efﬁciency of multi-agent collaboration.
Revised: Such an adaptive organization mechanism can improve the efficiency of multi-agent collaboration.

Original: To illustrate the coordination relations organized by SOP-CG, we visualize the learned graph structures on Chase.
Revised: To illustrate the coordination relations organized by SOP-CG, we visualize the learned graph structures on Chase.

Original: In Fig.
Revised: In Fig.

Original: 6, we take three snapshots chronologically from one episode.
Revised: 6, we take three snapshots chronologically from one episode.

Original: This episode is collected by SOP-CG ( G H ) using the tree-based graph class.
Revised: This episode is collected by SOP-CG ( G H ) using the tree-based graph class.

Original: It shows that SOP-CG can produce interpretable graph structures that characterize the ground-truth coordination dependencies.
Revised: It shows that SOP-CG can produce interpretable graph structures that characterize the ground-truth coordination dependencies.

Original: The learned coordination behaviors are interpreted as follows:
Revised: The learned coordination behaviors are interpreted as follows:

Original: Fig.
Revised: Fig.

Original: 6a presents the initial status of the game, where agents and adversaries are generated randomly on the map.
Revised: 6a presents the initial status of the game, where agents and adversaries are generated randomly on the map.

Original: As the three adversaries distribute in separate positions, the agents naturally form three groups to chase different adversaries.
Revised: As the three adversaries distribute in separate positions, the agents naturally form three groups to chase different adversaries.

Original: After several time steps, in Fig.
Revised: After several time steps, in Fig.

Original: 6b, two adversaries gather at the bottom left of the map.
Revised: 6b, two adversaries gather at the bottom left of the map.

Original: Note that, in this game, each agent can only observe objects within a small circular range.
Revised: Note that, in this game, each agent can only observe objects within a small circular range.

Original: Only agent 1 can observe the two adversaries simultaneously, which makes it connect to the majority of the rest of the agents and helps the other agents recognize the position of the adversaries.
Revised: Only agent 1 can observe the two adversaries simultaneously, which makes it connect to the majority of the rest of the agents and helps the other agents recognize the position of the adversaries.

Original: Finally, as illustrated by Fig.
Revised: Finally, as illustrated by Fig.

Original: 6c, the bottom adversary is surrounded by multiple agents, yielding an opportunity for the agents to win considerable rewards, which makes the collaboration structure concentrate around the adversary.
Revised: 6c, the bottom adversary is surrounded by multiple agents, yielding an opportunity for the agents to win considerable rewards, which makes the collaboration structure concentrate around the adversary

Original: As presented above, the graph structures learned by SOP-CGdeﬁnitely match the ground-truth demands for effective collaboration.
Revised: As presented above, the graph structures learned by SOP-CG match the ground-truth demands for effective collaboration on Chase.

Original: It demonstrates the ability of our approach to organizing coordination relations.
Revised: It demonstrates the ability of our approach to organizing coordination relations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper introduces SOP-CG, a novel multi-agent graph-based method that guarantees the polynomial-time greedy policy execution with expressive function representation of a structured graph class.
Revised: This paper introduces SOP-CG, a novel multi-agent graph-based method that guarantees the polynomial-time greedy policy execution with expressive function representation of a structured graph class.

Original: To enable end-to-end learning, SOP-CG introduces an imaginary agent to dynamically select the state-dependent graph and incorporates it into a uniﬁed Bellman optimality equation.
Revised: To enable end-to-end learning, SOP-CG introduces an imaginary agent to dynamically select the state-dependent graph and incorporates it into a unified Bellman optimality equation.

Original: We demonstrate that SOP-CG can learn interpretable graph topologies and outperform state-of-the-art baselines on several challenging tasks.
Revised: We demonstrate that SOP-CG can learn interpretable graph topologies and outperform state-of-the-art baselines on several challenging tasks.

Original: Although the graph class is exponentially large, we conduct two speciﬁc polynomial-time graph class instances to achieve impressive performance.
Revised: Although the graph class is exponentially large, we conduct two specific polynomial-time graph class instances to achieve impressive performance.

Original: Designing effective graph exploration methods to deal with larger graph classes will be an interesting future direction.
Revised: Designing effective graph exploration methods to deal with larger graph classes will be an interesting future direction.

Original: E THICS AND R EPRODUCIBILITY S TATEMENTS
Revised: E THICS AND R EPRODUCIBILITY S TATEMENTS

Original: The main contribution of our work locates on the research of a foundation theoretical problem in optimality theory and try to improve the problem with our method.
Revised: The main contribution of our work locates on the research of a foundation theoretical problem in optimality theory and try to improve the problem with our method.

Original: Our work does not have a bad social impact.
Revised: Our work does not have a bad social impact.

Original: The source code of our method for all the experiments is attached in supplementary material and the settings and parameters for all models and algorithms mentioned in the experiment section please refer to Appendix D
Revised: The source code of our method for all the experiments is attached in supplementary material and the settings and parameters for all models and algorithms mentioned in the experiment section please refer to Appendix D

Original: A E XPERIMENTAL D ETAILS OF THE M OTIVATING E XAMPLE
Revised: A E XPERIMENTAL D ETAILS OF THE M OTIVATING E XAMPLE

Original: Algorithms.
Revised: Algorithms.

Original: To get the correct result of DCOP, we use a brute force with complexity O
Revised: To get the correct result of DCOP, we use a brute force with complexity O

Original: ( | A | n ) , while we choose Max-sum algorithm (has an alternative implementation called Max-plus) (Pearl, 1988), which is commonly used by most of the coordination graph learning methods (Zhang &
Revised: ( | A | n ) , while we choose Max-sum algorithm (has an alternative implementation called Max-plus) (Pearl, 1988), which is commonly used by most of the coordination graph learning methods (Zhang &

Original: Lesser, 2013; B¨ohmer et al., 2020; Wang et al., 2021b) to test the accuracy and relative Q error.
Revised: Lesser, 2013; B¨ohmer et al., 2020; Wang et al., 2021b) to test the accuracy and relative Q error.

Original: The details of the algorithm are presented below (Zhang & Lesser, 2013).
Revised: The details of the algorithm are presented below (Zhang & Lesser, 2013).

Original: Max-sum constructs a bipartite graph G m = (cid:104)V a , V q , E m (cid:105) according the coordination graph (cid:104)V , E(cid:105) .
Revised: Max-sum constructs a bipartite graph G m = ⟨V a , V q , E m ⟩ according the coordination graph ⟨V , E⟩ .

Original: Each node v ∈ V m represents an agent who needs to do action selection, and each node u ∈ V q represents a (hyper-)edge function.
Revised: Each node v ∈ V m represents an agent who needs to do action selection, and each node u ∈ V q represents a (hyper-)edge function.

Original: Edges in E m connect u with the corresponding agent nodes.
Revised: Edges in E m connect u with the corresponding agent nodes.

Original: Max-sum algorithm will do multi-round message passing on this graph.
Revised: Max-sum algorithm will do multi-round message passing on this graph.

Original: The number of iterations is usually set smaller than 10 in previous works, while we set it 100 in this experiment.
Revised: The number of iterations is usually set smaller than 10 in previous work, while we set it 100 in this experiment.

Original: Message passing on this bipartite graph starts with sending messages from node v ∈ V a to node u ∈ V q :
Revised: Message passing on this bipartite graph starts with sending messages from node v ∈ V a to node u ∈ V q :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where F v represents the node connected to v , and c vu is the normalizing factor preventing the value of messages from growing arbitrarily large.
Revised: where F v represents the node connected to v , and c vu is the normalizing factor preventing the value of messages from growing arbitrarily large.

Original: The message from node u to v is:
Revised: The message from node u to v is:

Original: [Equation]
Revised: [Equation]

Original: where V u is the set of nodes connected to node u , a u = { a h | h ∈ V u } , a u \ a v = { a h | h ∈ V u \{ v }} , and f represents the value function of the (hyper-)edge.
Revised: where V u is the set of nodes connected to node u , a u = { a h | h ∈ V u } , a u \ a v = { a h | h ∈ V u \{ v }} , and f represents the value function of the (hyper-)edge.

Original: After iterations of message passing, each agent v can ﬁnd its optimal action by calculating a∗ v = argmax a v (cid:80) h ∈F v m h → v ( a v ) .
Revised: After iterations of message passing, each agent v can find its optimal action by calculating a

Original:  Test data generalization.
Revised: [Equation] Test data.

Original: We take 1 , 000 random seeds to test the problem.
Revised: We take 1 , 000 random seeds to test the problem.

Original: We ﬁx the number of actions 3 and move the number of agents from 2 to 18 .
Revised: We fix the number of actions 3 and move the number of agents from 2 to 18 .

Original: For each setting, we generate a fully connected graph with each value a uniformly random number in {− 1 , 0 , 1 } along with a noise drawn from the normal distribution.
Revised: For each setting, we generate a fully connected graph with each value a uniformly random number in {− 1 , 0 , 1 } along with a noise drawn from the normal distribution.

Original: B C OMPUTATIONS WITH CLASSES G P AND G H
Revised: B C OMPUTATIONS WITH CLASSES G P AND G H

Original: Here we present the details of algorithms we used to select graph G ( t ) from the graph class G .
Revised: Here we present the details of algorithms we used to select graph G ( t ) from the graph class G .

Original: When G = G P , pairwise grouping.
Revised: When G = G P , pairwise grouping.

Original: In this case, n agents are pairwise matched by (cid:98) n/ 2 (cid:99) edges.
Revised: In this case, n agents are pairwise matched by ⌊ n/ 2 ⌋ edges.

Original: One agent will be isolated if n is an odd number.
Revised: One agent will be isolated if n is an odd number.

Original: With such a coordination graph, each agent coordinates with exactly one agent, except for the isolated one.
Revised: With such a coordination graph, each agent coordinates with exactly one agent, except for the isolated one.

Original: If node i is matched with node j , it will contribute q i → j = q i ( τ i , a i )+ q j
Revised: If node i is matched with node j , it will contribute q i → [Equation] = [Equation]

Original: [Equation]
Revised: [Equation]

Original: +q ij ( τ i , τ
Revised: +

Original: [Equation]
Revised: [Equation]

Original: , a j ) to the total value when choosing action a i , which only contains the single utility function and one pairwise function.
Revised: a j ) to the total value when choosing action a i , which only contains the single utility function and one pairwise function.

Original: Thus the pair ( i, j ) can contribute at most
Revised: Thus the pair ( i, j ) can contribute at most

Original: [Equation] 
Revised: [Equation] [Equation]

Original: to the joint value without effect the action selection of other agents.
Revised: to the joint value without effect the action selection of other agents.

Original: Hence, we construct a undirected weighted graph G = (cid:104)V , E w (cid:105) , where V is the vertex set, and
Revised: Hence, we construct a undirected weighted graph G = ⟨V , E w ⟩ , where V is the vertex set, and

Original: E w =(cid:26)(cid:18)
Revised: E w =

Original: [Equation]
Revised: [Equation]

Original: A matching on such graph is a function mate ( v ) where mate ( mate ( v ))
Revised: A matching on such graph is a function mate ( v ) where mate ( mate ( v ))

Original: = v for all v ∈ V with at most one v satisﬁes that mate ( v ) = v .
Revised: = v for all v ∈ V with at most one v satisfies that mate ( v ) = v .

Original: The weight of the matching is deﬁned as (cid:80) v w ( v, mate ( v )) .
Revised: The weight of the matching is defined as (cid:80) v w ( v, mate ( v )) .

Original: Our goal is to ﬁnd the maximum weighted matching on this graph.
Revised: Our goal is to find the maximum weighted matching on this graph.

Original: This problem can be solved by the blossom algorithm (Edmonds, 1965) in O ( n 3 ) time.
Revised: This problem can be solved by the blossom algorithm (Edmonds, 1965) in O ( n 3 ) time.

Original: The brief steps of this algorithm are to iteratively ﬁnd augmentation paths on the graph and contract the oddcycle, which is called blossom in this algorithm.
Revised: The brief steps of this algorithm are to iteratively find augmentation paths on the graph and contract the oddcycle, which is called blossom in this algorithm.

Original: When G = G H , hierarchical organization .
Revised: When G = G H , hierarchical organization .

Original: In this case, the graph is an acyclic graph with n −edges, and all agents form a connected component.
Revised: In this case, the graph is an acyclic graph with n −edges, and all agents form a connected component.

Original: The distributed constraint optimization problem (DCOP) on such a graph can always be solved in the polynomial time (Fioretto et al., 2018).
Revised: The distributed constraint optimization problem (DCOP) on such a graph can always be solved in the polynomial time (Fioretto et al., 2018).

Original: Choosing the best graph G ( t ) in G H is non-trivial.
Revised: Choosing the best graph G ( t ) in G H is non-trivial.

Original: Therefore, we present a greedy algorithm to select the graph from the graph class.
Revised: Therefore, we present a greedy algorithm to select the graph from the graph class.

Original: The high level idea is to add edges greedily until all agents are connected.
Revised: The high level idea is to add edges greedily until all agents are connected.

Original: At each iteration, we add an edge across two different connected components to the coordination graph, which optimizes the increment of the maximal joint value.
Revised: At each iteration, we add an edge across two different connected components to the coordination graph, which optimizes the increment of the maximal joint value.

Original: For an edge set
Revised: For an edge set

Original: E ⊆ E , we deﬁne the value of a set f ( E ) to be the result of DCOP on graph (cid:104)V , E (cid:105) .
Revised: E ⊆ E , we define the value of a set f ( E ) to be the result of DCOP on graph ⟨V , E ⟩ .

Original: Initially we have E is an empty set.
Revised: Initially we have E is an empty set.

Original: We do the iteration n − 1 times.
Revised: We do the iteration n − 1 times.

Original: In each iteration, we ﬁnd
Revised: In each iteration, we find

Original: [Equation]
Revised: [Equation]

Original: (12) and add this edge to the current graph, i.e., E ← E ∪ { e } .
Revised: and add this edge to the current graph, i.e., E ← E ∪ { e } .

Revised: Once a graph from G H is selected, the joint action selection can be computed via dynamic programming.

Revised: Time complexity .

Revised: We summarize the time complexity (in worst case) of each component in Table 2.

Revised: For SOP-CG ( G H ), we use pruning techniques in our implementation, so the real-time cost of selecting graph G ( t ) is usually much smaller than the complexity reported in Table 2.

Revised: The detailed training time cost of SOP-CG and DCG can be found in Appendix D.

Original:  C G RAPH R ELABELING T ECHNIQUE
Revised: [Table] [Equation] [Table] C G RAPH R ELABELING T ECHNIQUE

Original: As studied by Hasselt (2010), vanilla Q-learning already has a tendency to overestimate, since the
Revised: As studied by Hasselt (2010), vanilla Q-learning already has a tendency to overestimate, since the

Original: TD target takes a max operator over a set of estimated action-values.
Revised: TD target takes a max operator over a set of estimated action-values.

Original: Notice that our framework introduces an additional max operator over graphs in class G , which will probably induce extra upward bias.
Revised: Notice that our framework introduces an additional max operator over graphs in class G , which will probably induce extra upward bias.

Original: To attenuate this problem, we use a graph relabeling technique to control the effects of overestimation errors.
Revised: To attenuate this problem, we use a graph relabeling technique to control the effects of overestimation errors.

Original: In our implementation, we modify the vanilla TD loss deﬁned in Eq.
Revised: In our implementation, we modify the vanilla TD loss defined in Eq.

Original: (7) to the following form:
Revised: (7) to the following form:

Original: [Equation]
Revised: [Equation]

Original:  where y cg is the one-step TD target deﬁned in Section 4.2 and the current value no longer takes the graph which is stored in the replay buffer with this transition.
Revised: (13) where y cg is the one-step TD target defined in Section 4.2 and the current value no longer takes the graph which is stored in the replay buffer with this transition.

Original: As G is the graph with largest estimated value, this loss achieves a soft constraint for the following target:
Revised: As G is the graph with largest estimated value, this loss achieves a soft constraint for the following target:

Original: [Equation]
Revised: [Equation]

Original:  which helps our method avoid overestimation.
Revised: 14) which helps our method avoid overestimation.

Revised: We conduct ablation studies on the tasks Pursuit, Chase and Postman to demonstrate the necessity of this component.

Revised: The results illustrated in Figure 7.

Revised: SOP-CG ( G H )

Revised: w/o graph relabeling is more likely to stick in suboptimal strategies and may even collapse on Pursuit.

Original: D I MPLEMENTATION D ETAILS A ND E XPERIMENT S ETTINGS
Revised: As analyzed before, these phenomenons are due to the dramatic overestimation caused by the additional max operator over graphs. By contrast, the performance of SOP-CG ( G H ) is significantly improved, highlighting the effectiveness of graph relabeling. 3.0 T (mil) T e s t R e t u r n

Revised: Pursuit     5 T (mil) 2.

Revised: T e s t R e t u r n

Revised: Chase 0.

Revised: 3.0 T (mil)

Original:  A shared GRU (Cho et al., 2014) is used to process sequential inputs and then output the encoded observation history for each agent.
Revised: T e s t R e t u r n [Figure] D I MPLEMENTATION D ETAILS A ND E XPERIMENT S ETTINGS A shared GRU (Cho et al., 2014) is used to process sequential inputs and then output the encoded observation history for each agent.

Original: A following fully-connected layer converts the observation histories to the individual utility function of each agent.
Revised: A following fully-connected layer converts the observation histories to the individual utility function of each agent.

Original: The pairwise payoff function is computed by another multi-layer perceptron, which takes the concatenation of two agents’ observation histories as input.
Revised: The pairwise payoff function is computed by another multi-layer perceptron, which takes the concatenation of two agents’ observation histories as input.

Original: Agents share parameters of the network for computing individual utility, and the parameters of the payoff network are also shared among different agent pairs.
Revised: Agents share parameters of the network for computing individual utility, and the parameters of the payoff network are also shared among different agent pairs.

Original: All tasks in this paper use a discount factor γ = 0 .
Revised: All tasks in this paper use a discount factor γ = 0 .

Original: We use (cid:15) -greedy exploration, and (cid:15) anneals linearly from 1.0 to 0.05 over 50000 time-steps.
Revised: We use ϵ -greedy exploration, and ϵ anneals linearly from 1.0 to 0.05 over 50000 time-steps.

Original: We use an RMSProp optimizer with a learning rate of 5 × 10 − 3 to train our network.
Revised: We use an RMSProp optimizer with a learning rate of 5 × 10 − 3 to train our network.

Original: A ﬁrst-in-ﬁrst-out (FIFO) replay buffer stores the experiences of at most 5000 episodes, and a batch of 32 episodes are sampled from the buffer during the training phase.
Revised: A first-in-first-out (FIFO) replay buffer stores the experiences of at most 5000 episodes, and a batch of 32 episodes are sampled from the buffer during the training phase.

Original: The target network is periodically updated every 200 episodes.
Revised: The target network is periodically updated every 200 episodes.

Original: Our method and all the baselines involved in this paper are implemented based on the open-sourced codebase PyMARL (Samvelyan et al., 2019), which ensures the fairness of comparison.
Revised: Our method and all the baselines involved in this paper are implemented based on the open-sourced codebase PyMARL (Samvelyan et al., 2019), which ensures the fairness of comparison.

Original: For the baselines (VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2020b), DCG (B¨ohmer et al., 2020),
Revised: For the baselines (VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2020b), DCG (B ¨ ohmer et al., 2020),

Original: CASEC (Wang et al., 2021b)), we adopt the default hyperparameter settings provided by the authors.
Revised: CASEC (Wang et al., 2021b)), we adopt the default hyperparameter settings provided by the authors.

Original: For evaluation, all learning curves in this paper are smoothed over 5 random seeds.
Revised: For evaluation, all learning curves in this paper are smoothed over 5 random seeds.

Original: The experiments are ﬁnished on NVIDIA RTX 2080TI GPU. 
Revised: The experiments are finished on NVIDIA RTX 2080TI GPU. The training time of SOP-CG and

Original: Training our algorithm on any task shown in  this paper can be ﬁnished within 2.5 days.
Revised: DCG are listed in Table D. SOP-CG is much faster than DCG. [Table] E S TAR C RAFT II MICROMANAGEMENT BENCHMARK In this section, we compare SOP-CG against DCG (B¨ohmer et al., 2020) and CASEC (Wang et al., 2021b) on StarCraft II micromanagement benchmark (Samvelyan et al., 2019).

Revised: The pairwise payoff function has A 2 output head, where A = | (cid:83) ni

Original: {+=
+}
Revised: {+=
+}

Revised: A i | .

Revised: DCG and CASEC adopt different techniques (e.g. low-rank approximation or learning action representations) to improve the learning efficiency of the payoff functions in environments with large action space.

Revised: These techniques can also be applied to our algorithm.

Revised: For fair evaluation, we test the vanilla versions of the algorithms without applying extra techniques on payoff networks.

Revised: The results are shown in Fig.

Revised: SOP-CG outperforms DCG and CASEC on 4 of the 6 scenarios and achieves comparable performance on the other scenarios.

Revised: This finding indicates that the class of polynomial-time coordination graphs is large enough to express coordinated policies in these tasks, and the accurate action selection further helps SOP-CG achieve higher peak performance than DCG and CASEC on the four maps.

Revised: [Figure]

Revised: [Figure]

Revised: F SC2 TASKS RESTRICTED OBSERVABILITY

Revised: In coordination-graph-based methods, agents communicate to optimize joint action selection.

Revised: As an alternative way, previous work studies multi-agent communication by learning message embedding (Singh et al., 2018; Das et al., 2019; Wang et al., 2020c).

Revised: We compare our algorithm against the state-of-the-art communication learning method NDQ (Wang et al., 2020c) on 2 StarCraft II tasks: 1o10b vs 1r and 12z vs 2ul.

Revised: These tasks are modified from the challenging maps introduced by the NDQ paper.

Revised: The sight range of agents is reduced from 9 to 2, which enforces the agents to coordinate their actions by communication.

Revised: To enable effective communication, we further augment the observation of each agent by its position on the map.

Revised: • 1o10b vs 1r: 10 Banelings spawn randomly on a map full of cliffs and need to kill an enemy Roach.

Revised: An allied Overseer detects the Roach and they spawn at the same random point.

Revised: Therefore, the Banelings can get the position of the Roach from the allied Overseer.

Revised: • 12z vs 2ul: 12 Zealots try to kill 2 powerful Ultralisk.

Revised: All units spawn randomly on the map.

Revised: The agents need to communicate the positions of enemies as well as coordinate the attack timing to reach high win rate.

Revised: The learning curves are shown in Figure 9.

Revised: On the map 1o10b vs 1r, a simple but effective communication strategy is to let the Overseer coordinate with all allied Banelings to guide their moving direction.

Revised: We observe that it’s not hard for NDQ to learn this mechanism and NDQ has a similar performance as SOP-CG ( G H ) on this map.

Revised: However, the performance of NDQ is worse than the methods based on coordination graphs on 12z vs 2ul.

Revised: As all units spawn randomly on this map, the agents need to learn more complex coordinated strategies to kill both of the enemies.

Revised: This finding suggests that directly coordinating actions based on coordination graphs may be more efficient than learning encoded messages.

Revised: Notably, SOP-CG ( G H ) also outperforms DCG and CASEC on both tasks, highlighting the effectiveness of our method in complex scenarios.

Revised: G M ETRICS OF TASK COMPLEXITY

Revised: To illustrate the algorithmic properties of SOP-CG, we propose two measures of task complexity and conduct analysis between SOP-CG and DCG according to these measures.

Revised: Furthermore, we design a new didactic example to empirically demonstrate our theoretical implications answering the following questions: when does SOP-CG work better and when does DCG work better?

Revised: We will introduce the two measures of task complexity as follows:

Revised: • Problem size in general Dec-POMDPs.

Revised: Solving a general single-agent MDP with state space S and action space A requires the polynomial-time complexity of | S | × | A | (i.e., its problem size), as shown in Sutton et al.

Revised: (Sutton & Barto, 2018).

Revised: In MARL tasks, A = (cid:83) A i , whose problem size is extended to | S | × | A i | n , exponential of the number of agents and the size of local action space, respectively.

Revised: To analyze the effect of problem size on SOP-CG and CG, we find that the time complexity of exact greedy action selection (solving DCOPs) is exponential-time for DCG (i.e., with a complete graph), and which is polynomial-time complexity for SOP-CG (i.e., with a class of polynomial-time graphs).

Revised: • Structural complexity of coordination within agents.

Revised: This measure is used to represent how much function expressiveness of coordination graph (i.e., which types of graph) is needed to solve a specific task.

Revised: Note that the function capacity of DCG (line or star), SOP-CG, and DCG (complete graph) forms a hierarchy, i.e., Q DCG (line or star)

Revised: ⊂ Q SOP-CG ⊂ Q DCG (complete graph) , where Q indicates the class of value function which can be represented by each algorithm.

Revised: Thus, we formally design the structural complexity of coordination as the necessary and sufficient function capacity induced by each specific task.

Revised: Along with this structural complexity increases, larger function capacity of algorithms is required.

Revised: Moreover, we hypothesize that in the realistic tasks, the coordination structure is sparse (Kok & Vlassis,

Revised: ; Zhang & Lesser, 2013) and the function class of SOP-CG would be sufficient for an extensive set of situations.

Revised: Using these two measures of task complexity, we compare SOP-CG with DCG in four cases that either these two complexities are high or low.

Revised: • Both two complexities are low.

Revised: Both SOP-CG and DCG (complete graph) can easily address this case.

Revised: • The problem size is large but the structural complexity is relatively low, which is a common case in real-world tasks (Kok & Vlassis, 2004; Zhang & Lesser, 2013).

Revised: SOPCG can significantly outperform DCG (complete graph) because of efficient computation of exact greedy action selection without sacrificing its expressiveness in this case.

Revised: The major strength of SOP-CG is its ability to model the value function using self-organized polynomial-time coordination graphs.

Revised: Solving the DCOPs induced by SOP-CG is computationally efficient but which is expensive for DCG (complete graph) and the approximation of its induced DCOPs.

Revised: We find that SOP-CG is not sensitive to the problem size, such as the number of agents and individual actions.

Revised: In comparison, it is NP-hard to find a constant-ratio approximation of the DCOPs induced by complete graphs (i

Revised: e., the default implementation of DCG).

Revised: • The problem size is small but the structural complexity is very high.

Revised: In this case, DCG (complete graph) may outperform SOP-CG as it is possible for DCG to compute nearly optimal joint action selection.

Revised: However, as we hypothesized that this case may not usual in the realistic tasks since multi-agent systems are often large and the sparse coordination structure commonly exists in such systems.

Revised: • Both two complexities are high.

Revised: SOP-CG and DCG (complete graph) may perform comparably in this case because they both introduce some approximation in learning.

Revised: SOPCG introduces approximation in value function expressiveness while DCG has inaccurate greedy action selection.

Revised: This case is super hard and would be studied as an important future direction of coordination graph methods.

Revised: To investigate the metrics of task complexity and our performance relative to the metrics, we design a series of tasks.

Revised: The tasks consist m × k agents separated into m groups with each group k agents.

Revised: [Figure]

Revised: N o r m a li z e d R e t u r n m

Revised: [Figure]

Revised: [Figure]

Revised: k

Revised: [Figure]

Revised: 50 T (k)

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: N o r m a li z e d R e t u r n m

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: N o r

Revised: [Figure]

Revised: li z

Revised: [Figure]

Revised: d R

Revised: [Figure]

Revised: t u r n

Revised: [Figure]

Revised: Each agent has to choose an action from the action space A , and one group wins if and only if all its members perform k distinct actions.

Revised: The global reward is the proportion of winning groups.

Revised: In this task the number of agents in one group measures the representation complexity that each agent has to cooperate with other agents within the same group, while the number of groups indicates the scale of the problem.

Revised: When the size of group grows, the coordination relationship becomes dense.

Revised: If the number of groups increases and the size of group is fixed, the coordination complexity of the task will remain the same, but the scale of the task can be larger.

Revised: Results illustrated in Figure 10 show that our method can work very well when the group size of not so large (i.e., k ≤ 3 ), being less sensitive to the number of groups.

Revised: While DCG (line) or DCG (star) cannot work when the total number of agents m × k exceeds a small threshold, which corresponds to the fact that the representation capacity of them is restricted below our method.

Revised: In contrast, the performance of our method decreases when the group size grows larger, while DCG performs better when k grows and m remains small.

Revised: The result also suggests that when k keeps small, both SOP-CG and DCG have enough representation capacity to solve the problem.

Revised: As the scale of the problem grows, the complexity of solving DCOPs of DCG becomes higher, so SOP-CG can significantly outperform DCG.

Revised: H S OME DETAIL DISCUSSIONS ABOUT TRADE - OFF IN SOP-CG H. C ONNECTION TO A R ECENT W ORK ON H YPER -G RAPHS

Revised: We note that a recent work on the expressiveness of hyper-graphs (Yoon et al., 2020) is relevant to our work.

Revised: The connections (i.e., the differences and relevance) between Yoon et al.

Revised: and our paper are summarized as follows:

Revised: • Different metric of expressiveness.

Revised: Yoon et al.

Revised: (2020) investigates the functionality of graph expressiveness in terms of the order of hyper-edges.

Revised: By contrast, our paper focuses on the expressiveness of different graph topologies.

Revised: • Similar motivation.

Revised: Both Yoon et al.

Revised: and our paper aim to study the trade-off between the graph expressiveness and the complexity of solving downstream tasks.

Revised: In our paper, the downstream task specifically refer to the DCOPs induced by graph-based value factorization.

Revised: • Different work flow and contributions.

Revised: Yoon et al.

Revised: (2020) establish a systematic experiment study to provide an empirical guideline on how to trade off between the graph expressiveness and downstream costs.

Revised: In comparison, our work proposes an algorithm to break this trade-off for improving MARL performance.

Revised: Our algorithm is able to improve the function expressiveness without largely increasing the time complexity of downstream DCOPs.

Revised: Despite the expressiveness metric being different, several conclusions reached by Yoon et al.

Revised: (2020) can be observed in our experiments.

Revised: More specifically, the key findings of Yoon et al.

Revised: (2020) (namely, three points, i.e, Diminishing returns , Troubleshooter , and Irreducibility ) correspond to the following observations in our paper:

Revised: • Diminishing returns.

Revised: When the task coordination structure is simple, a concise graph topology (e.g., pairwise matching) can achieve comparable performance with topologies using larger edge sets.

Revised: As presented in Figure 5, SOP-CG ( G P ) performs better than SOPCG ( G H ) and DCG in Pursuit, since pairwise matching is sufficient to express the underlying coordination relations in this task.

Revised: (Recall that G P uses pairwise matching and G H uses tree-based topology.)

Revised: • Troubleshooter and Irreducibility.

Revised: When the task requires a complicated coordination structure, the graph topology needs to have higher function expressiveness.

Revised: This finding can also be seen in Figure 5.

Revised: SOP-CG ( G H ) significantly outperforms SOP-CG ( G P ) in Chase where pairwise matching cannot express the underlying coordination relations.

Revised: H. P OSITION OF SOP-CG

Revised: The trade-off line from DCG with fixed polynomial graphs (i.e., line or star) to SOP-CG to DCG with complete graph (i.e., the default configuration) is similar to the VDN-QMIX-QTRAN methods, though SOP-CG lies in the coordination graph line of MARL (rather than the value factorization line).

Revised: In this analogy hierarchy, SOP-CG corresponds to QMIX where SOP-CG has much larger function expressiveness than DCG with line or star graph (which corresponds to VDN), and can derive the exact greedy action selection compared to DCG (complete graph) (which corresponds to

Revised: QTRAN).

Revised: Note that different from VDN which has achieved excellent performance on StarCraft II benchmark (Samvelyan et al., 2019), DCG with line or star graph performs poorly due to its limited function capacity, which is mentioned in its original paper and in our evaluation (Appendix G).

Revised: Similar to discussions in Mahajan et al.

Revised: (2019) and Wang et al.

Revised: (2021a) that QTRAN may suffer from its approximate greedy action selection, DCG may also suffer non-optimal action selection, while

Revised: SOP-CG also considers the advantage of exact greedy selection (i.e., the property of polynomialtime graph proved by Fioretto et al.

Revised: and empirically achieves significant outperformance than DCG (Figure 5 and Appendix E and F).

Revised: In comparison of function expressiveness, DCG (complete graph) achieves larger value function capacity than SOP-CG, but we hypothesize that in the realistic tasks, the coordination structure could be sparse (Kok & Vlassis, 2004; Zhang & Lesser, 2013) and the function class of SOP-CG will be sufficient to address a large suite of challenging tasks, e.g., StarCraft II, in Appendix E and F. Therefore, from the perspective of “VDN-QMIXQTRAN” line in coordination graphs, SOP-CG has a clear and important position in the literature.