Original: Non-Linear Coordination Graphs
Revised: Non-Linear Coordination Graphs

Original: Abstract
Revised: Abstract

Original: Value decomposition multi-agent reinforcement learning methods learn the globalvalue function as a mixing of each agent’s individual utility functions.
Revised: Value decomposition multi-agent reinforcement learning methods learn the globalvalue function as a mixing of each agent’s individual utility functions.

Original: Coordinationgraphs (CGs) represent a higher-order decomposition by incorporating pairwisepayoff functions and thus is supposed to have a more powerful representationalcapacity.
Revised: Coordinationgraphs (CGs) represent a higher-order decomposition by incorporating pairwisepayoff functions and thus is supposed to have a more powerful representationalcapacity.

Original: However, CGs decompose the global value function linearly over localvalue functions, severely limiting the complexity of the value function class thatcan be represented.
Revised: However, CGs decompose the global value function linearly over localvalue functions, severely limiting the complexity of the value function class thatcan be represented.

Original: In this paper, we propose the ﬁrst non-linear coordinationgraph by extending CG value decomposition beyond the linear case.
Revised: In this paper, we propose the first non-linear coordinationgraph by extending CG value decomposition beyond the linear case.

Original: One majorchallenge is to conduct greedy action selections in this new function class to whichcommonly adopted DCOP algorithms are no longer applicable.
Revised: One majorchallenge is to conduct greedy action selections in this new function class to whichcommonly adopted DCOP algorithms are no longer applicable.

Original: We study how tosolve this problem when mixing networks with LeakyReLU activation are used.
Revised: We study how tosolve this problem when mixing networks with LeakyReLU activation are used.

Original: Anenumeration method with a global optimality guarantee is proposed and motivatesan efﬁcient iterative optimization method with a local optimality guarantee.
Revised: Anenumeration method with a global optimality guarantee is proposed and motivatesan efficient iterative optimization method with a local optimality guarantee.

Original: Weﬁnd that our method can achieve superior performance on challenging multi-agentcoordination tasks like MACO.
Revised: Wefind that our method can achieve superior performance on challenging multi-agentcoordination tasks like MACO.

Original:  Introduction
Revised:  Introduction

Original: Cooperative multi-agent problems are ubiquitous in real-world applications, such as crewless aerialvehicles [15, 28] and sensor networks [30].
Revised: Cooperative multi-agent problems are ubiquitous in real-world applications, such as crewless aerialvehicles [18, 32] and sensor networks [34].

Original: However, learning control policies for such systemsremains a major challenge.
Revised: However, learning control policies for such systemsremains a major challenge.

Original: Recently, value decomposition methods [21] signiﬁcantly push forwardthe progress of multi-agent reinforcement learning [17, 19, 23, 24].
Revised: Recently, value decomposition methods [25] significantly push forwardthe progress of multi-agent reinforcement learning [20, 23, 27, 28].

Original: In these methods, a centralizedaction-value function is learned as a mixing of individual utility functions.
Revised: In these methods, a centralizedaction-value function is learned as a mixing of individual utility functions.

Original: The mixing functioncan be conditioned on global states [17] while individual utility functions are based on local actionobservation history.
Revised: The mixing functioncan be conditioned on global states [20] while individual utility functions are based on local actionobservation history.

Original: The advantage is that agents can utilize global information and avoid learningnon-stationarity [22] via centralized training while maintaining scalable decentralized execution.
Revised: The advantage is that agents can utilize global information and avoid learningnon-stationarity [26] via centralized training while maintaining scalable decentralized execution.

Original: Notably, the major focus of the value function decomposition research was on full decomposition, where local utility functions are based on actions and observations of a single agent.
Revised: Notably, the major focus of the value function decomposition research was on full decomposition, where local utility functions are based on actions and observations of a single agent.

Original: Fulldecomposition leads to several drawbacks, such as miscoordination problems in partially observableenvironments with stochastic transition functions [23, 25] and a game-theoretical pathology calledrelative overgeneralization [13, 3].
Revised: Fulldecomposition leads to several drawbacks, such as miscoordination problems in partially observableenvironments with stochastic transition functions [27, 29] and a game-theoretical pathology calledrelative overgeneralization [16, 4].

Original: Relative overgeneralization embodies that, without the knowledge of actions of other agents, the employed utility function may not be able to express optimaldecentralized policies when the local return is inﬂuenced signiﬁcantly by other agents.
Revised: Relative overgeneralization embodies that, due to the concurrentlearning and exploration of other agents, the employed utility function may not be able to expressoptimal decentralized policies and prefer suboptimal actions that give higher returns on average.

Original: Coordination graph (CG) learning [7] holds the promise to address these problems while preservingthe advantages of value decomposition methods.
Revised: Coordination graph (CG) learning [9] holds the promise to address these problems while preservingthe advantages of value decomposition methods.

Original: In a CG, each vertex represents an agent, and each(hyper-) edge stands for a payoff function that is deﬁned on the joint action-observation space of theconnected agents.
Revised: In a CG, each vertex represents an agent, and each(hyper-) edge stands for a payoff function that is defined on the joint action-observation space of theconnected agents.

Original: The existence of payoff functions increases the granularity of decomposition, andconnected agents explicitly coordinate with each other.
Revised: The existence of payoff functions increases the granularity of decomposition, andconnected agents explicitly coordinate with each other.

Original: As a result, a CG represents a higher-orderfactorization of the global value function and represents a much larger value function class.
Revised: As a result, a CG represents a higher-orderfactorization of the global value function and represents a much larger value function class.

Original: However, coordination graphs typically assume a linear decomposition of the value function amongsub-groups of agents, which is too simple to represent credit assignment in complex tasks.
Revised: However, coordination graphs typically assume a linear decomposition of the value function amongsub-groups of agents, which is too simple to represent credit assignment in complex tasks.

Original: In thispaper, we solve this long-standing problem and extend CG by introducing non-linear and learnablevalue decomposition.
Revised: In thispaper, we solve this long-standing problem and extend CG by introducing non-linear and learnablevalue decomposition.

Original: To our best knowledge, it is the ﬁrst study on non-linear coordination graphs.
Revised: To our best knowledge, it is the first study on non-linear coordination graphs.

Original: The major challenge against extending CGs beyond linear cases is the calculation of value-maximizingactions.
Revised: The major challenge against extending CGs beyond linear cases is the calculation of value-maximizingactions.

Original: When linearly decomposed, DCOP algorithms [ can ﬁnd a globally greedy action via
Revised: When linearly decomposed, DCOP algorithms [6] can find a globally greedy action via

Original: #" = %& '  '
Revised: 𝑄 "#" = %𝑞 ’  ’

Original: Fully decomposed value functions    ! " #" = ) * (& , ,⋯,& / )
Revised: Fully decomposed value functions    𝑄 "#" = 𝑓 * (𝑞 , ,⋯,𝑞 / )

Original: VDN
Revised: VDN

Original: QMIX    ! " #" = %& '  ',
Revised: QMIX    𝑄 "#" = %𝑞 ’  ’,

Original: Coordination graphs    ! "
Revised: Coordination graphs    𝑄 "#" =

Original: #" = ) * ({& '1 })
Revised: 𝑓 * ({𝑞 ’1 })

Original: DCG
Revised: DCG

Original: Ours message passing.
Revised: Ours message passing.

Original: However, when the mixing function is non-linear, DCOP algorithms are no longerapplicable.
Revised: However, when the mixing function is non-linear, DCOP algorithms are no longerapplicable.

Original: To address this problem, we develop a DCOP method for non-linear mixing functionsrepresented as a deep network with LeakyReLU (or ReLU) activation.
Revised: To address this problem, we develop a DCOP method for non-linear mixing functionsrepresented as a deep network with LeakyReLU (or ReLU) activation.

Original: The core idea of our non-linear DCOP algorithm is to exploit the fact that these neural networksare equivalent to piece-wise linear functions.
Revised: The core idea of our non-linear DCOP algorithm is to exploit the fact that these neural networksare equivalent to piece-wise linear functions.

Original: For each linear piece, a value-maximizing action canbe found by classic DCOP.
Revised: For each linear piece, a value-maximizing action canbe found by classic DCOP.

Original: However, this action may fall out of the domain of the linear piece andis thus infeasible.
Revised: However, this action may fall out of the domain of the linear piece andis thus infeasible.

Original: We ﬁrst prove that such a shifted action indicates a better solution in its domain.
Revised: We first prove that such a shifted action indicates a better solution in its domain.

Original: Based on this conclusion, we ﬁrst show how to ﬁnd a feasible joint action with the global optimalvalue, and then derive an iterative algorithm with local optimum convergence guarantee.
Revised: Based on this conclusion, we first show how to find a feasible joint action with the global optimalvalue, and then derive an iterative algorithm with local optimum convergence guarantee.

Original: We demonstrate the improved representational capacity of our Non-Linear Coordination Graphs(NL-CG) on a matrix game by comparing the learned Q functions to those learned by conventionalcoordination graphs.
Revised: We demonstrate the improved representational capacity of our Non-Linear Coordination Graphs(NL-CG) on a matrix game by comparing the learned Q functions to those learned by conventionalcoordination graphs.

Original: We then evaluate our method on the Multi-Agent COordination (MACO)
Revised: We then evaluate our method on the Multi-Agent COordination (MACO)

Original: Benchmark [26] for its high requirements on close inter-agent coordination.
Revised: Benchmark [30] for its high requirements on close inter-agent coordination.

Original: The experimental resultsshow the superior performance enabled by the non-linear value decomposition.
Revised: The experimental resultsshow the superior performance enabled by the non-linear value decomposition.

Original:  Preliminaries
Revised:  Preliminaries

Original: In this paper, we focus on fully cooperative multi-agent tasks that can be modelled as a DecPOMDP [12] consisting of a tuple G = h
Revised: In this paper, we focus on fully cooperative multi-agent tasks that can be modelled as a DecPOMDP [15] consisting of a tuple G = ⟨

Original: I, S, A, P, R, ⌦ , O, n,  i , where I is the ﬁnite set of n agents, 2 [0 , 1) is the discount factor, and s 2 S is the true state of the environment.
Revised: I, S, A, P, R, Ω , O, n, γ ⟩ , where I is the finite set of n agents,γ ∈ [0 , 1) is the discount factor, and s ∈ S is the true state of the environment.

Original: At each timestep,each agent i receives an observation o i 2 ⌦ drawn according to the observation function
Revised: At each timestep,each agent i receives an observation o i ∈ Ω drawn according to the observation function

Original: O ( s, i ) andselects an action a i 2 A .
Revised: O ( s, i ) andselects an action a i ∈ A .

Original: Individual actions form a joint action a 2 A n , which leads to a next states [-
-]
Revised: Individual actions form a joint action a ∈ A n , which leads to a next states [-
-]′

Original: according to the transition function P ( s 0 | s, a ) , a reward r = R ( s, a ) shared by all agents.
Revised: according to the transition function P ( s ′ | s, a ) , a reward r = R ( s, a ) shared by all agents.

Original: Eachagent has local action-observation history ⌧ i 2 T ⌘ ( ⌦ ⇥ A ) ⇤ ⇥ ⌦ .
Revised: Eachagent has local action-observation history τ i ∈ T ≡ (Ω × A ) ∗ × Ω .

Original: Agents learn to collectively 65 maximize the global return Q 
Revised: Agents learn to collectivelymaximize the global return Q tot ( s, a ) =

Original: [Equation]
Revised: E s 0: ∞ ,a

Original: {+0: ∞ [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | s 
 = s, a 
 = a ] .+} Estimating Q tot is at the core of both value-based [21, 17, 19, 23] and policy-based [10, 27] multiagent reinforcement learning (MARL).
Revised: {+0: ∞ [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | s 
 = s, a 
 = a ] .+} Estimating Q tot is at the core of multi-agent reinforcement learning (MARL) [25, 20, 23, 27, 13, 31].

Original: However, the complexity of calculating max Q tot grows exponentially ( | A | n ) with the number of agents.
Revised: The complexity of calculating max Q tot grows exponentially ( | A | n ) with the number of agents.

Original: To solve this problem, value-based MARL decomposes theglobal action-value function into local utility functions q i and guarantees that the global maximizer canbe obtained locally: arg max a Q tot ( s, a ) =
Revised: Tosolve this problem, value-based MARL decomposes the global action-value function into local utilityfunctions q i and guarantees that the global maximizer can be obtained locally: arg max a Q tot ( s, a ) =

Original:  arg max [Equation] ( ⌧ 1 , [Equation]
Revised: 70 (cid:0) arg max a 1 q 1 ( τ 1 , a 1 ) , . .

Original: . , arg max [Equation] n q n ( ⌧ n , a n )  T .
Revised: . , arg max a n q n ( τ n , a n ) (cid:1) T .

Original: Value decomposition network (VDN) [21] satisﬁes this condition by learning the global value function as a summation of local utilities (Fig. 
Revised: Value decomposition network (VDN) [25]satisfies this condition by learning the global value function as a summation of local utilities (Fig. 1left).

Original: 1-left): Q tot  ( s, a )=
Revised: QMIX [20] extends the function class of VDN by learning Q tot as a monotonic mixing of localutilities. The mixing function is parameterized so that: ∂Q tot ( [Equation] , a )

Original: P i q i ( ⌧ i , a i )  .
Revised: [Equation] i ( τ i ,a i ) ≥ 0 .

Original: QMIX [17] extendsthe function class of

Original: 2.1 Coordination Graphs
Revised: 2.1 Coordination Graphs

Original: For fully decomposed value functions, local utility functions are conditioned on local actionobservation history.
Revised: For fully decomposed value functions, local utility functions are conditioned on local actionobservation history.

Original: Coordination Graphs (CGs) [7] increase the representational capacity of fullydecomposed value functions by introducing higher-order payoff functions.
Revised: Coordination Graphs (CGs) [9] increase the representational capacity of fully decomposed value functions by introducing higher-order payoff functions.

Original: Speciﬁcally, a coordination graph is a tuple of a vertex set and an edge set: G = h V , E i .
Revised: Specifically, a coordination graph is a tuple of a vertex set and an edge set: G = ⟨V , E⟩ .

Original: Each vertex v i 2 V represents 80 an agent i , and (hyper-) edges in E represent coordination dependencies among agents.
Revised: Each vertex v i ∈ V representsan agent i , and (hyper-) edges in E represent coordination dependencies among agents.

Original: In previouswork, the global value functions are decomposed linearly based on the graph topology:
Revised: In previouswork, the global value functions are decomposed linearly based on the graph topology:

Original: [Equation]
Revised: [Equation]

Original: where q i and q ij is utility functions for individual agents and pairwise payoff functions, respectively. ⌧ ij = h ⌧ i , ⌧ j i and a ij = h a i , a j i is the joint action-observation history and action of agent i and j .
Revised: where q i and q ij is utility functions for individual agents and pairwise payoff functions, respectively.τ ij = ⟨ τ i , τ j ⟩ and a ij = ⟨ a i , a j ⟩ is the joint action-observation history and action of agent i and j .

Original: Previous work studies different aspects of such coordination graphs.
Revised: Previous work studies different aspects of such coordination graphs.

Revised: It is shown that higher-orderfactorization is important on tackling an exponential number of joint actions [5].

Original: Sparse cooperative Q-learning [9]learns value functions for sparse coordination graphs with pre-deﬁned and static topology.
Revised: Sparse cooperative Q-learning [11] learns value functions for sparse coordination graphs with pre-defined and statictopology.

Original: Zhang etal ..
Revised: Zhang et al .

Original: [30] propose to learn minimized dynamic coordination sets for eachagent, but the computationalcomplexity grows exponentially with the neighborhood size of an agent.
Revised: [34] propose to learn minimized dynamic coordination sets for each agent.

Original: DCG [3] incorporatesdeep function approximation and parameter sharing into coordination graphs and scales to largestate-action spaces.
Revised: DCG [4] incorporates deep function approximation and parameter sharing into coordination graphsand scales to large state-action spaces.

Original: CASEC [26] studies how to build sparse deep coordination graphs that areadaptive to the dynamic coordination requirements.
Revised: CASEC [30] studies how to build sparse deep coordinationgraphs that are adaptive to the dynamic coordination requirements.

Original: In all these works, the global value functionis represented as a summation of local value functions.
Revised: In all these works, the globalvalue function is represented as a summation of local value functions.

Original: To the best of our knowledge, this paperpresents the ﬁst coordination graph learning method with non-linear value decomposition.
Revised: To the best of our knowledge,this paper presents the fist coordination graph learning method with non-linear value decomposition.

Original: 2.2 Message Passing
Revised: 2.2 Message Passing

Original: Within a coordination graph, the greedy action selection required by Q-learning can not be completedby simply computing the maximum of individual utility and payoff functions.
Revised: Within a coordination graph, the greedy action selection required by Q-learning can not be completedby simply computing the maximum of individual utility and payoff functions.

Original: Instead, distributedconstraint optimization (DCOP) [4] techniques are used.
Revised: Instead, distributedconstraint optimization (DCOP) [6] techniques are used.

Original: Max-Sum [20] is a popular implementationof DCOP.
Revised: Max-Sum [24] is a popular implementationof DCOP.

Original: Max-Sum ﬁnds near-optimal actions on a CG G = h V , E i via multi-round message passing 98 on a bipartite graph G m = h V 
Revised: Max-Sum finds near-optimal actions on a CG G = ⟨V , E⟩ via multi-round message passingon a bipartite graph G m = ⟨V a , V q , E m ⟩ .

Original:  V 
Revised: Each node i ∈ V a represents an agent, and each nodeg ∈ V q represents a utility ( q i ) or payoff ( q ij ) function.

Original: , E m 
Revised: Edges in E m connect g with the associatedagent node(s).

Original: 99 g 
Revised: Message passing starts with sending messages from node i ∈ V a to node g ∈ V q :

Original: [Equation]
Revised: [Equation]

Original: where F i is the set of nodes in V q connected to node i , and c ig is a normalizing factor preventing thevalue of messages from growing arbitrarily large.
Revised: where F i is the set of nodes in V q connected to node i , and c ig is a normalizing factor preventing thevalue of messages from growing arbitrarily large.

Original: Messages are then sent from node g to node i :
Revised: Messages are then sent from node g to node i :

Original: [Equation]
Revised: [Equation]

Original: where V g is the set of nodes in V a connected to node g , a g = { a h | h 2 V g } , a g \ a i
Revised: where V g is the set of nodes in V a connected to node g , a g = { a h | h ∈ V g } , a g \ a i

Original: = { a h | h 2 V g \{ i }} , 104 and q represents utility or payoff functions conditioned on a g .
Revised: = { a h | h ∈ V g \{ i }} ,and q represents utility or payoff functions conditioned on a g .

Original: Eq.
Revised: Eq.

Original: 2 and 3 make up an iteration of message passing.
Revised: 2 and 3 make up an iteration of message passing.

Original: Each agent i can ﬁnd its local optimal action by calculating a ⇤ i =argmax a 
Revised: Each agent i can find its local optimal action by calculating a ∗ i =argmax a i (cid:80) h ∈F

Original: i P
Revised: i m h

Original: [Equation]
Revised: →

Original: i ( a i ) after several iterations of message passing.
Revised: i ( a i ) after several iterations of message passing.

Original: Notably, Max-Sum and 107 other DCOP algorithms are only applicable to linearly decomposed value functions.
Revised: Notably, Max-Sum andother DCOP algorithms are only applicable to linearly decomposed value functions.

Original: 2.3 Piece-Wise Linear Neural Networks
Revised: 2.3 Piece-Wise Linear Neural Networks

Original: Following the recent decade’s success of deep neural networks (DNNs), analysis works have beendone trying to explain the mechanism of the DNN black-boxes and assess their function approximationcapabilities.
Revised: Following the recent decade’s success of deep neural networks (DNNs), analysis works have beendone trying to explain the mechanism of the DNN black-boxes and assess their function approximationcapabilities.

Original: One conclusion is that a DNN with piece-wise linear (PWL) activation functions (e.g.
Revised: One conclusion is that a DNN with piece-wise linear (PWL) activation functions (e.g.

Original: ReLU, LeakyReLU, PReLU) is equivalent to a PWL function.
Revised: ReLU, LeakyReLU, PReLU) is equivalent to a PWL function.

Original: This kind of DNNs are called piecewise linear neural networks (PLNNs) [5].
Revised: This kind of DNNs are called piecewise linear neural networks (PLNNs) [7].

Original: Early papers [11, 14] assess the expressivity of PLNNsby the amount of linear pieces.
Revised: Early papers [14, 17] assess the expressivity of PLNNsby the amount of linear pieces.

Original: [1, 18, 8] give more theoretically grounded results about the upperand lower bounds for the amount of pieces.
Revised: [2, 22, 10] give more theoretically grounded results about the upperand lower bounds for the amount of pieces.

Original: In this sense, it is shown that increasing the depth ofa network can generally be exponentially more valuable than increasing the width [14, 6, 1].
Revised: In this sense, it is shown that increasing the depth ofa network can generally be exponentially more valuable than increasing the width [17, 8, 2].

Original: Chuet al .
Revised: Chuet al .

Original: [5] propose OPENBOX to compute the mathematically equivalent set of linear pieces, which provides an accurate and consistent interpretation of PLNNs.
Revised: [7] propose OPENBOX to compute the mathematically equivalent set of linear pieces, whichprovides an accurate and consistent interpretation of PLNNs.

Original: For neural networks with a single layerof hidden nodes, the problem can be reduced to hyper-plane arrangement [29], and linear functionsand their domain can be enumerated efﬁciently [16, 2].
Revised: For neural networks with a single layerof hidden nodes, the problem can be reduced to hyper-plane arrangement [33], and linear functionsand their domain can be enumerated efficiently [19, 3].

Original:  Method
Revised:  Method

Original: Our method extends the representational capability of coordination graphs by introducing non-linearfactorization of the global value function.
Revised: Our method extends the representational capability of coordination graphs by introducing non-linearfactorization of the global value function.

Original: Speciﬁcally, for a coordination graph G = h V , E i , we 124 decompose the global Q as:
Revised: Specifically, for a coordination graph G = ⟨V , E⟩ , wedecompose the global Q as:

Original: [Equation]
Revised: [Equation]

Original: where q i is the vector of all individual utilities, and q ij is the vector of all pairwise payoffs foredges in E .
Revised: where q i is the vector of all individual utilities, and q ij is the vector of all pairwise payoffs foredges in E .

Original: Similar to DCG [3], we learn a shared utility function f v , parameterized by ✓ v , and getthe individual utility q i ( ⌧ i , a i ) =
Revised: Similar to DCG [4], we learn a shared utility function f v , parameterized by θ v , and getthe individual utility q i ( τ i , a i ) =

Original: f v ( a i | ⌧ i ; ✓ v ) .
Revised: f v ( a i | τ i ; θ v ) .

Original: The payoffs are estimated by a shared function f eparameterized by ✓ e : q ij ( ⌧ i , ⌧ j , a i , a j ) =
Revised: The payoffs are estimated by a shared function f eparameterized by θ e : q ij ( τ i , τ j , a i , a j ) =

Original: f e ( a i , a j | ⌧ i , ⌧ j ; ✓ e ) .
Revised: f e ( a i , a j | τ i , τ j ; θ e ) .

Original: Different from conventional CGs (Eq. 1), our f n is a  mixing network whose parameters are generatedby a hypernet f h conditioned on the global state s and parameterized by ✓ h .
Revised: Different from conventional CGs (Eq. 1), our f n is a non-linear mixing network whose parametersare generated by a hypernet f h conditioned on the global state s and parameterized by θ h .

Original: Our discussion is basedon LeakyReLU networks with the slope coefﬁcient ↵ 2 [0 , 1] , but it can readily extended to otheractivation functions like ReLU.
Revised: Ourdiscussion is based on LeakyReLU networks with the slope coefficient α ∈ [0 , 1] .

Original: For efﬁciently calculating the maximizer of Q tot , we require the weights after the ﬁrst layer of themixing network to be non-negative.
Revised: For efficientlycalculating the maximizer of Q tot , we require the weights after the first layer of the mixing networkto be non-negative.

Original: The reason for this non-negativity constraint will be discussed in Lemma 1 and 2.
Revised: The reason for this non-negativity constraint will be discussed in Lemma 1and 2.

Revised: Such a mixing network is effectively a type of input convex neural networks (ICNN [1]).

Revised: Thenon-negativity constraint on parameters is somewhat constraint, but we can use the passthrough trickintroduced in Proposition 1 of [1] to maintain substantial representation power of the mixing network.

Original: The whole framework, including the utility function f v , the payoff function f e , andthe hypernet f h , is updated by minimizing the TD loss:
Revised: The whole framework, including the utility function f v , the payoff function f e , and the hypernet f h ,is updated by minimizing the TD loss:

Original: [Equation]
Revised: [Equation]

Original: where D is the replay buffer, and ˆ Q tot is a target function whose parameters are periodically copiedfrom the function Q tot .
Revised: where D is the replay buffer, and ˆ Q tot is a target function whose parameters are periodically copiedfrom the function Q tot .

Original: The main challenge left, for both the training and execution phase of deep Q-learning, is to selectactions that maximize the global Q value at each time step, arg max a Q tot ( ⌧ , a ) .
Revised: The main challenge left, for both the training and execution phase of deep Q-learning, is to selectactions that maximize the global Q value at each time step, arg max a Q tot ( τ , a ) .

Original: This problemis theoretically NP-hard when the mixing function is non-linear.
Revised: This problemis theoretically NP-hard when the mixing function is non-linear.

Original: In the following sections, weﬁrst provide a global optimal algorithm for this problem.
Revised: In the following sections, wefirst provide a global optimal algorithm for this problem.

Original: The complexity of this algorithm growspolynomially with the number of hidden units in the mixing network.
Revised: The complexity of this algorithm growspolynomially with the number of hidden units in the mixing network.

Original: To reduce time complexity, wepropose an iterative algorithm with local optimum convergence guarantee.
Revised: To reduce time complexity, wepropose an iterative algorithm with local optimum convergence guarantee.

Original: We use the following notations .
Revised: We use the following notations .

Original: The input to the mixing network is q =
Revised: The input to the mixing network is q =

Original: [ q i k q ij ] , where [ · k · ] 146 means vector concatenation.
Revised: [ q i ∥ q ij ] , where [ ·∥· ] meansvector concatenation.

Revised: We use q ( a ) to denote the utilities and payoffs corresponding to action a .

Original: The mixing network has Llinear layers with LeakyReLU activation.d = |V| + |E| is the input dimension.
Revised: Themixing network has L LeakyReLU linear layers. d = |V| + |E| is the input dimension.

Original: m i , W i , and b i 2 R m i are the width, weights, and biases of the148 i th layer.
Revised: m i , W i , andb i ∈ R m i are the width, weights, and biases of the i th layer.

Original: The weights after the ﬁrst layer are non-negative.
Revised: The weights after the first layer arenon-negative.

Original: The input to the activation units at i th layeris o i , and the corresponding output is h 
Revised: The input to the activation units at i th layer is z i , and the corresponding output is h i =

Original: i = LeakyReLU
Revised: LeakyReLU

Original: ( o i ) =
Revised: ( z i ) =

Original: c i  o i , where c i is the value of theactivation function and  is the element-wise multiplication.
Revised: c i ◦ o i , where c i is the value of LeakyReLU activation ( c = α when z < 0 andc = 1 when z ≥ 0 ) and ◦ is the element-wise multiplication.

Original: We call c =
Revised: We call c =

Original: [ c 1 k . . .
Revised: [ c 1 ∥ . . .

Original: k c L ] 2 { ↵ , 1 } m , 151 m = P i m i , a slope conﬁguration of the mixing network.
Revised: ∥ c L ] ∈ { α, 1 } m ,m = (cid:80) i m i , a slope configuration of the mixing network.

Original: Subscripts of W i , b i , o i , and h i meansindex.
Revised: Subscripts of W i , b i , z i , and h i meansindex.

Original: For example, o ij is the j th element of o i , and W ij is the j th row of W i .
Revised: For example, z ij is the j th element of z i , and W ij is the j th row of W i .

Original: 3.1 Piece-Wise Optimization
Revised: 3.1 Piece-Wise Optimization

Original: The general idea of our action selection algorithm is to utilize the piece-wise linear property of the
Revised: The general idea of our action selection algorithm is to utilize the piece-wise linear property of the

Original: LeakyReLU network.
Revised: LeakyReLU network.

Original: Given a slope conﬁguration c , the mixing network becomes linear, and we canrun a DCOP algorithm to get the corresponding maximizer of Q tot .
Revised: Given a slope configuration c , the mixing network becomes linear, and we canrun a DCOP algorithm to get the corresponding maximizer of Q tot .

Original: The question is that the obtainedsolution may be out of the domain of the linear piece.
Revised: The question is that the obtainedsolution may be out of the domain of the linear piece.

Original: We show that, when the weights after the ﬁrstlayer are non-negative, we can ignore whether the obtained local optimal solution is in the piece’sdomain, and the global maximizer of Q tot is the best of the local optima.
Revised: We show that, when the weights after the firstlayer are non-negative, we can ignore whether the obtained local optimal solution is in the piece’sdomain, and the global maximizer of Q tot is the best of the local optima.

Original: Formally, there are 2 m slope conﬁgurations in C all
Revised: Formally, there are 2 m slope configurations in C all

Original: = { c | c 2 { ↵ , 1 } m } , each of which makes themixing network an afﬁne function in P all = { ⇢ 1 , ⇢ 2 , . .
Revised: = { c | c ∈ { α, 1 } m } , each of which makesthe mixing network an affine function in P all = { ρ 1 , ρ 2 , . .

Original: . , ⇢ 2 m } .
Revised: , ρ 2 m } .

Original:  For each c k 2 C all , by running162 DCOP, we obtain the local maximum q k and the corresponding joint action a k and utilities/payoffsq k .
Revised: Each affine function ρ k has acorresponding cell P k where q ∈ P k yields c k in the forward pass. For each c k ∈ C all , by running DCOP, we obtain the local maximum q k and the corresponding joint action a k and utilities/payoffsq k .

Original: However, it is possible that a k falls out of ⇢ k and indeed lead to another slope conﬁgurationc r [-
-] = k .
Revised: However, it is possible that q k falls out of P k and indeed yields another slope configuration c r [-
-]̸ = k .

Original: We ﬁrst show that such a shifted solution indicates an equal or better solution in its domain.
Revised: We first show that such a shifted solution indicates an equal or better solution in its domain.

Original: Lemma 1.
Revised: Lemma 1.

Revised: Denote affine function pieces and their cells of a fully-connected feedforward mixingnetwork with LeakyReLU activation as P all = { ρ j } 2 m 1 and { P j } 2 m 1 .

Original: For q corresponding to afﬁne function ⇢ r , for 8 ⇢ s 6 = r 2 P all , we have ⇢ r ( q )  ⇢ s6 = r ( q ) .
Revised: For q in the cell of the r th piece, P r , and ∀ ρ s ∈ P all , we have ρ r ( q ) ≥ ρ s ( q ) .

Original: Proof.
Revised: Proof.

Original: We start with the ﬁrst difference between c r and c s .
Revised: We start with the first difference between c r and c s .

Original: Suppose that they differ at the j th unit atlayer i : c rij [-
-] = c sij . 
Revised: We denote it as c rij [-
-]̸ = c sij . Because thisis the first difference, the input to this unit, z ij , is the same.

Original: Since q is in the domain of ⇢ r , we have that c rij is 1 when o ij is positive and is↵ < 1 when o ij is negative.
Revised: Since q is in P r , we have that c rij is 1when z ij is positive and is α < 1 when z ij is negative.

Original: Therefore, the output of layer i satisﬁes h rij 
Revised: Therefore, the output satisfies that

Original: h sij .
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: because W i +1  0 .
Revised: because W i +1 ≥ 0 .

Original: Other differences in the i th layer will lead to the same conclusion.
Revised: Other differences in the i th layer will lead to the same conclusion.

Original: At layeri + 1 , we have
Revised: At layeri + 1 , we have

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequity is because q is in the domain of ⇢ r .
Revised: The first inequity is because q is in the cell of ρ r .

Original: The second inequity is because of Eq.
Revised: The second inequity is because of Eq.

Original: 6 andthat LeakyReLU is a monotonic increasing function.
Revised: 7 and that LeakyReLU is a monotonic increasing function.

Original: The proof can repeat for the following layersand thus holds for the last layer, i .
Revised: The proof can repeat for the following layers andthus holds for the last layer, i .

Original: e ., ⇢ r ( q )  ⇢ s
Revised: e ., ρ r ( q ) ≥

Original: 6 = r ( q ) .
Revised: ρ s ( q ) .

Original: Based on Lemma 1, when the local optimal solution, a k , of piece k returned by DCOP actuallycorresponds to piece r [-
-] = k , we have ⇢ r ( q ( a k )) 
Revised: Based on Lemma 1, when the local optimal solution, a k , of piece k returned by DCOP actually fallsin cell P r [-
-]̸ = k , we have ρ r ( q ( a k )) ≥ ρ k ( q ( a k )) .

Original:  ⇢ k ( q ( a k )) , where q ( a k ) is the vector of utilitiesand payoffs of a k .

Original: This indicates that, on piece r , we will get a solution at least as good as the one onpiece k .
Revised: This indicates that, on piece r , we will get a solutionat least as good as the one on piece k .

Original: Based on this conclusion, we now prove that the maximum of each piece’s local optimum isthe global optimum.
Revised: Based on this conclusion, we now prove that the maximum ofeach piece’s local optimum is the global optimum.

Original: Lemma 2.
Revised: Lemma 2.

Original: Running DCOP algorithm for each linear piece in P all , then take the maximum valueamong these pieces, we can get the global optimizer of Q tot .
Revised: Running DCOP algorithm for each linear piece in P all , then take the maximum valueamong these pieces, we can get the global optimizer of Q tot .

Original: Proof.
Revised: Proof.

Original: We have that max q f n ( q ) = max q max ⇢ p 2 P all ⇢ p ( q ) = max ⇢ p 2 P all max q ⇢ p ( q ) ( f n is the 182 mixing function, Eq. 4), which means the maximum of local optimal values is the global optimalvalue.
Revised: We have that max q f n ( q ) = max q max ρ p ∈P all ρ p ( q ) = max ρ p ∈P all max q ρ p ( q ) ( f n is themixing function, Eq. 4), which means the maximum of local optimal values is the global optimalvalue.

Original: The question is whether the corresponding action is infeasible, i .
Revised: Suppose the global optimal value Q max is found on piece r and the corresponding action is a r ,the question is whether q ( a r ) is feasible, i .

Original: e .,  a shifted solution.
Revised: e ., q ( a r ) ∈ P r .

Original: Supposethat the global optimum Q max is obtained in piece r , and running DCOP on this piece returns asolution a r  in the domain of piece s 6 = r .
Revised: Assume that q ( a r ) falls in P s ̸ = r .

Original: Then for piece s , a r is a feasible solution with the value Q max (due to Lemma 1).
Revised: Thenfor piece s , a r is a feasible solution with the value Q max (due to Lemma 1).

Original: This means the global optimal solution is always a feasible solution.
Revised: This means the globaloptimal solution is always a feasible solution.

Original: According to Lemma 2, we can run message passing on each piece (Algorithm 1) and then take thebest of these results to get the global optimum.
Revised: According to Lemma 2, we can run message passing on each piece (Algorithm 3 in Appx. B) and thentake the best of these results to get the global optimum.

Original: Detailed process can be found in Algorithm 2.
Revised: Detailed process can be found in Algorithm 1.

Original: The problem of Algorithm 2 is time complexity.
Revised: The problem of Algorithm 1 is time complexity.

Original: When m is small, we enumerate all 2 m slopeconﬁgurations.
Revised: When m is small, we enumerate all 2 m slopeconfigurations.

Original: For a large m , we can use the hyperplane arrangement algorithm [5] to calculateexact linear pieces and their domains.
Revised: For a large m , we can use the hyperplane arrangement algorithm [7] to calculateexact linear pieces and their domains.

Original: Speciﬁcally, there are
Revised: Specifically, there are

Original: [Equation]
Revised: [Equation]

Original: pieces need enumerating, where d is the input length.
Revised: pieces need enumerating, where d is the input length.

Original: This number is exponential to d .
Revised: This number is exponential to d .

Original: To reducetime complexity, we propose an iterative optimization method in the next section.
Revised: To reducetime complexity, we propose an iterative optimization method in the next section.

Original: 3.2 Iterative Optimization with Local Optimum Guarantee
Revised: 3.2 Iterative Optimization with Local Optimum Guarantee

Original: When m is large, enumerating either 2 m or n m,d pieces can be costly.
Revised: When m is large, enumerating either 2 m or n m,d pieces can be costly.

Original: We thus propose an iterativealgorithm with local optimum guarantee (Algorithm 3) to get approximate solutions.
Revised: We thus propose an iterativealgorithm with local optimum guarantee (Algorithm 2) to get approximate solutions.

Original: We begin withthe slope conﬁguration where all hidden units are activated ( c = { 1 } m ).
Revised: We begin withthe slope configuration where all hidden units are activated ( c = { 1 } m ).

Original: We run message passing onthe current conﬁguration c
Revised: We run message passing onthe current configuration c

Original: p and get a local optimum solution a p .
Revised: p and get a local optimum solution a p .

Original: If the real conﬁguration c real ofa p is not c p , we calculate the local optimum of c real .
Revised: If the real configuration c real ofa p is not c p , we calculate the local optimum of c real .

Original: The iteration continues until c real =
Revised: The iteration continues until c real =

Original: c p .
Revised: c p .

Original: Lemma 1 guarantees that the solution values in the iteration are monotonically increasing.
Revised: Lemma 1 guarantees that the solution values in the iteration are monotonically increasing.

Original: Sincethere is a ﬁnite number of conﬁgurations, our algorithm can converge to a local optimum.
Revised: Sincethere is a finite number of configurations, our algorithm can converge to a local optimum.

Original: With a very low probability, a loop would appear in the iteration.
Revised: With a very low probability, a loop would appear in the iteration.

Original: The appearance of a loop meansthat the equity in Lemma 1 holds.
Revised: The appearance of a loop meansthat the equity in Lemma 1 holds.

Original: In this case, the solutions on the loop have the same value, thus wecan stop our iteration when encountering a slope conﬁguration that has been searched.
Revised: In this case, the solutions on the loop have the same value, thus wecan stop our iteration when encountering a slope configuration that has been searched.

Original: To increase the possibility of ﬁnding a global optimum, we can introduce a simulated annealingmechanism.

Original: Each time we ﬁnd a better solution, with a probability of 1  ✏ , we move to thecorresponding piece, and with a probability of ✏ , we jump to a random piece that has not beensearched.

Original: ✏ decreases with the iteration number.

Original: This algorithm is called by Algorithm 2 and 3, and the deﬁnition of inputs can be found there.*/

Original: [Equation]

Original: /*Initialize forward messages ( µ ) and backward messages ( ¯ µ ).*/

Original: [Equation]

Original: V /*Initial “Q-value”.*/

Original: [Equation]

Original:

Original: [Equation]

Original: ⇤ /*Initialize

Original: the best found solution.*/ for t 2 { 1 , . . .

Original: , k } do /* k rounds of message passing.*/ for e =

Original: ( i, j ) 2 E do /*Update forward and backward messages.

Original: Subscripts of f , q , µ mean indexing.*/

Original: [Equation]

Original: /*Forward: maximize sender.*/ ¯

Original: [Equation]

Original: = max a

Original: [Equation]

Original: /*Backward: maximize receiver.*/

Original: [Equation]

Original: µ tea /*Normalize forward messages to ensure converging.*/ ¯ µ te ¯

Original: [Equation]

Original: te

Original: [Equation]

Original: i |

Original: P

Original: [Equation]

Original: ¯ µ tea /*Normalize backward messages to ensure converging.*/ end for for i 2 V do /*Update

Original: “Q-value” with messages.*/ q ti :=

Original: f V i +

Original: X e =( · ,i ) 2 E µ te +

Original: X e =( i, · ) 2 E ¯ µ te /*Utility plus incoming messages.*/ a ti := arg max a 2 A i { q tia } /*Select the greedy action of agent

Original: i .*/ end for q

Original: [-P |V| i =
-]
Revised: [-P |V| i =
-]

Original: f V a i +

Original: P ( i,j ) 2 E f E a i a j + b /*Get the Q-value of the greedy action.*/ if q 0 > q max then { a max a t ; q max q 0 q q ti } /*Remember

Original: the best action.*/ end if end for return q max , a max 2 A 1 ⇥ . . . ⇥

Original: A |V| , q /*Return the maximum Q value, the corresponding action, and utilities/payoffs.*/

Original: Algorithm
Revised: Algorithm

Original: /*Show the case for a two-layer mixing network, but can be easily extended to more layers.*/
Revised: /*Show the case for a two-layer mixing network, but can be easily extended to more layers.*/

Original: Input: f V 2 R |V| ⇥ [Equation] f [Equation]
Revised: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , W 0 , b 0 , W 1 , b 1 q max :=

Original: max := 1 ; a max : [Equation] ⇥ ⇤ /*Initialize
Revised: −∞ ; a max := (cid:2)(cid:3) /*Initialize

Original: the best found solution.*/ for c p 2 { ↵ , 1 } m do /*Calculate the equivalent weights and biases.*/
Revised: the best found solution.*/ for c p ∈ { α, 1 } m do /*Calculate the equivalent weights and biases.*/

Original: [Equation] p 
Revised: W ρ p :=

Original: [Equation]
Revised: W 0 · ( c p ◦ W 1 )

Original: q [Equation]
Revised: b ρ p :=

Original:  a , · WEIGHTED-MAX-SUM ( f V , f E , W ⇢ p , b ⇢ p ) if q > q max then a max  a q max  q /*Remember only the best actions.*/ end if end for return a max /*Return
Revised: ( c p ◦ W 1 ) · b T 0 + bq, a , · ← w -MAX-SUM ( f V , f E , W ρ p , b ρ p ) if q > q max then a max ← a q max ← q /*Remember only the best actions.*/ end if end for return a max /*Return

Original: the selected joint action a max .*/
Revised: the selected joint action a max .*/

Original: Algorithm
Revised: Algorithm

Original: /*Show the case for a two-layer mixing network, but can be easily extended to more layers.*/
Revised: /*Show the case for a two- mixing network, but can be easily extended to more layers.*/

Original: Input: [Equation]
Revised: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , W 0 , b 0 , W 1 , b 1 , n max , ϵ q max :=

Original: = ⇥ ⇤ , c p := { 1 } m for n 2 { 1 , . . . , n max } do /*Try
Revised: −∞ , a max := (cid:2)(cid:3) , c p := { 1 } m for n ∈ { 1 , . . . , n max } do /*Try

Original: one conﬁguration at a time.*/
Revised: one configuration at a time.*/

Original: W [Equation]
Revised: W ρ p :=

Original: W  1 )b ⇢ p [Equation]
Revised: W 0 · ( c p ◦ W 1 )

Original: [Equation] b 
Revised: b ρ p :=

Original: [Equation]
Revised: ( c p ◦ W 1 ) ·

Original: b 1 q, a , · WEIGHTED-MAX-SUM ( f V , f E , W ⇢ p , b ⇢ p ) if q > q max then a max  a q max  q /*Remember only the best action.*/ end if  c real :=
Revised: b T 0 + bq, a , · ← w -MAX-SUM ( f V , f E , W ρ p , b ρ p ) if q > q max then a max ← a q max ← q c real ← The real LeakyReLU slope configuration end if if c real ̸ = c p then c

Original: MIXING-NETWORK ( a ) /*Feed a back to the mixing network and get the real LeakyReLU slope conﬁguration.*/ if c real [-
-]
Revised: p ← c real [-
-]else with prob. ϵ break or continue with an unvisited c p . end for return a max To increase the possibility of finding a global optimum, we can introduce a simulated annealingmechanism.

Original: = c p then c p c real /*Under this condition, c real must lead to a better (or at least equal) Q than c p .*/ else
Revised: Each time we find a better solution, with a probability of 1 − ϵ , we move to thecorresponding piece, and with a probability of ϵ , we jump to a random piece that has not beensearched.

Original: c p an unsearched conﬁguration end if end for return a max /*Return
Revised: ϵ decreases with the iteration number. Discussion about loopy graph topology In this paper, we consider complete graphs when studyingnon-linear coordination graphs. A concern is that message passing algorithms like Max-Sum maynot converge to the optimal solutions in loopy graphs and has an error rate of e .

Original:  the selected joint action a max .*/
Revised: Lemma 1 is notaffected because it is a property of LeakyReLU Networks. For Lemma 2, the maximum of solutionsfound by message passing in all slope configurations is the global optimum with a probability of1 − e . An error occurs when message passing cannot find the right solution on the piece where theglobal optimum is located. Our iterative method may stop earlier when message passing returns awrong solution located in the current cell.

Revised: The probability of this situation is less than e .

Revised: Thus wehave at least a probability of (1 − e ) n ( n is the number of iterations) to find the piece where the localoptimum is located, and the final probability of finding the local optimum is larger than (1 − e ) n +1 .

Original:  Representational Capability
Revised:  Representational Capability

Original: In this section, we compare the representational capacity of our model against conventional coordination graphs.
Revised: In this section, we compare the representational capacity of our model against conventional coordination graphs.

Original: The comparison is carried out on a two-step cooperative matrix game with four playersand two actions.
Revised: The comparison is carried out on a two-step cooperative matrix game with four playersand two actions.

Original: At the ﬁrst step, Agent 1’s action decides which of the two matrix games (Table.
Revised: At the first step, Agent 1’s action decides which of the two matrix games (Table.

Original: 1)to play in the next timestep.
Revised: 1)to play in the next timestep.

Original: In the second step, the number of agents taking Action B determines theglobal reward received by the agent team (Table. 1).
Revised: In the second step, the number of agents taking Action B determines theglobal reward received by the agent team (Table. 1).

Original: [Table]
Revised: [Table]

Revised: step game after Agent 1 chooses the first action.

Revised: Action A takes the agents to State 2A, while action B takes them to State 2B.

Revised: The team reward depends on the number of agents taking Action B.

Original: We ﬁrst theoretically prove that the representational capacity of conventional coordination graphsis unable to represent the Q -function of this game.
Revised: We first theoretically prove that the representational capacity of conventional coordination graphsis unable to represent the Q -function of this game.

Original: Since the reward is invariant to the identityof agents, i .
Revised: Since the reward is invariant to the identity ofagents, i .

Original: e .,
Revised: e .,

Original: R ( a 1 , a 2 , a 3 , a 4 ) =
Revised: R ( a 1 , a 2 , a 3 , a 4 ) =

Original: R ( p ( a 1 , a 2 , a 3 , a 4 )) , where p is an arbitrary permutation, thelearned action-value function should also be permutation invariant.
Revised: R ( p ( a 1 , a 2 , a 3 , a 4 )) , where p is an arbitrary permutation, the learnedaction-value function should also be permutation invariant.

Original: Therefore we can ignore theorder of actions in value functions.
Revised: Therefore we can ignore the order of actions in value functions.

Original: Now let’s focus on State 2B ( s 2 B ).
Revised: Now let’s focus on State 2B ( s 2 B ).

Original: For conventional coordinationgraphs, there are ﬁve unknowns: q i ( s 2 B , A ) = 0 , q i ( s 2 B , B ) , q ij ( s 2 B , AA ) , q ij ( s 2 B , AB ) , andq ij ( s 2 B , BB ) . Ignoring the order of actions, we get a  system  of ﬁve equations.
Revised: Conventional coordination graphs needto solve a linear system consisting of five unknowns and five equations.

Original: However, for thissystem, the rank of the augmented matrix  is greater than the rank of the coefﬁcient matrix.
Revised: However, for this system, therank of the augmented matrix (4) is greater than the rank of the coefficient matrix (3).

Original: Therefore,this system has no solution, and it is impossible for conventional CGs to learn a correct Q function.
Revised: Details can befound in Appx. C. Therefore, this system has no solution, and it is impossible for conventional CGsto learn a correct Q function.

Original: We then empirically demonstrate our idea.
Revised: We then empirically demonstrate our idea.

Original: We train NL-CG and DCG on the task for 5000 episodesunder full exploration ( ✏ = 1 ) and examine the learned value functions.
Revised: We train NL-CG and DCG on the task for 5000 episodesunder full exploration ( ϵ = 1 ) and examine the learned value functions.

Original: Full exploration ensuresthat both methods explore all state-action pairs.
Revised: Full exploration ensuresthat both methods explore all state-action pairs.

Original: In such a case, the representational capacity of theaction-value function approximator remains the only challenge of learning accurate Q functions.
Revised: In such a case, the representational capacity of theaction-value function approximator remains the only challenge of learning accurate Q functions.

Original: Forour algorithm, the utility and payoff function is fully connected networks with a single hidden layerof 64 units with a ReLU non-linearity.
Revised: Forour algorithm, the utility and payoff function is fully connected networks with a single hidden layerof 64 units with a ReLU non-linearity.

Original:  is 0.99, and the replay buffer stores the last 500 episodes,from which we uniformly sample batches of size 32 for training.
Revised: γ is 0.99, and the replay buffer stores the last 500 episodes,from which we uniformly sample batches of size 32 for training.

Original: The target network is updated every100 episodes.
Revised: The target network is updated every100 episodes.

Original: The learning rate of RMSprop is set to 5 ⇥ 10  4 .
Revised: The learning rate of RMSprop is set to 5 × 10 − 4 .

Original: Agents receive the full state asobservation, which is represented as an one-hot vector.
Revised: Agents receive the full state asobservation, which is represented as an one-hot vector.

Original: Table 2 and 3 show the learned Q by DCG and NL-CG.
Revised: Table 2 and 3 show the learned Q by DCG and NL-CG.

Original: In line with our theoretical analysis, we cansee that DCG learns a sub-optimal strategy of selecting Action A in the ﬁrst step.
Revised: In line with our theoretical analysis, we cansee that DCG learns a sub-optimal strategy of selecting Action A in the first step.

Original: By contrast, ourmethod learns the accurate value of Action B in State A and get the optimal strategy.
Revised: By contrast, ourmethod learns the accurate value of Action B in State A and get the optimal strategy.

Original: Furthermore,the Q values for State 2B learned by NL-CG is more accurate than those learned by DCG.
Revised: Furthermore,the Q values for State 2B learned by NL-CG is more accurate than those learned by DCG.

Original: Theseresults demonstrate that NL-CG’s higher representational capacity allows it to accurately estimate thevalue function of this game whereas DCG cannot.
Revised: Theseresults demonstrate that NL-CG’s higher representational capacity allows it to accurately estimate thevalue function of this game whereas DCG cannot.

Original: We also note that such an example is common fortask with few actions because there are typically more equations than unknowns.
Revised: We also note that such an example is common fortask with few actions because there are typically more equations than unknowns.

Original: [Table]
Revised: [Table]

Revised: Action

Original: [Table]
Revised: A B Q 6. State 2B

Original: by our method for the matrix game.
Revised: # Action B Q 1. [Table] [Table]

Original:  Experiments
Revised:  Experiments

Original: In this section, we conduct experiments to show the effectiveness of our method on complex tasks.
Revised: In this section, we conduct experiments to show the effectiveness of our method on complex tasks.

Original: We benchmark our method on the Multi-Agent COordination (MACO) benchmark [26], whichcovers various classic coordination tasks in the literature of multi-agent learning and increases theircomplexity to better evaluate the performance of different algorithms.
Revised: We benchmark our method on the Multi-Agent COordination (MACO) benchmark [30], whichcovers various classic coordination tasks in the literature of multi-agent learning and increases theircomplexity to better evaluate the performance of different algorithms.

Original: The MACO benchmark ischaracterized by a high demand on the sophistication of agent coordination.
Revised: The MACO benchmark ischaracterized by a high demand on the sophistication of agent coordination.

Original: For NL-CG, we use a mixing network that has one hidden layer with  widths.
Revised: For NL-CG, we use amixing network that has one hidden layer with different widths.

Original: Detailed hyperparameter settingsof our method can be found in Appendix A. For fair comparison, we run all our experiments with 5random seeds and show the mean performance with a 95% conﬁdence interval.
Revised: Detailed hyperparameter settingsof our method can be found in Appendix A. For fair comparison, we run all our experiments with 5random seeds and show the mean performance with a 95% confidence interval.

Original: In Fig.
Revised: In Fig.

Original: 2, we compare NL-CG against the state-of-the-art coordination graph learning method(DCG [3]) anda fully decomposed value function (QMIX [17]).
Revised: 2, we compare NL-CG against the state-of-the-art coordination graph learning method(DCG [4]) and fully decomposed methods (QMIX [20], DICG [12]).

Original: For NL-CG, we (1) use a hidden layerof three units and enumerate all 4 slope conﬁgurations and (2) use a hidden layer of four units and runour iterative method with n max = 4 .
Revised: For NL-CG, we (1) use ahidden layer of three units and enumerate all 8 slope configurations and (2) use a hidden layerof four units and run our iterative method with n max = 4 .

Original: The result shows that our algorithm can outperform conventional CGs signiﬁcantly.
Revised: The result shows that our algorithmcan outperform conventional CGs significantly.

Original: Moreover, our iterative optimization method has comparable performance withthe enumeration method, showing its effectiveness.
Revised: Moreover, our iterative optimization method hascomparable performance with the enumeration method, showing its effectiveness.

Original: QMIX struggles on these tasks, indicating thatthese tasks are beyond the representation capacity of a fully decomposed function.
Revised: QMIX struggles onthese tasks, indicating that these tasks are beyond the representation capacity of a fully decomposedfunction.

Original: These resultsdemonstrate the advantage of non-linear CGs on tasks that need sophisticate agent coordination.
Revised: These results demonstrate the advantage of non-linear CGs on tasks that need sophisticateagent coordination.

Original: We further investigate the the inﬂuence of the mixing network’s width.
Revised: We further investigate the the influence of the mixing network’s width.

Original: Speciﬁcally, we test NLCG with a width of 2, 3, 4, and 10 and compare their performance in Fig.
Revised: Specifically, we test NLCG with a width of 2, 3, 4, and 10 and compare their performance in Fig.

Original: It can be observedthat, generally, more units (10) in the hidden layer lead to better or at least equal performance thanother conﬁgurations.
Revised: It can be observedthat, generally, more units (10) in the hidden layer lead to better or at least equal performance thanother configurations.

Original: This result is in line with our motivation: a powerful non-linear mixing networkincreases the capability of CGs.
Revised: This result is in line with our motivation: a powerful non-linear mixing networkincreases the capability of CGs.

Original: In Fig.
Revised: In Fig.

Original: 4, we compare our enumerative and iterative method.
Revised: 4, we compare our enumerative and iterative methods.

Original: These two methods check the samenumber (16) of linear pieces, but the iterative method can use a hidden layer of 10 .
Revised: These two methods checkthe same number (16) of linear pieces, but the iterativemethod can use a hidden layerof 10 .

Original: As a result, theiterative method outperforms the enumerative method.
Revised: As a result, the iterativemethod outperforms the enumerative method.

Revised: Additionally, in Fig. 5, we compare NL-CG against DCG [4],

Revised: QMIX [20], and DICG [12] on

Revised: Pursuit and MMM2 from the

Revised: SMAC benchmark.

Original:  5.1 Analysis of the optimality and efﬁciency of the iterative optimization method
Revised: We find NL-CG generally outperforms previous methods. On Pursuit , NL-CG with wider mixers use very few (20K-30K) samples toachieve DCG’s performance after convergence. 5.1 Analysis of the optimality and efficiency of the iterative optimization method

Revised: [Figure]

Original: Although we have checked the performance of our iterative optimization method, learning curves can not fully reveal its optimality and efﬁciency.
Revised: Although we have checked the performance of our iterative optimization method, learning curves can not fully reveal its optimality and efficiency.

Original: In this section, we provide optimality and efﬁciencyanalyses by checking the action selection results and time costs in detail.
Revised: In this section, we provide optimality and efficiencyanalyses by checking the action selection results and time costs in detail.

Original: On Aloha , Sensor and Gather , we use our piece-wise enumeration method to train non-linear CGmodels with a width of 4.
Revised: On Aloha , Sensor and Gather , we use our piece-wise enumeration method to train non-linear CGmodels with a width of 4.

Original: Then we use the enumerative and iterative optimization method to selectgreedy actions.
Revised: Then we use the enumerative and iterative optimization method to selectgreedy actions.

Original: We compare the Q tot of the actions selected by two methods in Fig.
Revised: We compare the Q tot of the actions selected by two methods in Fig.

Original: Also shownthere is the Q-values of the actions selected randomly and by DCG.
Revised: Also shownthere is the Q-values of the actions selected randomly and by DCG.

Original: We can see that the iterativemethod is near-optimal after checking only 4 slope conﬁgurations: the Q-values of its actions are veryclose to those selected by enumeration and are better than those selected by DCG.
Revised: We can see that the iterativemethod is near-optimal after checking only 4 slope configurations: the Q-values of its actions are veryclose to those selected by enumeration and are better than those selected by DCG.

Original: Since, theoretically,enumerative optimization can ﬁnd global optimum, points that are closer to the line y
Revised: Since, theoretically,enumerative optimization can find global optimum, points that are closer to the line y

Original: = x indicatebetter solutions.
Revised: = x indicatebetter solutions.

Original: We also note that, in practice, Max-Sum is an approximate algorithm on cyclicgraphs, and enumeration cannot guarantee global optimality.
Revised: We also note that, in practice, Max-Sum is an approximate algorithm on cyclicgraphs, and enumeration cannot guarantee global optimality.

Original: Some actions selected by the iterativemethod are even better than those selected by enumeration.
Revised: Some actions selected by the iterativemethod are even better than those selected by enumeration.

Original: In Fig.
Revised: In Fig.

Original: 6, we compare the time spent byenumerative and iterative optimization methods.
Revised: 7, we compare the time spent byenumerative and iterative optimization methods.

Original: It can be found that the iterative method saves 50%to 65% of running time.
Revised: It can be found that the iterative method saves 50%to 65% of running time.

Original: We can thus conclude that the iterative optimization method provides a goodtrade-off between complexity and optimality.
Revised: We can thus conclude that the iterative optimization method provides a goodtrade-off between complexity and optimality.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we extend coordination graphs beyond linear decomposition by introducing non-linearmixing networks.
Revised: In this paper, we extend coordination graphs beyond linear decomposition by introducing non-linearmixing networks.

Original: Experiments manifest its superior representation power on complex tasks thatconventional CGs are not able to solve.
Revised: Experiments manifest its superior representation power on complex tasks thatconventional CGs are not able to solve.

Original: An important research direction is the stability of non-linear
Revised: An important research direction is the stability of non-linear

Original: CGs and to get rid of the non-negative constraint on weights of the mixing network.
Revised: CGs and to get rid of the non-negative constraint on weights of the mixing network.

Original: The authors donot see obvious negative societal impacts of our method.
Revised: The authors donot see obvious negative societal impacts of our method.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: [Yes](d) Did you include the total amount of compute and the type of resources used (e

Original: of GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: A Hyperparameters and Infrastructure

Revised: When comparing the performance of NL-CG against DCG [4] and QMIX [20] on the MACObenchmark, we adopted the following hyper-parameter settings.

Revised: For all tasks, we used a discountfactor γ = 0 .

Revised: 99 and ϵ -greedy exploration, where ϵ was linearly decayed from 1 to 0 .

Revised: within the first50,000 training time steps.

Revised: The replay buffer stored the last 5000 episodes, from which we uniformlysampled batches of size 32 for training.

Revised: The target network was updated every 200 episodes.

Revised: Thelearning rate of RMSprop was set to 5 × 10 − 4 .

Revised: Every 10000 time steps we paused training andevaluated the model with 300 greedy test trajectories sampled with ϵ = 0 .

Revised: Each agent processed its local action-observation history at each time step using a linear layer of64 neurons, followed by a ReLU activation, a GRU of the same dimensionality, and finally a linearlayer of | A | neurons.

Revised: The output served as an individual feature vector and was fed into the utilityand payoff function.

Revised: The parameters of the utility and payoff function were shared amongst agents,who were identified by a one-hot encoded ID in the input.

Revised: All message passing procedures in

Revised: NL-CG iterated for 4 rounds.

Revised: The weights of the mixing network in NL-CG were generated by a2-layer hyper-network with a hidden layer of 64 ReLU neurons.

Revised: Biases in the mixing network weretransformed linearly from state vectors.

Revised: Our implementation of NL-CG was based on the PyMARL 1 [21] framework.

Revised: We used NVIDIA

Revised: GeForce RTX 3090 GPUs for training and evaluation.

Revised: B Weighted Max-Sum

Revised: Algorithm 1 and Algorithm 2 find global and approximate greedy actions for a non-linear coordinationgraph, respectively.

Revised: Both of these two algorithms rely on the weighted Max-Sum algorithm to findlocal optimal action on a linear piece.

Revised: We present the weighted Max-Sum algorithm in Algorithm 3.

Revised: C Representational Capability

Revised: In Sec. 4, we use a matrix game to show the representational capability of non-linear coordinationgraphs.

Revised: For conventional coordination graphs on State 2B of this game, since the value functionshould be permutation invariant, there are five unknowns q 1 = q i ( s 2 B , A ) , q 2 = q i ( s 2 B , B ) , q 3 =q ij ( s 2 B , AA ) , q 4 = q ij ( s 2 B , AB ) , and q 5 = q ij ( s 2 B , BB ) and five equations:

Revised: [Equation]

Revised: The augmented matrix of this system has a higher rank (4) than its coefficient matrix (3).

Revised: Therefore,this linear system does not have a solution, which means that a conventional coordination graphcannot represent the accurate value function for this task.

Revised: Algorithm

Revised: /*Greedy action selection with k message passing for one linear piece of the mixing network f m .

Revised: This algorithm is called by Algorithm 1 and 2, and the definition of inputs can be found there.*/

Revised: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , ( W V , W E

Revised: ) ∈ R |V| + |E| , b ∈ R f V :=

Revised: W V ◦ f V f E :=

Revised: W E ◦ f E µ 0 , ¯ µ 0 :=

Revised: 0 ∈ R

Revised: |E|× A /*Initialize forward messages ( µ ) and backward messages ( ¯ µ ).*/

Revised: q 0 :

Revised: = f V /*Initial “Q-value”.*/ q max :=

Revised: −∞ ; a max := (cid:2) arg max a ∈A i q 0 ai (cid:12)(cid:12) i ∈ V (cid:3) /*Initialize

Revised: the best found solution.*/ for t ∈ { 1 , . . .

Revised: , k } do /* k rounds of message passing.*/ for e =

Revised: ( i, j ) ∈

Revised: E do /*Update forward and backward messages.

Revised: Subscripts of f , q , µ mean indexing.*/ µ te := max

Revised: [Equation]

Revised: (cid:8) ( q

Revised: [Equation]

Revised: + f

Revised: [Equation]

Revised: (cid:9) /*Forward: maximize sender.*/ ¯ µ te := max

Revised: [Equation]

Revised: (cid:8) (

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: (cid:9) /*Backward

Revised: : maximize receiver.*/

Revised: [Equation]

Revised: tea /*Normalize forward messages to ensure converging.*/ ¯

Revised: [Equation]

Revised: ←

Revised: [Equation]

Revised: tea /*Normalize backward messages to ensure converging.*/ end for for i ∈ V do /*Update

Revised: “Q-value” with messages.*/ q ti :=

Revised: f V i + (cid:88)

Revised: e =(

Revised: · ,i ) ∈E µ te + (cid:88) e =( i, · ) ∈E ¯ µ te /*Utility plus incoming messages.*/ a ti := arg max a ∈A

Original: {+i { q tia } /*Select the greedy action of agent i .*/ end for q ′ ← (cid:80) |V| i =
+}
Revised: {+i { q tia } /*Select the greedy action of agent i .*/ end for q ′ ← (cid:80) |V| i =
+}

Revised: f V a i

Revised: + (cid:80) ( i,j ) ∈E f E a i a j + b /*Get the Q-value of the greedy action.*/ if q ′ >

Revised: q max then { a max ← a t ; q max ← q ′ q ← q ti } /*Remember

Revised: the best action.*/ end if end for return q max , a max ∈

Revised: A 1 × . . .

Revised: × A |V| , q /*Return the maximum Q value, the corresponding action, and utilities/payoffs.*/