Original: Non-Linear Coordination Graphs
Revised: Non-Linear Coordination Graphs

Original: Abstract
Revised: Abstract

Original: Value decomposition multi-agent reinforcement learning methods learn the globalvalue function as a mixing of each agent’s individual utility functions.
Revised: Value decomposition multi-agent reinforcement learning methods learn the globalvalue function as a mixing of each agent’s individual utility functions.

Original: Coordinationgraphs (CGs) represent a higher-order decomposition by incorporating pairwisepayoff functions and thus is supposed to have a more powerful representationalcapacity.
Revised: Coordinationgraphs (CGs) represent a higher-order decomposition by incorporating pairwisepayoff functions and thus is supposed to have a more powerful representationalcapacity.

Original: However, CGs decompose the global value function linearly over localvalue functions, severely limiting the complexity of the value function class thatcan be represented.
Revised: However, CGs decompose the global value function linearly over localvalue functions, severely limiting the complexity of the value function class thatcan be represented.

Original: In this paper, we propose the first non-linear coordinationgraph by extending CG value decomposition beyond the linear case.
Revised: In this paper, we propose the first non-linear coordinationgraph by extending CG value decomposition beyond the linear case.

Original: One majorchallenge is to conduct greedy action selections in this new function class to whichcommonly adopted DCOP algorithms are no longer applicable.
Revised: One majorchallenge is to conduct greedy action selections in this new function class to whichcommonly adopted DCOP algorithms are no longer applicable.

Original: We study how tosolve this problem when mixing networks with LeakyReLU activation are used.
Revised: We study how tosolve this problem when mixing networks with LeakyReLU activation are used.

Original: Anenumeration method with a global optimality guarantee is proposed and motivatesan efficient iterative optimization method with a local optimality guarantee.
Revised: Anenumeration method with a global optimality guarantee is proposed and motivatesan efficient iterative optimization method with a local optimality guarantee.

Original: Wefind that our method can achieve superior performance on challenging multi-agentcoordination tasks like MACO.
Revised: Wefind that our method can achieve superior performance on challenging multi-agentcoordination tasks like MACO.

Original:  Introduction
Revised:  Introduction

Original: Cooperative multi-agent problems are ubiquitous in real-world applications, such as crewless aerialvehicles [18, 32] and sensor networks [34].
Revised: Cooperative multi-agent problems are ubiquitous in real-world applications, such as crewless aerialvehicles [18, 32] and sensor networks [34].

Original: However, learning control policies for such systemsremains a major challenge.
Revised: However, learning control policies for such systemsremains a major challenge.

Original: Recently, value decomposition methods [25] significantly push forwardthe progress of multi-agent reinforcement learning [20, 23, 27, 28].
Revised: Recently, value decomposition methods [25] significantly push forwardthe progress of multi-agent reinforcement learning [20, 23, 27, 28].

Original: In these methods, a centralizedaction-value function is learned as a mixing of individual utility functions.
Revised: In these methods, a centralizedaction-value function is learned as a mixing of individual utility functions.

Original: The mixing functioncan be conditioned on global states [20] while individual utility functions are based on local actionobservation history.
Revised: The mixing functioncan be conditioned on global states [20] while individual utility functions are based on local actionobservation history.

Original: The advantage is that agents can utilize global information and avoid learningnon-stationarity [26] via centralized training while maintaining scalable decentralized execution.
Revised: The advantage is that agents can utilize global information and avoid learningnon-stationarity [26] via centralized training while maintaining scalable decentralized execution.

Original: Notably, the major focus of the value function decomposition research was on full decomposition, where local utility functions are based on actions and observations of a single agent.
Revised: Notably, the major focus of the value function decomposition research was on full decomposition, where local utility functions are based on actions and observations of a single agent.

Original: Fulldecomposition leads to several drawbacks, such as miscoordination problems in partially observableenvironments with stochastic transition functions [27, 29] and a game-theoretical pathology calledrelative overgeneralization [16, 4].
Revised: Fulldecomposition leads to several drawbacks, such as miscoordination problems in partially observableenvironments with stochastic transition functions [27, 29] and a game-theoretical pathology calledrelative overgeneralization [16, 4].

Original: Relative overgeneralization embodies that, due to the concurrentlearning and exploration of other agents, the employed utility function may not be able to expressoptimal decentralized policies and prefer suboptimal actions that give higher returns on average.
Revised: Relative overgeneralization embodies that, due to the concurrentlearning and exploration of other agents, the employed utility function may not be able to expressoptimal decentralized policies and prefer suboptimal actions that give higher returns on average.

Original: Coordination graph (CG) learning [9] holds the promise to address these problems while preservingthe advantages of value decomposition methods.
Revised: Coordination graph (CG) learning [9] holds the promise to address these problems while preservingthe advantages of value decomposition methods.

Original: In a CG, each vertex represents an agent, and each(hyper-) edge stands for a payoff function that is defined on the joint action-observation space of theconnected agents.
Revised: In a CG, each vertex represents an agent, and each(hyper-) edge stands for a payoff function that is defined on the joint action-observation space of theconnected agents.

Original: The existence of payoff functions increases the granularity of decomposition, andconnected agents explicitly coordinate with each other.
Revised: The existence of payoff functions increases the granularity of decomposition, andconnected agents explicitly coordinate with each other.

Original: As a result, a CG represents a higher-orderfactorization of the global value function and represents a much larger value function class.
Revised: As a result, a CG represents a higher-orderfactorization of the global value function and represents a much larger value function class.

Original: However, coordination graphs typically assume a linear decomposition of the value function amongsub-groups of agents, which is too simple to represent credit assignment in complex tasks.
Revised: However, coordination graphs typically assume a linear decomposition of the value function amongsub-groups of agents, which is too simple to represent credit assignment in complex tasks.

Original: In thispaper, we solve this long-standing problem and extend CG by introducing non-linear and learnablevalue decomposition.
Revised: In thispaper, we solve this long-standing problem and extend CG by introducing non-linear and learnablevalue decomposition.

Original: To our best knowledge, it is the first study on non-linear coordination graphs.
Revised: To our best knowledge, it is the first study on non-linear coordination graphs.

Original: The major challenge against extending CGs beyond linear cases is the calculation of value-maximizingactions.
Revised: The major challenge against extending CGs beyond linear cases is the calculation of value-maximizingactions.

Original: When linearly decomposed, DCOP algorithms [6] can find a globally greedy action via
Revised: When linearly decomposed, DCOP algorithms [6] can find a globally greedy action via

Original: 𝑄 "#" = %𝑞 ’  ’
Revised: 𝑄 "#" = %𝑞 ’  ’

Original: Fully decomposed value functions    𝑄 "#" = 𝑓 * (𝑞 , ,⋯,𝑞 / )
Revised: Fully decomposed value functions    𝑄 "#" = 𝑓 * (𝑞 , ,⋯,𝑞 / )

Original: VDN
Revised: VDN

Original: QMIX    𝑄 "#" = %𝑞 ’  ’,
Revised: QMIX    𝑄 "#" = %𝑞 ’  ’,

Original: Coordination graphs    𝑄 "#" =
Revised: Coordination graphs    𝑄 "#" =

Original: 𝑓 * ({𝑞 ’1 })
Revised: 𝑓 * ({𝑞 ’1 })

Original: DCG
Revised: DCG

Original: Ours message passing.
Revised: Ours message passing.

Original: However, when the mixing function is non-linear, DCOP algorithms are no longerapplicable.
Revised: However, when the mixing function is non-linear, DCOP algorithms are no longerapplicable.

Original: To address this problem, we develop a DCOP method for non-linear mixing functionsrepresented as a deep network with LeakyReLU (or ReLU) activation.
Revised: To address this problem, we develop a DCOP method for non-linear mixing functionsrepresented as a deep network with LeakyReLU (or ReLU) activation.

Original: The core idea of our non-linear DCOP algorithm is to exploit the fact that these neural networksare equivalent to piece-wise linear functions.
Revised: The core idea of our non-linear DCOP algorithm is to exploit the fact that these neural networksare equivalent to piece-wise linear functions.

Original: For each linear piece, a value-maximizing action canbe found by classic DCOP.
Revised: For each linear piece, a value-maximizing action canbe found by classic DCOP.

Original: However, this action may fall out of the domain of the linear piece andis thus infeasible.
Revised: However, this action may fall out of the domain of the linear piece andis thus infeasible.

Original: We first prove that such a shifted action indicates a better solution in its domain.
Revised: We first prove that such a shifted action indicates a better solution in its domain.

Original: Based on this conclusion, we first show how to find a feasible joint action with the global optimalvalue, and then derive an iterative algorithm with local optimum convergence guarantee.
Revised: Based on this conclusion, we first show how to find a feasible joint action with the global optimalvalue, and then derive an iterative algorithm with local optimum convergence guarantee.

Original: We demonstrate the improved representational capacity of our Non-Linear Coordination Graphs(NL-CG) on a matrix game by comparing the learned Q functions to those learned by conventionalcoordination graphs.
Revised: We demonstrate the improved representational capacity of our Non-Linear Coordination Graphs(NL-CG) on a matrix game by comparing the learned Q functions to those learned by conventionalcoordination graphs.

Original: We then evaluate our method on the Multi-Agent COordination (MACO)
Revised: We then evaluate our method on the Multi-Agent COordination (MACO)

Original: Benchmark [30] for its high requirements on close inter-agent coordination.
Revised: Benchmark [30] for its high requirements on close inter-agent coordination.

Original: The experimental resultsshow the superior performance enabled by the non-linear value decomposition.
Revised: The experimental resultsshow the superior performance enabled by the non-linear value decomposition.

Original:  Preliminaries
Revised:  Preliminaries

Original: In this paper, we focus on fully cooperative multi-agent tasks that can be modelled as a DecPOMDP [15] consisting of a tuple G = ⟨
Revised: In this paper, we focus on fully cooperative multi-agent tasks that can be modelled as a DecPOMDP [15] consisting of a tuple G = ⟨

Original: I, S, A, P, R, Ω , O, n, γ ⟩ , where I is the finite set of n agents,γ ∈ [0 , 1) is the discount factor, and s ∈ S is the true state of the environment.
Revised: I, S, A, P, R, Ω , O, n, γ ⟩ , where I is the finite set of n agents,γ ∈ [0 , 1) is the discount factor, and s ∈ S is the true state of the environment.

Original: At each timestep,each agent i receives an observation o
Revised: At each timestep,each agent i receives an observation o

Original: i ∈ Ω drawn according to the observation function
Revised: i ∈ Ω drawn according to the observation function

Original: O ( s, i ) andselects an action a i ∈ A .
Revised: O ( s, i ) andselects an action a i ∈ A .

Original: Individual actions form a joint action a ∈ A n , which leads to a next states ′
Revised: Individual actions form a joint action a ∈ A n , which leads to a next states ′

Original: according to the transition function P ( s ′ | s, a ) , a reward r = R ( s, a ) shared by all agents.
Revised: according to the transition function P ( s ′ | s, a ) , a reward r = R ( s, a ) shared by all agents.

Original: Eachagent has local action-observation history τ i ∈ T ≡ (Ω × A ) ∗ × Ω .
Revised: Eachagent has local action-observation history τ i ∈ T ≡ (Ω × A ) ∗ × Ω .

Original: Agents learn to collectivelymaximize the global return Q tot ( s, a ) =
Revised: Agents learn to collectivelymaximize the global return Q tot ( s, a ) =

Original: E s 0: ∞ ,a
Revised: E s 0: ∞ ,a

Original: 0: ∞ [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | s 

Revised: 0: ∞ [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | s

Original: = s, a 

Revised: = s, a

Original: = a ] .
Revised: = a ] .

Original: Estimating Q tot is at the core of multi-agent reinforcement learning (MARL) [25, 20, 23, 27, 13, 31].
Revised: Estimating Q tot is at the core of multi-agent reinforcement learning (MARL) [25, 20, 23, 27, 13, 31].

Original: The complexity of calculating max Q tot grows exponentially ( | A | n ) with the number of agents.
Revised: The complexity of calculating max Q tot grows exponentially ( | A | n ) with the number of agents.

Original: Tosolve this problem, value-based MARL decomposes the global action-value function into local utilityfunctions q i and guarantees that the global maximizer can be obtained locally: arg max a Q tot ( s, a ) =
Revised: Tosolve this problem, value-based MARL decomposes the global action-value function into local utilityfunctions q i and guarantees that the global maximizer can be obtained locally: arg max a Q tot ( s, a ) =

Original: 70 (cid:0) arg max a 1 q 1 ( τ 1 , a 1 ) , . .
Revised: 70 (cid:0) arg max a 1 q 1 ( τ 1 , a 1 ) , . .

Original: . , arg max a n q n ( τ n , a n ) (cid:1) T .
Revised: . , arg max a n q n ( τ n , a n ) (cid:1) T .

Original: Value decomposition network (VDN) [25]satisfies this condition by learning the global value function as a summation of local utilities (Fig. 1left).
Revised: Value decomposition network (VDN) [25]satisfies this condition by learning the global value function as a summation of local utilities (Fig. 1left).

Original: QMIX [20] extends the function class of VDN by learning Q tot as a monotonic mixing of localutilities.
Revised: QMIX [20] extends the function class of VDN by learning Q tot as a monotonic mixing of localutilities.

Original: The mixing function is parameterized so that: ∂Q tot (
Revised: The mixing function is parameterized so that: ∂Q tot (

Original: [Equation]
Revised: [Equation]

Original: , a )
Revised: , a )

Original: [Equation]
Revised: [Equation]

Original: i ( τ i ,a i ) ≥ 0 .
Revised: i ( τ i ,a i ) ≥ 0 .

Original: 2.1 Coordination Graphs
Revised: 2.1 Coordination Graphs

Original: For fully decomposed value functions, local utility functions are conditioned on local actionobservation history.
Revised: For fully decomposed value functions, local utility functions are conditioned on local actionobservation history.

Original: Coordination Graphs (CGs) [9] increase the representational capacity of fully decomposed value functions by introducing higher-order payoff functions.
Revised: Coordination Graphs (CGs) [9] increase the representational capacity of fully decomposed value functions by introducing higher-order payoff functions.

Original: Specifically, a coordination graph is a tuple of a vertex set and an edge set: G = ⟨V , E⟩ .
Revised: Specifically, a coordination graph is a tuple of a vertex set and an edge set: G = ⟨V , E⟩ .

Original: Each vertex v i ∈ V representsan agent i , and (hyper-) edges in E represent coordination dependencies among agents.
Revised: Each vertex v i ∈ V representsan agent i , and (hyper-) edges in E represent coordination dependencies among agents.

Original: In previouswork, the global value functions are decomposed linearly based on the graph topology:
Revised: In previouswork, the global value functions are decomposed linearly based on the graph topology:

Original: [Equation]
Revised: [Equation]

Original: where q i and q ij is utility functions for individual agents and pairwise payoff functions, respectively.τ ij = ⟨ τ i , τ j ⟩ and a ij = ⟨ a i , a j ⟩ is the joint action-observation history and action of agent i and j .
Revised: where q i and q ij is utility functions for individual agents and pairwise payoff functions, respectively.τ ij = ⟨ τ i , τ j ⟩ and a ij = ⟨ a i , a j ⟩ is the joint action-observation history and action of agent i and j .

Original: Previous work studies different aspects of such coordination graphs.
Revised: Previous work studies different aspects of such coordination graphs.

Original: It is shown that higher-orderfactorization is important on tackling an exponential number of joint actions [5].
Revised: It is shown that higher-orderfactorization is important on tackling an exponential number of joint actions [5].

Original: Sparse cooperative
Revised: Sparse cooperative

Original: Q-learning [11] learns value functions for sparse coordination graphs with pre-defined and statictopology.
Revised: Q-learning [11] learns value functions for sparse coordination graphs with pre-defined and statictopology.

Original: Zhang et al .
Revised: Zhang et al .

Original: [34] propose to learn minimized dynamic coordination sets for each agent.
Revised: [34] propose to learn minimized dynamic coordination sets for each agent.

Original: DCG [4] incorporates deep function approximation and parameter sharing into coordination graphsand scales to large state-action spaces.
Revised: DCG [4] incorporates deep function approximation and parameter sharing into coordination graphsand scales to large state-action spaces.

Original: CASEC [30] studies how to build sparse deep coordinationgraphs that are adaptive to the dynamic coordination requirements.
Revised: CASEC [30] studies how to build sparse deep coordinationgraphs that are adaptive to the dynamic coordination requirements.

Original: In all these works, the globalvalue function is represented as a summation of local value functions.
Revised: In all these works, the globalvalue function is represented as a summation of local value functions.

Original: To the best of our knowledge,this paper presents the fist coordination graph learning method with non-linear value decomposition.
Revised: To the best of our knowledge,this paper presents the fist coordination graph learning method with non-linear value decomposition.

Original: 2.2 Message Passing
Revised: 2.2 Message Passing

Original: Within a coordination graph, the greedy action selection required by Q-learning can not be completedby simply computing the maximum of individual utility and payoff functions.
Revised: Within a coordination graph, the greedy action selection required by Q-learning can not be completedby simply computing the maximum of individual utility and payoff functions.

Original: Instead, distributedconstraint optimization (DCOP) [6] techniques are used.
Revised: Instead, distributedconstraint optimization (DCOP) [6] techniques are used.

Original: Max-Sum [24] is a popular implementationof DCOP.
Revised: Max-Sum [24] is a popular implementationof DCOP.

Original: Max-Sum finds near-optimal actions on a CG G = ⟨V , E⟩ via multi-round message passingon a bipartite graph G m = ⟨V a , V q , E m ⟩ .
Revised: Max-Sum finds near-optimal actions on a CG G = ⟨V , E⟩ via multi-round message passingon a bipartite graph G m = ⟨V a , V q , E m ⟩ .

Original: Each node i ∈ V a represents an agent, and each nodeg ∈ V q represents a utility ( q i ) or payoff ( q ij ) function.
Revised: Each node i ∈ V a represents an agent, and each nodeg ∈ V q represents a utility ( q i ) or payoff ( q ij ) function.

Original: Edges in E m connect g with the associatedagent node(s).
Revised: Edges in E m connect g with the associatedagent node(s).

Original: Message passing starts with sending messages from node i ∈ V a to node g ∈ V q :
Revised: Message passing starts with sending messages from node i ∈ V a to node g ∈ V q :

Original: [Equation]
Revised: [Equation]

Original: where F i is the set of nodes in V q connected to node i , and c ig is a normalizing factor preventing thevalue of messages from growing arbitrarily large.
Revised: where F i is the set of nodes in V q connected to node i , and c ig is a normalizing factor preventing thevalue of messages from growing arbitrarily large.

Original: Messages are then sent from node g to node i :
Revised: Messages are then sent from node g to node i :

Original: [Equation]
Revised: [Equation]

Original: where V g is the set of nodes in V a connected to node g , a g = { a h | h ∈ V g } , a g \ a i
Revised: where V g is the set of nodes in V a connected to node g , a g = { a h | h ∈ V g } , a g \ a i

Original: = { a h | h ∈ V g \{ i }} ,and q represents utility or payoff functions conditioned on a g .
Revised: = { a h | h ∈ V g \{ i }} ,and q represents utility or payoff functions conditioned on a g .

Original: Eq.
Revised: Eq.

Original: 2 and 3 make up an iteration of message passing.
Revised: 2 and 3 make up an iteration of message passing.

Original: Each agent i can find its local optimal action by calculating a ∗ i =argmax a i (cid:80) h
Revised: Each agent i can find its local optimal action by calculating a ∗ i =argmax a i (cid:80) h

Original: ∈F
Revised: ∈F

Original: i m h
Revised: i m h

Original: →
Revised: →

Original: i ( a i ) after several iterations of message passing.
Revised: i ( a i ) after several iterations of message passing.

Original: Notably, Max-Sum andother DCOP algorithms are only applicable to linearly decomposed value functions.
Revised: Notably, Max-Sum andother DCOP algorithms are only applicable to linearly decomposed value functions.

Original: 2.3 Piece-Wise Linear Neural Networks
Revised: 2.3 Piece-Wise Linear Neural Networks

Original: Following the recent decade’s success of deep neural networks (DNNs), analysis works have beendone trying to explain the mechanism of the DNN black-boxes and assess their function approximationcapabilities.
Revised: Following the recent decade’s success of deep neural networks (DNNs), analysis works have beendone trying to explain the mechanism of the DNN black-boxes and assess their function approximationcapabilities.

Original: One conclusion is that a DNN with piece-wise linear (PWL) activation functions (e.g.
Revised: One conclusion is that a DNN with piece-wise linear (PWL) activation functions (e.g.

Original: ReLU, LeakyReLU, PReLU) is equivalent to a PWL function.
Revised: ReLU, LeakyReLU, PReLU) is equivalent to a PWL function.

Original: This kind of DNNs are called piecewise linear neural networks (PLNNs) [7].
Revised: This kind of DNNs are called piecewise linear neural networks (PLNNs) [7].

Original: Early papers [14, 17] assess the expressivity of PLNNsby the amount of linear pieces.
Revised: Early papers [14, 17] assess the expressivity of PLNNsby the amount of linear pieces.

Original: [2, 22, 10] give more theoretically grounded results about the upperand lower bounds for the amount of pieces.
Revised: [2, 22, 10] give more theoretically grounded results about the upperand lower bounds for the amount of pieces.

Original: In this sense, it is shown that increasing the depth ofa network can generally be exponentially more valuable than increasing the width [17, 8, 2].
Revised: In this sense, it is shown that increasing the depth ofa network can generally be exponentially more valuable than increasing the width [17, 8, 2].

Original: Chuet al .
Revised: Chuet al .

Original: [7] propose OPENBOX to compute the mathematically equivalent set of linear pieces, whichprovides an accurate and consistent interpretation of PLNNs.
Revised: [7] propose OPENBOX to compute the mathematically equivalent set of linear pieces, whichprovides an accurate and consistent interpretation of PLNNs.

Original: For neural networks with a single layerof hidden nodes, the problem can be reduced to hyper-plane arrangement [33], and linear functionsand their domain can be enumerated efficiently [19, 3].
Revised: For neural networks with a single layerof hidden nodes, the problem can be reduced to hyper-plane arrangement [33], and linear functionsand their domain can be enumerated efficiently [19, 3].

Original:  Method
Revised:  Method

Original: Our method extends the representational capability of coordination graphs by introducing non-linearfactorization of the global value function.
Revised: Our method extends the representational capability of coordination graphs by introducing non-linearfactorization of the global value function.

Original: Specifically, for a coordination graph G = ⟨V , E⟩ , wedecompose the global Q as:
Revised: Specifically, for a coordination graph G = ⟨V , E⟩ (we studycomplete coordination graphs in this paper), we decompose the global Q as:

Original: [Equation]
Revised: [Equation]

Original: where q i is the vector of all individual utilities, and q ij is the vector of all pairwise payoffs foredges in E .
Revised: where q i is the vector of all individual utilities, and q ij is the vector of all pairwise payoffs foredges in E .

Original: Similar to DCG [4], we learn a shared utility function f v , parameterized by θ v , and getthe individual utility q i ( τ i , a i ) =
Revised: Similar to DCG [4], we learn a shared utility function f v , parameterized by θ v , and getthe individual utility q i ( τ i , a i ) =

Original: f v ( a i | τ i ; θ v ) .
Revised: f v ( a i | τ i ; θ v ) .

Original: The payoffs are estimated by a shared function f eparameterized by θ e : q ij ( τ i , τ j , a i , a j ) =
Revised: The payoffs are estimated by a shared function f eparameterized by θ e : q ij ( τ i , τ j , a i , a j ) =

Original: f e ( a i , a j | τ i , τ j ; θ e ) .
Revised: f e ( a i , a j | τ i , τ j ; θ e ) .

Original: Different from conventional CGs (Eq. 1), our f n is a non-linear mixing network whose parametersare generated by a hypernet f h conditioned on the global state s and parameterized by θ h .
Revised: Different from conventional CGs (Eq. 1), our f n is a non-linear mixing network whose parametersare generated by a hypernet f h conditioned on the global state s and parameterized by θ h .

Original: Ourdiscussion is based on LeakyReLU networks with the slope coefficient α ∈ [0 , 1] .
Revised: Ourdiscussion is based on LeakyReLU networks with the slope coefficient α ∈ [0 , 1] .

Original: For efficientlycalculating the maximizer of Q tot , we require the weights after the first layer of the mixing networkto be non-negative.
Revised: For efficientlycalculating the maximizer of Q tot , we require the weights after the first layer of the mixing networkto be non-negative.

Original: The reason for this non-negativity constraint will be discussed in Lemma 1and 2.
Revised: The reason for this non-negativity constraint will be discussed in Lemma 1and 2.

Original: Such a mixing network is effectively a type of input convex neural networks (ICNN [1]).
Revised: Such a mixing network is effectively a type of input convex neural networks (ICNN [1]).

Original: Thenon-negativity constraint on parameters is somewhat constraint, but we can use the passthrough trickintroduced in Proposition 1 of [1] to maintain substantial representation power of the mixing network.
Revised: Thenon-negativity constraint on parameters is somewhat constraint, but we can use the passthrough trickintroduced in Proposition 1 of [1] to maintain substantial representation power of the mixing network.

Original: The whole framework, including the utility function f v , the payoff function f e , and the hypernet f h ,is updated by minimizing the TD loss:
Revised: The whole framework, including the utility function f v , the payoff function f e , and the hypernet f h ,is updated by minimizing the TD loss:

Original: [Equation]
Revised: [Equation]

Original: where D is the replay buffer, and ˆ Q tot is a target function whose parameters are periodically copiedfrom the function Q tot .
Revised: where D is the replay buffer, and ˆ Q tot is a target function whose parameters are periodically copiedfrom the function Q tot .

Original: The main challenge left, for both the training and execution phase of deep Q-learning, is to selectactions that maximize the global Q value at each time step, arg max a Q tot ( τ , a ) .
Revised: The main challenge left, for both the training and execution phase of deep Q-learning, is to selectactions that maximize the global Q value at each time step, arg max a Q tot ( τ , a ) .

Original: This problemis theoretically NP-hard when the mixing function is non-linear.
Revised: This problemis theoretically NP-hard when the mixing function is non-linear.

Original: In the following sections, wefirst provide a global optimal algorithm for this problem.
Revised: In the following sections, wefirst provide a global optimal algorithm for this problem.

Original: The complexity of this algorithm growspolynomially with the number of hidden units in the mixing network.
Revised: The complexity of this algorithm growspolynomially with the number of hidden units in the mixing network.

Original: To reduce time complexity, wepropose an iterative algorithm with local optimum convergence guarantee.
Revised: To reduce time complexity, wepropose an iterative algorithm with local optimum convergence guarantee.

Original: We use the following notations .
Revised: We use the following notations .

Original: The input to the mixing network is q =
Revised: The input to the mixing network is q =

Original: [ q i ∥ q ij ] , where [ ·∥· ] meansvector concatenation.
Revised: [ q i ∥ q ij ] , where [ ·∥· ] meansvector concatenation.

Original: We use q ( a ) to denote the utilities and payoffs corresponding to action a .
Revised: We use q ( a ) to denote the utilities and payoffs corresponding to action a .

Original: Themixing network has L LeakyReLU linear layers.
Revised: Themixing network has L LeakyReLU linear layers.

Original: d = |V| + |E| is the input dimension.
Revised: d = |V| + |E| is the input dimension.

Original: m i , W i , andb i ∈
Revised: m i , W i , andb i ∈

Original: R m i are the width, weights, and biases of the i th layer.
Revised: R m i are the width, weights, and biases of the i th layer.

Original: The weights after the first layer arenon-negative.
Revised: The weights after the first layer arenon-negative.

Original: The input to the activation units at i th layer is z i , and the corresponding output is h i =
Revised: The input to the activation units at i th layer is z i , and the corresponding output is h i =

Original: LeakyReLU
Revised: LeakyReLU

Original: ( z i ) =
Revised: ( z i ) =

Original: c i ◦
Revised: c i ◦

Original: o i , where c i is the value of LeakyReLU activation ( c = α
Revised: o i , where c i is the value of LeakyReLU activation ( c = α

Original: when z < 0 andc = 1 when z ≥ 0 ) and ◦ is the element-wise multiplication.
Revised: when z < 0 andc = 1 when z ≥ 0 ) and ◦ is the element-wise multiplication.

Original: We call c =
Revised: We call c =

Original: [ c 1 ∥ . . .
Revised: [ c 1 ∥ . . .

Original: ∥ c L ] ∈ { α, 1 } m ,m = (cid:80) i m i , a slope configuration of the mixing network.
Revised: ∥ c L ] ∈ { α, 1 } m ,m = (cid:80) i m i , a slope configuration of the mixing network.

Original: Subscripts of W i , b i , z i , and h i meansindex.
Revised: Subscripts of W i , b i , z i , and h i meansindex.

Original: For example, z ij is the j th element of z i , and W ij is the j th row of W i .
Revised: For example, z ij is the j th element of z i , and W ij is the j th row of W i .

Original: 3.1 Piece-Wise Optimization
Revised: 3.1 Piece-Wise Optimization

Original: The general idea of our action selection algorithm is to utilize the piece-wise linear property of the
Revised: The general idea of our action selection algorithm is to utilize the piece-wise linear property of the

Original: LeakyReLU network.
Revised: LeakyReLU network.

Original: Given a slope configuration c , the mixing network becomes linear, and we canrun a DCOP algorithm to get the corresponding maximizer of Q tot .
Revised: Given a slope configuration c , the mixing network becomes linear, and we canrun a DCOP algorithm to get the corresponding maximizer of Q tot .

Original: The question is that the obtainedsolution may be out of the domain of the linear piece.
Revised: The question is that the obtainedsolution may be out of the domain of the linear piece.

Original: We show that, when the weights after the firstlayer are non-negative, we can ignore whether the obtained local optimal solution is in the piece’sdomain, and the global maximizer of Q tot is the best of the local optima.
Revised: We show that, when the weights after the firstlayer are non-negative, we can ignore whether the obtained local optimal solution is in the piece’sdomain, and the global maximizer of Q tot is the best of the local optima.

Original: Formally, there are 2 m slope configurations in C all
Revised: Formally, there are 2 m slope configurations in C all

Original: = { c | c ∈ { α, 1 } m } , each of which makesthe mixing network an affine function in P all = { ρ 1 , ρ 2 , . .
Revised: = { c | c ∈ { α, 1 } m } , each of which makesthe mixing network an affine function in P all = { ρ 1 , ρ 2 , . .

Original: , ρ 2 m } .
Revised: , ρ 2 m } .

Original: Each affine function ρ k has acorresponding cell P k where q ∈ P k yields c k in the forward pass.
Revised: Each affine function ρ k has acorresponding cell P k where q ∈ P k yields c k in the forward pass.

Original: For each c k ∈ C all , by running
Revised: For each c k ∈ C all , by running

Original: DCOP, we obtain the local maximum q k and the corresponding joint action a k and utilities/payoffsq k .
Revised: DCOP, we obtain the local maximum q k and the corresponding joint action a k and utilities/payoffsq k .

Original: However, it is possible that q k falls out of P k and indeed yields another slope configuration c r ̸ = k .
Revised: However, it is possible that q k falls out of P k and indeed yields another slope configuration c r ̸ = k .

Original: We first show that such a shifted solution indicates an equal or better solution in its domain.
Revised: We first show that such a shifted solution indicates an equal or better solution in its domain.

Original: Lemma 1.
Revised: Lemma 1.

Original: Denote affine function pieces and their cells of a fully-connected feedforward mixingnetwork with LeakyReLU activation as P all = { ρ j } 2 m 1 and { P j } 2 m 1 .
Revised: Denote affine function pieces and their cells of a fully-connected feedforward mixingnetwork with LeakyReLU activation as P all = { ρ j } 2 m 1 and { P j } 2 m 1 .

Original: For q in the cell of the r th piece,
Revised: For q in the cell of the r th piece,

Original: P r , and ∀ ρ s ∈ P all , we have ρ r ( q ) ≥
Revised: P r , and ∀ ρ s ∈ P all , we have ρ r ( q ) ≥

Original: ρ s ( q ) .
Revised: ρ s ( q ) .

Original: Proof.
Revised: Proof.

Original: We start with the first difference between c r and c s .
Revised: We start with the first difference between c r and c s .

Original: We denote it as c rij ̸ = c sij .
Revised: We denote it as c rij ̸ = c sij .

Original: Because thisis the first difference, the input to this unit, z ij , is the same.
Revised: Because thisis the first difference, the input to this unit, z ij , is the same.

Original: Since q is in P r , we have that c rij is 1when z ij is positive and is α < 1 when z ij is negative.
Revised: Since q is in P r , we have that c rij is 1when z ij is positive and is α < 1 when z ij is negative.

Original: Therefore, the output satisfies that
Revised: Therefore, the output satisfies that

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: because W i +1 ≥ 0 .
Revised: because W i +1 ≥ 0 .

Original: Other differences in the i th layer will lead to the same conclusion.
Revised: Other differences in the i th layer will lead to the same conclusion.

Original: At layeri + 1 , we have
Revised: At layeri + 1 , we have

Original: [Equation]
Revised: [Equation]

Original: The first inequity is because q is in the cell of ρ r .
Revised: The first inequity is because q is in the cell of ρ r .

Original: The second inequity is because of Eq.
Revised: The second inequity is because of Eq.

Original: 7 and that
Revised: 7 and that

Original: LeakyReLU is a monotonic increasing function.
Revised: LeakyReLU is a monotonic increasing function.

Original: The proof can repeat for the following layers andthus holds for the last layer, i .
Revised: The proof can repeat for the following layers andthus holds for the last layer, i .

Original: e ., ρ r ( q ) ≥
Revised: e ., ρ r ( q ) ≥

Original: ρ s ( q ) .
Revised: ρ s ( q ) .

Original: Based on Lemma 1, when the local optimal solution, a k , of piece k returned by DCOP actually fallsin cell P r ̸ = k , we have ρ r ( q ( a k )) ≥ ρ k ( q ( a k )) .
Revised: Based on Lemma 1, when the local optimal solution, a k , of piece k returned by DCOP actually fallsin cell P r ̸ = k , we have ρ r ( q ( a k )) ≥ ρ k ( q ( a k )) .

Original: This indicates that, on piece r , we will get a solutionat least as good as the one on piece k .
Revised: This indicates that, on piece r , we will get a solutionat least as good as the one on piece k .

Original: Based on this conclusion, we now prove that the maximum ofeach piece’s local optimum is the global optimum.
Revised: Based on this conclusion, we now prove that the maximum ofeach piece’s local optimum is the global optimum.

Original: Lemma 2.
Revised: Lemma 2.

Original: Running DCOP algorithm for each linear piece in P all , then take the maximum valueamong these pieces, we can get the global optimizer of Q tot .
Revised: Running DCOP algorithm for each linear piece in P all , then take the maximum valueamong these pieces, we can get the global optimizer of Q tot .

Original: Proof.
Revised: Proof.

Original: We have that max q f n ( q ) = max q max ρ p ∈P all ρ p ( q ) = max ρ p ∈P all max q ρ p ( q ) ( f n is themixing function, Eq. 4), which means the maximum of local optimal values is the global optimalvalue.
Revised: We have that max q f n ( q ) = max q max ρ p ∈P all ρ p ( q ) = max ρ p ∈P all max q ρ p ( q ) ( f n is themixing function, Eq. 4), which means the maximum of local optimal values is the global optimalvalue.

Original: Suppose the global optimal value Q max is found on piece r and the corresponding action is a r ,the question is whether q ( a r ) is feasible, i .
Revised: Suppose the global optimal value Q max is found on piece r and the corresponding action is a r ,the question is whether q ( a r ) is feasible, i .

Original: e ., q ( a r ) ∈ P r .
Revised: e ., q ( a r ) ∈ P r .

Original: Assume that q ( a r ) falls in P s ̸ = r .
Revised: Assume that q ( a r ) falls in P s ̸ = r .

Original: Thenfor piece s , a r is a feasible solution with the value Q max (due to Lemma 1).
Revised: Thenfor piece s , a r is a feasible solution with the value Q max (due to Lemma 1).

Original: This means the globaloptimal solution is always a feasible solution.
Revised: This means the globaloptimal solution is always a feasible solution.

Original: According to Lemma 2, we can run message passing on each piece (Algorithm 3 in Appx.
Revised: According to Lemma 2, we can run message passing on each piece (Algorithm 3 in Appx.

Original: B) and thentake the best of these results to get the global optimum.
Revised: B) and thentake the best of these results to get the global optimum.

Original: Detailed process can be found in Algorithm 1.
Revised: Detailed process can be found in Algorithm 1.

Original: The problem of Algorithm 1 is time complexity.
Revised: The problem of Algorithm 1 is time complexity.

Original: When m is small, we enumerate all 2 m slopeconfigurations.
Revised: When m is small, we enumerate all 2 m slopeconfigurations.

Original: For a large m , we can use the hyperplane arrangement algorithm [7] to calculateexact linear pieces and their domains.
Revised: For a large m , we can use the hyperplane arrangement algorithm [7] to calculateexact linear pieces and their domains.

Original: Specifically, there are
Revised: Specifically, there are

Original: [Equation]
Revised: [Equation]

Original: pieces need enumerating, where d is the input length.
Revised: pieces need enumerating, where d is the input length.

Original: This number is exponential to d .
Revised: This number is exponential to d .

Original: To reducetime complexity, we propose an iterative optimization method in the next section.
Revised: To reducetime complexity, we propose an iterative optimization method in the next section.

Original: 3.2 Iterative Optimization with Local Optimum Guarantee
Revised: 3.2 Iterative Optimization with Local Optimum Guarantee

Original: When m is large, enumerating either 2 m or n m,d pieces can be costly.
Revised: When m is large, enumerating either 2 m or n m,d pieces can be costly.

Original: We thus propose an iterativealgorithm with local optimum guarantee (Algorithm 2) to get approximate solutions.
Revised: We thus propose an iterativealgorithm with local optimum guarantee (Algorithm 2) to get approximate solutions.

Original: We begin withthe slope configuration where all hidden units are activated ( c = { 1 } m ).
Revised: We begin withthe slope configuration where all hidden units are activated ( c = { 1 } m ).

Original: We run message passing onthe current configuration c
Revised: We run message passing onthe current configuration c

Original: p and get a local optimum solution a p .
Revised: p and get a local optimum solution a p .

Original: If the real configuration c real ofa p is not c p , we calculate the local optimum of c real .
Revised: If the real configuration c real ofa p is not c p , we calculate the local optimum of c real .

Original: The iteration continues until c real =
Revised: The iteration continues until c real =

Original: c p .
Revised: c p .

Original: Lemma 1 guarantees that the solution values in the iteration are monotonically increasing.
Revised: Lemma 1 guarantees that the solution values in the iteration are monotonically increasing.

Original: Sincethere is a finite number of configurations, our algorithm can converge to a local optimum.
Revised: Sincethere is a finite number of configurations, our algorithm can converge to a local optimum.

Original: With a very low probability, a loop would appear in the iteration.
Revised: With a very low probability, a loop would appear in the iteration.

Original: The appearance of a loop meansthat the equity in Lemma 1 holds.
Revised: The appearance of a loop meansthat the equity in Lemma 1 holds.

Original: In this case, the solutions on the loop have the same value, thus wecan stop our iteration when encountering a slope configuration that has been searched.
Revised: In this case, the solutions on the loop have the same value, thus wecan stop our iteration when encountering a slope configuration that has been searched.

Original: Algorithm
Revised: Algorithm

Original: /*Show the case for a two-layer mixing network, but can be easily extended to more layers.*/
Revised: /*Show the case for a two-layer mixing network, but can be easily extended to more layers.*/

Original: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , W 0 , b 0 , W 1 , b 1 q max :=
Revised: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , W 0 , b 0 , W 1 , b 1 q max :=

Original: −∞ ; a max := (cid:2)(cid:3) /*Initialize
Revised: −∞ ; a max := (cid:2)(cid:3) /*Initialize

Original: the best found solution.*/ for c p ∈ { α, 1 } m do /*Calculate the equivalent weights and biases.*/
Revised: the best found solution.*/ for c p ∈ { α, 1 } m do /*Calculate the equivalent weights and biases.*/

Original: W ρ p :=
Revised: W ρ p :=

Original: W 0 · ( c p ◦ W 1 )
Revised: W 0 · ( c p ◦ W 1 )

Original: b ρ p :=
Revised: b ρ p :=

Original: ( c p ◦ W 1 ) ·
Revised: ( c p ◦ W 1 ) ·

Original: b T 0 + bq, a , · ← w -MAX-SUM
Revised: b T 0 + bq, a , · ← w -MAX-SUM

Original: ( f V , f E , W ρ p , b ρ p )
Revised: ( f V , f E , W ρ p , b ρ p )

Original: if q > q max then a max ← a q max ← q /*Remember only the best actions.*/ end if end for return a max /*Return
Revised: if q > q max then a max ← a q max ← q /*Remember only the best actions.*/ end if end for return a max /*Return

Original: the selected joint action a max .*/
Revised: the selected joint action a max .*/

Original: Algorithm
Revised: Algorithm

Original: /*Show the case for a two- mixing network, but can be easily extended to more layers.*/
Revised: /*Show the case for a two- mixing network, but can be easily extended to more layers.*/

Original: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , W 0 , b 0 , W 1 , b 1 , n max , ϵ q max :=
Revised: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , W 0 , b 0 , W 1 , b 1 , n max , ϵ q max :=

Original: −∞ , a max := (cid:2)(cid:3) , c p := { 1 } m for n ∈ { 1 , . . . , n max } do /*Try
Revised: −∞ , a max := (cid:2)(cid:3) , c p := { 1 } m for n ∈ { 1 , . . . , n max } do /*Try

Original: one configuration at a time.*/
Revised: one configuration at a time.*/

Original: W ρ p :=
Revised: W ρ p :=

Original: W 0 · ( c p ◦ W 1 )
Revised: W 0 · ( c p ◦ W 1 )

Original: b ρ p :=
Revised: b ρ p :=

Original: ( c p ◦ W 1 ) ·
Revised: ( c p ◦ W 1 ) ·

Original: b T 0 + bq, a , · ← w -MAX-SUM
Revised: b T 0 + bq, a , · ← w -MAX-SUM

Original: ( f V , f E , W ρ p , b ρ p )
Revised: ( f V , f E , W ρ p , b ρ p )

Original: if q > q max then a max ← a q max ← q c real ←
Revised: if q > q max then a max ← a q max ← q c real ←

Original: The real LeakyReLU slope configuration end if if c real ̸ = c p then c
Revised: The real LeakyReLU slope configuration end if if c real ̸ = c p then c

Original: p ← c real else with prob.
Revised: p ← c real else with prob.

Original: ϵ break or continue with an unvisited c p .
Revised: ϵ break or continue with an unvisited c p .

Original: end for return a max
Revised: end for return a max

Original: To increase the possibility of finding a global optimum, we can introduce a simulated annealingmechanism.
Revised: To increase the possibility of finding a global optimum, we can introduce a simulated annealingmechanism.

Original: Each time we find a better solution, with a probability of 1 − ϵ , we move to thecorresponding piece, and with a probability of ϵ , we jump to a random piece that has not beensearched.
Revised: Each time we find a better solution, with a probability of 1 − ϵ , we move to thecorresponding piece, and with a probability of ϵ , we jump to a random piece that has not beensearched.

Original: ϵ decreases with the iteration number.
Revised: ϵ decreases with the iteration number.

Original: Discussion about loopy graph topology In this paper, we consider complete graphs when studyingnon-linear coordination graphs.
Revised: Discussion about loopy graph topology In this paper, we consider complete graphs when studyingnon-linear coordination graphs.

Original: A concern is that message passing algorithms like Max-Sum maynot converge to the optimal solutions in loopy graphs and has an error rate of e .
Revised: A concern is that message passing algorithms like Max-Sum maynot converge to the optimal solutions in loopy graphs and has an error rate of e .

Original: Lemma 1 is notaffected because it is a property of LeakyReLU Networks.
Revised: Lemma 1 is notaffected because it is a property of LeakyReLU Networks.

Original: For Lemma 2, the maximum of solutionsfound by message passing in all slope configurations is the global optimum with a probability of1 − e .
Revised: For Lemma 2, the maximum of solutionsfound by message passing in all slope configurations is the global optimum with a probability of1 − e .

Original: An error occurs when message passing cannot find the right solution on the piece where theglobal optimum is located.
Revised: An error occurs when message passing cannot find the right solution on the piece where theglobal optimum is located.

Original: Our iterative method may stop earlier when message passing returns awrong solution located in the current cell.
Revised: Our iterative method may stop earlier when message passing returns awrong solution located in the current cell.

Original: The probability of this situation is less than e .
Revised: The probability of this situation is less than e .

Original: Thus wehave at least a probability of (1 − e ) n ( n is the number of iterations) to find the piece where the localoptimum is located, and the final probability of finding the local optimum is larger than (1 − e ) n +1 .
Revised: Thus wehave at least a probability of (1 − e ) n ( n is the number of iterations) to find the piece where the localoptimum is located, and the final probability of finding the local optimum is larger than (1 − e ) n +1 .

Original:  Representational Capability
Revised:  Representational Capability

Original: In this section, we compare the representational capacity of our model against conventional coordination graphs.
Revised: In this section, we compare the representational capacity of our model against conventional coordination graphs.

Original: The comparison is carried out on a two-step cooperative matrix game with four playersand two actions.
Revised: The comparison is carried out on a two-step cooperative matrix game with four playersand two actions.

Original: At the first step, Agent 1’s action decides which of the two matrix games (Table.
Revised: At the first step, Agent 1’s action decides which of the two matrix games (Table.

Original: 1)to play in the next timestep.
Revised: 1)to play in the next timestep.

Original: In the second step, the number of agents taking Action B determines theglobal reward received by the agent team (Table. 1).
Revised: In the second step, the number of agents taking Action B determines theglobal reward received by the agent team (Table. 1).

Original: [Table]
Revised: [Table]

Original: step game after Agent 1 chooses the first action.
Revised: step game after Agent 1 chooses the first action.

Original: Action A takes the agents to State 2A, while action B takes them to State 2B.
Revised: Action A takes the agents to State 2A, while action B takes them to State 2B.

Original: The team reward depends on the number of agents taking Action B.
Revised: The team reward depends on the number of agents taking Action B.

Original: We first theoretically prove that the representational capacity of conventional coordination graphsis unable to represent the Q -function of this game.
Revised: We first theoretically prove that the representational capacity of conventional coordination graphsis unable to represent the Q -function of this game.

Original: Since the reward is invariant to the identity ofagents, i .
Revised: Since the reward is invariant to the identity ofagents, i .

Original: e .,
Revised: e .,

Original: R ( a 1 , a 2 , a 3 , a 4 ) =
Revised: R ( a 1 , a 2 , a 3 , a 4 ) =

Original: R ( p ( a 1 , a 2 , a 3 , a 4 )) , where p is an arbitrary permutation, the learnedaction-value function should also be permutation invariant.
Revised: R ( p ( a 1 , a 2 , a 3 , a 4 )) , where p is an arbitrary permutation, the learnedaction-value function should also be permutation invariant.

Original: Therefore we can ignore the order of actions in value functions.
Revised: Therefore we can ignore the order of actions in value functions.

Original: Now let’s focus on State 2B ( s 2 B ).
Revised: Now let’s focus on State 2B ( s 2 B ).

Original: Conventional coordination graphs needto solve a linear system consisting of five unknowns and five equations.
Revised: Conventional coordination graphs needto solve a linear system consisting of five unknowns and five equations.

Original: However, for this system, therank of the augmented matrix (4) is greater than the rank of the coefficient matrix (3).
Revised: However, for this system, therank of the augmented matrix (4) is greater than the rank of the coefficient matrix (3).

Original: Details can befound in Appx.
Revised: Details can befound in Appx.

Original: C. Therefore, this system has no solution, and it is impossible for conventional CGsto learn a correct Q function.
Revised: C. Therefore, this system has no solution, and it is impossible for conventional CGsto learn a correct Q function.

Original: We then empirically demonstrate our idea.
Revised: We then empirically demonstrate our idea.

Original: We train NL-CG and DCG on the task for 5000 episodesunder full exploration ( ϵ = 1 ) and examine the learned value functions.
Revised: We train NL-CG and DCG on the task for 5000 episodesunder full exploration ( ϵ = 1 ) and examine the learned value functions.

Original: Full exploration ensuresthat both methods explore all state-action pairs.
Revised: Full exploration ensuresthat both methods explore all state-action pairs.

Original: In such a case, the representational capacity of theaction-value function approximator remains the only challenge of learning accurate Q functions.
Revised: In such a case, the representational capacity of theaction-value function approximator remains the only challenge of learning accurate Q functions.

Original: Forour algorithm, the utility and payoff function is fully connected networks with a single hidden layerof 64 units with a ReLU non-linearity.
Revised: Forour algorithm, the utility and payoff function is fully connected networks with a single hidden layerof 64 units with a ReLU non-linearity.

Original: γ is 0.99, and the replay buffer stores the last 500 episodes,from which we uniformly sample batches of size 32 for training.
Revised: γ is 0.99, and the replay buffer stores the last 500 episodes,from which we uniformly sample batches of size 32 for training.

Original: The target network is updated every100 episodes.
Revised: The target network is updated every100 episodes.

Original: The learning rate of RMSprop is set to 5 × 10 − 4 .
Revised: The learning rate of RMSprop is set to 5 × 10 − 4 .

Original: Agents receive the full state asobservation, which is represented as an one-hot vector.
Revised: Agents receive the full state asobservation, which is represented as an one-hot vector.

Original: Table 2 and 3 show the learned Q by DCG and NL-CG.
Revised: Table 2 and 3 show the learned Q by DCG and NL-CG.

Original: In line with our theoretical analysis, we cansee that DCG learns a sub-optimal strategy of selecting Action A in the first step.
Revised: In line with our theoretical analysis, we cansee that DCG learns a sub-optimal strategy of selecting Action A in the first step.

Original: By contrast, ourmethod learns the accurate value of Action B in State A and get the optimal strategy.
Revised: By contrast, ourmethod learns the accurate value of Action B in State A and get the optimal strategy.

Original: Furthermore,the Q values for State 2B learned by NL-CG is more accurate than those learned by DCG.
Revised: Furthermore,the Q values for State 2B learned by NL-CG is more accurate than those learned by DCG.

Original: Theseresults demonstrate that NL-CG’s higher representational capacity allows it to accurately estimate thevalue function of this game whereas DCG cannot.
Revised: Theseresults demonstrate that NL-CG’s higher representational capacity allows it to accurately estimate thevalue function of this game whereas DCG cannot.

Original: We also note that such an example is common fortask with few actions because there are typically more equations than unknowns.
Revised: We also note that such an example is common fortask with few actions because there are typically more equations than unknowns.

Original: [Table]
Revised: [Table]

Original: Action
Revised: Action

Original: A
Revised: A

Original: B
Revised: B

Original: Q 6.
Revised: Q 6.

Original: State 2B
Revised: State 2B

Original: # Action B
Revised: # Action B

Original: Q 1.
Revised: Q 1.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Experiments
Revised:  Experiments

Original: In this section, we conduct experiments to show the effectiveness of our method on complex tasks.
Revised: In this section, we conduct experiments to show the effectiveness of our method on complex tasks.

Original: We benchmark our method on the Multi-Agent COordination (MACO) benchmark [30], whichcovers various classic coordination tasks in the literature of multi-agent learning and increases theircomplexity to better evaluate the performance of different algorithms.
Revised: We benchmark our method on the Multi-Agent COordination (MACO) benchmark [30], whichcovers various classic coordination tasks in the literature of multi-agent learning and increases theircomplexity to better evaluate the performance of different algorithms.

Original: The MACO benchmark ischaracterized by a high demand on the sophistication of agent coordination.
Revised: The MACO benchmark ischaracterized by a high demand on the sophistication of agent coordination.

Original: For NL-CG, we use amixing network that has one hidden layer with different widths.
Revised: For NL-CG, we use amixing network that has one hidden layer with different widths.

Original: Detailed hyperparameter settingsof our method can be found in Appendix A. For fair comparison, we run all our experiments with 5random seeds and show the mean performance with a 95% confidence interval.
Revised: Detailed hyperparameter settingsof our method can be found in Appendix A. For fair comparison, we run all our experiments with 5random seeds and show the mean performance with a 95% confidence interval.

Original: In Fig.
Revised: In Fig.

Original: 2, we compare NL-CG against the state-of-the-art coordination graph learning method(DCG [4]) and fully decomposed methods (QMIX [20], DICG [12]).
Revised: 2, we compare NL-CG against the state-of-the-art coordination graph learning method(DCG [4]) and fully decomposed methods (QMIX [20], DICG [12]).

Revised: For both DCG and our method,we use the complete graphs for all experiments in the paper.

Original: For NL-CG, we (1) use ahidden layer of three units and enumerate all8 slope configurations and (2) use a hidden layerof four units and run our iterative method with n max = 4 . 
Revised: For NL-CG, we (1) set the hidden widthto 3 and enumerate all configurations and (2) set the hidden width to 4 and run our iterative methodwith n max = 4 . We stop iteration when n max slope configurations are visited.

Original: The result shows that our algorithmcan outperform conventional CGs significantly.
Revised: The result shows thatour algorithm can outperform conventional CGs significantly.

Original: Moreover, our iterative optimization method hascomparable performance with the enumeration method, showing its effectiveness.
Revised: Moreover, our iterative optimizationmethod has comparable performance with the enumeration method, showing its effectiveness.

Original: QMIX struggles onthese tasks, indicating that these tasks are beyond the representation capacity of a fully decomposedfunction.
Revised: QMIXstruggles on these tasks, indicating that these tasks are beyond the representation capacity of a fullydecomposed function.

Original: These results demonstrate the advantage of non-linear CGs on tasks that need sophisticateagent coordination.

Original: We further investigate the the influence of the mixing network’s width.
Revised: We further investigate the the influence of the mixing network’s width.

Original: Specifically, we test NLCG with a width of 2, 3, 4, and 10 and compare their performance in Fig.
Revised: Specifically, we test NLCG with a width of 2, 3, 4, and 10 and compare their performance in Fig.

Original: It can be observedthat, generally, more units (10) in the hidden layer lead to better or at least equal performance thanother configurations.
Revised: It can be observedthat, generally, more units (10) in the hidden layer lead to better or at least equal performance thanother configurations.

Original: This result is in line with our motivation: a powerful non-linear mixing networkincreases the capability of CGs.
Revised: This result is in line with our motivation: a powerful non-linear mixing networkincreases the capability of CGs.

Original: In Fig.
Revised: In Fig.

Original: 4, we compare our enumerative and iterative methods.
Revised: 4, we compare our enumerative and iterative methods.

Original: These two methods checkthe same number (16) of linear pieces, but the iterativemethod can use a hidden layerof 10 .
Revised: These two methods checkthe same number (16) of linear pieces, but the iterativemethod can use a hidden layerof 10 .

Original: As a result, the iterativemethod outperforms the enumerative method.
Revised: As a result, the iterativemethod outperforms the enumerative method.

Original: Additionally, in Fig. 5, we compare NL-CG against DCG [4],
Revised: Additionally, in Fig. 5, we compare NL-CG against DCG [4],

Original: QMIX [20], and DICG [12] on
Revised: QMIX [20], and DICG [12] on

Original: Pursuit and MMM2 from the
Revised: Pursuit and MMM2 from the

Original: SMAC benchmark.
Revised: SMAC benchmark.

Original: We find
Revised: We find

Original: NL-CG generally outperforms previous methods.
Revised: NL-CG generally outperforms previous methods.

Original: On Pursuit , NL-CG with wider mixers use very few (20K-30K) samples toachieve DCG’s performance after convergence.
Revised: On Pursuit , NL-CG with wider mixers use very few (20K-30K) samples toachieve DCG’s performance after convergence.

Original: 5.1 Analysis of the optimality and efficiency of the iterative optimization method
Revised: 5.1 Analysis of the optimality and efficiency of the iterative optimization method

Original: [Figure]
Revised: [Figure]

Original: Although we have checked the performance of our iterative optimization method, learning curves can not fully reveal its optimality and efficiency.
Revised: Although we have checked the performance of our iterative optimization method, learning curves can not fully reveal its optimality and efficiency.

Original: In this section, we provide optimality and efficiencyanalyses by checking the action selection results and time costs in detail.
Revised: In this section, we provide optimality and efficiency analyses by checking the action selection results and time costs in detail.

Original: On Aloha , Sensor and Gather , we use our piece-wise enumeration method to train non-linear CGmodels with a width of 4.
Revised: In Fig.

Original: Then weuse the enumerative and iterative optimization method to selectgreedy actions. We compare the Q tot  of the actions selected by two methods in Fig. Also shownthere is the Q-values of the actions selected randomly and by DCG.
Revised: 6-middle, we compare the Q tot value of our enumerative and iterative optimization methods.

Original: We can see that the iterativemethod is near-optimal after checking only 4  slope configurations: the Q-values of its actions are veryclose to those selected by enumeration and are better than those selected by DCG. 
Revised: We can see that the iterative method is near-optimal after checking only 4 (embed=8) slope configurations: the Q-values of its actions are very close to those selected by enumeration and are better thanthose selected by DCG. In Fig.

Original: Since, theoretically,enumerative optimization can find global optimum, points that are closer to the line y = x indicatebetter solutions.
Revised: 6-left, we compare the ground truth Q estimates of DCG and NL-CG(embed=3, iterative, n max = 4 ). The result shows that NL-CG learns a policy with higher value.

Original: We also note that, in practice, Max-Sum is an approximate algorithm on cyclicgraphs, and enumeration cannot guarantee global optimality.
Revised: In Fig. 6-right, we compare the Q tot values estimated by NL-CG (embed=3, iterative, n max = 4 )against ground truth Q estimates.

Original: Some actions selected by the iterativemethod are even better than those selected by enumeration.
Revised: The estimation errors on all tested state-action pairs are less than20%.

Original: In Fig.
Revised: In Fig.

Original: 7, we compare the time spent byenumerative and iterative optimization methods.
Revised: 7, we compare the time spent by enumerative and iterative optimization methods.

Original: It can be found that the iterative method saves 50%to 65% of running time.
Revised: It can befound that the iterative method saves 50% to 65% of running time.

Original: We can thus conclude that the iterative optimization method provides a goodtrade-off between complexity and optimality.
Revised: We can thus conclude that theiterative optimization method provides a good trade-off between complexity and optimality.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we extend coordination graphs beyond linear decomposition by introducing non-linearmixing networks.
Revised: In this paper, we extend coordination graphs beyond linear decomposition by introducing non-linearmixing networks.

Original: Experiments manifest its superior representation power on complex tasks thatconventional CGs are not able to solve.
Revised: Experiments manifest its superior representation power on complex tasks thatconventional CGs are not able to solve.

Original: An important research direction is the stability of non-linear
Revised: An important research direction is the stability of non-linear

Original: CGs and to get rid of the non-negative constraint on weights of the mixing network.
Revised: CGs and to get rid of the non-negative constraint on weights of the mixing network.

Original: The authors donot see obvious negative societal impacts of our method.
Revised: The authors donot see obvious negative societal impacts of our method.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes](d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: A Hyperparameters and Infrastructure

Revised: When comparing the performance of NL-CG against DCG [4] and QMIX [20] on the MACObenchmark, we adopted the following hyper-parameter settings.

Revised: For all tasks, we used a discountfactor γ = 0 .

Revised: 99 and ϵ -greedy exploration, where ϵ was linearly decayed from 1 to 0 .

Revised: within the first50,000 training time steps.

Revised: The replay buffer stored the last 5000 episodes, from which we uniformlysampled batches of size 32 for training.

Revised: The target network was updated every 200 episodes.

Revised: Thelearning rate of RMSprop was set to 5 × 10 − 4 .

Revised: Every 10000 time steps we paused training andevaluated the model with 300 greedy test trajectories sampled with ϵ = 0 .

Revised: Each agent processed its local action-observation history at each time step using a linear layer of64 neurons, followed by a ReLU activation, a GRU of the same dimensionality, and finally a linearlayer of | A | neurons.

Revised: The output served as an individual feature vector and was fed into the utilityand payoff function.

Revised: The parameters of the utility and payoff function were shared amongst agents,who were identified by a one-hot encoded ID in the input.

Revised: All message passing procedures in

Revised: NL-CG iterated for 4 rounds.

Revised: The weights of the mixing network in NL-CG were generated by a2-layer hyper-network with a hidden layer of 64 ReLU neurons.

Revised: Biases in the mixing network weretransformed linearly from state vectors.

Revised: Our implementation of NL-CG was based on the PyMARL 1 [21] framework.

Revised: We used NVIDIA

Revised: GeForce RTX 3090 GPUs for training and evaluation.

Revised: B Weighted Max-Sum

Revised: Algorithm 1 and Algorithm 2 find global and approximate greedy actions for a non-linear coordinationgraph, respectively.

Revised: Both of these two algorithms rely on the weighted Max-Sum algorithm to findlocal optimal action on a linear piece.

Revised: We present the weighted Max-Sum algorithm in Algorithm 3.

Revised: C Representational Capability

Revised: In Sec. 4, we use a matrix game to show the representational capability of non-linear coordinationgraphs.

Revised: For conventional coordination graphs on State 2B of this game, since the value functionshould be permutation invariant, there are five unknowns q 1 = q i ( s 2 B , A ) , q 2 = q i ( s 2 B , B ) , q 3 =q ij ( s 2 B , AA ) , q 4 = q ij ( s 2 B , AB ) , and q 5 = q ij ( s 2 B , BB ) and five equations:

Revised: [Equation]

Revised: The augmented matrix of this system has a higher rank (4) than its coefficient matrix (3).

Revised: Therefore,this linear system does not have a solution, which means that a conventional coordination graphcannot represent the accurate value function for this task.

Revised: Algorithm

Revised: /*Greedy action selection with k message passing for one linear piece of the mixing network f m .

Revised: This algorithm is called by Algorithm 1 and 2, and the definition of inputs can be found there.*/

Revised: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , ( W V , W E

Revised: ) ∈ R |V| + |E| , b ∈ R f V :=

Revised: W V ◦ f V f E :=

Revised: W E ◦ f E µ 0 , ¯ µ 0 :=

Revised: 0 ∈ R

Revised: |E|× A /*Initialize forward messages ( µ ) and backward messages ( ¯ µ ).*/

Revised: q 0 :

Revised: = f V /*Initial “Q-value”.*/ q max :=

Revised: −∞ ; a max := (cid:2) arg max a ∈A i q 0 ai (cid:12)(cid:12) i ∈ V (cid:3) /*Initialize

Revised: the best found solution.*/ for t ∈ { 1 , . . .

Revised: , k } do /* k rounds of message passing.*/ for e =

Revised: ( i, j ) ∈

Revised: E do /*Update forward and backward messages.

Revised: Subscripts of f , q , µ mean indexing.*/ µ te := max

Revised: [Equation]

Revised: (cid:8) ( q

Revised: [Equation]

Revised: + f

Revised: [Equation]

Revised: (cid:9) /*Forward: maximize sender.*/ ¯ µ te := max

Revised: [Equation]

Revised: (cid:8) (

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: (cid:9) /*Backward

Revised: : maximize receiver.*/

Revised: [Equation]

Revised: tea /*Normalize forward messages to ensure converging.*/ ¯

Revised: [Equation]

Revised: ←

Revised: [Equation]

Revised: tea /*Normalize backward messages to ensure converging.*/ end for for i ∈ V do /*Update

Revised: “Q-value” with messages.*/ q ti :=

Revised: f V i + (cid:88)

Revised: e =(

Revised: · ,i ) ∈E µ te + (cid:88) e =( i, · ) ∈E ¯ µ te /*Utility plus incoming messages.*/ a ti := arg max a ∈A

Original: {+i { q tia } /*Select the greedy action of agent i .*/ end for q ′ ← (cid:80) |V| i =
+}
Revised: {+i { q tia } /*Select the greedy action of agent i .*/ end for q ′ ← (cid:80) |V| i =
+}

Revised: f V a i

Revised: + (cid:80) ( i,j ) ∈E f E a i a j + b /*Get the Q-value of the greedy action.*/ if q ′ >

Revised: q max then { a max ← a t ; q max ← q ′ q ← q ti } /*Remember

Revised: the best action.*/ end if end for return q max , a max ∈

Revised: A 1 × . . .

Revised: × A |V| , q /*Return the maximum Q value, the corresponding action, and utilities/payoffs.*/