Original: Non-Linear Coordination Graphs
Revised: Non-Linear Coordination Graphs

Original: Abstract
Revised: Abstract

Original: Value decomposition multi-agent reinforcement learning methods learn the global value function as a mixing of each agent’s individual utility functions.
Revised: Value decomposition multi-agent reinforcement learning methods learn the global value function as a mixing of each agent’s individual utility functions.

Original: Coordination graphs (CGs) represent a higher-order decomposition by incorporating pairwise payoff functions and thus is supposed to have a more powerful representational capacity.
Revised: Coordination graphs (CGs) represent a higher-order decomposition by incorporating pairwise payoff functions and thus is supposed to have a more powerful representational capacity.

Original: However, CGs decompose the global value function linearly over local value functions, severely limiting the complexity of the value function class that can be represented.
Revised: However, CGs decompose the global value function linearly over local value functions, severely limiting the complexity of the value function class that can be represented.

Original: In this paper, we propose the first non-linear coordination graph by extending CG value decomposition beyond the linear case.
Revised: In this paper, we propose the first non-linear coordination graph by extending CG value decomposition beyond the linear case.

Original: One major challenge is to conduct greedy action selections in this new function class to which commonly adopted DCOP algorithms are no longer applicable.
Revised: One major challenge is to conduct greedy action selections in this new function class to which commonly adopted DCOP algorithms are no longer applicable.

Original: We study how to solve this problem when mixing networks with LeakyReLU activation are used.
Revised: We study how to solve this problem when mixing networks with LeakyReLU activation are used.

Original: An enumeration method with a global optimality guarantee is proposed and motivates an efficient iterative optimization method with a local optimality guarantee.
Revised: An enumeration method with a global optimality guarantee is proposed and motivates an efficient iterative optimization method with a local optimality guarantee.

Original: We find that our method can achieve superior performance on challenging multi-agent coordination tasks like MACO.
Revised: We find that our method can achieve superior performance on challenging multi-agent coordination tasks like MACO.

Original:  Introduction
Revised:  Introduction

Original: Cooperative multi-agent problems are ubiquitous in real-world applications, such as crewless aerial vehicles [20, 36] and sensor networks [39].
Revised: Cooperative multi-agent problems are ubiquitous in real-world applications, such as crewless aerial vehicles [20, 35] and sensor networks [38].

Original: However, learning control policies for such systems remains a major challenge.
Revised: However, learning control policies for such systems remains a major challenge.

Original: Recently, value decomposition methods [27] significantly push forward the progress of multi-agent reinforcement learning [22, 25, 29, 30, 31].
Revised: Recently, value decomposition methods [26] significantly push forward the progress of multi-agent reinforcement learning [22, 24, 28, 29, 30].

Original: In these methods, a centralized action-value function is learned as a mixing of individual utility functions.
Revised: In these methods, a centralized action-value function is learned as a mixing of individual utility functions.

Original: The mixing function can be conditioned on global states [22] while individual utility functions are based on local actionobservation history.
Revised: The mixing function can be conditioned on global states [22] while individual utility functions are based on local actionobservation history.

Original: The advantage is that agents can utilize global information and avoid learning non-stationarity [28] via centralized training, while maintaining scalable decentralized execution.
Revised: The advantage is that agents can utilize global information and avoid learning non-stationarity [27] via centralized training, while maintaining scalable decentralized execution.

Original: Notably, the major focus of the value function decomposition research was on full decomposition, where local utility functions are based on actions and observations of a single agent.
Revised: Notably, the major focus of the value function decomposition research was on full decomposition, where local utility functions are based on actions and observations of a single agent.

Original: Full decomposition leads to several drawbacks, such as miscoordination problems in partially observable environments with stochastic transition functions [29, 33] and a game-theoretical pathology called relative overgeneralization [18, 4].
Revised: Full decomposition leads to several drawbacks, such as miscoordination problems in partially observable environments with stochastic transition functions [28, 32] and a game-theoretical pathology called relative overgeneralization [18, 4].

Original: Relative overgeneralization embodies that, due to the concurrent learning and exploration of other agents, the employed utility function may not be able to express optimal decentralized policies and prefer suboptimal actions that give higher returns on average.Coordination graph (CG) learning [9] holds the promise to address these problems while preserving the advantages of value decomposition methods. In a CG, each vertex represents an agent, and each
Revised: Relative overgeneralization embodies that, due to the concurrent learning and exploration of other agents, the employed utility function may not be able to express optimal decentralized policies and prefer suboptimal actions that give higher returns on average.

Original: 𝑄 "#" = %𝑞 ’  ’
Revised: 𝑄 "#" = %𝑞 ’  ’

Original: Fully decomposed value functions    𝑄 "#" = 𝑓 * (𝑞 , ,⋯,𝑞 / )
Revised: Fully decomposed value functions    𝑄 "#" = 𝑓 * (𝑞 , ,⋯,𝑞 / )

Original: VDN
Revised: VDN

Original: QMIX    𝑄 "#" = %𝑞 ’  ’,
Revised: QMIX    𝑄 "#" = %𝑞 ’  ’,

Original: Coordination graphs    𝑄 "#" =
Revised: Coordination graphs    𝑄 "#" =

Original: 𝑓 * ({𝑞 ’1 })
Revised: 𝑓 * ({𝑞 ’1 })

Original: DCG
Revised: DCG

Original: Ours
Revised: Ours

Revised: Coordination graph (CG) learning [9] holds the promise to address these problems while preserving the advantages of value decomposition methods.

Original:  (hyper-) edge stands for a payoff function that is defined on the joint action-observation space of the connected agents.
Revised: In a CG, each vertex represents an agent, and each (hyper-) edge stands for a payoff function that is defined on the joint action-observation space of the connected agents.

Original: The existence of payoff functions increases the granularity of decomposition, and connected agents explicitly coordinate with each other.
Revised: The existence of payoff functions increases the granularity of decomposition, and connected agents explicitly coordinate with each other.

Original: As a result, a CG represents a higher-order factorization of the global value function and represents a much larger value function class.
Revised: As a result, a CG represents a higher-order factorization of the global value function and represents a much larger value function class.

Original: However, coordination graphs typically assume a linear decomposition of the value function among sub-groups of agents, which is too simple to represent credit assignment in complex tasks.
Revised: However, coordination graphs typically assume a linear decomposition of the value function among sub-groups of agents, which is too simple to represent credit assignment in complex tasks.

Original: In this paper, we solve this long-standing problem and extend CG by introducing non-linear and learnable value decomposition.
Revised: In this paper, we solve this long-standing problem and extend CG by introducing non-linear and learnable value decomposition.

Original: To our best knowledge, it is the first study on non-linear coordination graphs.
Revised: To our best knowledge, it is the first study on non-linear coordination graphs.

Original: The major challenge against extending CGs beyond linear cases is the calculation of value-maximizing actions.
Revised: The major challenge against extending CGs beyond linear cases is the calculation of value-maximizing actions.

Original: When linearly decomposed, DCOP algorithms [6] can find a globally greedy action via message passing.
Revised: When linearly decomposed, DCOP algorithms [6] can find a globally greedy action via message passing.

Original: However, when the mixing function is non-linear, DCOP algorithms are no longer applicable.
Revised: However, when the mixing function is non-linear, DCOP algorithms are no longer applicable.

Original: To address this problem, we develop a DCOP method for non-linear mixing functions represented as a deep network with LeakyReLU (or ReLU) activation.
Revised: To address this problem, we develop a DCOP method for non-linear mixing functions represented as a deep network with LeakyReLU (or ReLU) activation.

Original: The core idea of our non-linear DCOP algorithm is to exploit the fact that these neural networks are equivalent to piece-wise linear functions.
Revised: The core idea of our non-linear DCOP algorithm is to exploit the fact that these neural networks are equivalent to piece-wise linear functions.

Original: For each linear piece, a value-maximizing action can be found by classic DCOP.
Revised: For each linear piece, a value-maximizing action can be found by classic DCOP.

Original: However, this action may fall out of the domain of the linear piece and is thus infeasible.
Revised: However, this action may fall out of the domain of the linear piece and is thus infeasible.

Original: We first prove that such a shifted action indicates a better solution in its domain.
Revised: We first prove that such a shifted action indicates a better solution in its domain.

Original: Based on this conclusion, we first show how to find a feasible joint action with the global optimal value, and then derive an iterative algorithm with local optimum convergence guarantee.
Revised: Based on this conclusion, we first show how to find a feasible joint action with the global optimal value, and then derive an iterative algorithm with local optimum convergence guarantee.

Original: We demonstrate the improved representational capacity of our Non-Linear Coordination Graphs (NL-CG) on a matrix game by comparing the learned Q functions to those learned by conventional coordination graphs.
Revised: We demonstrate the improved representational capacity of our Non-Linear Coordination Graphs (NL-CG) on a matrix game by comparing the learned Q functions to those learned by conventional coordination graphs.

Original: We then evaluate our method on the Multi-Agent COordination (MACO)
Revised: We then evaluate our method on the Multi-Agent COordination (MACO)

Original: Benchmark [34] for its high requirements on close inter-agent coordination.
Revised: Benchmark [33] for its high requirements on close inter-agent coordination.

Original: The experimental results show the superior performance enabled by the non-linear value decomposition.
Revised: The experimental results show the superior performance enabled by the non-linear value decomposition.

Original:  Preliminaries
Revised:  Preliminaries

Original: In this paper, we focus on fully cooperative multi-agent tasks that can be modelled as a DecPOMDP [17] consisting of a tuple G = ⟨
Revised: In this paper, we focus on fully cooperative multi-agent tasks that can be modelled as a DecPOMDP [17] consisting of a tuple G = ⟨

Original: I, S, A, P, R, Ω , O, n, γ ⟩ , where I is the finite set of n agents, γ ∈ [0 , 1) is the discount factor, and s ∈ S is the true state of the environment.
Revised: I, S, A, P, R, Ω , O, n, γ ⟩ , where I is the finite set of n agents, γ ∈ [0 , 1) is the discount factor, and s ∈ S is the true state of the environment.

Original: At each timestep, each agent i receives an observation o
Revised: At each timestep, each agent i receives an observation o

Original: i ∈ Ω drawn according to the observation function
Revised: i ∈ Ω drawn according to the observation function

Original: O ( s, i ) and selects an action a i ∈ A .
Revised: O ( s, i ) and selects an action a i ∈ A .

Original: Individual actions form a joint action a ∈ A n , which leads to a next state s ′ according to the transition function P ( s ′ | s, a ) , a reward r = R ( s, a ) shared by all agents.
Revised: Individual actions form a joint action a ∈ A n , which leads to a next state s ′ according to the transition function P ( s ′ | s, a ) , a reward r = R ( s, a ) shared by all agents.

Original: Each agent has local action-observation history τ i ∈ T ≡ (Ω × A ) ∗ × Ω .
Revised: Each agent has local action-observation history τ i ∈ T ≡ (Ω × A ) ∗ × Ω .

Original: Agents learn to collectively maximize the global return Q tot ( s, a ) =
Revised: Agents learn to collectively maximize the global return Q tot ( s, a ) =

Original: E s 0: ∞ ,a
Revised: E s 0: ∞ ,a

Original: 0: ∞ [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | s 

Revised: 0: ∞ [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | s

Original: = s, a 

Revised: = s, a

Original: = a ] .
Revised: = a ] .

Original: Estimating Q tot is at the core of multi-agent reinforcement learning (MARL) [27, 25, 14, 35, 12].
Revised: Estimating Q tot is at the core of multi-agent reinforcement learning (MARL) [26, 24, 14, 34, 12].

Original: The complexity of calculating max Q tot grows exponentially ( | A | n ) with the number of agents [15, 32].
Revised: The complexity of calculating max Q tot grows exponentially ( | A | n ) with the number of agents [15, 31].

Original: To solve this problem, value-based MARL decomposes the global action-value function into local utility functions q i and guarantees that the global maximizer can be obtained locally: arg max a Q tot ( s, a )
Revised: To solve this problem, value-based MARL decomposes the global action-value function into local utility functions q i and guarantees that the global maximizer can be obtained locally: arg max a Q tot ( s, a )

Original: =
Revised: =

Original: (cid:0) arg max [Equation] [Equation] , a 
Revised: (cid:0) arg max a 1 q 1 ( τ 1 , a 1 ) , . .

Original: [Equation]

Original: ) , . . . , arg max a n q n ( τ n , a n ) (cid:1) T .
Revised: . , arg max a n q n ( τ n , a n ) (cid:1) T .

Original: Value decomposition network (VDN) [27] satisfies this condition by learning the global value function as a summation of local utilities (Fig. 1left).
Revised: Value decomposition network (VDN) [26] satisfies this condition by learning the global value function as a summation of local utilities (Fig. 1left).

Original: QMIX [22] extends the function class of VDN by learning Q tot as a monotonic mixing of local utilities.
Revised: QMIX [22] extends the function class of VDN by learning Q tot as a monotonic mixing of local utilities.

Original: The mixing function is parameterized so that: ∂Q tot ( s, a ) ∂q i ( τ i ,a i ) ≥ 0 .
Revised: The mixing function is parameterized so that: ∂Q tot ( s, a ) ∂q i ( τ i ,a i ) ≥ 0 .

Original: 2.1 Coordination Graphs
Revised: 2.1 Coordination Graphs

Original: For fully decomposed value functions, local utility functions are conditioned on local actionobservation history.
Revised: For fully decomposed value functions, local utility functions are conditioned on local actionobservation history.

Original: Coordination Graphs (CGs) [9] increase the representational capacity of fully decomposed value functions by introducing higher-order payoff functions.
Revised: Coordination Graphs (CGs) [9] increase the representational capacity of fully decomposed value functions by introducing higher-order payoff functions.

Original: Specifically, a coordination graph is a tuple of a vertex set and an edge set: G = ⟨V , E⟩ .
Revised: Specifically, a coordination graph is a tuple of a vertex set and an edge set: G = ⟨V , E⟩ .

Original: Each vertex v i ∈ V represents an agent i , and (hyper-) edges in E represent coordination dependencies among agents.
Revised: Each vertex v i ∈ V represents an agent i , and (hyper-) edges in E represent coordination dependencies among agents.

Original: In previous work, the global value functions are decomposed linearly based on the graph topology:
Revised: In previous work, the global value functions are decomposed linearly based on the graph topology:

Original: [Equation]
Revised: [Equation]

Original: where q i and q ij is utility functions for individual agents and pairwise payoff functions, respectively.
Revised: where q i and q ij is utility functions for individual agents and pairwise payoff functions, respectively.

Original: τ ij = ⟨ τ i , τ j ⟩ and a ij = ⟨ a i , a j ⟩ is the joint action-observation history and action of agent i and j .
Revised: τ ij = ⟨ τ i , τ j ⟩ and a ij = ⟨ a i , a j ⟩ is the joint action-observation history and action of agent i and j .

Original: Previous work studies different aspects of such coordination graphs.
Revised: Previous work studies different aspects of such coordination graphs.

Original: It is shown that higher-order factorization is important on tackling an exponential number of joint actions [5].
Revised: It is shown that higher-order factorization is important on tackling an exponential number of joint actions [5].

Original: Sparse cooperative
Revised: Sparse cooperative

Original: Q-learning [11] learns value functions for sparse coordination graphs with pre-defined and static topology.
Revised: Q-learning [11] learns value functions for sparse coordination graphs with pre-defined and static topology.

Original: Zhang et al .
Revised: Zhang et al .

Original: [39] propose to learn minimized dynamic coordination sets for each agent.
Revised: [38] propose to learn minimized dynamic coordination sets for each agent.

Original: DCG [4] incorporates deep function approximation and parameter sharing into coordination graphs and scales to large state-action spaces.
Revised: DCG [4] incorporates deep function approximation and parameter sharing into coordination graphs and scales to large state-action spaces.

Original: CASEC [34] and SOP-CG [37] studies how to build sparse deep coordination graphs that are adaptive to the dynamic coordination requirements.
Revised: CASEC [33] and SOP-CG [36] studies how to build sparse deep coordination graphs that are adaptive to the dynamic coordination requirements.

Original: In all these works, the global value function is represented as a summation of local value functions.
Revised: In all these works, the global value function is represented as a summation of local value functions.

Original: To the best of our knowledge, this paper presents the fist CG learning method with non-linear value decomposition.
Revised: To the best of our knowledge, this paper presents the fist CG learning method with non-linear value decomposition.

Original: 2.2 Message Passing
Revised: 2.2 Message Passing

Original: Within a coordination graph, the greedy action selection required by Q-learning can not be completed by simply computing the maximum of individual utility and payoff functions.
Revised: Within a coordination graph, the greedy action selection required by Q-learning can not be completed by simply computing the maximum of individual utility and payoff functions.

Original: Instead, distributed constraint optimization (DCOP) [6] techniques are used.
Revised: Instead, distributed constraint optimization (DCOP) [6] techniques are used.

Original: Max-Sum [26] is a popular implementation of DCOP.
Revised: Max-Sum [25] is a popular implementation of DCOP.

Original: Max-Sum finds near-optimal actions on a CG G = ⟨V , E⟩ via multi-round message passing on a bipartite graph G m = ⟨V a , V q , E m ⟩ .
Revised: Max-Sum finds near-optimal actions on a CG G = ⟨V , E⟩ via multi-round message passing on a bipartite graph G m = ⟨V a , V q , E m ⟩ .

Original: Each node i ∈ V a represents an agent, and each node g ∈ V q represents a utility ( q i ) or payoff ( q ij ) function.
Revised: Each node i ∈ V a represents an agent, and each node g ∈ V q represents a utility ( q i ) or payoff ( q ij ) function.

Original: Edges in E m connect g with the associated agent node(s).
Revised: Edges in E m connect g with the associated agent node(s).

Original: Message passing starts with sending messages from node i ∈ V a to node g ∈ V q :
Revised: Message passing starts with sending messages from node i ∈ V a to node g ∈ V q :

Original: [Equation]
Revised: [Equation]

Original: where F i is the set of nodes in V q connected to node i , and c ig is a normalizing factor preventing the value of messages from growing arbitrarily large.
Revised: where F i is the set of nodes in V q connected to node i , and c ig is a normalizing factor preventing the value of messages from growing arbitrarily large.

Original: Messages are then sent from node g to node i :
Revised: Messages are then sent from node g to node i :

Original: [Equation]
Revised: [Equation]

Original: where V g is the set of nodes in V a connected to node g , a g [Equation] { a 
Revised: where V g is the set of nodes in V a connected to node g , a g = { a h | h ∈ V g } , a g \ a i

Original: [Equation] }} , and q represents utility or payoff functions conditioned on a g .
Revised: = { a h | h ∈ V g \{ i }} , and q represents utility or payoff functions conditioned on a g .

Original: Eq.
Revised: Eq.

Original: 2 and 3 make up an iteration of message passing.
Revised: 2 and 3 make up an iteration of message passing.

Original: Each agent i can find its local optimal action by calculating a ∗ i = argmax a i (cid:80) h
Revised: Each agent i can find its local optimal action by calculating a ∗ i = argmax a i (cid:80) h

Original: ∈F
Revised: ∈F

Original: i m h
Revised: i m h

Original: →
Revised: →

Original: i ( a i ) after several iterations of message passing.
Revised: i ( a i ) after several iterations of message passing.

Original: Notably, Max-Sum and other DCOP algorithms are only applicable to linearly decomposed value functions.
Revised: Notably, Max-Sum and other DCOP algorithms are only applicable to linearly decomposed value functions.

Original: 2.3 Piece-Wise Linear Neural Networks
Revised: 2.3 Piece-Wise Linear Neural Networks

Original: Following the recent decade’s success of deep neural networks (DNNs), analysis works have been done trying to explain the mechanism of the DNN black-boxes and assess their function approximation capabilities.
Revised: Following the recent decade’s success of deep neural networks (DNNs), analysis works have been done trying to explain the mechanism of the DNN black-boxes and assess their function approximation capabilities.

Original: One conclusion is that a DNN with piece-wise linear (PWL) activation functions (e.g.
Revised: One conclusion is that a DNN with piece-wise linear (PWL) activation functions (e.g.

Original: ReLU, LeakyReLU, PReLU) is equivalent to a PWL function.
Revised: ReLU, LeakyReLU, PReLU) is equivalent to a PWL function.

Original: This kind of DNNs are called piecewise linear neural networks (PLNNs) [7].
Revised: This kind of DNNs are called piecewise linear neural networks (PLNNs) [7].

Original: Early papers [16, 19] assess the expressivity of PLNNs by the amount of linear pieces.
Revised: Early papers [16, 19] assess the expressivity of PLNNs by the amount of linear pieces.

Original: [2, 24, 10] give more theoretically grounded results about the upper and lower bounds for the amount of pieces.
Revised: [2, 23, 10] give more theoretically grounded results about the upper and lower bounds for the amount of pieces.

Original: In this sense, it is shown that increasing the depth of a network can generally be exponentially more valuable than increasing the width [19, 8, 2].
Revised: In this sense, it is shown that increasing the depth of a network can generally be exponentially more valuable than increasing the width [19, 8, 2].

Original: Chu et al .
Revised: Chu et al .

Original: [7] propose OPENBOX to compute the mathematically equivalent set of linear pieces, which provides an accurate and consistent interpretation of PLNNs.
Revised: [7] propose OPENBOX to compute the mathematically equivalent set of linear pieces, which provides an accurate and consistent interpretation of PLNNs.

Original: For neural networks with a single layer of hidden nodes, the problem can be reduced to hyper-plane arrangement [38], and linear functions and their domain can be enumerated efficiently [21, 3].
Revised: For neural networks with a single layer of hidden nodes, the problem can be reduced to hyper-plane arrangement [37], and linear functions and their domain can be enumerated efficiently [21, 3].

Original:  Method
Revised:  Method

Original: Our method extends the representational capability of coordination graphs by introducing non-linear factorization of the global value function.
Revised: Our method extends the representational capability of coordination graphs by introducing non-linear factorization of the global value function.

Original: Specifically, for a coordination graph G = ⟨V , E⟩ (we study complete coordination graphs in this paper), we decompose the global Q as:
Revised: Specifically, for a coordination graph G = ⟨V , E⟩ (we study complete coordination graphs in this paper), we decompose the global Q as:

Original: [Equation]
Revised: [Equation]

Original: where q i is the vector of all individual utilities, and q ij is the vector of all pairwise payoffs for edges in E .
Revised: where q i is the vector of all individual utilities, and q ij is the vector of all pairwise payoffs for edges in E .

Original: Similar to DCG [4], we learn a shared utility function f v , parameterized by θ v , and get the individual utility q i ( τ i , a i ) =
Revised: Similar to DCG [4], we learn a shared utility function f v , parameterized by θ v , and get the individual utility q i ( τ i , a i ) =

Original: f v ( a i | τ i ; θ v ) .
Revised: f v ( a i | τ i ; θ v ) .

Original: The payoffs are estimated by a shared function f e parameterized by θ e : q ij ( τ i , τ j , a i , a j ) =
Revised: The payoffs are estimated by a shared function f e parameterized by θ e : q ij ( τ i , τ j , a i , a j ) =

Original: f e ( a i , a j | τ i , τ j ; θ e ) .
Revised: f e ( a i , a j | τ i , τ j ; θ e ) .

Original: Different from conventional CGs (Eq. 1), our f n is a non-linear mixing network whose parameters are generated by a hypernet f h conditioned on the global state s and parameterized by θ h .
Revised: Different from conventional CGs (Eq. 1), our f n is a non-linear mixing network whose parameters are generated by a hypernet f h conditioned on the global state s and parameterized by θ h .

Original: Our discussion is based on LeakyReLU networks with the slope coefficient α ∈ [0 , 1] .
Revised: Our discussion is based on LeakyReLU networks with the slope coefficient α ∈ [0 , 1] .

Original: For efficiently calculating the maximizer of Q tot , we require the weights after the first layer of the mixing network to be non-negative.
Revised: For efficiently calculating the maximizer of Q tot , we require the weights after the first layer of the mixing network to be non-negative.

Original: The reason for this non-negativity constraint will be discussed in Lemmaand 2.
Revised: The reason for this non-negativity constraint will be discussed in Lemmaand 2.

Original: Such a mixing network is effectively a type of input convex neural networks (ICNN [1]).
Revised: Such a mixing network is effectively a type of input convex neural networks (ICNN [1]).

Original: The non-negativity constraint on parameters is somewhat constraint, but we can use the passthrough trick introduced in Proposition 1 of [1] to maintain substantial representation power of the mixing network.
Revised: The non-negativity constraint on parameters is somewhat constraint, but we can use the passthrough trick introduced in Proposition 1 of [1] to maintain substantial representation power of the mixing network.

Original: The whole framework, including the utility function f v , the payoff function f e , and the hypernet f h , is updated by minimizing the TD loss:
Revised: The whole framework, including the utility function f v , the payoff function f e , and the hypernet f h , is updated by minimizing the TD loss:

Original: [Equation]
Revised: [Equation]

Original: where D is the replay buffer, and ˆ Q tot is a target function whose parameters are periodically copied from the function Q tot .
Revised: where D is the replay buffer, and ˆ Q tot is a target function whose parameters are periodically copied from the function Q tot .

Original: The main challenge left, for both the training and execution phase of deep Q-learning, is to select actions that maximize the global Q value at each time step, arg max a Q tot ( τ , a ) .
Revised: The main challenge left, for both the training and execution phase of deep Q-learning, is to select actions that maximize the global Q value at each time step, arg max a Q tot ( τ , a ) .

Original: This problem is theoretically NP-hard when the mixing function is non-linear.
Revised: This problem is theoretically NP-hard when the mixing function is non-linear.

Original: In the following sections, we first provide a global optimal algorithm for this problem.
Revised: In the following sections, we first provide a global optimal algorithm for this problem.

Original: The complexity of this algorithm grows polynomially with the number of hidden units in the mixing network.
Revised: The complexity of this algorithm grows polynomially with the number of hidden units in the mixing network.

Original: To reduce time complexity, we propose an iterative algorithm with local optimum convergence guarantee.
Revised: To reduce time complexity, we propose an iterative algorithm with local optimum convergence guarantee.

Original: We use the following notations .
Revised: We use the following notations .

Original: The input to the mixing network is q =
Revised: The input to the mixing network is q =

Original: [ q i ∥ q ij ] , where [ ·∥· ] means vector concatenation.
Revised: [ q i ∥ q ij ] , where [ ·∥· ] means vector concatenation.

Original: We use q ( a ) to denote the utilities and payoffs corresponding to action a .
Revised: We use q ( a ) to denote the utilities and payoffs corresponding to action a .

Original: The mixing network has L LeakyReLU linear layers.
Revised: The mixing network has L LeakyReLU linear layers.

Original: d = |V| + |E| is the input dimension.
Revised: d = |V| + |E| is the input dimension.

Original: m i , W i , and b i ∈
Revised: m i , W i , and b i ∈

Original: R m i are the width, weights, and biases of the i th layer.
Revised: R m i are the width, weights, and biases of the i th layer.

Original: The weights after the first layer are non-negative.
Revised: The weights after the first layer are non-negative.

Original: The input to the activation units at i th layer is z i , and the corresponding output is h i =
Revised: The input to the activation units at i th layer is z i , and the corresponding output is h i =

Original: LeakyReLU
Revised: LeakyReLU

Original: ( z i ) =
Revised: ( z i ) =

Original: c i ◦
Revised: c i ◦

Original: o i , where c i is the value of LeakyReLU activation ( c = α when z < 0 and c = 1 when z ≥ 0 ) and ◦ is the element-wise multiplication.
Revised: o i , where c i is the value of LeakyReLU activation ( c = α when z < 0 and c = 1 when z ≥ 0 ) and ◦ is the element-wise multiplication.

Original: We call c =
Revised: We call c =

Original: [ c 1 ∥ . . .
Revised: [ c 1 ∥ . . .

Original: ∥ c L ] ∈ { α, 1 } m , m = (cid:80) i m i , a slope configuration of the mixing network.
Revised: ∥ c L ] ∈ { α, 1 } m , m = (cid:80) i m i , a slope configuration of the mixing network.

Original: Subscripts of W i , b i , z i , and h i means index.
Revised: Subscripts of W i , b i , z i , and h i means index.

Original: For example, z ij is the j th element of z i , and W ij is the j th row of W i .
Revised: For example, z ij is the j th element of z i , and W ij is the j th row of W i .

Original: 3.1 Piece-Wise Optimization
Revised: 3.1 Piece-Wise Optimization

Original: The general idea of our action selection algorithm is to utilize the piece-wise linear property of the
Revised: The general idea of our action selection algorithm is to utilize the piece-wise linear property of the

Original: LeakyReLU network.
Revised: LeakyReLU network.

Original: Given a slope configuration c , the mixing network becomes linear, and we can run a DCOP algorithm to get the corresponding maximizer of Q tot .
Revised: Given a slope configuration c , the mixing network becomes linear, and we can run a DCOP algorithm to get the corresponding maximizer of Q tot .

Original: The question is that the obtained solution may be out of the domain of the linear piece.
Revised: The question is that the obtained solution may be out of the domain of the linear piece.

Original: We show that, when the weights after the first layer are non-negative, we can ignore whether the obtained local optimal solution is in the piece’s domain, and the global maximizer of Q tot is the best of the local optima.
Revised: We show that, when the weights after the first layer are non-negative, we can ignore whether the obtained local optimal solution is in the piece’s domain, and the global maximizer of Q tot is the best of the local optima.

Original: Formally, there are 2 m slope configurations in C all
Revised: Formally, there are 2 m slope configurations in C all

Original: = { c | c ∈ { α, 1 } m } , each of which makes the mixing network an affine function in P all = { ρ 1 , ρ 2 , . .
Revised: = { c | c ∈ { α, 1 } m } , each of which makes the mixing network an affine function in P all = { ρ 1 , ρ 2 , . .

Original: , ρ 2 m } .
Revised: , ρ 2 m } .

Original: Each affine function ρ k has a corresponding cell P k where q ∈ P k yields c k in the forward pass.
Revised: Each affine function ρ k has a corresponding cell P k where q ∈ P k yields c k in the forward pass.

Original: For each c k ∈ C all , by running
Revised: For each c k ∈ C all , by running

Original: DCOP, we obtain the local maximum q k and the corresponding joint action a k and utilities/payoffs q k .
Revised: DCOP, we obtain the local maximum q k and the corresponding joint action a k and utilities/payoffs q k .

Original: However, it is possible that q k falls out of P k and indeed yields another slope configuration c r ̸ = k .
Revised: However, it is possible that q k falls out of P k and indeed yields another slope configuration c r ̸ = k .

Original: We first show that such a shifted solution indicates an equal or better solution in its domain.
Revised: We first show that such a shifted solution indicates an equal or better solution in its domain.

Original: Lemma 1.
Revised: Lemma 1.

Original: Denote affine function pieces and their cells of a fully-connected feedforward mixing network with LeakyReLU activation as P all = { ρ j } 2 m 1 and { P j } 2 m 1 .
Revised: Denote affine function pieces and their cells of a fully-connected feedforward mixing network with LeakyReLU activation as P all = { ρ j } 2 m 1 and { P j } 2 m 1 .

Original: For q in the cell of the r th piece,
Revised: For q in the cell of the r th piece,

Original: P r , and ∀ ρ s ∈ P all , we have ρ r ( q ) ≥
Revised: P r , and ∀ ρ s ∈ P all , we have ρ r ( q ) ≥

Original: ρ s ( q ) .
Revised: ρ s ( q ) .

Original: Proof.
Revised: Proof.

Original: We start with the first difference between c r and c s .
Revised: We start with the first difference between c r and c s .

Original: We denote it as c rij ̸ = c sij .
Revised: We denote it as c rij ̸ = c sij .

Original: Because this is the first difference, the input to this unit, z ij , is the same.
Revised: Because this is the first difference, the input to this unit, z ij , is the same.

Original: Since q is in P r , we have that c rij iswhen
Revised: Since q is in P r , we have that c rij iswhen

Original: z ij is positive and is α < 1 when z ij is negative.
Revised: z ij is positive and is α < 1 when z ij is negative.

Original: Therefore, the output satisfies that
Revised: Therefore, the output satisfies that

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: because W i +1 ≥ 0 .
Revised: because W i +1 ≥ 0 .

Original: Other differences in the i th layer will lead to the same conclusion.
Revised: Other differences in the i th layer will lead to the same conclusion.

Original: At layer i + 1 , we have
Revised: At layer i + 1 , we have

Original: [Equation]
Revised: [Equation]

Original: The first inequity is because q is in the cell of ρ r .
Revised: The first inequity is because q is in the cell of ρ r .

Original: The second inequity is because of Eq.
Revised: The second inequity is because of Eq.

Original: 7 and that
Revised: 7 and that

Original: LeakyReLU is a monotonic increasing function.
Revised: LeakyReLU is a monotonic increasing function.

Original: The proof can repeat for the following layers and thus holds for the last layer, i .
Revised: The proof can repeat for the following layers and thus holds for the last layer, i .

Original: e ., ρ r ( q ) ≥
Revised: e ., ρ r ( q ) ≥

Original: ρ s ( q ) .
Revised: ρ s ( q ) .

Original: Based on Lemma 1, when the local optimal solution, a k , of piece k returned by DCOP actually falls in cell P r ̸
Revised: Based on Lemma 1, when the local optimal solution, a k , of piece k returned by DCOP actually falls in cell P r ̸

Original: = k , we have ρ r ( q ( a k )) ≥ ρ k ( q ( a k )) .
Revised: = k , we have ρ r ( q ( a k )) ≥ ρ k ( q ( a k )) .

Original: This indicates that, on piece r , we will get a solution at least as good as the one on piece k .
Revised: This indicates that, on piece r , we will get a solution at least as good as the one on piece k .

Original: Based on this conclusion, we now prove that the maximum of each piece’s local optimum is the global optimum.
Revised: Based on this conclusion, we now prove that the maximum of each piece’s local optimum is the global optimum.

Original: Lemma 2.
Revised: Lemma 2.

Original: Running DCOP algorithm for each linear piece in P all , then take the maximum value among these pieces, we can get the global optimizer of Q tot .
Revised: Running DCOP algorithm for each linear piece in P all , then take the maximum value among these pieces, we can get the global optimizer of Q tot .

Original: Proof.
Revised: Proof.

Revised: We have that max

Revised: [Equation]

Original: We have that max q f n ( q ) = max q max ρ p ∈P all ρ p ( q ) = max ρ p ∈P all max q ρ p ( q ) ( f n is the mixing function, Eq. 4), which means the maximum of local optimal values is the global optimal value.
Revised: [Equation] ) = [Equation] ρ p ( q ) = max ρ p ∈P all max q ρ p ( q ) ( f n is the mixing function, Eq. 4), which means the maximum of local optimal values is the global optimal value.

Original: Suppose the global optimal value Q max is found on piece r and the corresponding action is a r , the question is whether q ( a r ) is feasible, i .
Revised: Suppose the global optimal value Q max is found on piece r and the corresponding action is a r , the question is whether q ( a r ) is feasible, i .

Original: e ., q ( a r ) ∈ P r .
Revised: e ., q ( a r ) ∈ P r .

Original: Assume that q ( a r ) falls in P s ̸ = r .
Revised: Assume that q ( a r ) falls in P s ̸ = r .

Original: Then for piece s , a r is a feasible solution with the value Q max (due to Lemma 1).
Revised: Then for piece s , a r is a feasible solution with the value Q max (due to Lemma 1).

Original: This means the global optimal solution is always a feasible solution.
Revised: This means the global optimal solution is always a feasible solution.

Original: According to Lemma 2, we can run message passing on each piece (Algorithm 3 in Appx. 
Revised: According to Lemma 2, we can run message passing on each piece (Algorithm ?? in Appx. ?? )

Original: B) and then take the best of these results to get the global optimum.
Revised: and then take the best of these results to get the global optimum.

Original: Detailed process can be found in Algorithm 1.
Revised: Detailed process can be found in Algorithm 1.

Original: The problem of Algorithm 1 is time complexity.
Revised: The problem of Algorithm 1 is time complexity.

Original: When m is small, we enumerate all 2 m slope configurations.
Revised: When m is small, we enumerate all 2 m slope configurations.

Original: For a large m , we can use the hyperplane arrangement algorithm [7] to calculate exact linear pieces and their domains.
Revised: For a large m , we can use the hyperplane arrangement algorithm [7] to calculate exact linear pieces and their domains.

Original: Specifically, there are
Revised: Specifically, there are

Original: [Equation]
Revised: [Equation]

Original: pieces need enumerating, where d is the input length.
Revised: pieces need enumerating, where d is the input length.

Original: This number is exponential to d .
Revised: This number is exponential to d .

Original: To reduce time complexity, we propose an iterative optimization method in the next section.
Revised: To reduce time complexity, we propose an iterative optimization method in the next section.

Original: 3.2 Iterative Optimization with Local Optimum Guarantee
Revised: 3.2 Iterative Optimization with Local Optimum Guarantee

Original: When m is large, enumerating either 2 m or n m,d pieces can be costly.
Revised: When m is large, enumerating either 2 m or n m,d pieces can be costly.

Original: We thus propose an iterative algorithm with local optimum guarantee (Algorithm 2) to get approximate solutions.
Revised: We thus propose an iterative algorithm with local optimum guarantee (Algorithm 2) to get approximate solutions.

Original: We begin with the slope configuration where all hidden units are activated ( c = { 1 } m ).
Revised: We begin with the slope configuration where all hidden units are activated ( c = { 1 } m ).

Original: We run message passing on the current configuration c
Revised: We run message passing on the current configuration c

Original: p and get a local optimum solution a p .
Revised: p and get a local optimum solution a p .

Original: If the real configuration c real of a p is not c p , we calculate the local optimum of c real .
Revised: If the real configuration c real of a p is not c p , we calculate the local optimum of c real .

Original: The iteration continues until c real =
Revised: The iteration continues until c real =

Original: c p .
Revised: c p .

Original: Lemma 1 guarantees that the solution values in the iteration are monotonically increasing.
Revised: Lemma 1 guarantees that the solution values in the iteration are monotonically increasing.

Original: Since there is a finite number of configurations, our algorithm can converge to a local optimum.
Revised: Since there is a finite number of configurations, our algorithm can converge to a local optimum.

Original: With a very low probability, a loop would appear in the iteration.
Revised: With a very low probability, a loop would appear in the iteration.

Original: The appearance of a loop means that the equity in Lemma 1 holds.
Revised: The appearance of a loop means that the equity in Lemma 1 holds.

Original: In this case, the solutions on the loop have the same value, thus we can stop our iteration when encountering a slope configuration that has been searched.
Revised: In this case, the solutions on the loop have the same value, thus we can stop our iteration when encountering a slope configuration that has been searched.

Original: Algorithm
Revised: Algorithm

Original: /*Show the case for a two-layer mixing network, but can be easily extended to more layers.*/
Revised: /*Show the case for a two-layer mixing network, but can be easily extended to more layers.*/

Original: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , W 0 , b 0 , W 1 , b 1 q max :=
Revised: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , W 0 , b 0 , W 1 , b 1 q max :=

Original: −∞ ; a max := (cid:2)(cid:3) /*Initialize
Revised: −∞ ; a max := (cid:2)(cid:3) /*Initialize

Original: the best found solution.*/ for c p ∈ { α, 1 } m do /*Calculate the equivalent weights and biases.*/
Revised: the best found solution.*/ for c p ∈ { α, 1 } m do /*Calculate the equivalent weights and biases.*/

Original: W ρ p :=
Revised: W ρ p :=

Original: W 0 · ( c p ◦ W 1 )
Revised: W 0 · ( c p ◦ W 1 )

Original: b ρ p :=
Revised: b ρ p :=

Original: ( c p ◦ W 1 ) ·
Revised: ( c p ◦ W 1 ) ·

Original: b T 0 + bq, a , · ← w -MAX-SUM
Revised: b T 0 + bq, a , · ← w -MAX-SUM

Original: ( f V , f E , W ρ p , b ρ p )
Revised: ( f V , f E , W ρ p , b ρ p )

Original: if q > q max then a max ← a q max ← q /*Remember only the best actions.*/ end if end for return a max /*Return
Revised: if q > q max then a max ← a q max ← q /*Remember only the best actions.*/ end if end for return a max /*Return

Original: the selected joint action a max .*/
Revised: the selected joint action a max .*/

Original: Algorithm
Revised: Algorithm

Original: /*Show the case for a two-layer mixing network, but can be easily extended to more layers.*/
Revised: /*Show the case for a two-layer mixing network, but can be easily extended to more layers.*/

Original: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , W 0 , b 0 , W 1 , b 1 , n max , ϵ q max :=
Revised: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , W 0 , b 0 , W 1 , b 1 , n max , ϵ q max :=

Original: −∞ , a max := (cid:2)(cid:3) , c p := { 1 } m for n ∈ { 1 , . . . , n max } do /*Try
Revised: −∞ , a max := (cid:2)(cid:3) , c p := { 1 } m for n ∈ { 1 , . . . , n max } do /*Try

Original: one configuration at a time.*/
Revised: one configuration at a time.*/

Original: W ρ p :=
Revised: W ρ p :=

Original: W 0 · ( c p ◦ W 1 )
Revised: W 0 · ( c p ◦ W 1 )

Original: b ρ p :=
Revised: b ρ p :=

Original: ( c p ◦ W 1 ) ·
Revised: ( c p ◦ W 1 ) ·

Original: b T 0 + bq, a , · ← w -MAX-SUM
Revised: b T 0 + bq, a , · ← w -MAX-SUM

Original: ( f V , f E , W ρ p , b ρ p )
Revised: ( f V , f E , W ρ p , b ρ p )

Original: if q > q max then a max ← a q max ← q c real ←
Revised: if q > q max then a max ← a q max ← q c real ←

Original: The real LeakyReLU slope configuration end if if c real ̸ = c p then c
Revised: The real LeakyReLU slope configuration end if if c real ̸ = c p then c

Original: p ← c real else with prob.
Revised: p ← c real else with prob.

Original: ϵ break or continue with an unvisited c p .
Revised: ϵ break or continue with an unvisited c p .

Original: end for return a max
Revised: end for return a max

Original: To increase the possibility of finding a global optimum, we can introduce a simulated annealing mechanism.
Revised: To increase the possibility of finding a global optimum, we can introduce a simulated annealing mechanism.

Original: Each time we find a better solution, with a probability of 1 − ϵ , we move to the corresponding piece, and with a probability of ϵ , we jump to a random piece that has not been searched.
Revised: Each time we find a better solution, with a probability of 1 − ϵ , we move to the corresponding piece, and with a probability of ϵ , we jump to a random piece that has not been searched.

Original: ϵ decreases with the iteration number.
Revised: ϵ decreases with the iteration number.

Original: Discussion about loopy graph topology In this paper, we consider complete graphs when studying non-linear coordination graphs.
Revised: Discussion about loopy graph topology In this paper, we consider complete graphs when studying non-linear coordination graphs.

Original: A concern is that message passing algorithms like Max-Sum may not converge to the optimal solutions in loopy graphs and has an error rate of e .
Revised: A concern is that message passing algorithms like Max-Sum may not converge to the optimal solutions in loopy graphs and has an error rate of e .

Original: Lemma 1 is not affected because it is a property of LeakyReLU Networks.
Revised: Lemma 1 is not affected because it is a property of LeakyReLU Networks.

Original: For Lemma 2, the maximum of solutions found by message passing in all slope configurations is the global optimum with a probability of 1 − e .
Revised: For Lemma 2, the maximum of solutions found by message passing in all slope configurations is the global optimum with a probability of 1 − e .

Original: An error occurs when message passing cannot find the right solution on the piece where the global optimum is located.
Revised: An error occurs when message passing cannot find the right solution on the piece where the global optimum is located.

Original: Our iterative method may stop earlier when message passing returns a wrong solution located in the current cell.
Revised: Our iterative method may stop earlier when message passing returns a wrong solution located in the current cell.

Original: The probability of this situation is less than e .
Revised: The probability of this situation is less than e .

Original: Thus we have at least a probability of (1 − e ) n ( n is the number of iterations) to find the piece where the local optimum is located, and the final probability of finding the local optimum is larger than (1 − e ) n +1 .
Revised: Thus we have at least a probability of (1 − e ) n ( n is the number of iterations) to find the piece where the local optimum is located, and the final probability of finding the local optimum is larger than (1 − e ) n +1 .

Original:  Representational Capability
Revised:  Representational Capability

Original: In this section, we compare the representational capacity of our model against conventional coordination graphs.
Revised: In this section, we compare the representational capacity of our model against conventional coordination graphs.

Original: The comparison is carried out on a two-step cooperative matrix game with four players and two actions.
Revised: The comparison is carried out on a two-step cooperative matrix game with four players and two actions.

Original: At the first step, Agent 1’s action decides which of the two matrix games (Table. 1) to play in the next timestep.
Revised: At the first step, Agent 1’s action decides which of the two matrix games (Table. 1) to play in the next timestep.

Original: In the second step, the number of agents taking Action B determines the global reward received by the agent team (Table. 1).
Revised: In the second step, the number of agents taking Action B determines the global reward received by the agent team (Table. 1).

Original: [Table]
Revised: [Table]

Original: We first theoretically prove that the representational capacity of conventional coordination graphs is unable to represent the Q -function of this game.
Revised: We first theoretically prove that the representational capacity of conventional coordination graphs is unable to represent the Q -function of this game.

Original: Since the reward is invariant to the identity of agents, i .
Revised: Since the reward is invariant to the identity of agents, i .

Original: e .,
Revised: e .,

Original: R ( a 1 , a 2 , a 3 , a 4 ) =
Revised: R ( a 1 , a 2 , a 3 , a 4 ) =

Original: R ( p ( a 1 , a 2 , a 3 , a 4 )) , where p is an arbitrary permutation, the learned action-value function should also be permutation invariant.
Revised: R ( p ( a 1 , a 2 , a 3 , a 4 )) , where p is an arbitrary permutation, the learned action-value function should also be permutation invariant.

Original: Therefore we can ignore the order of actions in value functions.
Revised: Therefore we can ignore the order of actions in value functions.

Original: Now let’s focus on State 2B ( s 2 B ).
Revised: Now let’s focus on State 2B ( s 2 B ).

Original: Conventional coordination graphs need to solve a linear system consisting of five unknowns and five equations.
Revised: Conventional coordination graphs need to solve a linear system consisting of five unknowns and five equations.

Original: However, for this system, the rank of the augmented matrix (4) is greater than the rank of the coefficient matrix (3).
Revised: However, for this system, the rank of the augmented matrix (4) is greater than the rank of the coefficient matrix (3).

Original: Details can be found in Appx. 
Revised: Details can be found in Appx. ?? .

Original: C. Therefore, this system has no solution, and it is impossible for conventional CGs to learn a correct Q function.
Revised: Therefore, this system has no solution, and it is impossible for conventional CGs to learn a correct Q function.

Original: We then empirically demonstrate our idea.
Revised: We then empirically demonstrate our idea.

Original: We train NL-CG and DCG on the task for 5000 episodes under full exploration ( ϵ = 1 ) and examine the learned value functions.
Revised: We train NL-CG and DCG on the task for 5000 episodes under full exploration ( ϵ = 1 ) and examine the learned value functions.

Original: Full exploration ensures that both methods explore all state-action pairs.
Revised: Full exploration ensures that both methods explore all state-action pairs.

Original: In such a case, the representational capacity of the action-value function approximator remains the only challenge of learning accurate Q functions.
Revised: In such a case, the representational capacity of the action-value function approximator remains the only challenge of learning accurate Q functions.

Original: For our algorithm, the utility and payoff function is fully connected networks with a single hidden layer of 64 units with a ReLU non-linearity.
Revised: For our algorithm, the utility and payoff function is fully connected networks with a single hidden layer of 64 units with a ReLU non-linearity.

Original: γ is 0.99, and the replay buffer stores the last 500 episodes, from which we uniformly sample batches of size 32 for training.
Revised: γ is 0.99, and the replay buffer stores the last 500 episodes, from which we uniformly sample batches of size 32 for training.

Original: The target network is updated every 100 episodes.
Revised: The target network is updated every 100 episodes.

Original: The learning rate of RMSprop is set to 5 × 10 − 4 .
Revised: The learning rate of RMSprop is set to 5 × 10 − 4 .

Original: Agents receive the full state as observation, which is represented as an one-hot vector.
Revised: Agents receive the full state as observation, which is represented as an one-hot vector.

Original: Table 2 and 3 show the learned Q by DCG and NL-CG.
Revised: Table 2 and 3 show the learned Q by DCG and NL-CG.

Original: In line with our theoretical analysis, we can see that DCG learns a sub-optimal strategy of selecting Action A in the first step.
Revised: In line with our theoretical analysis, we can see that DCG learns a sub-optimal strategy of selecting Action A in the first step.

Original: By contrast, our method learns the accurate value of Action B in State A and get the optimal strategy.
Revised: By contrast, our method learns the accurate value of Action B in State A and get the optimal strategy.

Original: Furthermore, the Q values for State 2B learned by NL-CG is more accurate than those learned by DCG.
Revised: Furthermore, the Q values for State 2B learned by NL-CG is more accurate than those learned by DCG.

Original: These results demonstrate that NL-CG’s higher representational capacity allows it to accurately estimate the value function of this game whereas DCG cannot.
Revised: These results demonstrate that NL-CG’s higher representational capacity allows it to accurately estimate the value function of this game whereas DCG cannot.

Original: We also note that such an example is common for task with few actions because there are typically more equations than unknowns.
Revised: We also note that such an example is common for task with few actions because there are typically more equations than unknowns.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Experiments
Revised:  Experiments

Original: In this section, we conduct experiments to show the effectiveness of our method on complex tasks.
Revised: In this section, we conduct experiments to show the effectiveness of our method on complex tasks.

Original: We benchmark our method on the Multi-Agent COordination (MACO) benchmark [34], which covers various classic coordination tasks in the literature of multi-agent learning and increases their complexity to better evaluate the performance of different algorithms.
Revised: We benchmark our method on the Multi-Agent COordination (MACO) benchmark [33], which covers various classic coordination tasks in the literature of multi-agent learning and increases their complexity to better evaluate the performance of different algorithms.

Original: The MACO benchmark is characterized by a high demand on the sophistication of agent coordination.
Revised: The MACO benchmark is characterized by a high demand on the sophistication of agent coordination.

Original: For NL-CG, we use a mixing network that has one hidden layer with different widths.
Revised: For NL-CG, we use a mixing network that has one hidden layer with different widths.

Original: Detailed hyperparameter settings of our method can be found in Appendix A. For fair comparison, we run all our experiments withrandom seeds and show the meanperformance with [Table]
Revised: Detailed hyperparameter settings of our method can be found in Appendix ?? . For fair comparison, we run all our experiments withrandom seeds and show the mean

Original: In Fig.
Revised: In Fig.

Original: 2, we compare NL-CG against the state-of-the-art coordination graph learning method (DCG [4]) and fully decomposed methods (QMIX [22], DICG [13]).
Revised: 2, we compare NL-CG against the state-of-the-art coordination graph learning method (DCG [4]) and fully decomposed methods (QMIX [22], DICG [13]).

Original: For both
Revised: For both

Original: DCG and our method, we use the complete graphs for all experiments in the paper.
Revised: DCG and our method, we use the complete graphs for all experiments in the paper.

Original: For NL-CG, we (1) set the hidden width to 3 and enumerate all configurations and (2) set the hidden width to 4 and run our iterative method with n max = 4 .
Revised: For NL-CG, we (1) set the hidden width to 3 and enumerate all configurations and (2) set the hidden width to 4 and run our iterative method with n max = 4 .

Original: We stop iteration when n max slope configurations are visited.
Revised: We stop iteration when n max slope configurations are visited.

Original: The result shows that our algorithm can outperform conventional CGs significantly.
Revised: The result shows that our algorithm can outperform conventional CGs significantly.

Original: Moreover, our iterative optimization method has comparable performance with the enumeration method, showing its effectiveness.
Revised: Moreover, our iterative optimization method has comparable performance with the enumeration method, showing its effectiveness.

Original: QMIX struggles on these tasks, indicating that these tasks are beyond the representation capacity of a fully decomposed function.
Revised: QMIX struggles on these tasks, indicating that these tasks are beyond the representation capacity of a fully decomposed function.

Original: We further investigate the the influence of the mixing network’s width.
Revised: We further investigate the the influence of the mixing network’s width.

Original: Specifically, we test NLCG with a width of 2, 3, 4, and 10 and compare their performance in Fig.
Revised: Specifically, we test NLCG with a width of 2, 3, 4, and 10 and compare their performance in Fig.

Original: It can be observed that, generally, more units (10) in the hidden layer lead to better or at least equal performance than other configurations.
Revised: It can be observed that, generally, more units (10) in the hidden layer lead to better or at least equal performance than other configurations.

Original: This result is in line with our motivation: a powerful non-linear mixing network increases the capability of CGs.
Revised: This result is in line with our motivation: a powerful non-linear mixing network increases the capability of CGs.

Original: In Fig.
Revised: In Fig.

Original: we compare our enumerative and iterative methods.
Revised: we compare our enumerative and iterative methods.

Original: These two methods check the same number (16) of linear pieces, but the iterative method can use a hidden layer of 10 .
Revised: These two methods check the same number (16) of linear pieces, but the iterative method can use a hidden layer of 10 .

Original: As a result, the iterative method outperforms the enumerative method.
Revised: As a result, the iterative method outperforms the enumerative method.

Original: Additionally, in Fig. 5, we compare NL-CG against DCG [4], QMIX [22], and DICG [13] on
Revised: Additionally, in Fig. 5, we compare NL-CG against DCG [4], QMIX [22], and DICG [13] on

Original: Pursuit .
Revised: Pursuit .

Original: We find NL-CG generally outperforms previous methods.
Revised: We find NL-CG generally outperforms previous methods.

Original: 5.1 Analysis of the optimality and efficiency of the iterative optimization method
Revised: 5.1 Analysis of the optimality and efficiency of the iterative optimization method

Original: Although we have checked the performance of our iterative optimization method, learning curves can not fully reveal its optimality and efficiency.
Revised: Although we have checked the performance of our iterative optimization method, learning curves can not fully reveal its optimality and efficiency.

Original: In this section, we provide optimality and efficiency analyses by checking the action selection results and time costs in detail.
Revised: In this section, we provide optimality and efficiency analyses by checking the action selection results and time costs in detail.

Original: In Fig.
Revised: In Fig.

Original: 6-middle, we compare the Q tot value of our enumerative and iterative optimization methods.
Revised: 6-middle, we compare the Q tot value of our enumerative and iterative optimization methods.

Original: We can see that the iterative method is near-optimal after checking only 4 (embed=3) slope configurations: the Q-values of its actions are very close to those selected by enumeration and are better than those selected by DCG.
Revised: We can see that the iterative method is near-optimal after checking only 4 (embed=3) slope configurations: the Q-values of its actions are very close to those selected by enumeration and are better than those selected by DCG.

Original: In Fig.
Revised: In Fig.

Original: 6-left, we compare the ground truth Q estimates of DCG and NL-CG (embed=3, iterative, n max = 4 ).
Revised: 6-left, we compare the ground truth Q estimates of DCG and NL-CG (embed=3, iterative, n max = 4 ).

Original: The result shows that NL-CG learns a policy with higher value.
Revised: The result shows that NL-CG learns a policy with higher value.

Original: In Fig.
Revised: In Fig.

Original: 6-right, we compare the Q tot values estimated by NL-CG (embed=3, iterative, n max = 4 ) against ground truth Q estimates.
Revised: 6-right, we compare the Q tot values estimated by NL-CG (embed=3, iterative, n max = 4 ) against ground truth Q estimates.

Original: The estimation errors on all tested state-action pairs are less than 20%.
Revised: The estimation errors on all tested state-action pairs are less than 20%.

Original: In Fig.
Revised: In Fig.

Original: 7, we compare the time spent by enumerative and iterative optimization methods.
Revised: 7, we compare the time spent by enumerative and iterative optimization methods.

Original: It can be found that the iterative method saves 50% to 65% of running time.
Revised: It can be found that the iterative method saves 50% to 65% of running time.

Original: We can thus conclude that the iterative optimization method provides a good trade-off between complexity and optimality.
Revised: We can thus conclude that the iterative optimization method provides a good trade-off between complexity and optimality.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we extend coordination graphs beyond linear decomposition by introducing non-linear mixing networks.
Revised: In this paper, we extend coordination graphs beyond linear decomposition by introducing non-linear mixing networks.

Original: Experiments manifest its superior representation power on complex tasks that conventional CGs are not able to solve.
Revised: Experiments manifest its superior representation power on complex tasks that conventional CGs are not able to solve.

Original: An important research direction is the stability of non-linear
Revised: An important research direction is the stability of non-linear

Original: CGs and to get rid of the non-negative constraint on weights of the mixing network.
Revised: CGs and to get rid of the non-negative constraint on weights of the mixing network.

Original: The authors do not see obvious negative societal impacts of our method.Checklist For all authors..
Revised: The authors do not see obvious negative societal impacts of our method.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: A Hyperparameters and Infrastructure

Original: When comparing the performance of NL-CG against DCG [4] and QMIX [22] on the MACO benchmark, we adopted the following hyper-parameter settings.

Original: For all tasks, we used a discount factor γ = 0 .

Original: 99 and ϵ -greedy exploration, where ϵ was linearly decayed from 1 to 0 .

Original: within the first 50,000 training time steps.

Original: The replay buffer stored the last 5000 episodes, from which we uniformly sampled batches of size 32 for training.

Original: The target network was updated every 200 episodes.

Original: The learning rate of RMSprop was set to 5 × 10 − 4 , except for the Pursuit experiment, which used 1 × 10 − 2 for faster convergence.

Original: Every 10000 time steps we paused training and evaluated the model with 300 greedy test trajectories sampled with ϵ = 0 .

Original: Each agent processed its local action-observation history at each time step using a linear layer of 64 neurons, followed by a ReLU activation, a GRU of the same dimensionality, and finally a linear layer of | A | neurons.

Original: The output served as an individual feature vector and was fed into the utility and payoff function.

Original: The parameters of the utility and payoff function were shared amongst agents, who were identified by a one-hot encoded ID in the input.

Original: All message passing procedures in NLCG iterated for 4 rounds.

Original: The weights and bias of the mixing network in NL-CG were generated by a 2-layer hyper-network with a hidden layer of 64 ReLU neurons, except for the Aloha experiment, which uses one linear layer.

Original: Our implementation of NL-CG was based on the PyMARL * [23] framework.

Original: We used NVIDIA

Original: GeForce RTX 3090 GPUs for training and evaluation.

Original: B Weighted Max-Sum

Original: Algorithm 1 and Algorithm 2 find global and approximate greedy actions for a non-linear coordination graph, respectively.

Original: Both of these two algorithms rely on the weighted Max-Sum algorithm to find local optimal action on a linear piece.

Original: We present the weighted Max-Sum algorithm in Algorithm 3.

Original: C Representational Capability

Original: In Sec. 4, we use a matrix game to show the representational capability of non-linear coordination graphs.

Original: For conventional coordination graphs on State 2B of this game, since the value function should be permutation invariant, there are five unknowns q 1 = q i ( s 2 B , A ) , q 2 = q i ( s 2 B , B ) , q 3 = q ij ( s 2 B , AA ) , q 4 = q ij ( s 2 B , AB ) , and q 5 = q ij ( s 2 B , BB ) and five equations:

Original: [Equation]

Original: The augmented matrix of this system has a higher rank (4) than its coefficient matrix (3).

Original: Therefore, this linear system does not have a solution, which means that a conventional coordination graph cannot represent the accurate value function for this task.

Original: Algorithm

Original: /*Greedy action selection with k message passing for one linear piece of the mixing network f m .

Original: This algorithm is called by Algorithm 1 and 2, and the definition of inputs can be found there.*/

Original: Input: f V ∈ R |V|× A , f E ∈ R |E|× A × A , ( W V , W E

Original: ) ∈ R |V| + |E| , b ∈ R f V :=

Original: W V ◦ f V f E :=

Original: W E ◦ f E µ 0 , ¯ µ 0 :=

Original: 0 ∈ R

Original: |E|× A /*Initialize forward messages ( µ ) and backward messages ( ¯ µ ).*/

Original: q 0 :

Original: = f V /*Initial “Q-value”.*/ q max :=

Original: −∞ ; a max := (cid:2) arg max a ∈A i q 0 ai (cid:12)(cid:12) i ∈ V (cid:3) /*Initialize

Original: the best found solution.*/ for t ∈ { 1 , . . .

Original: , k } do /* k rounds of message passing.*/ for e =

Original: ( i, j ) ∈

Original: E do /*Update forward and backward messages.

Original: Subscripts of f , q , µ mean indexing.*/ µ te := max

Original: [Equation]

Original: (cid:8) ( q

Original: [Equation]

Original: + f

Original: [Equation]

Original: (cid:9) /*Forward: maximize sender.*/ ¯ µ te := max

Original: [Equation]

Original: (cid:8) (

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: (cid:9) /*Backward

Original: : maximize receiver.*/

Original: [Equation]

Original: tea /*Normalize forward messages to ensure converging.*/ ¯

Original: [Equation]

Original: ←

Original: [Equation]

Original: tea /*Normalize backward messages to ensure converging.*/ end for for i ∈ V do /*Update

Original: “Q-value” with messages.*/ q ti :=

Original: f V i + (cid:88)

Original: e =(

Original: · ,i ) ∈E µ te + (cid:88) e =( i, · ) ∈E ¯ µ te /*Utility plus incoming messages.*/ a ti := arg max a ∈A

Original: [-i { q tia } /*Select the greedy action of agent i .*/ end for q ′ ← (cid:80) |V| i =
-]
Revised: [-i { q tia } /*Select the greedy action of agent i .*/ end for q ′ ← (cid:80) |V| i =
-]

Original: f V a i

Original: + (cid:80) ( i,j ) ∈E f E a i a j + b /*Get the Q-value of the greedy action.*/ if q ′ >

Original: q max then { a max ← a t ; q max ← q ′ q ← q ti } /*Remember

Original: the best action.*/ end if end for return q max , a max ∈

Original: A 1 × . . .

Original: × A |V| , q /*Return the maximum Q value, the corresponding action, and utilities/payoffs.*/