Original: N EGOTIATING T EAM F ORMATION U SING D EEP R EINFORCEMENT L EARNING
Revised: N EGOTIATING T EAM F ORMATION U SING D EEP R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: When autonomous agents interact in the same environment, they must often cooperate to achieve their goals.
Revised: When autonomous agents interact in the same environment, they must often cooperate to achieve their goals.

Original: One way for agents to cooperate effectively is to form a team, make a binding agreement on a joint plan, and execute it.
Revised: One way for agents to cooperate effectively is to form a team, make a binding agreement on a joint plan, and execute it.

Original: However, when agents are self-interested, the gains from team formation must be allocated appropriately to incentivize agreement.
Revised: However, when agents are self-interested, the gains from team formation must be allocated appropriately to incentivize agreement.

Original: Various approaches for multi-agent negotiation have been proposed, but typically only work for particular negotiation protocols.
Revised: Various approaches for multi-agent negotiation have been proposed, but typically only work for particular negotiation protocols.

Original: More general methods usually require human input or domain-speciﬁc data, and so do not scale.
Revised: More general methods usually require human input or domain-speciﬁc data, and so do not scale.

Original: To address this, we propose a framework for training agents to negotiate and form teams using deep reinforcement learning.
Revised: To address this, we propose a framework for training agents to negotiate and form teams using deep reinforcement learning.

Original: Importantly, our method makes no assumptions about the speciﬁc negotiation protocol, and is instead completely experience driven.
Revised: Importantly, our method makes no assumptions about the speciﬁc negotiation protocol, and is instead completely experience driven.

Original: We evaluate our approach on both non-spatial and spatially extended team-formation negotiation environments, demonstrating that our agents beat hand-crafted bots and reach negotiation outcomes consistent with fair solutions predicted by cooperative game theory.
Revised: We evaluate our approach on both non-spatial and spatially extended team-formation negotiation environments, demonstrating that our agents beat hand-crafted bots and reach negotiation outcomes consistent with fair solutions predicted by cooperative game theory.

Original: Additionally, we investigate how the physical location of agents inﬂuences negotiation outcomes.
Revised: Additionally, we investigate how the physical location of agents inﬂuences negotiation outcomes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multiple agents inhabiting the same environment affect each other, and may gain by coordinating their actions.
Revised: Multiple agents inhabiting the same environment affect each other, and may gain by coordinating their actions.

Original: Indeed, many tasks are effectively intractable for any single agent, and so can only be solved by a team of collaborators.
Revised: Indeed, many tasks are effectively intractable for any single agent, and so can only be solved by a team of collaborators.

Original: Examples include search and rescue (Kitano et al., 1999), multirobot patrolling (Agmon et al., 2008), security (Tambe, 2011) and multiplayer ﬁrst-person video games (Jaderberg et al., 2018).
Revised: Examples include search and rescue (Kitano et al., 1999), multirobot patrolling (Agmon et al., 2008), security (Tambe, 2011) and multiplayer ﬁrst-person video games (Jaderberg et al., 2018).

Original: Despite the need to cooperate, stakeholders have different abilities and preferences which affect the chosen course of action.
Revised: Despite the need to cooperate, stakeholders have different abilities and preferences which affect the chosen course of action.

Original: Agents must therefore negotiate to form teams that are both fairly aligned with individual interests and capable of achieving the task at hand.
Revised: Agents must therefore negotiate to form teams that are both fairly aligned with individual interests and capable of achieving the task at hand.

Original: This problem can formalized as a team-formation negotiation task as follows (Kraus, 1997; Shehory & Kraus, 1998).
Revised: This problem can formalized as a team-formation negotiation task as follows (Kraus, 1997; Shehory & Kraus, 1998).

Original: By deﬁnition, no single agent can perform the task on their own, but there may be several teams of agents who are capable of doing so, and therefore each agent must decide who to collaborate with.
Revised: By deﬁnition, no single agent can perform the task on their own, but there may be several teams of agents who are capable of doing so, so each agent must decide who to collaborate with.

Original: The reward for accomplishing the task is awarded to the ﬁrst team that solves it.
Revised: The reward for accomplishing the task is awarded to the ﬁrst team that solves it.

Original: Hence, agents need to interact with one another to simultaneously form a team and agree on how to share the joint reward.
Revised: Hence, agents need to interact with one another to simultaneously form a team and agree on how to share the joint reward.

Original: To solve this abstract problem, one must provide a concrete environment in which agents can negotiate and reach an agreement. More speciﬁcally, we must specify a negotiation protocol that encodes the allowed negotiation actions and determines the agreement reached (Rosenschein & Zlotkin, 1994).
Revised: To solve this abstract problem, one must provide a concrete environment where agents can negotiate and reach an agreement; We must specify a negotiation protocol that encodes the allowed negotiation actions and determines the agreement reached (Rosenschein & Zlotkin, 1994).

Original: Team-formation negotiation tasks are natural objects of study in game theory.
Revised: Team-formation negotiation tasks are natural objects of study in game theory.

Original: More precisely, cooperative game theory focuses on interactions between agents who form teams and make enforceable agreements about outcomes (Brandenburger, 2007; Chalkiadakis et al., 2011).
Revised: More precisely, cooperative game theory focuses on interactions between agents who form teams and make enforceable agreements about outcomes (Brandenburger, 2007; Chalkiadakis et al., 2011).

Original: Weighted voting games are an archetypal problem, in which every agent has a weight and a team of agents is successful if the sum of the weights of its participants exceeds a ﬁxed threshold (Banzhaf III, 1964; Holler, 1982).
Revised: Weighted voting games are an archetypal problem, in which every agent has a weight and a team of agents is successful if the sum of the weights of its participants exceeds a ﬁxed threshold (Banzhaf III, 1964; Holler, 1982).

Original: Weighted voting games also offer a simple model of coalition formation in legislative bodies (Leech, 2002; Felsenthal et al., 1998).
Revised: Weighted voting games also offer a simple model of coalition formation in legislative bodies (Leech, 2002; Felsenthal et al., 1998).

Original: Cooperative game theory seeks to predict the agreements negotiated by agents in such settings, proposing several solution concepts.
Revised: Cooperative game theory seeks to predict the agreements negotiated by agents in such settings, proposing several solution concepts.

Original: Some solutions, such as the core and nucleolus (Schmeidler, 1969), have focused on identifying stable agreements.
Revised: Some solutions, such as the core and nucleolus (Schmeidler, 1969), have focused on identifying stable agreements.

Original: Other solutions have tried to identify fair agreements between agents, reﬂecting the strength of the  negotiation position of each agent, or measuring their relative ability to affect the outcome of the game.
Revised: Other solutions, known as power indices , have tried to measure the objective negotiation position of agents, quantifying their relative ability to affect the outcome of the game, or the fair share of the joint reward they should receive (Chalkiadakis et al., 2011).

Original: The most prominent of these is the Shapley value (Shapley, 1953b) which has been widely studied for weighted voting games (Shapley & Shubik, 1954; Strafﬁn, 1988).
Revised: The most prominent of these is the Shapley value (Shapley, 1953b) which has been widely studied for weighted voting games (Shapley & Shubik, 1954; Strafﬁn, 1988).

Original: In particular, it has been used to estimate political power (Leech, 2002; Felsenthal et al., 1998).
Revised: In particular, it has been used to estimate political power (Leech, 2002; Felsenthal et al., 1998).

Original: In Appendix A we provide a detailed motivating example, showing how the Shapley value fairly measures power in such settings.
Revised: In Appendix A we provide a detailed motivating example, showing how the Shapley value fairly measures power in such settings.

Original: There remains a pragmatic question for the design of multi-agent systems.
Revised: There remains a pragmatic question for the design of multi-agent systems.

Original: How should one construct a negotiating agent that maximizes the reward obtained?
Revised: How should one construct a negotiating agent that maximizes the reward obtained?

Original: Many researchers have borrowed ideas from cooperative game theory to hand-craft bots (Zlotkin & Rosenschein, 1989; Aknine et al., 2004;
Revised: Many researchers have borrowed ideas from cooperative game theory to hand-craft bots (Zlotkin & Rosenschein, 1989; Aknine et al., 2004;

Original: Ito et al., 2007), often requiring additional human data (Oliver, 1996; Lin & Kraus, 2010).
Revised: Ito et al., 2007), often requiring additional human data (Oliver, 1996; Lin & Kraus, 2010).

Original: Such bots are tailored to speciﬁc negotiation protocols, so modifying the protocol or switching to a different protocol requires manually re-writing the bot (Jennings et al., 2001; An et al., 2010).
Revised: Such bots are tailored to speciﬁc negotiation protocols, so modifying the protocol or switching to a different protocol requires manually re-writing the bot (Jennings et al., 2001; An et al., 2010).

Original: As a result, algorithms based purely on cooperative game theory are neither generally applicable nor scalable.
Revised: As a result, algorithms based purely on cooperative game theory are neither generally applicable nor scalable.

Original: Moreover, negotiation and team formation in the real world is signiﬁcantly more complex than in the game theoretic setting, for several reasons: (1) negotiation protocols can be arbitrarily complicated and are rarely ﬁxed; (2) enacting a negotiation requires a temporally extended policy; (3) the idiosyncrasies of the environment affect the negotiation mechanics; (4) Players must make decisions based on incomplete information about others’ policies.
Revised: Moreover, negotiation and team formation in the real world is signiﬁcantly more complex than in the game theoretic setting, for several reasons: (1) negotiation protocols can be arbitrarily complicated and are rarely ﬁxed; (2) enacting a negotiation requires a temporally extended policy; (3) the idiosyncrasies of the environment affect the negotiation mechanics; (4) Players must make decisions based on incomplete information about others’ policies.

Original: We propose multi-agent reinforcement learning as an alternative paradigm which may be applied to arbitrary negotiation protocols in complex environments.
Revised: We propose multi-agent reinforcement learning as an alternative paradigm which may be applied to arbitrary negotiation protocols in complex environments.

Original: Here, individual agents must learn how to solve team formation tasks based on their experiences interacting with others, rather than via hand-crafted algorithms.
Revised: Here, individual agents must learn how to solve team formation tasks based on their experiences interacting with others, rather than via hand-crafted algorithms.

Original: Our RL approach is automatically applicable to Markov games (Shapley, 1953a; Littman, 1994), which are temporally and spatially extended, similar to recent work in the non-cooperative case (Leibo et al., 2017; Lerer & Peysakhovich, 2017; Foerster et al., 2017).
Revised: Our RL approach is automatically applicable to Markov games (Shapley, 1953a; Littman, 1994), which are temporally and spatially extended, similar to recent work in the non-cooperative case (Leibo et al., 2017; Lerer & Peysakhovich, 2017; Foerster et al., 2017).

Revised: In contrast to earlier work on multi-agent RL in non-cooperative games, the key novelty of our work is comparing the behaviour of negotiating RL agents with solutions from cooperative game theory .

Original: Some previous work in multi-agent (deep) reinforcement learning for negotiation has cast the problem as one of communication, rather than team formation (e.g. Georgila & Traum (2011); Lewis et al.
Revised: Some previous work in multi-agent (deep) reinforcement learning for negotiation has cast the problem as one of communication, rather than team formation (e.g. Georgila & Traum (2011); Lewis et al.

Original: (2017); Cao et al.
Revised: (2017); Cao et al.

Original: (2018b)).
Revised: (2018b)).

Original: In particular, the environments considered involved only two agents, sidestepping the issue of coalition selection.
Revised: In particular, the environments considered involved only two agents, sidestepping the issue of coalition selection.

Original: Closer to our perspective is the work of Chalkiadakis & Boutilier (2004); Matthews et al.
Revised: Closer to our perspective is the work of Chalkiadakis & Boutilier (2004); Matthews et al.

Original: (2012), which propose a Bayesian reinforcement learning framework for team formation.
Revised: (2012), which propose a Bayesian reinforcement learning framework for team formation.

Original: However, they do not consider spatially extended environments and the computational cost of the Bayesian calculation is signiﬁcant.
Revised: However, they do not consider spatially extended environments and the computational cost of the Bayesian calculation is signiﬁcant.

Original: We evaluate our approach on a team formation negotiation task using a direct negotiation protocol, showing that agents trained via independent reinforcement learning outperform hand-crafted bots based on game-theoretic principles.
Revised: We evaluate our approach on a team formation negotiation task using a direct negotiation protocol, showing that agents trained via independent reinforcement learning outperform hand-crafted bots based on game-theoretic principles.

Original: We analyze the reward distribution, showing a high correspondence with the Shapley value solution from cooperative game theory.
Revised: We analyze the reward distribution, showing a high correspondence with the Shapley value solution from cooperative game theory.

Original: We show that the slight deviation is not due to lack of neural network capacity by training a similar-sized supervised model to predict the Shapley value.
Revised: We show that the slight deviation is not due to lack of neural network capacity by training a similar-sized supervised model to predict the Shapley value.

Original: We also introduce a more complicated spatial grid-world environment in which agents must move around to form teams.
Revised: We also introduce a more complicated spatial grid-world environment in which agents must move around to form teams.

Original: We show that the correspondence with the Shapley value persists in this case, and investigate how spatial perturbations inﬂuence agents’ rewards.
Revised: We show that the correspondence with the Shapley value persists in this case, and investigate how spatial perturbations inﬂuence agents’ rewards.

Original:  D EFINITIONS
Revised:  D EFINITIONS

Original: Cooperative Games We providekey deﬁnitions from cooperative game theory on which our analysis relies (for a full review see Osborne & Rubinstein (1994) and Chalkiadakis et al.
Revised: Cooperative Games We provide deﬁnitions from cooperative game theory that our analysis uses (for a full review see Osborne & Rubinstein (1994) and Chalkiadakis et al.

Original: A (transferable-utility) cooperative game consists of a set A of n agents, and a characteristic function v : P ( A ) → R mapping any subset C ⊆ A of agents to a real value, reﬂecting the total utility that these agents can achieve when working together.
Revised: A (transferableutility) cooperative game consists of a set A of n agents, and a characteristic function v : P ( A ) → R mapping any subset C ⊆ A of agents to a real value, reﬂecting the total utility that these agents can achieve when working together.

Original: We refer to a subset of agents C ⊆
Revised: We refer to a subset of agents C ⊆

Original: A as a team or coalition .
Revised: A as a team or coalition .

Original: A simple cooperative game has v take values in { 0 , 1 } , where v ( C )
Revised: A simple cooperative game has v take values in { 0 , 1 } , where v ( C )

Original: = 1 iff C can achieve the task, modelling the situation where only certain subsets are viable teams.
Revised: = 1 iff C can achieve the task, modelling the situation where only certain subsets are viable teams.

Original: In such games we refer to a team X with v ( X ) = 1 as a winning team , and a team Y with v ( Y ) = 0 as a losing team .
Revised: In such games we refer to a team X with v ( X ) = 1 as a winning team , and a team Y with v ( Y ) = 0 as a losing team .

Original: Given a winning team C and an agent a ∈ C , we say that a is pivotal in C if v ( C \ { a } ) = 0 , i.e. a ’s removal from C turns it from winning to losing.
Revised: Given a winning team C and an agent a ∈ C , we say that a is pivotal in C if v ( C \{ a } ) = 0 , i.e. a ’s removal from C turns it from winning to losing.

Original: An agent is pivotal in a permutation π of agents if they are pivotal in the set of agents occurring before them in the permutation union themselves.
Revised: An agent is pivotal in a permutation π of agents if they are pivotal in the set of agents occurring before them in the permutation union themselves.

Original: Formally, let S π ( i ) =
Revised: Formally, let S π ( i ) =

Original: { j | π ( j ) <
Revised: { j | π ( j ) <

Original: π ( i ) } .
Revised: π ( i ) } .

Original: Agent i iscalled pivotal in π if they are pivotal for the set S π ( i ) ∪ { i } .
Revised: Agent i is pivotal in π if they are pivotal for the set S π ( i ) ∪ { i } .

Original: Shapley Value
Revised: Shapley Value

Original: The Shapley value characterizes fair agreements in cooperative games (Shapley, 1953b).
Revised: The Shapley value characterizes fair agreements in cooperative games (Shapley, 1953b).

Original: It is the only solution concept fulﬁlling important fairness axioms, and is therefore an important quantity in cooperative games (Dubey, 1975; Dubey & Shapley, 1979; Strafﬁn, 1988).
Revised: It is the only solution concept fulﬁlling important fairness axioms, and is therefore an important quantity in cooperative games (Dubey, 1975; Dubey & Shapley, 1979; Strafﬁn, 1988).

Original: The Shapley value measures the proportion of all permutations in which an agent is pivotal, and is given by the vector φ ( v ) =
Revised: The Shapley value measures the proportion of all permutations in which an agent is pivotal, and is given by the vector φ ( v ) =

Original: ( φ 1 ( v ) , φ 2 ( v ) , . . .
Revised: ( φ 1 ( v ) , φ 2 ( v ) , . . .

Original: , φ n ( v ))
Revised: , φ n ( v ))

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Weighted voting games A weighted voting game [ w 1 , w 2 , . .
Revised: Weighted voting games A weighted voting game [ w 1 , w 2 , . .

Original: . , w n ; q ] is a simple cooperative game described by a vector of weights ( w 1 , w 2 , . . .
Revised: . , w n ; q ] is a simple cooperative game described by a vector of weights ( w 1 , w 2 , . . .

Original: , w n ) and a quota (threshold) q .
Revised: , w n ) and a quota (threshold) q .

Original: A coalition
Revised: A coalition

Original: C wins iff its total weight (the sum of the weight of its participants) meets or exceeds the quota.
Revised: C wins iff its total weight (the sum of the weight of its participants) meets or exceeds the quota.

Original: Formally v ( C ) =
Revised: Formally v ( C ) =

Original: 1 iff (cid:80) i ∈
Revised: 1 iff (cid:80) i ∈

Original: C w i ≥ q .
Revised: C w i ≥ q .

Original: By abuse of notation, we identify the game description with the characteristic function, writing v =
Revised: By abuse of notation, we identify the game description with the characteristic function, writing v =

Original: [ w 1 , w 2 , . .
Revised: [ w 1 , w 2 , . .

Original: . , w n ; q ] .
Revised: . , w n ; q ] .

Original: The Shapley value of weighted voting games has been used to analyze political power in legislative bodies, and for such settings it is referred to as the Shapley-Shubik power index (Shapley & Shubik, 1954).
Revised: The Shapley value of weighted voting games has been used to analyze political power in legislative bodies, and for such settings it is referred to as the Shapley-Shubik power index (Shapley & Shubik, 1954).

Original: Multi-agent Reinforcement Learning An n -player Markov game speciﬁes how the state of the an environment changes as the result of the joint actions of n individuals, as follows.
Revised: Multi-agent Reinforcement Learning An n -player Markov game speciﬁes how the state of the an environment changes as the result of the joint actions of n individuals.

Original: The game has a ﬁnite set of states S .
Revised: The game has a ﬁnite set of states S .

Original: The observation function O : S × { 1 , . . . , n } → R d speciﬁes each player’s d -dimensional view of the state space.
Revised: The observation function O : S × { 1 , . . . , n } → R d speciﬁes each player’s d -dimensional view of the state space.

Original: We write O i =
Revised: We write O i =

Original: { o i | s ∈ S , o i
Revised: { o i | s ∈ S , o i

Original: = O
Revised: = O

Original: ( s, i ) } to denote the observation space of player i .
Revised: ( s, i ) } to denote the observation space of player i .

Original: From each state, players take actions from the set
Revised: From each state, players take actions from the set

Original: A 1 , . . . ,
Revised: A 1 , . . . ,

Original: A n (one for each player).
Revised: A n (one per player).

Original: The state changes as a result of the joint action a 1 , . .
Revised: The state changes as a result of the joint action a 1 , . .

Original: . , a n ∈ A 1 , . . .
Revised: . , a n ∈ A 1 , . . .

Original: , A n , according to a stochastic transition function T : S ×
Revised: , A n , according to a stochastic transition function T : S ×

Original: A 1 × · · · × A n → ∆( S ) , where ∆( S ) denotes the set ofdiscrete probability distributions over S .
Revised: A 1 × · · · × A n → ∆( S ) , where ∆( S ) denotes the set of probability distributions over S .

Original: Each player receives an individual reward deﬁned as r i : S ×
Revised: Each player receives an individual reward deﬁned as r i : S ×

Original: A 1 × · · · × A n → R for player i .
Revised: A 1 × · · · × A n → R for player i .

Original: In reinforcement learning each agent independently learns, through its own experience of the environment, a behavior policy π i : O i → ∆( A i ) (denoted π ( a i | o i ) )
Revised: In reinforcement learning each agent independently learns, through its own experience, a behavior policy π i : O i → ∆( A i ) (denoted π ( a i | o i ) )

Original: based on itsown observation o i and reward r i .
Revised: based on its observation o i and reward r i .

Original: Each agent’s goal is to maximize a long term γ -discounted payoff (Sutton & Barto, 2018).
Revised: Each agent’s goal is to maximize a long term γ -discounted payoff (Sutton & Barto, 2018).

Original: Since agents are independent,  the learningprocess iscompletely decentralized (Bernstein et al., 2002).
Revised: Agents are independent, so the learning is decentralized (Bernstein et al., 2002).

Original:  G AME S ETTING 3. O VERVIEW
Revised:  G AME S ETTING 3. O VERVIEW

Original: We propose a method for training agents to negotiate team formation under a diverse set of negotiation protocols.
Revised: We propose a method for training agents to negotiate team formation under a diverse set of negotiation protocols.

Original: In this setting, many different combinations of agents can successfully perform a task.
Revised: In this setting, many different combinations of agents can successfully perform a task.

Original: This is captured by an underlying cooperative game , given by a characteristic function v : P ( A ) → { 0 , 1 } (which maps viable teams C ⊆
Revised: This is captured by an underlying cooperative game , given by a characteristic function v : P ( A ) → { 0 , 1 } (which maps viable teams C ⊆

Original: A to the value 1, and non-viable teams to 0).
Revised: A to the value 1, and non-viable teams to 0).

Original: When a viable team is formed, it obtains a reward r , to be shared between the team’s individual members.
Revised: When a viable team is formed, it obtains a reward r , to be shared between the team’s individual members.

Original: The outcome of the team formation process is an agreement between agents of a viable team regarding how they share the reward r (with each agent trying to maximize its share).
Revised: The outcome of the team formation process is an agreement between agents of a viable team regarding how they share the reward r (with each agent trying to maximize its share).

Original: Cooperative game theory can characterize how agents would share the joint reward, by applying solution concepts such as the Shapley value.
Revised: Cooperative game theory can characterize how agents would share the joint reward, by applying solution concepts such as the Shapley value.

Original: However, cooperative game theory abstracts away the mechanism by which agents negotiate.
Revised: However, cooperative game theory abstracts away the mechanism by which agents negotiate.

Original: To allow agents to interact, form teams and reach an agreement regarding sharing the reward, one must use a negotiation protocol , forming an environment with speciﬁc rules governing how agents interact; this environment consists of the actions agents can take, and their semantics, determining which team is formed, and how the reward is shared.
Revised: To allow agents to interact, form teams and reach an agreement regarding sharing the reward, one must use a negotiation protocol , forming an environment with speciﬁc rules governing how agents interact; this environment consists of the actions agents can take, and their semantics, determining which team is formed, and how the reward is shared.

Original: We examine two simple negotiation protocols, a non-spatial environment where agents take turns making offers and accepting or declining them, and a spatial environment where agents control their demanded share of the reward in a grid-world setting.
Revised: We examine two simple negotiation protocols, a non-spatial environment where agents take turns making offers and accepting or declining them, and a spatial environment where agents control their demanded share of the reward in a grid-world setting.

Original: Overlaying the underlying cooperative game with a speciﬁc negotiation protocol yields a negotiation environment ; this is a Markov game, which may be analyzed by non-cooperative game theory, identifying the equilibrium strategies of agents.
Revised: Overlaying the underlying cooperative game with a speciﬁc negotiation protocol yields a negotiation environment ; this is a Markov game, which may be analyzed by non-cooperative game theory, identifying the equilibrium strategies of agents.

Original: However, solving even the simplest such games is computationally infeasible: even the restricted case of an unrepeated two-player general-sum game is computationally hard to solve, being PPADcomplete (Chen & Deng, 2006)
Revised: However, solving even the simplest such games is computationally infeasible: even the restricted case of an unrepeated two-player general-sum game is computationally hard to solve, being PPADcomplete (Chen & Deng, 2006)

Original: (see Appendix C).
Revised: (see Appendix C).

Original: Instead, we propose training independent RL agents in the negotiation environment.
Revised: Instead, we propose training independent RL agents in the negotiation environment.

Original: While our methodology can be applied to any cooperative game, our experiments are based on a weighted voting game as the underlying cooperative game.
Revised: While our methodology can be applied to any cooperative game, our experiments are based on a weighted voting game as the underlying cooperative game.

Original: We 
Revised: We examine the relation between the negotiation outcome that RL agents arrive at and the cooperative game theoretic solution, as is illustrated in Figure 1.

Original: [Figure]
Revised: [Figure]

Original: 3. N EGOTIATION E NVIRONMENTS
Revised: 3. N EGOTIATION E NVIRONMENTS

Original: Our negotiation environments use a weighted voting game [ w 1 , . .
Revised: Our negotiation environments use a weighted voting game [ w 1 , . .

Original: . , w n ; q ] as the underlying game, offering a total reward r ∈ N .
Revised: . , w n ; q ] as the underlying game, offering a total reward r ∈ N .

Original: Each agent i is assigned a weight w i ∈ R at the beginning of each episode, and the goal of the agents is to construct a team C whose total weight (cid:80) i ∈
Revised: Each agent i is assigned a weight w i ∈ R at the beginning of each episode, and the goal of the agents is to construct a team C whose total weight (cid:80) i ∈

Original: C w i exceeds a ﬁxed quota q .
Revised: C w i exceeds a ﬁxed quota q .

Original: If a successful team forms, the entire team is assigned the ﬁxed reward r .
Revised: If a successful team forms, the entire team is assigned the ﬁxed reward r .

Original: The agents comprising the team must decide how to allocate this reward amongst themselves by agreeing on shares { r i }
Revised: The agents comprising the team must decide how to allocate this reward amongst themselves by agreeing on shares { r i }

Original: i ∈ C such that (cid:80)
Revised: i ∈ C such that (cid:80)

Original: i ∈
Revised: i ∈

Original: C r i = r , with r i ∈ N 0 .
Revised: C r i = r , with r i ∈ N 0 .

Original: We say a team C ⊆
Revised: We say a team C ⊆

Original: A is viable if (cid:80) i ∈
Revised: A is viable if (cid:80) i ∈

Original: C w i ≥ q .
Revised: C w i ≥ q .

Original: Though there are many viable teams, not all teams are viable (depending on the weights).
Revised: Though there are many viable teams, not all teams are viable (depending on the weights).

Original: Only one viable team is chosen, and non-members all get a reward of 0 .
Revised: Only one viable team is chosen, and non-members all get a reward of 0 .

Original: In such settings, agents face opportunities to reach agreements for forming teams, along with their share of the gains.
Revised: In such settings, agents face opportunities to reach agreements for forming teams, along with their share of the gains.

Original: If they do not agree, they stand the risk that some other team would form without them (in which case they get no reward); on the other hand, if they agree to a low share of the reward, they miss out on the opportunity to reach an agreement with others giving them a potentially higher share.
Revised: If they do not agree, they stand the risk that some other team would form without them (in which case they get no reward); on the other hand, if they agree to a low share of the reward, they miss out on the opportunity to reach an agreement with others giving them a potentially higher share.

Original: Our two negotiation environments have an identical underlying game, but employ different protocols for negotiating over the formed team and reward allocation.
Revised: Our two negotiation environments have an identical underlying game, but employ different protocols for negotiating over the formed team and reward allocation.

Original: The non-spatial environment uses direct proposals regarding formed teams and reward allocations, while our spatial environment has a similar interpretation to that of recent work applying multi-agent deep reinforcement learning to non-cooperative game theory settings like spatially extended seqential social dilemmas (e.g. Leibo et al.
Revised: The non-spatial environment uses direct proposals regarding formed teams and reward allocations, while our spatial environment has a similar interpretation to that of recent work applying multi-agent deep reinforcement learning to non-cooperative game theory settings like spatially extended seqential social dilemmas (e.g. Leibo et al.

Original: ; Perolat et al.
Revised: ; Perolat et al.

Original: ; Hughes et al.
Revised: ; Hughes et al.

Original: 3.2. P ROPOSE -A CCEPT (N ON -S PATIAL )
Revised: 3.2. P ROPOSE -A CCEPT (N ON -S PATIAL )

Original: In the Propose-Accept environment, agents take turns in proposing an agreement and accepting or declining such proposals.
Revised: In the Propose-Accept environment, agents take turns in proposing an agreement and accepting or declining such proposals.

Original: The underlying game, consisting of the weights and threshold, is public knowledge, so all agents are aware of v = [ w 1 , . . .
Revised: The underlying game, consisting of the weights and threshold, is public knowledge, so all agents are aware of v = [ w 1 , . . .

Original: , w n ; q ] chosen for the episode.
Revised: , w n ; q ] chosen for the episode.

Original: Each turn within the episode, an agent is chosen uniformly at random to be the proposer .
Revised: Each turn within the episode, an agent is chosen uniformly at random to be the proposer .

Original: The proposer chooses a viable team and an integer allocation of the total reward between the agents.
Revised: The proposer chooses a viable team and an integer allocation of the total reward between the agents.

Original: The set of actions for the proposer agent consists of all n-tuples ( r 1 , r 2 , . . .
Revised: The set of actions for the proposer agent consists of all n-tuples ( r 1 , r 2 , . . .

Original: , r n ) , where r i ∈ N 0 and (cid:80) ni
Revised: , r n ) , where r i ∈ N 0 and (cid:80) ni

Original: =1 r i = r .
Revised: =1 r i = r .

Original: By convention, the selected team C consists of the agents with non-zero reward under this allocation; that is to say C
Revised: By convention, the selected team C consists of the agents with non-zero reward under this allocation; that is to say C

Original: = { i | r i > 0 } .
Revised: = { i | r i > 0 } .

Original: We refer to the agents in the team chosen by the proposer as the proposees .
Revised: We refer to the agents in the team chosen by the proposer as the proposees .

Original: Once the proposer has made a proposal, every proposee has to either accept or decline the proposal.
Revised: Once the proposer has made a proposal, every proposee has to either accept or decline the proposal.

Original: If all proposees accept, the episode terminates, with the rewards being the ones in the proposed allocation; if one or more of the proposees decline, the entire proposal is declined.
Revised: If all proposees accept, the episode terminates, with the rewards being the ones in the proposed allocation; if one or more of the proposees decline, the entire proposal is declined.

Original: When a proposal is declined, with a ﬁxed probability p a new round begins (with another proposer chosen uniformly at random), and with probability 1 − p the game terminates with a reward of zero for all agents, in which case we say the episode terminated with agents failing to reach an agreement .
Revised: When a proposal is declined, with a ﬁxed probability p a new round begins (with another proposer chosen uniformly at random), and with probability 1 − p the game terminates with a reward of zero for all agents, in which case we say the episode terminated with agents failing to reach an agreement .

Original: All proposals consist of a viable team and an allocation ( r 1 , . . .
Revised: All proposals consist of a viable team and an allocation ( r 1 , . . .

Original: , r n ) such that (cid:80) ni
Revised: , r n ) such that (cid:80) ni

Original: =1 r i = r , so the total reward all the agents obtain in each episode, is either exactly r (when some proposal is accepted), or exactly zero (when agents fail to reach an agreement).
Revised: =1 r i = r , so the total reward all the agents obtain in each episode, is either exactly r (when some proposal is accepted), or exactly zero (when agents fail to reach an agreement).

Original: Interactions in the Propose- Accept environment can be viewed as a non-cooperative game, however solving this game for the equilibrium behavior is intractable (see Appendix C for details).
Revised: Interactions in the ProposeAccept environment can be viewed as a non-cooperative game, however solving this game for the equilibrium behavior is intractable (see Appendix C for details).

Original: 3.2. T EAM P ATCHES (S PATIAL )
Revised: 3.2. T EAM P ATCHES (S PATIAL )

Original: We construct a spatial negotiation environment, based on the same underlying weighted voting game as in the Propose-Accept environment, called the Team Patches environment (shown in Figure 2).
Revised: We construct a spatial negotiation environment, based on the same underlying weighted voting game as in the Propose-Accept environment, called the Team Patches environment (shown in Figure 2).

Original: This is a 15 × 15 grid-world that agent can navigate, including several colored rectangular areas called patches .
Revised: This is a 15 × 15 grid-world that agent can navigate, including several colored rectangular areas called patches .

Original: Agents can form a team with other agents occupying the same patch, demanding a share of the available reward to be a part of the team.
Revised: Agents can form a team with other agents occupying the same patch, demanding a share of the available reward to be a part of the team.

Original: Similar to Propose-Accept, the underlying game (i.e. the agent weights and the quota) are fully observable by all agents.
Revised: Similar to Propose-Accept, the underlying game (i.e. the agent weights and the quota) are fully observable by all agents.

Original: Additionally, as the environment is spatial, agents also observe their surrounding area (see the Appendix for details).
Revised: Additionally, as the environment is spatial, agents also observe their surrounding area (see the Appendix for details).

Original: At the start of an episode, agents are randomly initialized in the center of the map.
Revised: At the start of an episode, agents are randomly initialized in the center of the map.

Original: Each step, they can move around the environment (forwards, backwards, left, right), rotate (left, right), and set their demand ( d i ∈ { 1 , 2 . . .
Revised: Each step, they can move around the environment (forwards, backwards, left, right), rotate (left, right), and set their demand ( d i ∈ { 1 , 2 . . .

Original: , r } ), indicating the minimal share of the reward they are willing to take to join the team.
Revised: , r } ), indicating the minimal share of the reward they are willing to take to join the team.

Original: To form teams, agents must move into patches, and we refer to all agents occupying the same patch as the patch’s team.
Revised: To form teams, agents must move into patches, and we refer to all agents occupying the same patch as the patch’s team.

Original: The team T j in patch j is viable if the total weight of the agents in T j is greater than the threshold ( (cid:80) i ∈ T
Revised: The team T j in patch j is viable if the total weight of the agents in T j is greater than the threshold ( (cid:80) i ∈ T

Original: j w i ≥ q ).
Revised: j w i ≥ q ).

Original: The demands of the patch’s team are valid if the total demanded is less than the totalreward available ( (cid:80) i ∈ T j d i ≤ r ).
Revised: The demands of the patch’s team are valid if the total demanded is less than the total available ( (cid:80) i ∈ T j d i ≤ r ).

Original: An agreement is reached once there is a patch whose team is both viable and with a valid reward allocation.
Revised: An agreement is reached once there is a patch whose team is both viable and with a valid reward allocation.

Original: We show an exampleof this in Figure 2(b).
Revised: We show an example in Figure 2(b).

Original: An episode terminates when an agreement is reached on one of the patches j , giving each agent i in the team
Revised: An episode terminates when an agreement is reached on one of the patches j , giving each agent i in the team

Original: T j their demanded reward ( d i ).
Revised: T j their demanded reward ( d i ).

Original: Unlike the ProposeAccept environment, the agents don’t necessarily use up all the reward available, since (cid:80) i d i < r is allowed.
Revised: Unlike the Propose-Accept environment, the agents don’t necessarily use up all the reward available, since (cid:80) i d i < r is allowed.

Original: If the agents fail to reach an agreement after 100 steps, the episode is terminated, and all agents receive 0 reward. 
Revised: If the agents fail to reach an agreement after 100 steps, the episode is terminated, and all agents receive 0 reward. [Figure]

Original: AgentDemands  =
Revised: AgentDemands [Figure] =

Original: ====(
Revised: = ===(

Original: a) Environment.
Revised: a) Environment.

Original: AgentDemands =
Revised: AgentDemands =

Original: == == [Figure] formation.
Revised: ====(b) Example team formation.

Original: 3. L EARNING A GENTS
Revised: 3. L EARNING A GENTS

Original: For Propose-Accept, each agent independently learns a policy usingthe SARSA( λ )algorithm (A. Rummery & Niranjan, 1994) with λ = 0 .
Revised: For Propose-Accept, each agent independently learns a policy using SARSA( λ ) (A. Rummery & Niranjan, 1994) with λ = 0 .

Original: We use a function approximator to learn the Q function, reducing the intractable state space to manageable number of features.
Revised: We use a function approximator to learn the Q function.

Original: The weights of the network are trained with the the Adam optimizer (Kingma & Ba, 2014) to minimize theerror associated with the online temporal difference.
Revised: We train using the Adam optimizer (Kingma & Ba, 2014) to minimize the online temporal difference error.

Original: The function approximator is a multi-layer perceptron with 3 hidden layers, each of size 64 .
Revised: The function approximator is a multi-layer perceptron with 3 hidden layers, each of size 64 .

Original: For Team Patches, we use an advantage actor-critic algorithm (Mnih et al., 2016) with the V-trace correction (Espeholt et al., 2018), allowing us to learn on experience from 16 parallel copies of the environment using an on-policy algorithm.
Revised: For Team Patches, we use an advantage actor-critic algorithm (Mnih et al., 2016) with the V-trace correction (Espeholt et al., 2018), learning from 16 parallel copies of the environment with an on-policy algorithm.

Original: Each agent is trained independently according to their own experience of states and actions.

Original: We use the Adam optimizer (Kingma & Ba, 2014) to adjust the weights of a neural network.

Original: The neural network uses a convolutional layer with 6 channels, kernel size 3 and stride 1 , followed by a multi-layer perspective with 2 hidden layers of size 32 .
Revised: The neural network uses a convolutional layer with 6 channels, kernel size 3 and stride 1 , followed by a multi-layer perspective with 2 hidden layers of size 32 .

Original: The policy and value function heads are linear layers.
Revised: The policy and value function heads are linear layers.

Original: Ourtraining setup is analogous tothat of Jaderberg et al.
Revised: Our setup is analogous to Jaderberg et al.

Original: (2018), butwith the population size being the number of players, and using no evolution.
Revised: (2018), but using no evolution.

Original:  E XPERIMENTS4. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS

Original: Our experiments are based on a distribution D over underlying weighted voting games.
Revised: Our experiments are based on a distribution D over underlying weighted voting games.

Original: We sample k = 20 weighted voting games, ( v 1 , . . , v k ) , and refer to them as the experiment’s boards ; each board consists of its weights and threshold ( v i =
Revised: We sample games from this distribution, and refer to them as the experiment’s boards ; each board consists of its weights and threshold ( v i =

Original: [ w i 1 , . . .
Revised: [ w i 1 , . . .

Original: , w in ; q i ]) .
Revised: , w in ; q i ]) .

Original: Each such board has n =agents, a threshold of q = 15 , and weights sampled from a Gaussian distribution w i ∼ N (6 , 1) .
Revised: Each such board has n = 5 agents, a threshold of q = 15 , and weights sampled from a Gaussian distribution w i ∼ N (6 , 1) , and exclude boards where all players have identical power.

Original: We  train  and evaluate our agents across all k = 20 boards, requiring them to learn a general strategy for negotiating under different weights and players for each weight.
Revised: We partition the boards sampled from the distribution to to a train set and a test set . Agents are trained for 500 , 000 games using train set boards and are evaluated on the test set, requiring them to learn a general strategy for negotiating under different weights, and to be have to generalize to negotiation situations they have not encountered during training. We use a train set of 150 unique boards and a test set of k = 50 unique boards.

Original: 4. E XPERIMENT 1: C OMPARISON WITH H AND -C RAFTED B OTS
Revised: 4. E XPERIMENT 1: C OMPARISON WITH H AND -C RAFTED B OTS

Original: We compare the negotiation performance of RL agents trained in our framework with hand-crafted bots.
Revised: We compare the negotiation performance of RL agents trained in our framework with hand-crafted bots.

Original: While hand-crafted bots can form excellent negotiators, they can only be used for a speciﬁc negotiation protocol.
Revised: While hand-crafted bots can form excellent negotiators, they can only be used for a speciﬁc negotiation protocol.

Original: For simplicity, we examine the Propose-Accept protocol where the bot faces two kinds of decisions: (1) what offer to put in as a proposer, and (2) which offers to accept as a proposee.
Revised: For simplicity, we examine the Propose-Accept protocol where the bot faces two kinds of decisions: (1) what offer to put in as a proposer, and (2) which offers to accept as a proposee.

Original: As our baseline, we use a weight-proportional bot.
Revised: As our baseline, we use a weight-proportional bot.

Original: Given a proposal, the bot uses the proportion of its weight in the proposed team as a “target” share.
Revised: Given a proposal, the bot uses the proportion of its weight in the proposed team as a “target” share.

Original: The more that is proposed beyond this target, the more likely it is to accept.
Revised: The more that is proposed beyond this target, the more likely it is to accept.

Original: Bots with a similar design have shown good negotiation performance against both other bots and people (Lin et al., 2008; Lin & Kraus, 2010; Baarslag et al., 2012; Mash et al., 2017).
Revised: Bots with a similar design have shown good negotiation performance against both other bots and people (Lin et al., 2008; Lin & Kraus, 2010; Baarslag et al., 2012; Mash et al., 2017).

Original: As a proposer, bot i chooses a viable team C (cid:51) i with the weightproportional allocation p
Revised: As a proposer, bot i chooses a viable team C (cid:51) i with the weight-proportional allocation p

Original: i
Revised: i

Original: = rw i /
Revised: = rw i /

Original: ( (cid:80) i ∈
Revised: ( (cid:80) i ∈

Original: C w i ) .
Revised: C w i ) .

Original: As a proposee, bot i computes its share of the same target allocation p
Revised: As a proposee, bot i computes its share of the same target allocation p

Original: i and compares it with the amount r i offered to it, and accepts with probability σ ( c ( r i − p i )) where σ is the logistic function (see the Appendix for full details of the bot).
Revised: i and compares it with the amount r i offered to it, and accepts with probability σ ( c ( r i − p i )) where σ is the logistic function (see the Appendix for full details of the bot).

Revised: As a more sophisticated baseline, we use a Shapley-proportional bot, which follows the same design as the weight-proportional bot, except it sets the target allocation to be the one proportional to the Shapley values rather than the weights, i.e. p i = rφ

Revised: i / ( (cid:80) i ∈

Revised: C φ i ) (where φ i denotes the Shapley value of agent i ).

Revised: We note that as computing the Shapley values is an NP-hard problem (Elkind et al., 2009) this method is tractable for 5 agents, but does not scale to games with many agents.

Original: We create two agent teams (called a team pair ), one consisting of 5 RL agents (all-RL group), and one consisting of 4 RL agents and one bot (bot group).
Revised: We create two agent teams (called a team pair ), one with of 5 RL agents (all-RL group), and one with of 4 RL agents and one bot (bot group).

Original: Each group is co-trained over random boards from the distribution, and evaluated for 500 episodes on an evaluation board.
Revised: Each group is co-trained over training set boards, and evaluated for 500 episodes on an evaluation board.

Original: We compare the amount won by the bot and RL agent with the same weight in the board.
Revised: We compare the amount won by the bot and RL agent with the same weight in the board.

Original: We repeat the analysis over our 20 evaluation boards, creating 200 team pairs for eachevaluation board.
Revised: We repeat the analysis for all evaluation boards, creating 200 team pairs for each board.

Original: In the all-RL group, each agent makes on average 0 . 

Revised: In the all-RL group, each agent makes on average 0 .

Original: · r , whereas in the one-bot group the bot makes on average 0 . 
 
Revised: · r , whereas in the one-bot group, a weight-proportional bot makes on average 0 . 
 · r and a Shapley-proportional bot makes on average 0 .

Original:  · r .
Revised: 185 · r .

Original: The difference isstatistically signiﬁcant at the p < 0 .
Revised: The difference is signiﬁcant at the p < 0 .

Original: 005 level, using a Mann-WhitneyWilcoxon test.
Revised: 005 level, using a MannWhitney-Wilcoxon test.

Original: Additionally, against a random bot that selects valid actions uniformly we geteven more signiﬁcant results ( p < 0 . 001 ).
Revised: Also, against a random bot that selects valid actions uniformly we get more signiﬁcant results ( p < 0 . 001 ).

Original: These results indicate that RL-trained agents can outperform sensible hand-crafted bots, without requiring any tailoring for a speciﬁc negotiation protocol.
Revised: One may construct more sophisticated bots or tune their parameters to improve performance, but these results indicate that RL agents can outperform sensible handcrafted bots.

Revised: The above results are robust to the hyper-parameter choices for our RL agents; RL agents outperform the bots, at the same statistical signiﬁcance, even when perturbing the learning rate, hidden layer sizes and the λ parameters by ± 25% .

Original: 4. E XPERIMENT 2: C ONSISTENCY WITH G AME -T HEORETIC P REDICTIONS
Revised: 4. E XPERIMENT 2: C ONSISTENCY WITH G AME -T HEORETIC P REDICTIONS

Original: In this experiment, we investigate whether our trained agents negotiate in a way that is consistent with solution concepts from cooperative game theory, focusing on the Shapley value.
Revised: In this experiment, we investigate whether our trained agents negotiate in a way that is consistent with solution concepts from cooperative game theory, focusing on the Shapley value.

Original: As the Shapley value was proposed as a “fair” solution, one can view this as studying whether independent-RL agents are likely share the joint gains from cooperation in a fair manner.
Revised: As the Shapley value was proposed as a “fair” solution, one can view this as studying whether independent-RL agents are likely share the joint gains from cooperation in a fair manner.

Original: To investigate the negotiation policies of our RL agents, we train and evaluate them on our two environments using the same set of k = 20 boards.
Revised: To investigate the negotiation policies of our RL agents, we train and evaluate them on our two environments using the same set of k = 20 boards.

Original: We denote the average amounts won by weight i on board j as s ji , and refer to these as empirical agent rewards .
Revised: We denote the average amounts won by weight i on board j as s ji , and refer to these as empirical agent rewards .

Original: Given the game v j we can compute the Shapley values, denoted as φ ( v j ) =
Revised: Given the game v j we can compute the Shapley values, denoted as φ ( v j ) =

Original: ( φ j 1 , . ..
Revised: ( φ j 1 , . .

Original:  , φ jn ) .
Revised: . , φ j n ) .

Original: We view φ ji as the game theoretic prediction of the fair share the agent with weight w ji should get in a negotiation in the game v j .
Revised: We view φ j i as the game theoretic prediction of the fair share the agent with weight w ji should get in a negotiation in the game v j .

Original: We apply the analysis to each of our k = 20 boards and obtain k · n pairs { ( φ ji , s ji ) } i ∈ [ n ] ,j ∈ [ k ] of Shapley value predictions and empirical agent rewards.
Revised: We apply the analysis to each of our k = 20 boards and obtain k · n pairs { ( φ ji , s ji ) } i ∈ [ n ] ,j ∈ [ k ] of Shapley value predictions and empirical agent rewards.

Original: In Figure 3, we scatter and density-plot these pairs.
Revised: In Figure 3, we scatter and density-plot these pairs.

Revised: 0.3 Predicted (Shapley value) 0.

Original: [Figure]
Revised: E m p i r i c a l ( R L a g e n t s ) 0.

Revised: 0.3 Predicted (Shapley value) 0.

Revised: E m p

Revised: i r i c a l

Revised: ( R

Revised: L a g e n t s ) (a) Propose-Accept.

Revised: 0.3 Predicted (Shapley value) 0.

Revised: E m p

Revised: i r i c a l ( R L a g e n t s )

Revised: y=0.33 x + 0.

Revised: 0.3 Predicted (Shapley value) 0.

Original:  Our results indicate a strong correlation between the Shapley value based predictions and the empirical gains of our independent RL trained negotiator agents.
Revised: E m p i r i c a l ( R L a g e n t s ) (b) Team Patches. Our results indicate a strong correlation between the Shapley value based predictions and the empirical gains of our independent RL trained negotiator agents.

Original: We see that the majority of the mass occurs in close proximity to the line y
Revised: We see that the majority of the mass occurs in close proximity to the line y

Original: =
Revised: =

Original: x , indicating a good ﬁt between the game theoretic prediction and the point to which our RL agents converge.
Revised: x , indicating a good ﬁt between the game theoretic prediction and the point to which our RL agents converge.

Original: RL agents exhibit a stronger deviation from the game theoretic “fair” reward on the boards where there are players with signiﬁcantly more power or signiﬁcantly less power than others, i.e. boards where there is a high inequality in Shapley values.
Revised: RL agents exhibit a stronger deviation from the game theoretic “fair” reward on the boards where there are players with signiﬁcantly more power or signiﬁcantly less power than others, i.e. boards where there is a high inequality in Shapley values.

Original: This deviation is marginally stronger in our spatial Team Patches environment.

Original: While for these cases the high-Shapley weights do empirically get a higher share of the rewards than low-Shapley weights, the gap in empirical reward is lower than the one predicted by game theory. In other words, “strong” players (according to the game theoretic prediction) do get a higher share than “weak” players, but not to the extent predicted; for these rare boards, the empirical allocation of reward achieved by the 
Revised: In these boards, “strong” players (according to the Shapley value) do get a higher share than “weak” players, but not to the extent predicted; for these rare boards, the empirical allocation of reward achieved by the RL-agents is more equal than predicted by the Shapley value. Further evaluation in

Original: RL-agents is more equal than predicted by the Shapley value.
Revised: Appendix E shows that having a lower weight variance in the board distribution leads to a higher correspondence between the Shapley value and the empirical gains of RL agents.

Original: In summary, our independent RL negotiation agents empirically achieve rewards that are consistent with the outcomes predicted by the Shapley value from cooperative game theory.
Revised: In summary, our independent RL negotiation agents empirically achieve rewards that are consistent with the outcomes predicted by the Shapley value from cooperative game theory.

Original: Notably, this consistency extends to our spatial negotiation environment even with its different negotiation protocol and spatial interactions, with deviations occurring on boards where players have signiﬁcantly more or less power than others.
Revised: Notably, this consistency extends to our spatial negotiation environment even with its different negotiation protocol and spatial interactions.

Original: In Section 4.5 we investigate potential reasons for these variations.
Revised: In Section 4.4 we investigate potential reasons for these variations.

Original: 4. E XPERIMENT 3: I NFLUENCE OF SPATIAL PERTURBATIONS
Revised: 4. E XPERIMENT 3: I NFLUENCE OF SPATIAL PERTURBATIONS

Original: As the Team Patches environment is a spatial one, we can examine how spatial aspects, that are abstracted away in the underlying cooperative game, affect negotiation outcomes.
Revised: As the Team Patches environment is a spatial one, we can examine how spatial aspects, that are abstracted away in the underlying cooperative game, affect negotiation outcomes.

Original: How does changing the spatial structure of the environment inﬂuence the “power” of agents?
Revised: How does changing the spatial structure of the environment inﬂuence the “power” of agents?

Original: Can we reduce the share of the total reward received by the “most powerful” agent (with the highest weight)?
Revised: Can we reduce the share of the total reward received by the “most powerful” agent (with the highest weight)?

Original: We consider two patches and vary the starting position of the agent with the highest weight.
Revised: We consider two patches and vary the starting position of the maximal weight agent.

Original: Specifically, we measure how the agent’s share of the total reward changes as weincrementally change its starting position from directly touching a patch to being 10 steps away from the average patch location.
Revised: We measure how the agent’s share of the total reward changes as we change its starting position from directly touching a patch to being 10 steps away from the average patch location.

Original: Every other agent always starts 3 steps away from the nearest patch.
Revised: Every other agent always starts 3 steps away from the nearest patch.

Original: We visualize the resultsof this in Figure 4 (more details are given in Appendix D).
Revised: We visualize the results in Figure 4 (more details are given in Appendix D).

Original: When the highest-weight agent is closer to the patch than the other agents, it can start negotiating a team earlier, and due to its high weight it can demand a high reward.
Revised: When the highest-weight agent is closer to the patch than the other agents, it can start negotiating a team earlier, and due to its high weight it can demand a high reward.

Original: However, as we move the agent further away, it takes longer for the agent to reach a patch, so the other (weaker) agents have more time to form a team; thus, its share of the reward signiﬁcantly drops.
Revised: However, as we move the agent further away, it takes longer for the agent to reach a patch, so the other (weaker) agents have more time to form a team; thus, its share of the reward signiﬁcantly drops.

Original: 4. E XPERIMENT 4: R EPRESENTATIONAL CAPACITY OR LEARNING DYNAMICS ?
Revised: 4. E XPERIMENT 4: W HY DO RL A GENTS D EVIATE F ROM THE S HAPLEY V ALUE ?

Original: While for most boards our RL agents do indeed converge to an outcome that approximates the Shapley value, there is a deviation on boards where  players have particularly high Shapley values (“strong [Figure] [Figure] P r e d i c t
Revised: While for most boards our RL agents do indeed converge to an outcome that approximates the Shapley value, there is a deviation on boards where some players have particularly strong or particularly weak negotiation positions (Shapley values).

Original: [Figure] d ( [Figure] u p
Revised: Multiple factors may contribute to this.

Original: [Figure] P [Figure] e d
Revised: One potential factor relates to the representational capacity of the RL agents; computing the Shapley value in a weighted voting game is an NP-hard problem (Elkind et al., 2009), so perhaps such a function cannot be easily induced with a neural network.

Original: [Figure] d ( [Figure] p
Revised: Even if a neural network can compute the negotiation position in the underlying weighted voting game, we may still have an optimization error; RL agents optimize for their individual reward when negotiating under a speciﬁc protocol, and it might

Original: [Figure]
Revised: [Figure]

Original: players”) or have particularly low Shapley values (“weak players”).

Original: We observe that very weak players get more than predicted by the Shapley value, while very strong players get less.

Original: Multiple factors may contribute to this.

Original: One potential factor relates to the representational capacity of the RL agents; computing the Shapley value in a weighted voting game is known to be an NP-hard problem (Elkind et al., 2009), so perhaps such a function cannot be easily induced with a neural network.

Original: Second, it may be difﬁcult to express a policy exactly ﬁtting the Shapley value in the negotiation protocols we’ve selected.
Revised: be difﬁcult to express a policy exactly ﬁtting the Shapley value in the negotiation protocols we’ve selected.

Original: Finally, the deviation might be caused by the learning dynamics of independent RL agents; the highly non-stationary environment induced by agents learning at the same time may lead agents to agreements deviating from the cooperative game theoretic predictions for extreme boards.
Revised: Finally, the deviation might be caused by the learning dynamics of independent RL agents; the highly non-stationary environment induced by agents learning at the same time may lead agents to agreements deviating from the cooperative game theoretic predictions.

Original: We rule out the representational capacity factor by conducting a  supervised learning experiment.
Revised: We show that a neural network can approximate the Shapley value under supervised learning (i.e. when it does not need to consider the environment resulting from a speciﬁc negotiation protocol).

Original: We train a model to take the parameters of a weighted voting game (i.e. the weightsof the agents and the threshold), and output the Shapley values of each player.
Revised: We train a model to take the parameters of a weighted voting game (i.e. agent weights and the threshold), and output the Shapley values of each player.

Original: We generate 3 , 000 unique boards from the same distribution with Gaussian weights used for the experiments in Section 4.3, and apply a train/test partition (80%/20% of the data).
Revised: We generate 3 , 000 unique boards from the same distribution with Gaussian weights used for the experiments in Section 4.2, and apply a train/test partition (80%/20% of the data).

Original: We then ﬁt a simple 3-layer MLP with 20 hidden units (smaller than used in the RL agents), minimizing the mean-squared error loss between the model’s predictions and the Shapley values.
Revised: We then ﬁt a simple 3-layer MLP with 20 hidden units (smaller than used in the RL agents), minimizing the mean-squared error loss between the model’s predictions and the Shapley values.

Original: The results of this are shown in Figure 5.
Revised: The results of this are shown in Figure 5.

Original: We ﬁnd that the MLP is not only able to generalize to unseen boards, but moreover is able to better match the game theoretic predictionsfor a wider distribution of boards.
Revised: We ﬁnd that the MLP generalizes to unseen boards, and better matches the game theoretic predictions of the Shapley value. We have also conducted another experiment where we let RL agents observe not only their weights, but also their Shapley values in the underlying game.

Original: As the deviations from the 
Revised: However, even in this setting we note a discrepancy between the Shapley value and the outcomes that RL agents arrive at (see Appendix F for full details).

Original: Shapley value have a similar shape for both the two negotiation protocols we used, we conjecture that the key reason why our RL  negotiation  agents deviate from the Shapley value for extreme boards is the non-stationary environment induced by the learning dynamics of independent RL agents.
Revised: We thus believe that the deviation is not caused by the RL agents’ inability to determine their relative negotiation power; The RL agents either do not manage to ﬁnd a strong policy under the speciﬁc negotiation protocol, or converge on an outcome different from the Shapley value due to having a non-stationary environment with independent learners.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: Team formation is an important problem for multi-agent systems, since many real-world tasks are impossible without the cooperation and coordination of multiple agents.
Revised: Team formation is an important problem for multi-agent systems, since many real-world tasks are impossible without the cooperation and coordination of multiple agents.

Original: Our contributions are as follows: (1) we introduced a scalable method for team-formation negotiation based on deep reinforcement learning which generalizes to new negotiation protocols and does not require human data, (2) we showed that negotiator agents derived by this method outperform simple hand-crafted bots, and produce results consistent with cooperative game theory, 3) we applied our method to spatially and temporally extended team-formation negotiation environments, where solving for the equilibrium behavior is hard, and (4) we showed that our method makes sensible predictions about the effect of spacial changes on agent behavioral and negotiation outcomes.
Revised: Our contributions are as follows: (1) we introduced a scalable method for team-formation negotiation based on deep reinforcement learning which generalizes to new negotiation protocols and does not require human data, (2) we showed that negotiator agents derived by this method outperform simple hand-crafted bots, and produce results consistent with cooperative game theory, 3) we applied our method to spatially and temporally extended team-formation negotiation environments, where solving for the equilibrium behavior is hard, and (4) we showed that our method makes sensible predictions about the effect of spacial changes on agent behavioral and negotiation outcomes.

Original: This work opens up a new avenue of research applying deep learning to team-formation negotiation tasks.
Revised: This work opens up a new avenue of research applying deep learning to team-formation negotiation tasks.

Original: In particular, it would be interesting to analyze how team formation dynamics affect emergent language in reinforcement learning agents, naturally extending the work of (Cao et al., 2018a) and (Lazaridou et al., 2016).
Revised: In particular, it would be interesting to analyze how team formation dynamics affect emergent language in reinforcement learning agents, naturally extending the work of (Cao et al., 2018a) and (Lazaridou et al., 2016).

Original: Indeed, it has been suggested that the human ability to negotiate and form teams was critical in the evolution of language (Thomas & Kirby, 2018).
Revised: Indeed, it has been suggested that the human ability to negotiate and form teams was critical in the evolution of language (Thomas & Kirby, 2018).

Original: One might also consider creating tasks that interpolate between the fully cooperative game-theoretic setting and the purely non-cooperative one.
Revised: One might also consider creating tasks that interpolate between the fully cooperative game-theoretic setting and the purely non-cooperative one.

Original: Fundamentally, binding contracts are managed by dynamic institutions, whose behavior is also determined by learning.
Revised: Fundamentally, binding contracts are managed by dynamic institutions, whose behavior is also determined by learning.

Original: In principle, we could extend our method to this hierarchical case, perhaps along the lines of (Greif, 2006).
Revised: In principle, we could extend our method to this hierarchical case, perhaps along the lines of (Greif, 2006).

Original: Perhaps such an analysis would even have implications for the social and political sciences.
Revised: Perhaps such an analysis would even have implications for the social and political sciences.

Original: A P OWER INDICES POLITICAL SETTINGS : A MOTIVATING EXAMPLE
Revised: A M OTIVATING E XAMPLES FOR A PPLICATIONS OF THE S HAPLEY V ALUE

Original: Our results compare the agreements of RL agents to the Shapley value in a weighted voting game setting.
Revised: Our results compare the agreements of RL agents to the Shapley value in a weighted voting game setting.

Original: We provide a simple illustration of how such indices formalize power in a way that depends on the possible teams that can form, rather than the direct parameters of the game (such as the weights).
Revised: To illustrate and motivate the importance of the Shapley value and other power indices, let us discuss a few example domains. Consider multiple providers such as travel agencies or airline carriers which can allow a person to travel between various destinations, and a client who is willing to pay a certain amount to get to a desired destination; while there is no direct ﬂight to the client’s destination, there are multiple routing options, using different carrier combinations. How should the carriers share the customer’s payment? Similarly, consider a manufacturing scenario where multiple companies can provide subsets of components required to manufacture an end product, and where each company has only some of the components.

Revised: How should they share the proﬁt from selling the end product?

Original: Cooperative game theory has been applied to  measuring agent power in decision making bodies (Strafﬁn, 1988; Shapley & Shubik, 1954; Felsenthal et al., 1998).One example of such a domain that has been widely investigated is the formation of a coalition government following an election.
Revised: Both the above scenarios can be captured as a cooperative game (and one can follow our paradigm to train RL agents to negotiate in such domains). Solution concepts from cooperative game theory can analyze such domain, and make predictions regarding how agents are likely to share the joint gains. However, the most prominent example for applying cooperative game theory to analyze the negotiation power of agents originates from measuring political power in decision making bodies (Strafﬁn, 1988; Shapley & Shubik, 1954; Felsenthal et al., 1998).

Revised: We illustrate how power indices, and the

Revised: Shapley value in particular, formalize power in a way that depends on the possible teams that can form, rather than the direct parameters of the game (such as the weights).

Original: Consider  a set of political partieswho each obtained a certain number of parliament seats in an election, and where a quota of the majority of the seats is required to form a ruling government.
Revised: Consider the formation of a coalition government following an election, where a set of political parties each obtained a certain number of parliament seats in an election, and where a quota of the majority of the seats is required to form a ruling government.

Original: If no single party has won the majority of the votes on its own, multiple parties would have to join so as to form a ruling coalition government. 
Revised: If no single party has won the majority of the votes on its own, multiple parties would have to join so as to form a ruling coalition government. Parties in this setting would have to negotiate to form a ruling government.

Original: This is rare in the UK, and has never happened in the USA, but is the typical situation in many other countries.
Revised: This is rare in the UK, and has never happened in the USA, but is the typical situation in many other countries.

Original: For intuition, consider a parliament of 100 seats, two big parties with 49 seats each and a small party with 2 seats; a majority requires 50 seats (half of the total number of seats), so no party can form a government on its own.
Revised: For intuition, consider a parliament of 100 seats, two big parties with 49 seats each and a small party with 2 seats; a majority requires 50 seats (half of the total number of seats), so no party can form a government on its own.

Original: While each big party has far more seats than the small party, any team of two or more parties has the required majority of seats.
Revised: While each big party has far more seats than the small party, any team of two or more parties has the required majority of seats.

Original: Under such a framing of the domain, any team of at least two out the three parties is sufﬁcient for completing the task of forming a government, and due to this symmetry between agents one may claim they all have equal power.
Revised: Under such a framing of the domain, any team of at least two out the three parties is sufﬁcient for completing the task of forming a government, and due to this symmetry between agents one may claim they all have equal power.

Original: In other words, what matters in making a party powerful is not its number of parliament seats, but rather the number of opportunities it has to form teams.
Revised: In other words, what matters in making a party powerful is not its number of parliament seats, but rather the number of opportunities it has to form teams.

Original: Intuitively, one would say that the small party has a strong negotiation position.
Revised: Intuitively, one would say that the small party has a strong negotiation position.

Original: It could, for instance, demand control of a dis-proportionally high part of the budget (i.e. although it only has 2% of the seats, it is likely to get control of a much larger share of a the budget).
Revised: It could, for instance, demand control of a dis-proportionally high part of the budget (i.e. although it only has 2% of the seats, it is likely to get control of a much larger share of a the budget).

Original: We note that for the above example, the Shapley value would give each party the same share.
Revised: We note that for the above example, the Shapley value would give each party the same share.

Original: In fact, the Shapley value as a function does not even take the weights or threshold as input, but rather the characteristic function of the game.
Revised: In fact, the Shapley value as a function does not even take the weights or threshold as input, but rather the characteristic function of the game.

Original: We say that agents i, j are equivalent in a game with characteristic function v if for any coalition C that contains neither i nor j (i.e. any C such that i / ∈ C and j / ∈ C ) we can add either i or j to the coalition and obtain the same value (i.e. v ( C ∪ { i } )
Revised: We say that agents i, j are equivalent in a game with characteristic function v if for any coalition C that contains neither i nor j (i.e. any C such that i / ∈ C and j / ∈ C ) we can add either i or j to the coalition and obtain the same value (i.e. v ( C ∪ { i } )

Original: = v ( C ∪ { j } ) ).
Revised: = v ( C ∪ { j } ) ).

Original: We note that in the above example, all agents are equivalent, and that the Shapley value allocates them the same share.
Revised: We note that in the above example, all agents are equivalent, and that the Shapley value allocates them the same share.

Original: This is not a coincidence, one of the fairness axioms characterizing the Shapley value is that equivalent agents get the same share (Shapley, 1953b; 
Revised: This is not a coincidence — one of the fairness axioms characterizing the Shapley value is that equivalent agents get the same share (Shapley, 1953b; Dubey, 1975).

Original: Dubey, 1975).
Revised: Indeed, the Shapley value is the only index fulﬁlling a small set of fairness axioms such as this one. Two additional axioms that fully characterize the Shapley value, in addition to the above equivalence axiom, are that of giving null players no reward, and the additivity axiom. A null player is one which adds no value to any coalition, and the null player axioms states that null players would get no reward; the additivity axioms relates to the sum of multiple games, stating that the value allocated to any player in the sum-game would be the sum of the values in the individual composing games.

Original: Indeed, the Shapley value is the only index fulﬁlling a small set offairness axioms such as this one (see textbooks and papers on  axiomatic characterization of the Shapley valuefor a more detailed picture (Strafﬁn, 1988; Chalkiadakis et al., 2011)).
Revised: For a more detailed discussion of such axioms, see textbooks and papers on the axiomatic characterization of the Shapley value (Strafﬁn, 1988; Chalkiadakis et al., 2011).

Original: B B EATING BASELINE BOTS : FULL EXPERIMENT DETAILS
Revised: B B EATING BASELINE BOTS : FULL EXPERIMENT DETAILS

Original: Section 4.2 describes our experiment comparing the negotiation performance of RL agents and handcrafted bots.
Revised: Section 4.1 describes our experiment comparing the negotiation performance of RL agents and handcrafted bots.

Original: We now provide a more detailed discussion of this experiment.
Revised: We now provide a more detailed discussion of this experiment.

Original: We ﬁrst note that although the negotiation protocols differ across environments, the essence of the decisions agent face is similar.
Revised: We ﬁrst note that although the negotiation protocols differ across environments, the essence of the decisions agent face is similar.

Original: When agents fail to reach an agreement they obtain no reward, so agents who almost never reach an agreement would be bad negotiators.
Revised: When agents fail to reach an agreement they obtain no reward, so agents who almost never reach an agreement would be bad negotiators.

Original: On the other hand, reaching agreement easily by itself does not make an agent a strong negotiator.
Revised: On the other hand, reaching agreement easily by itself does not make an agent a strong negotiator.

Original: For instance, an agent who accepts any offer is likely to reach agreement easily but perform poorly, as they would take even very low offers (even when they have a high relative weight and thus a strong negotiation position).
Revised: For instance, an agent who accepts any offer is likely to reach agreement easily but perform poorly, as they would take even very low offers (even when they have a high relative weight and thus a strong negotiation position).

Original: Although the essence of the task is similar across environments, the observations, action spaces and semantics of interaction differ considerably across negotiation protocols.
Revised: Although the essence of the task is similar across environments, the observations, action spaces and semantics of interaction differ considerably across negotiation protocols.

Original: The key advantage of using RL to train negotiation agents is not having to hand-craft rules relating to the speciﬁcs of the negotiation environment.
Revised: The key advantage of using RL to train negotiation agents is not having to hand-craft rules relating to the speciﬁcs of the negotiation environment.

Original: Indeed, as hand-crafting a bot is a time consuming process, we focused on the simpler propose-accept environment; creating a hand-crafted bot for the team patches environment is more difﬁcult, as there are many more decisions to be made (how to ﬁnd a good patch, which share to demand, how to respond if the total reward demanded is too high, etc.)
Revised: Indeed, as hand-crafting a bot is a time consuming process, we focused on the simpler propose-accept environment; creating a hand-crafted bot for the team patches environment is more difﬁcult, as there are many more decisions to be made (how to ﬁnd a good patch, which share to demand, how to respond if the total reward demanded is too high, etc.)

Original: Our simplest baseline is a random bot .
Revised: Our simplest baseline is a random bot .

Original: As a proposer, it selects an allowed proposal uniformly at random from the set of all such proposals.
Revised: As a proposer, it selects an allowed proposal uniformly at random from the set of all such proposals.

Original: As a proposee it accepts with probability 12 and rejects with probability 1 2 .
Revised: As a proposee it accepts with probability 12 and rejects with probability 12 .

Original: A more sensible basedline is the weight-proportional bot , which is an adaptation of previously proposed negotiation agents to the propose-accept protocol (Lin et al., 2008; Lin & Kraus, 2010;
Revised: A more sensible basedline is the weight-proportional bot , which is an adaptation of previously proposed negotiation agents to the propose-accept protocol (Lin et al., 2008; Lin & Kraus, 2010;

Original: Baarslag et al., 2012; Mash et al., 2017) (the propose-accept protocol itself bears some similarity to the negotiation protocols used in such earlier work on negotiation agents).
Revised: Baarslag et al., 2012; Mash et al., 2017) (the propose-accept protocol itself bears some similarity to the negotiation protocols used in such earlier work on negotiation agents).

Revised: An even stronger baseline is the Shapley-proportional bot , which is similar to the weight-proportional bot, but computes the

Revised: Shapley values of all the agents (which is computationally tractable only for games with few agents).

Original: Section 4.2 provides a short description of the actions taken by the weight-proportional bot.
Revised: Section 4.1 provides a short description of the actions taken by the weight-proportional and Shapley proportional bots.

Original: We provide a longer description in this appendix, trying to elaborate on the intuition and principles behind the design of this bot.
Revised: We provide a longer description in this appendix, trying to elaborate on the intuition and principles behind the design of these bots.

Original: As a proposer, the weight-proportional bot randomly chooses a viable team C which contains itself from the set of all such teams.
Revised: As a proposer, the weight-proportional bot randomly chooses a viable team C which contains itself from the set of all such teams.

Original: It then proposes an allocation of the reward that is proportional to each the team agents’ weights.
Revised: It then proposes an allocation of the reward that is proportional to each the team agents’ weights.

Original: Formally, given a board [( w 1 , . . . , w n ); q ] , we denote the total weight of a team C by w ( C ) =
Revised: Formally, given a board [( w 1 , . . . , w n ); q ] , we denote the total weight of a team C by w ( C ) =

Original: (cid:80) i ∈
Revised: (cid:80) i ∈

Original: C w i .
Revised: C w i .

Original: For the team it chose, the bot uses the target allocation p i =
Revised: For the team it chose, the bot uses the target allocation p i =

Original: w i w ( C ) · r where r is the ﬁxed total reward.
Revised: w i w ( C ) · r where r is the ﬁxed total reward.

Original: As a proposee, the  bot computes its share of the same target allocation p i =
Revised: As a proposee, the weight-proportional bot computes its share of the same target allocation p i =

Original: w i w ( C ) · r in the proposed team C , and compares it with the amount r i offered to it in the proposal.
Revised: w i w ( C ) · r in the proposed team C , and compares it with the amount r i offered to it in the proposal.

Original: We denote by g i =
Revised: We denote by g i =

Original: r i − p
Revised: r i − p

Original: i the amount by which the offer r i exceeds p i , the bot’s expectations under the weight-proportional allocation.
Revised: i the amount by which the offer r i exceeds p i , the bot’s expectations under the weight-proportional allocation.

Original: A high positive amount g i indicates that the bot is offered much more than it believes it deserves according to the weight-proportional allocation, while a negative amount indicates it is offered less than it thinks it deserves (when offered exactly the weight-proportional share we have g i = 0 ).
Revised: A high positive amount g i indicates that the bot is offered much more than it believes it deserves according to the weight-proportional allocation, while a negative amount indicates it is offered less than it thinks it deserves (when offered exactly the weightproportional share we have g i = 0 ).

Original: The probability of the bot accepting the offer is σ ( c · g i ) where σ denotes the logistic function σ ( x ) = 1 1+ e − x , and where c = 5 is a constant correcting the fact that g i only ranges between − 1 and +1 rather than between −∞ and + ∞ .
Revised: The probability of the bot accepting the offer is σ ( c · g i ) where σ denotes the logistic function σ ( x ) = 1 1+ e − x , and where c = 5 is a constant correcting the fact that g i only ranges between − 1 and +1 rather than between −∞ and + ∞ .

Original: Thus the bot accepts a “fair” offer (according to the weight-proportional target) with probability of 12 , and the probability convexly increases as g i increases (and decreases as g i decreases). 
Revised: Thus the bot accepts a “fair” offer (according to the weight-proportional target) with probability of 12 , and the probability convexly increases as g i increases (and decreases as g i decreases). The more sophisticated Shapley-proportional bot follows the same design as the weight-proportional bot, except it sets the target allocation to be the one proportional to the Shapley values rather than the weights, i.e. p i = rφ i / ( (cid:80)

Revised: i ∈ C φ i ) (where φ i denotes the Shapley value of agent i ).

Original: As discussed in Section 4.2, we compare the performance of the bot and RL trained agents by creating many pairs of groups.
Revised: As discussed in Section 4.1, we compare the performance of the bot and RL trained agents by creating many pairs of groups.

Original: We examine each of the20 evaluation boards (sampled from the distribution of Gaussian weigh boards).
Revised: We examine each of the evaluation boards (sampled from the distribution of Gaussian weigh boards).

Original: For each evaluation board, we create t = 200 pairs of agent groups, where in each pair we have one group of n = 5 independent RL agents (the all-RL group ), and one set consisting of 1 bot and 4 independent RL agents (the single-bot group ).
Revised: For each evaluation board, we create t = 200 pairs of agent groups, where in each pair we have one group of n = 5 independent RL agents (the all-RL group ), and one set consisting of 1 bot and 4 independent RL agents (the single-bot group ).

Original: Each group is co-trained for m = 500 , 000 episodes, where during training each episode uses a different boardsampled at random from the board distribution D .
Revised: Each group is co-trained for m = 500 , 000 episodes, where during training each episode uses a different board from the train set.

Original: During the evaluation step, we let each agent group play 5 , 000 gamesepisodes on the evaluation board.
Revised: During the evaluation step, we let each agent group play 5 , 000 games on each evaluation board in the test set.

Original: We repeat the same analysis, each time allowing the bot to play a different weight in the board (i.e. letting the bot be the agent in the ﬁrst position, in the second position and so on).
Revised: We repeat the same analysis, each time allowing the bot to play a different weight in the board (i.e. letting the bot be the agent in the ﬁrst position, in the second position and so on).

Original: We investigate the performance of the RL agent from the all-RL group and the bot from the singlebot group, when playing the same weight on the same evaluation board.
Revised: We investigate the performance of the RL agent from the all-RL group and the bot from the singlebot group, when playing the same weight on the same evaluation board.

Original: We average the fraction of the total ﬁxed reward achieved by the RL-agent and bot over the t = 200 group pairs and over the 5000 episodes played with each evaluation board, and examine the difference d between the amount won by the RL-agent and the amount won by the bot.
Revised: We average the fraction of the total ﬁxed reward achieved by the RL-agent and bot over the t = 200 group pairs and over the 5000 episodes played with each evaluation board, and examine the difference d between the amount won by the RL-agent and the amount won by the bot.

Original: A positive value for d indicates the RL-agent has a better performance than the bot, and a negative number shows a higher performance for the bot.
Revised: A positive value for d indicates the RL-agent has a better performance than the bot, and a negative number shows a higher performance for the bot.

Original: On average (across boards and episodes played on each evaluation board), the RL-agent outperforms the  bot, achieving 0 .
Revised: On average (across boards and episodes played on each evaluation board), the RL-agent outperforms the weight-proportional bot, achieving 0 .

Original: 025  r more of the total reward.
Revised: 025 · r more of the total reward.

Original: In the all-RL group, each agent makes on average 0 . 

Revised: In the all-RL group, each agent makes on average 0 .

Original: · r , whereas in the one-bot group the bot makes on average 0 . 

Revised: · r , whereas in the one-bot group the bot makes on average 0 .

Original: · r .
Revised: · r .

Original: In other words, the RL-agent obtains 10% more reward than the bot.
Revised: In other words, the RL-agent obtains 10% more reward than the bot.

Original: We performed a Mann-Whitney-Wilcoxon test, which shows the difference is signiﬁcant at the p < 0 .
Revised: We performed a Mann-WhitneyWilcoxon test, which shows the difference is signiﬁcant at the p < 0 .

Original: 005 level.
Revised: 005 level.

Revised: The results are similar for the Shapley-proportional bot, although the performance gap is smaller, with the Shapleyproportional bot making on average 0 .

Revised: 185 · r (the RL agent outperforms the Shapley-proportional bot at the same statistical signiﬁcance level as for the weight-proportional bot).

Original: 4 Unsurprisingly, performing a similar analysis for the random bot, that selects valid actions uniformly at random, we get even more signiﬁcant results ( p < 0 . 001 ).
Revised: Unsurprisingly, performing a similar analysis for the random bot, that selects valid actions uniformly at random, we get even more signiﬁcant results ( p < 0 . 001 ).

Revised: We have performed an additional experiment, where we train the RL-agents in the one-bot group with a weight-proportional bot, but use a Shapley-proportional bot during their evaluation.

Revised: In other words, their adapt their policy for one type of bot, but are measured against another type of bot.

Revised: This hindrance sightly lowers the average reward share of RL agents, allowing the bot to gain a slightly larger reward share of 0 .

Revised: 188 · r .

Revised: However, even in this case the RL-agents outperform the bot ( p < 0 . 005 ).

Original: We note that our result indicates that RL agents can outperform some simple heuristic hand-crafted bots.
Revised: We note that our results indicate that RL agents can outperform some simple heuristic hand-crafted bots.

Original: This result certainly does not mean that it is impossible to create well-crafted bots, tailored to a speciﬁc negotiation protocol, that would out-negotiate RL agents.
Revised: This result certainly does not mean that it is impossible to create well-crafted bots, tailored to a speciﬁc negotiation protocol, that would out-negotiate RL agents.

Revised: For instance, even for the weight-proportional and Shapley-proportional bots we can tune the distribution parameter c discussed above, and possible improve the negotiation performance.

Original: In other words, we view this analysis as an indication to RL-agents make at least somewhat sensible negotiation decisions.
Revised: We view this analysis as an indication to RL-agents make at least somewhat sensible negotiation decisions.

Original: The key advantage of the methodology we propose is its generality and ability to tackle new negotiation protocols, without relying on hand-crafted solutions.
Revised: The key advantage of the methodology we propose is its generality and ability to tackle new negotiation protocols, without relying on hand-crafted solutions.

Original: In other words, our technique offers a way of automatically constructing at least reasonable negotiators, without requiring ﬁtting a bot to a given negotiation protocol.
Revised: In other words, our technique offers a way of automatically constructing at least reasonable negotiators, without requiring ﬁtting a bot to a given negotiation protocol.

Original: C O N THE C OMPLEXITY OF C OMPUTING N ASH E QUILIBRIA IN THE N EGOTIATION E NVIRONMENTS
Revised: C O N THE C OMPLEXITY OF C OMPUTING N ASH E QUILIBRIA IN THE N EGOTIATION E NVIRONMENTS

Original: We used the Shapley value from cooperative game theory to examine likely outcomes in the underlying weighted voting game.
Revised: We used the Shapley value from cooperative game theory to examine likely outcomes in the underlying weighted voting game.

Original: The Propose-Accept environment and the Team Patches environment are both based on this underlying game, but deﬁne the possible actions agents can take and the outcomes for various action proﬁles (i.e. they employ a speciﬁc negotiation protocol).
Revised: The Propose-Accept environment and the Team Patches environment are both based on this underlying game, but deﬁne the possible actions agents can take and the outcomes for various action proﬁles (i.e. they employ a speciﬁc negotiation protocol).

Original: Thus, they can be viewed as a non-cooperative (Markov) game.
Revised: Thus, they can be viewed as a non-cooperative (Markov) game.

Original: For such setting, one can also examine solution concepts from non-cooperative game theory, such as the Nash equilibrium (Nash, 1950).
Revised: For such setting, one can also examine solution concepts from non-cooperative game theory, such as the Nash equilibrium (Nash, 1950).

Original: The resulting Markov game is not computationally tractable to solve for the Nash equilibrium, as it is an inﬁnitely-repeated n -player general-sum extensive-form game.
Revised: The resulting Markov game is not computationally tractable to solve for the Nash equilibrium, as it is an inﬁnitely-repeated n -player general-sum extensive-form game.

Original: This is among the hardest class of games to compute an equilibrium for.
Revised: This is among the hardest class of games to compute an equilibrium for.

Original: Even the restricted case of an unrepeated two-player general-sum game is “hopelessly impractical to solve exactly” (Shoham & Leyton-Brown, 2009,
Revised: Even the restricted case of an unrepeated two-player general-sum game is “hopelessly impractical to solve exactly” (Shoham & Leyton-Brown, 2009,

Original: Section 4.3), being PPAD-complete (Chen & Deng, 2006).
Revised: Section 4.3), being PPAD-complete (Chen & Deng, 2006).

Original: We thus choose to apply cooperative game theoretic solutions, namely the Shapley value.
Revised: We thus choose to apply cooperative game theoretic solutions, namely the Shapley value.

Original: D T EAM P ATCHES E NVIRONMENT
Revised: D T EAM P ATCHES E NVIRONMENT

Original: Our spatial Team Patches environment is a 15 × 15 grid-world where each entity has an assigned color.
Revised: Our spatial Team Patches environment is a 15 × 15 grid-world where each entity has an assigned color.

Original: The agents observe this world from their own perspective, as shown in Figure 6(a), and also observe the weights and demands of every other agent, as well as their own index.
Revised: The agents observe this world from their own perspective, as shown in Figure 6(a), and also observe the weights and demands of every other agent, as well as their own index.

Original: In Experiment 3 (Section 4.4) we change this world in two ways:
Revised: In Experiment 3 (Section 4.3) we change this world in two ways:

Original: We set the total number of patches in the world to 2 (red and blue).
Revised: We set the total number of patches in the world to 2 (red and blue).

Original: We modify the starting position of the agent with the highest weight, investigating how this spatial perturbation inﬂuences their share of the total reward.
Revised: We modify the starting position of the agent with the highest weight, investigating how this spatial perturbation inﬂuences their share of the total reward.

Original: In Figure 6(b) we visualize this with a full view of the environment where the white agent is moved between squares 0 and 10, corresponding to being N steps away from the nearest patch in Ldistance.
Revised: In Figure 6(b) we visualize this with a full view of the environment where the white agent is moved between squares 0 and 10, corresponding to being N steps away from the nearest patch in Ldistance.

Original: [Figure]
Revised: [Figure]

Revised: E T HE I MPACT OF W EIGHT AND P OWER I NEQUALITY ON THE C ORRESPONDANCE WITH THE S HAPLEY V ALUE

Revised: Section 4.2 discussed the correspondence between the outcomes achived by RL agents and the Shapley value from cooperative game theory, indicating that deviations occur mostly in boards where some agents have a particularly weak or particularly strong negotiation position, as measured by the

Revised: Shapley value.

Revised: The board distribution D that we have there used ruled-out boards where all agents have identical weights (resulting in identical power).

Revised: To investigate the impact of the weight and power inequality on the correspondence with the Shapley value, we examined an alternative board distribution D (cid:48) , which did not rule-out equal-weight boards (the remaining weight sampling procedure was identical to the original distribution D ).

Revised: We generated a new train set and test set of boards from D (cid:48) .

Revised: The sampled weights under D had a standard deviation of STD D = 1 .

Revised: 65 whereas the standard deviation of weights under D (cid:48) was STD D (cid:48)

Revised: = 1 . 1 .

Revised: We repeated the analysis of Section 4.2 for these reduced variance boards, resulting in Figure 7.

Revised: Figure 7 shows that the mass is much more concentrated around the area with equal Shapley values (points with x ≈ 0 . 2 ), where the outcome achieved by RL agents is also very close to the same value (i.e. y ≈ 0 . 2 ).

Revised: As the Figure shows, this results in having a stronger correspondence between the outcome obtained by RL agents and the Shapley value.

Revised: In other words, high inequality in agent power (negotiation position) results in a larger discrepancy between the Shapley value and outcomes that RL agents arrive at.

Revised: [Figure]

Revised: i

Revised: [Figure]

Revised: g

Revised: [Figure]

Revised: F L ETTING A GENTS O BSERVE THEIR N EGOTIATION P OSITION

Revised: In Section 4.4 we discuss potential reasons for the empirical gains of our RL agents deviating from the Shapley value.

Revised: The analysis there shows that given a direct supervision signal (boards labeled with the Shapley values), a small neural net can approximate the Shapley value well.

Revised: Our RL agents have a more challenging task for two reasons: (1) They have to take into account not only their negotiating position but also protocol details.

Revised: (2) Their RL supervision is weaker: they only know how successful a whole sequence of actions was, and not the “correct” action they should have taken at every timestep.

Revised: Our conclusion from Figure 5 is that at least the basic supervised learning task can be accomplished with a small neural network, so the agents network has the capacity required to estimate their raw negotiating power, abstracting away protocol details.

Revised: We now show that even when given the Shapley values in the underlying game, RL agents reach outcomes that may deviate from the Shapley value.

Revised: We use an experimental setup that is identical to that used to produce Figure 3a, except we provide the Shapley values of all the agents as a part of the observation (in every state).

Revised: The outcome of the experiment is shown in Figure 8.

Revised: [Figure]

Revised: p i r i c a l

Revised: ( R

Revised: L a g e n t s )

Revised: Figure 8 shows the same pattern as Figure 3a.

Revised: The deviation from game theoretic predictions thus stems not from being unable to identify the “high-level” negotiation position of agents, but rather from the RL procedure we applied; RL agents attempt to maximize their own share of the gains (rather than for obtaining their “fair” share of the gains, as captured by the Shapley value), and are forced to deal with a the need to ﬁnd strong policies taking into account the speciﬁc negotiation protocol (under a non-stationary environment, where their peers constantly adapt their behavior).