Original: E MERGENT COMMUNICATION THROUGH NEGOTIATION
Revised: E MERGENT COMMUNICATION THROUGH NEGOTIATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multi-agent reinforcement learning offers a way to study how communication could emerge in communities of agents needing to solve speciﬁc problems.
Revised: Multi-agent reinforcement learning offers a way to study how communication could emerge in communities of agents needing to solve speciﬁc problems.

Original: In this paper , we study the emergence of communication in the negotiation environment , a semi-cooperative model of agent interaction.
Revised: In this paper, we study the emergence of communication in the negotiation environment, a semi-cooperative model of agent interaction.

Original: We introduce two communication protocols – one grounded in the semantics of the game and one which is a priori ungrounded and is a form of cheap talk.
Revised: We introduce two communication protocols – one grounded in the semantics of the game and one which is a priori ungrounded and is a form of cheap talk.

Original: We show that self-interested agents can use the pre-grounded communication channel to negotiate fairly but are unable to effectively use the ungrounded channel to do the same.
Revised: We show that self-interested agents can use the pre-grounded communication channel to negotiate fairly, but are unable to effectively use the ungrounded channel.

Original: However, prosocial agents do learn to use cheap talk to ﬁnd an optimal negotiating strategy, suggesting that cooperation is necessary for language to emerge.
Revised: However, prosocial agents do learn to use cheap talk to ﬁnd an optimal negotiating strategy, suggesting that cooperation is necessary for language to emerge.

Original: We also study communication behaviour in a setting where one agent interacts with agents in a community with different levels of prosociality and show how agent identiﬁability can aid negotia- tion.
Revised: We also study communication behaviour in a setting where one agent interacts with agents in a community with different levels of prosociality and show how agent identiﬁability can aid negotiation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: How can communication emerge?
Revised: How can communication emerge?

Original: A necessary prerequisite is a task that requires coordination between multiple agents to solve, and some communication protocol for the agents to exchange messages through (see a review by Wagner et al.
Revised: A necessary prerequisite is a task that requires coordination between multiple agents to solve, and some communication protocol for the agents to exchange messages through (see a review by Wagner et al.

Original: (2003) on earlier work on emergent communication as well as recent deep reinforcement learning methods by Foerster et al.
Revised: (2003) on earlier work on emergent communication as well as recent deep reinforcement learning methods by Foerster et al.

Original: and Sukhbaatar et al.
Revised: and Sukhbaatar et al.

Original: Given these basic requirements, an interesting question to ask is what task structures aid the emergence of communication and how different communication protocols affect task success.
Revised: Given these basic requirements, an interesting question to ask is what task structures aid the emergence of communication and how different communication protocols affect task success.

Original: In the context of linguistic communication, previous work on this subject has mainly studied the emergence of communication in co-operative games like referential games, variants of the Lewis signaling game (Lewis, 1969), where messages are used to disambiguate between different possible referents (Goldman et al., 2007;
Revised: In the context of linguistic communication, previous work on this subject has mainly studied the emergence of communication in co-operative games like referential games, variants of the Lewis signaling game (Lewis, 1969), where messages are used to disambiguate between different possible referents (Goldman et al., 2007;

Original: Lazaridou et al., 2016; Evtimova et al., 2017).
Revised: Lazaridou et al., 2016; Evtimova et al., 2017).

Original: Human language, though, is not merely a referential tool.
Revised: Human language, though, is not merely a referential tool.

Original: Amongst other things, we communicate private information and thoughts, discuss plans, ask questions and tell jokes.
Revised: Amongst other things, we communicate private information and thoughts, discuss plans, ask questions and tell jokes.

Original: Moreover, many human interactions are not fully cooperative, yet we can still successfully use language to communicate in these situations.
Revised: Moreover, many human interactions are not fully cooperative, yet we can still successfully use language to communicate in these situations.

Original: In this paper, we study communication in the negotiation game (see Figure 1), an established model of non-cooperative games in classical game theory (Nash, 1950b; Neumann & Morgenstern, 1944;
Revised: In this paper, we study communication in the negotiation game (see Figure 1), an established model of non-cooperative games in classical game theory (Nash, 1950b; Neumann & Morgenstern, 1944;

Original: Nash, 1950a; 1951; Schelling, 1960; Binmore et al., 1986; Peters, 2008).
Revised: Nash, 1950a; 1951; Schelling, 1960; Binmore et al., 1986; Peters, 2008).

Original: In this game, agents are asked to establish a mutually acceptable division of a common pool of items while having their own hidden utilities for each of them.
Revised: In this game, agents are asked to establish a mutually acceptable division of a common pool of items while having their own hidden utilities for each of them.

Original: Effective communication is crucial in this game, as the agents need to exchange strategic information about their desires, infer their opponent’s desires from communication, and balance between the two.
Revised: Effective communication is crucial in this game, as the agents need to exchange strategic information about their desires, infer their opponent’s desires from communication, and balance between the two.

Original: Work in classical game theory on negotiation typically uses simple forms of offer / counter-offer bargaining games that do not explicitly address the question of emergent communication (Rubinstein, 1982).
Revised: Work in classical game theory on negotiation typically uses simple forms of offer / counter-offer bargaining games that do not explicitly address the question of emergent communication (Rubin- stein, 1982).

Original: Recent work on deep multi-agent reinforcement learning (MARL) has shown great success in teaching agents complex behaviour without a complex environment simulator or demonstration data (Panait & Luke, 2005; Busoniu et al., 2008; Tuyls & Weiss, 2012; Silver et al., 2017;
Revised: Recent work on deep multi-agent reinforcement learning (MARL) has shown great success in teaching agents complex behaviour without a complex environment simulator or demonstration data (Panait & Luke, 2005; Busoniu et al., 2008; Tuyls & Weiss, 2012; Silver et al., 2017;

Original: Leibo et al., 2017).
Revised: Leibo et al., 2017).

Original: By repeatedly interacting with other agents learning at the same time, agents can gradually bootstrap complex behaviour, including motor skills (Bansal et al., 2017) and linguistic communication (Lazaridou et al., 2016; Havrylov & Titov, 2017).
Revised: By repeatedly interacting with other agents learning at the same time, agents can gradually bootstrap complex behaviour, including motor skills (Bansal et al., 2017) and linguistic communication (Lazaridou et al., 2016; Havrylov & Titov, 2017).

Original: We apply techniques from the MARL literature and train agents to negotiate using task success as the only supervision signal.
Revised: We apply techniques from the MARL literature and train agents to negotiate using task success as the only supervision signal.

Original: We show that, when communicating via a task-speciﬁc communication channel with inherent semantics, selﬁsh agents can learn to negotiate fairly and divide up the item pool to the agents’ mutual satisfaction.
Revised: We show that, when communicating via a task-speciﬁc communication channel with inherent semantics, selﬁsh agents can learn to negotiate fairly and divide up the item pool to the agents’ mutual satisfaction.

Original: However, when communicating via cheap talk (Crawford &
Revised: However, when communicating via cheap talk (Crawford &

Original: Sobel, 1982; Farrell & Rabin, 1996), a task-independent communication channel consisting of sequences of arbitrary symbols similar to language, selﬁsh agents fail to exhibit negotiating behaviour at all.
Revised: Sobel, 1982; Farrell & Rabin, 1996), a task-independent communication channel consisting of sequences of arbitrary symbols similar to language, selﬁsh agents fail to exhibit negotiating behaviour at all.

Original: On the other hand, we show that cheap talk can facilitate effective negotiation in prosocial agents, which take the other agent’s reward into consideration, providing experimental evidence that cooperation is necessary for language emergence (Nowak & Krakauer, 1999).
Revised: On the other hand, we show that cheap talk can facilitate effective negotiation in prosocial agents, which take the other agent’s reward into consideration, providing experimental evidence that cooperation is necessary for language emergence (Nowak & Krakauer, 1999).

Original: The above results are obtained from paired agents interacting exclusively with each other.
Revised: The above results are obtained from paired agents interacting exclusively with each other.

Original: In more realistic multi-agent scenarios, agents may interact with many other agents within a society.
Revised: In more realistic multi-agent scenarios, agents may interact with many other agents within a society.

Original: In these cases, cheap talk can have a signiﬁcant effect on the evolutionary dynamics of the population (Robson, 1990) as well as the equilibria, stability, and basins of attractions (Skyrms, 2002).
Revised: In these cases, cheap talk can have a signiﬁcant effect on the evolutionary dynamics of the population (Robson, 1990) as well as the equilibria, stability, and basins of attractions (Skyrms, 2002).

Original: Furthermore, it is well-known that, unless trained in a diverse environment, agents overﬁt to the their speciﬁc opponent or teammate (Lanctot et al., 2017).
Revised: Furthermore, it is well-known that, unless trained in a diverse environment, agents overﬁt to the their speciﬁc opponent or teammate (Lanctot et al., 2017).

Original: Inspired by these considerations, we perform experiments where agents interact with many agents having different prosociality levels, and ﬁnd that being able to identify and model other agents’ beliefs aids the negotiation success.
Revised: Inspired by these considerations, we perform experiments where agents interact with many agents having different prosociality levels, and ﬁnd that being able to identify and model other agents’ beliefs aids the negotiation success.

Original: This is consistent with experiments using models based on Theory of Mind: boundedly rational agents can collectively beneﬁt by making inferences about the sophistication levels and beliefs of their opponents, and there is evidence that this occurs in human behavior (Yoshida et al., 2008).
Revised: This is consistent with experiments using models based on Theory of Mind: boundedly rational agents can collectively beneﬁt by making inferences about the sophistication levels and beliefs of their opponents, and there is evidence that this occurs in human behavior (Yoshida et al., 2008).

Revised: Moody & Saffell (2001) also applied reinforcement learning with non-linear function approximators to economic problems.

Revised: They trained a recurrent agent to trade securities on the open market using policy gradient methods, similarly to how we train agents.

Revised: However, our agents must create the market from scratch - as there are no established buyers and sellers, the agents participating in the exchange must learn how to make optimal bids.

Original:  G AME SETTING 2. N EGOTIATION ENVIRONMENT
Revised:  G AME SETTING 2. N EGOTIATION ENVIRONMENT

Original: The negotiation task is based on the set-up of Lewis et al.
Revised: The negotiation task is based on the set-up of Lewis et al.

Original: (2017), itself based on DeVault et al.
Revised: (2017), itself based on DeVault et al.

Original: Agents are presented with three types of items: peppers , cherries and strawberries .
Revised: Agents are presented with three types of items: peppers , cherries and strawberries .

Original: At each round (i) an item pool is sampled uniformly, instantiating a quantity (between 0 and 5) for each of the types and represented as a vector i ∈ { 0 ... 5 } 3 and (ii) each agent j receives a utility function sampled uniformly, which speciﬁes how rewarding one unit of each item is (with item rewards between 0 and 10, and with the constraint that there is at least one item with non-zero utility), represented as a vector u j ∈ { 0 ... 10 } 3 .
Revised: At each round (i) an item pool is sampled uniformly, instantiating a quantity (between 0 and 5) for each of the types and represented as a vector i ∈ { 0 ... 5 } 3 and (ii) each agent j receives a utility function sampled uniformly, which speciﬁes how rewarding one unit of each item is (with item rewards between 0 and 10, and with the constraint that there is at least one item with non-zero utility), represented as a vector u j ∈ { 0 ... 10 } 3 .

Original: These utilities are hidden , i.e., each agent only has access to its own utilities.
Revised: These utilities are hidden , i.e., each agent only has access to its own utilities.

Original: Following these assignments, the agents start “negotiating” for N timesteps by exchanging messages m jt and proposals p jt ∈ { 0 ... 5 } 3 for each timestep 1 ≤ t ≤ N .
Revised: Following these assignments, the agents start “negotiating” for N timesteps by exchanging messages m jt and proposals p jt ∈ { 0 ... 5 } 3 for each timestep 1 ≤ t ≤ N .

Original: Agents alternate between timesteps, meaning that agent A always acts on the odd turns while agent
Revised: Agents alternate between timesteps, meaning that agent A always acts on the odd turns while agent

Original: B on the even turns.
Revised: B on the even turns.

Original: Either agent can terminate the negotiation at any timestep t with a special action, which signiﬁes agreement with the most recent proposal made by the other agent.
Revised: Either agent can terminate the negotiation at any timestep t with a special action, which signiﬁes agreement with the most recent proposal made by the other agent.

Original: For example, if agent B terminates the negotiation at t = 4 , agent A gets rewarded by R A =
Revised: For example, if agent B terminates the negotiation at t = 4 , agent A gets rewarded by R A =

Original: u A · ( p At − 1 ) while agent B receives reward
Revised: u A · ( p At − 1 ) while agent B receives reward

Original: R B = u B · ( i − p At − 1 ) , where · denotes the dot product of vectors.
Revised: R B = u B · ( i − p At − 1 ) , where · denotes the dot product of vectors.

Original: Note that, if an agent makes an invalid proposal, such as asking for more items than exist in the item pool, and this is accepted, then both agents receive R A
Revised: Note that, if an agent makes an invalid proposal, such as asking for more items than exist in the item pool, and this is accepted, then both agents receive R A

Original: = R B = 0 .
Revised: = R B = 0 .

Original: We impose an upper limit, N , on the number of negotiation turns allowed.
Revised: We impose an upper limit, N , on the number of negotiation turns allowed.

Original: If the agents get to the limit without agreement, then both agents receive no reward.
Revised: If the agents get to the limit without agreement, then both agents receive no reward.

Original: We experimented with having a ﬁxed upper limit of 10 turns.
Revised: We experimented with having a ﬁxed upper limit of 10 turns.

Original: However, this led to a “ﬁrst-mover” advantage: if the agents know in advance how many turns the negotiation lasts, there is a degenerate policy in which agent
Revised: However, this led to a “ﬁrst-mover” advantage: if the agents know in advance how many turns the negotiation lasts, there is a degenerate policy in which agent

Original: A can wait until the last timestep N − 1 before giving a lopsided offer, which agent B is compelled to accept, a situation similar to the ultimatum game (G¨uth et al., 1982).
Revised: A can wait until the last timestep N − 1 before giving a lopsided offer, which agent B is compelled to accept, a situation similar to the ultimatum game (G ¨ uth et al., 1982).

Original: To eliminate this effect, we sample N between 4 and 10 at each round, according to a truncated Poisson distribution with mean 7.
Revised: To eliminate this effect, we sample N between 4 and 10 at each round, according to a truncated Poisson distribution with mean 7.

Original: 2. C OMMUNICATION CHANNELS
Revised: 2. C OMMUNICATION CHANNELS

Original: To achieve negotiation, the agents need to communicate.
Revised: To achieve negotiation, the agents need to communicate.

Original: One obvious communication protocol is to directly transmit the proposed division of items p j .
Revised: One obvious communication protocol is to directly transmit the proposed division of items p j .

Original: We refer to this as the proposal channel.
Revised: We refer to this as the proposal channel.

Original: This communication channel is task-speciﬁc and pertains to the action space of the agents used to derive their pay-offs.
Revised: This communication channel is task-speciﬁc and pertains to the action space of the agents used to derive their pay-offs.

Original: Consequently, the information bandwidth is restricted and grounded in the action space of the game.
Revised: Consequently, the information bandwidth is restricted and grounded in the action space of the game.

Original: Motivated by work on emergent communication, we also give our agents the ability of transmitting strings of arbitrary symbols with no a priori grounding.
Revised: Motivated by work on emergent communication, we also give our agents the ability of transmitting strings of arbitrary symbols with no a priori grounding.

Original: We call this the linguistic channel, a speciﬁc instantiation of the more general concept of cheap talk.
Revised: We call this the linguistic channel, a speciﬁc instantiation of the more general concept of cheap talk.

Original: This channel differs from the proposal channel in two key properties:
Revised: This channel differs from the proposal channel in two key properties:

Original: • Non-bindingness: Messages sent via this channel do not commit the sender to any course of action, unlike directly transmitting a proposal which binds the sender to the proposal.
Revised: • Non-bindingness: Messages sent via this channel do not commit the sender to any course of action, unlike directly transmitting a proposal which binds the sender to the proposal.

Original: • Unveriﬁability: There is no inherent link between the linguistic utterance and the private proposal made, meaning that the agents could potentially lie.
Revised: • Unveriﬁability: There is no inherent link between the linguistic utterance and the private proposal made, meaning that the agents could potentially lie.

Original: We are interested in whether and under what circumstances the agents can make use of this channel to facilitate negotiation and establish a common ground for the symbols.
Revised: We are interested in whether and under what circumstances the agents can make use of this channel to facilitate negotiation and establish a common ground for the symbols.

Original: In our experiments, we consider 4 separate communication conﬁgurations: only the proposal channel, only the linguistic channel, both channels open, and no communication at all.
Revised: In our experiments, we consider 4 separate communication conﬁgurations: only the proposal channel, only the linguistic channel, both channels open, and no communication at all.

Original: 1 If a channel is closed, we replace any messages in that channel with a ﬁxed dummy symbol to ensure no information is transmitted.
Revised: 1 If a channel is closed, we replace any messages in that channel with a ﬁxed dummy symbol to ensure no information is transmitted.

Original: If the proposal channel is closed, we still use the decoded proposal to calculate the item division, but this information is not revealed to the opponent agent.
Revised: If the proposal channel is closed, we still use the decoded proposal to calculate the item division, but this information is not revealed to the opponent agent.

Original: 2. A GENT SOCIALITY AND REWARD SCHEMES
Revised: 2. A GENT SOCIALITY AND REWARD SCHEMES

Original: While purely self-interested agents may learn to divide up items so that only each individual is satisﬁed, this may not lead to an optimal joint allocation of items, where each item goes to the agent with the highest utility for that item.
Revised: While purely self-interested agents may learn to divide up items so that only each individual is satisﬁed, this may not lead to an optimal joint allocation of items, where each item goes to the agent with the highest utility for that item.

Original: We introduce a “prosocial” reward R that is shared across A and B, which is the sum of the agents’ individual (selﬁsh) rewards,
Revised: We introduce a “prosocial” reward R that is shared across A and B, which is the sum of the agents’ individual (selﬁsh) rewards,

Original: i.e., R = R A + R B .
Revised: i.e., R = R A + R B .

Revised: baseline to test whether the agents learn to improve over simple heuristics, particularly in the case of two prosocial agents, where a simple strategy such as one agent taking all items with non-zero utility may do unexpectedly well.

Original: Prosocial agents are incentivised to communicate, as ﬁnding the optimal joint allocation requires communicating the hidden values between the agents.
Revised: Prosocial agents are incentivised to communicate, as ﬁnding the optimal joint allocation requires communicating the hidden values between the agents.

Original: These different reward schemes can be seen as speciﬁc instantiations of a more general reward formula of the form R = αR
Revised: These different reward schemes can be seen as speciﬁc instantiations of a more general reward formula of the form R = αR

Original: A + βR B (Peysakhovich & Lerer, 2017).
Revised: A + βR B (Peysakhovich & Lerer, 2017).

Original: For the selﬁsh agent, α = 1 and β = 0 , while for the prosocial agent α = β = 1 .
Revised: For the selﬁsh agent, α = 1 and β = 0 , while for the prosocial agent α = β = 1 .

Original: 2. A GENT ARCHITECTURE AND LEARNING
Revised: A GENT ARCHITECTURE AND LEARNING

Original: At each timestep t , the proposer receives three inputs:
Revised: At each timestep t , the proposer receives three inputs:

Original: • The item context c j =
Revised: • The item context c j =

Original: [ i ; u j ] , a concatenation of the item pool and the proposer’s utilities.
Revised: [ i ; u j ] , a concatenation of the item pool and the proposer’s utilities.

Original: • The utterance m t − 1 produced by the other agent in the previous timestep t − 1 .
Revised: • The utterance m t − 1 produced by the other agent in the previous timestep t − 1 .

Original: If the linguistic channel is closed, this is simply a dummy message.
Revised: If the linguistic channel is closed, this is simply a dummy message.

Original: • The proposal p t − 1 made by the other agent in the previous timestep t − 1 .
Revised: • The proposal p t − 1 made by the other agent in the previous timestep t − 1 .

Original: If the proposal channel is closed, this is again a dummy proposal.
Revised: If the proposal channel is closed, this is again a dummy proposal.

Original: First, the discrete inputs are turned into dense vectors through an embedding table.
Revised: First, the discrete inputs are turned into dense vectors through an embedding table.

Original: We use two embedding tables, one for the item context and the previous proposal, and a separate one for the previous utterance.
Revised: We use two embedding tables, one for the item context and the previous proposal, and a separate one for the previous utterance.

Original: This is essential as the item context and the previous proposal have predeﬁned numeric semantics, distinct from the linguistic semantics of the utterances’ symbols.
Revised: This is essential as the item context and the previous proposal have predeﬁned numeric semantics, distinct from the linguistic semantics of the utterances’ symbols.

Original: Then, each of the input sequences is encoded using an LSTM (Hochreiter & Schmidhuber, 1997), one for each input.
Revised: Then, each of the input sequences is encoded using an LSTM (Hochreiter & Schmidhuber, 1997), one for each input.

Original: This results in 3 ﬁxed-size vectors: h ct , h mt and h pt .
Revised: This results in 3 ﬁxed-size vectors: h ct , h mt and h pt .

Original: These vectors are concatenated and fed through a feedforward layer, followed by a ReLU non-linearity (Nair & Hinton, 2010), to give h t , the hidden state of the agent at timestep t .
Revised: These vectors are concatenated and fed through a feedforward layer, followed by a ReLU non-linearity (Nair & Hinton, 2010), to give h t , the hidden state of the agent at timestep t .

Original: This hidden state is then used to initialise the policies for the actions the agent can take:
Revised: This hidden state is then used to initialise the policies for the actions the agent can take:

Original: • π term is the policy for the termination action.
Revised: • π term is the policy for the termination action.

Original: If this action is taken by an agent, both agents receive reward according to the last proposal made by the other agent.
Revised: If this action is taken by an agent, both agents receive reward according to the last proposal made by the other agent.

Original: This is a binary decision, and we parametrise π term as a single feedforward layer, with the hidden state as input, followed by a sigmoid function, to represent the probability of termination.
Revised: This is a binary decision, and we parametrise π term as a single feedforward layer, with the hidden state as input, followed by a sigmoid function, to represent the probability of termination.

Original: • π utt is the policy for the linguistic utterances.
Revised: • π utt is the policy for the linguistic utterances.

Original: This is parametrised by an LSTM, which takes the hidden state of the agent as the initial hidden state.
Revised: This is parametrised by an LSTM, which takes the hidden state of the agent as the initial hidden state.

Original: For the ﬁrst timestep, a dummy symbol is fed in as input; subsequently, the model prediction from the previous timestep is fed in as input at the next timestep, in order to predict the next symbol.
Revised: For the ﬁrst timestep, a dummy symbol is fed in as input; subsequently, the model prediction from the previous timestep is fed in as input at the next timestep, in order to predict the next symbol.

Original: • π prop is the policy for the proposals the agent generates.
Revised: • π prop is the policy for the proposals the agent generates.

Original: This is parametrised by 3 separate feedforward neural networks, one for each item type, which each take as input h t and output a distribution over { 0 ... 5 } indicating the proposal for that item.
Revised: This is parametrised by 3 separate feedforward neural networks, one for each item type, which each take as input h t and output a distribution over { 0 ... 5 } indicating the proposal for that item.

Original: The overall policy for the agent, π ∗ , is a combination of the separate policies.
Revised: The overall policy for the agent, π ∗ , is a combination of the separate policies.

Original: The action that the agent takes at turn t can be summarised by a triple ( e t , m t , p t ) , where e t is a binary variable indicating whether the agent took the termination action, m t is a sequence of symbols produced from π utt , and p t is a proposal produced by π prop .
Revised: The action that the agent takes at turn t can be summarised by a triple ( e t , m t , p t ) , where e t is a binary variable indicating whether the agent took the termination action, m t is a sequence of symbols produced from π utt , and p t is a proposal produced by π prop .

Original: A negotiation can then be thought of as a sequence of triples τ = (cid:104) ( e t , u t , p t ) | 1 ≤ t ≤ N (cid:105) , with the agents alternating turns.
Revised: A negotiation can then be thought of as a sequence of triples τ = (cid:104) ( e t , u t , p t ) | 1 ≤ t ≤ N (cid:105) , with the agents alternating turns.

Original: During training, each agent i tries to independently ﬁnd the policy π ∗ i that maximises the following objective function:
Revised: During training, each agent i tries to independently ﬁnd the policy π ∗ i that maximises the following objective function:

Original: [Equation]
Revised: [Equation]

Original:  where R i ( τ ) is the reward agent i receives from the trajectory τ and H ( π i ) is an entropy regularisation term to encourage the agent to explore during training.
Revised: (1) where R i ( τ ) is the reward agent i receives from the trajectory τ and H ( π i ) is an entropy regularisation term to encourage the agent to explore during training.

Original: Note that each agent has its own objective function, and the objectives of both agents are coupled by the trajectory sampled from both agents’ policies.
Revised: Note that each agent has its own objective function, and the objectives of both agents are coupled by the trajectory sampled from both agents’ policies.

Original: At test time, instead of sampling from the policy π , we take the action with the highest probability.
Revised: At test time, instead of sampling from the policy π , we take the action with the highest probability.

Original: Parameters are updated using the REINFORCE (Williams, 1992) update rule with an exponentially smoothed mean baseline.
Revised: Parameters are updated using the REINFORCE (Williams, 1992) update rule with an exponentially smoothed mean baseline.

Original: Full hyperparameter details are in the appendix.
Revised: Full hyperparameter details are in [Figure]

Original:  E XPERIMENT 1: C AN SELF - INTERESTED AGENTS LEARN TO NEGOTIATE ? 3. E XPERIMENT DESCRIPTION
Revised:  E XPERIMENT 1: C AN SELF - INTERESTED AGENTS LEARN TO NEGOTIATE ? 3. E XPERIMENT DESCRIPTION

Original: In our ﬁrst experiment, we test whether purely self-interested agents can learn to negotiate and divide up items fairly, and investigate the effect of the various communication channels on negotiation success.
Revised: In our ﬁrst experiment, we test whether purely self-interested agents can learn to negotiate and divide up items fairly, and investigate the effect of the various communication channels on negotiation success.

Original: We train self-interested agents to negotiate for 500k episodes.
Revised: We train self-interested agents to negotiate for 500k episodes.

Original: Each episode corresponds to a batch of 128 games, each with item pools and hidden utilities generated as described in Section 2.1.
Revised: Each episode corresponds to a batch of 128 games, each with item pools and hidden utilities generated as described in Section 2.1.

Original: We also hold out 5 batches worth of games as a test set, and test the agents every 50 episodes, in order to measure the robustness of the negotiating behaviour to unseen environments.[Figure]
Revised: We also hold out 5 batches worth of games as a test set, and test the agents every 50 episodes, in order to measure the robustness of the negotiating behaviour to unseen environments.

Original: 3. R ESULTS
Revised: 3. R ESULTS

Original: Self-interested agents can learn to negotiate fairly Figure 2a shows that self-interested agents learn to divide up the items fairly when they exchange proposals directly.
Revised: Self-interested agents can learn to negotiate fairly Figure 2a shows that self-interested agents learn to divide up the items fairly when they exchange proposals directly.

Original: We see that the proportion of total utility each agent receives is roughly equal, and above 50%, suggesting that the agents have learnt to keep items with higher utility, while giving away items with low utility.
Revised: We see that the proportion of total utility each agent receives is roughly equal, and above 50%, suggesting that the agents have learnt to keep items with higher utility, while giving away items with low utility.

Original: In addition, the agents seem to show evidence of compromise: there is a drop of 0.24 between the value of the proposal that agent 1 makes at the beginning of negotiation, and the ﬁnal reward it receives; for agent 2, the corresponding drop is 0.18.
Revised: In addition, the agents seem to show evidence of compromise: there is a drop of 0.24 between the value of the proposal that agent 1 makes at the beginning of negotiation, and the ﬁnal reward it receives; for agent 2, the corresponding drop is 0.18.

Original: Furthermore, Figure 2b shows the mean optimality of the proposal made at each turn of the negotiation averaged over 1280 games, calculated as the sum of the reward of both agents divided by the maximum possible reward.
Revised: Furthermore, Figure 2b shows the mean optimality of the proposal made at each turn of the negotiation averaged over 1280 games, calculated as the sum of the reward of both agents divided by the maximum possible reward.

Original: This value increases with more rounds of negotiation, suggesting that the agents are adjusting their proposals based on information from the other agent.
Revised: This value increases with more rounds of negotiation, suggesting that the agents are adjusting their proposals based on information from the other agent.

Original: We show an illustrative example of two self-interested agents negotiating with the proposal channel in Table 2.
Revised: Table 1 presents an illustrative example of two self-interested agents negotiating using the proposal channel.

Original: We observe that the initial proposals made by both agents are over-optimistic, and that they compromise more with more negotiating turns as they discover what the other agent values.
Revised: While the initial proposals made by both agents are over-optimistic, over the course of the negotiation, they compromise to less items.

Original: This allows them to make more jointly optimal proposals, as seen in Figure 2b.
Revised: This allows them to achieve higher joint reward, as seen in Figure 2b, a clear example of “negotiation”.

Original: Self-interested agents do not appear to ground cheap talk When using the linguistic channel, agents do not negotiate optimally.
Revised: Self-interested agents do not appear to ground cheap talk When using the linguistic channel, agents do not negotiate optimally.

Original: Instead, the agents randomly alternate taking all the items, which is borne out by the oscillations in Figure 2a.
Revised: Instead, the agents randomly alternate taking all the items, which is borne out by the oscillations in Figure 2a.

Original: Indeed, examination of the messages exchanged by self-interested agents show they predominantly consist of a single symbol – see Figures 5a and 5b in the appendix for more details.
Revised: Indeed, examination of the messages exchanged by self-interested agents show they predominantly consist of a single symbol – see Figures 5a and 5b in the appendix for more details.

Original: This result suggests that the self-interested agents we implement  ﬁnd it difﬁcult to cheap talk to exchange meaningful information about how to divide up the items, which is in line with the theoretical analysis of cheap talk in Crawford & Sobel (1982), who ﬁnd that, once agent interests diverge by a ﬁnite amount, no communication is to be expected.
Revised: This result suggests that the self-interested agents we implement [Table] [Table] ﬁnd it difﬁcult to cheap talk to exchange meaningful information about how to divide up the items, which is in line with the theoretical analysis of cheap talk in Crawford & Sobel (1982), who ﬁnd that, once agent interests diverge by a ﬁnite amount, no communication is to be expected.

Original: Our ﬁndings also corroborate the hypothesis found in the language evolution literature that cooperation is a prerequisite for the emergence of language (Nowak & Krakauer, 1999).
Revised: Our ﬁndings also corroborate the hypothesis found in the language evolution literature that cooperation is a prerequisite for the emergence of language (Nowak & Krakauer, 1999).

Original: We conjecture that this ﬁnding is partially due to the fact that the agents operate in a non-iterated environment (i.e., they have no explicit memory of previous interactions).
Revised: We conjecture that this ﬁnding is partially due to the fact that the agents operate in a non-iterated environment (i.e., they have no explicit memory of previous interactions).

Original: In the classic prisoner’s dilemma game, rational agents playing a single-shot version of the game learn to always defect, whereas rational agents playing iterated prisoner’s dilemma can learn more mutually beneﬁcial cooperative strategies under the right conditions (Sandholm & Crites, 1996; Wunder et al., 2010).
Revised: In the classic prisoner’s dilemma game, rational agents playing a single-shot version of the game learn to always defect, whereas rational agents playing iterated prisoner’s dilemma can learn more mutually beneﬁcial cooperative strategies under the right conditions (Sandholm & Crites, 1996; Wunder et al., 2010).

Original: More sophisticated players will often have an incentive to take a short-term loss in reward to teach[Table] [Table] or punish another player for long-term gain in the repeated setting (Fudenberg & Levine, 1998).
Revised: More sophisticated players will often have an incentive to take a short-term loss in reward to teach or punish another player for long-term gain in the repeated setting (Fudenberg & Levine, 1998).

Original: We suspect that, by iterating the game, there is an incentive for the cheap talk to be more veriﬁable, as there is a more credible threat of punishment if the accepted proposal and the linguistic utterance do not correspond.
Revised: We suspect that, by iterating the game, there is an incentive for the cheap talk to be more veriﬁable, as there is a more credible threat of punishment if the accepted proposal and the linguistic utterance do not correspond.

Original:  E XPERIMENT 2 - C AN PROSOCIAL AGENTS LEARN TO COORDINATE ? 4. E XPERIMENT DESCRIPTION
Revised:  E XPERIMENT 2 - C AN PROSOCIAL AGENTS LEARN TO COORDINATE ? 4. E XPERIMENT DESCRIPTION

Original: In the previous experiment, we showed that self-interested agents can divide up items fairly such that each agent gets equal reward.
Revised: In the previous experiment, we showed that self-interested agents can divide up items fairly such that each agent gets equal reward.

Original: However, this does not correspond to an optimal joint allocation where items go to the agent with the higher utility for that item (equivalently, maximising the sum of both agents’ rewards).
Revised: However, this does not correspond to an optimal joint allocation where items go to the agent with the higher utility for that item (equivalently, maximising the sum of both agents’ rewards).

Original: The prosocial reward scheme aligns exactly with this optimal outcome, as learning to negotiate equates to ﬁnding the optimal joint item allocation.
Revised: The prosocial reward scheme aligns exactly with this optimal outcome, as learning to negotiate equates to ﬁnding the optimal joint item allocation.

Original: In this setting, there is an in-built bias towards communication, as solving the game optimally requires pooling hidden utilities.
Revised: In this setting, there is an in-built bias towards communication, as solving the game optimally requires pooling hidden utilities.

Original: Table 1 reports the turns taken as well as the joint reward optimality , i.e., the sum of both agents’ rewards, divided by the maximum possible reward given the item context, where the latter corresponds to each agent getting all the items for which they have a higher utility.[Table]
Revised: Table 2 reports the turns taken as well as the joint reward optimality , i.e., the sum of both agents’ rewards, divided by the maximum possible reward given the item context, where the latter corresponds to each agent getting all the items for which they have a higher utility.

Original: [Table]
Revised: [Table]

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Cheap talk helps agents coordinate For prosocial agents with aligned interests, theoretical results suggest that communication using cheap talk is a Nash equilibrium.
Revised: Cheap talk helps agents coordinate For prosocial agents with aligned interests, theoretical results suggest that communication using cheap talk is a Nash equilibrium.

Original: This is indeed what we observe, as the linguistic channel results in much better task success than any other communication scheme.
Revised: This is indeed what we observe, as the linguistic channel results in much better task success than any other communication scheme.

Original: The information bandwidth of the proposal channel is limited by the type and number of items, whereas the linguistic channel’s bandwidth is unconstrained and can (in theory) be arbitrarily large, helping the effective exchange of task-speciﬁc information.
Revised: The information bandwidth of the proposal channel is limited by the type and number of items, whereas the linguistic channel’s bandwidth is unconstrained and can (in theory) be arbitrarily large, helping the effective exchange of task-speciﬁc information.

Original: In addition, the variance in joint optimality is much lower for the linguistic channel, suggesting it is more robust.
Revised: In addition, the variance in joint optimality is much lower for the linguistic channel, suggesting it is more robust.

Original: We suspect that difﬁculties in optimisation cause the results for both channels open to be similar to those for only the proposal channel open, as the proposal channel is pre-grounded, which poses a strong local maxima.
Revised: We suspect that difﬁculties in optimisation cause the results for both channels open to be similar to those for only the proposal channel open, as the proposal channel is pre-grounded, which poses a strong local maxima.

Original: With better optimisation, we believe that the performance of both channels open will more closely match those of just the linguistic channel open.
Revised: With better optimisation, we believe that the performance of both channels open will more closely match those of just the linguistic channel open.

Original: Table 3 illustrates an example of negotiation between the two prosocial agents.
Revised: Table 3 illustrates an example of negotiation between the two prosocial agents.

Original: Interestingly, we observe that agent A over the course of the negotiation game adjusts its proposal (which is however hidden and not revealed) based on the utterances of agent B.
Revised: Interestingly, we observe that agent A over the course of the negotiation game adjusts its proposal (which is however hidden and not revealed) based on the utterances of agent B.

Original: Prosocial agents can still use the proposal channel to co-ordinate With random termination, agents communicating with the proposal channel do even worse than the no-communication baseline.
Revised: Prosocial agents can still use the proposal channel to co-ordinate With random termination, agents communicating with the proposal channel do even worse than the no-communication baseline.

Original: However, when given the full 10 turns to exchange information (see Figure 7 in the appendix), prosocial agents manage to outperform the no-communication baseline.
Revised: However, when given the full 10 turns to exchange information (see Figure 7 in the appendix), prosocial agents manage to outperform the no-communication baseline.

Original: This suggests that the agents are slowly learning to repurpose the proposal channel to transmit information, i.e., they communicate directly on the action space.
Revised: This suggests that the agents are slowly learning to repurpose the proposal channel to transmit information, i.e., they communicate directly on the action space.

Original: Reusing task-speciﬁc actions to transmit information has precedent, such as in bridge bidding (Simon, 1949), and it is interesting to observe that agents learn to develop a codebook without the need for prior agreement.
Revised: Reusing task-speciﬁc actions to transmit information has precedent, such as in bridge bidding (Simon, 1949), and it is interesting to observe that agents learn to develop a codebook without the need for prior agreement.

Original:  A NALYSIS OF LINGUISTIC COMMUNICATION 5. S YMBOL USAGE
Revised:  A NALYSIS OF LINGUISTIC COMMUNICATION 5. S YMBOL USAGE

Original: The symbol unigram and bigram distributions of messages exchanged by prosocial agents (see Figures 3a and 3b respectively) show that agent A, the agent who initiates the negotiation, is not transmitting any information using the linguistic channel.
Revised: The symbol unigram and bigram distributions of messages exchanged by prosocial agents (see Figures 3a and 3b respectively) show that agent A, the agent who initiates the negotiation, is not transmitting any information using the linguistic channel.

Original: On the other hand, agent B uses a diversity of symbols resulting in a long-tailed bigram symbol distribution, reminiscent of the Zipﬁan distribution found in natural languages.
Revised: On the other hand, agent B uses a diversity of symbols resulting in a long-tailed bigram symbol distribution, reminiscent of the Zipﬁan distribution found in natural languages.

Original: This suggests that, even though the task is symmetric, the agents differentiate into speaker and listener roles, similar to other co-ordination games of communication, such as the reference game.
Revised: This suggests that, even though the task is symmetric, the agents differentiate into speaker and listener roles, similar to other co-ordination games of communication, such as the reference game.

Original: Thus, they have adopted a simple strategy for solving the game: an agent shares its utilities (the speaker), while the other (the listener) compares the shared utilities with their own and generates the optimal proposal.
Revised: Thus, they have adopted a simple strategy for solving the game: an agent shares its utilities (the speaker), while the other (the listener) compares the shared utilities with their own and generates the optimal proposal.

Original: In contrast, the selﬁsh agents do not show evidence of grounded symbol usage.
Revised: In contrast, the selﬁsh agents do not show evidence of grounded symbol usage.

Original: The unigram and bigram statistics show that most messages consist solely of strings of a single ﬁxed symbol, regardless of the item context, and hence no information is being exchanged.
Revised: The unigram and bigram statistics show that most messages consist solely of strings of a single ﬁxed symbol, regardless of the item context, and hence no information is being exchanged.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: 5. C ONTENT OF THE MESSAGES
Revised: 5. C ONTENT OF THE MESSAGES

Original: To interpret the information that is transmitted through the linguistic channel, we use the message transcript and the item pool to create probe classiﬁers that predict the hidden utilities of each agent and the (hidden) accepted proposal.
Revised: To interpret the information that is transmitted through the linguistic channel, we use the message transcript and the item pool to create probe classiﬁers that predict the hidden utilities of each agent and the (hidden) accepted proposal.

Original: We use an LSTM to encode the sequence of symbols belonging to the message transcript into a vector, and another LSTM to encode the item pool into a vector.
Revised: We use an LSTM to encode the sequence of symbols belonging to the message transcript into a vector, and another LSTM to encode the item pool into a vector.

Original: The resulting two vectors are then concatenated and used to predict the hidden utility functions and the accepted proposal using 9 linear classiﬁers, one for each item (3 each for agent A and agent B’s utilities and 3 for the proposal).
Revised: The resulting two vectors are then concatenated and used to predict the hidden utility functions and the accepted proposal using 9 linear classiﬁers, one for each item (3 each for agent A and agent B’s utilities and 3 for the proposal).

Original: We conduct 10-fold cross-validation and report averaged per-item accuracy (see Table 4).
Revised: We conduct 10-fold cross-validation and report averaged per-item accuracy (see Table 4).

Original: We also include two baselines.
Revised: We also include two baselines.

Original: The ﬁrst, reported as the “random baseline” in Table 4, attempts to predict the accepted proposal and agent utilities from a message transcript and item pool of all 0’s (indicating chance performance at the task).
Revised: The ﬁrst, reported as the “random baseline” in Table 4, attempts to predict the accepted proposal and agent utilities from a message transcript and item pool of all 0’s (indicating chance performance at the task).

Original: The other predicts the accepted proposal from the item pool and a message transcript of all 0’s.
Revised: The other predicts the accepted proposal from the item pool and a message transcript of all 0’s.

Original: This shows how much additional information about the proposal is contained in the message transcript.
Revised: This shows how much additional information about the proposal is contained in the message transcript.

Original: We can see from Table 4 that not only is there variety in the symbols used, but the messages also have semantic content: they contain information about the hidden utilities of agent B, and about the proposal that was made.
Revised: We can see from Table 4 that not only is there variety in the symbols used, but the messages also have semantic content: they contain information about the hidden utilities of agent B, and about the proposal that was made.

Original: This shows that our agents have learned to give meaning to the symbols, and can use them to transmit information.
Revised: This shows that our agents have learned to give meaning to the symbols, and can use them to transmit information.

Original: However, the purely self-interested agents do not seem to transmit meaningful information using the linguistic channel, and thus do not appear to ground the symbols.
Revised: However, the purely self-interested agents do not seem to transmit meaningful information using the linguistic channel, and thus do not appear to ground the symbols.

Original:  E XPERIMENT 3: A SOCIETY OF AGENTS
Revised:  E XPERIMENT 3: A SOCIETY OF AGENTS

Original: In more realistic scenarios, we learn and practice negotiation in environments with diverse agent populations and levels of prosociality.
Revised: In more realistic scenarios, we learn and practice negotiation in environments with diverse agent populations and levels of prosociality.

Original: In this setting, maximising one’s reward requires identifying which agents are most useful in achieving one’s aims.
Revised: In this setting, maximising one’s reward requires identifying which agents are most useful in achieving one’s aims.

Original: In practice, this involves identifying which agents are the most prosocial, as they are both the most exploitable for self-interested agents, and the most cooperative for other prosocial agents.
Revised: In practice, this involves identifying which agents are the most prosocial, as they are both the most exploitable for self-interested agents, and the most cooperative for other prosocial agents.

Original: 6. E XPERIMENTAL PROTOCOL
Revised: 6. E XPERIMENTAL PROTOCOL

Original: We trained a ﬁxed agent against a community of 10 agents, with varying proportions of selﬁsh and prosocial agents.
Revised: We trained a ﬁxed agent against a community of 10 agents, with varying proportions of selﬁsh and prosocial agents.

Original: For each training episode, we randomly sample one agent from the community, play a batch of negotiation games, and then update both agents.
Revised: For each training episode, we randomly sample one agent from the community, play a batch of negotiation games, and then update both agents.

Original: We do this for both ﬁxed agent A (the agent who goes ﬁrst) and for ﬁxed agent B (the agent who goes second).
Revised: We do this for both ﬁxed agent A (the agent who goes ﬁrst) and for ﬁxed agent B (the agent who goes second).

Original: We experiment with either 1 or 5 prosocial agents, which communication channels are open, and also whether the agents are identiﬁable.
Revised: We experiment with either 1 or 5 prosocial agents, which communication channels are open, and also whether the agents are identiﬁable.

Original: At test time, we care primarily about whether self-interested agents can exploit prosocial agents and whether prosocial agents can cooperate with other prosocial agents.
Revised: At test time, we care primarily about whether self-interested agents can exploit prosocial agents and whether prosocial agents can cooperate with other prosocial agents.

Original: Hence, we only test the ﬁxed agent against the prosocial agents in the community.
Revised: Hence, we only test the ﬁxed agent against the prosocial agents in the community.

Original: We generate 10 batches of 128 games each batch, and play each game with every pair of ﬁxed agent and community prosocial agent.
Revised: We generate 10 batches of 128 games each batch, and play each game with every pair of ﬁxed agent and community prosocial agent.

Original: We average the results across all pairs and all games, and show the mean and standard deviations of the rewards obtained in this experiment in Table 5.
Revised: We average the results across all pairs and all games, and show the mean and standard deviations of the rewards obtained in this experiment in Table 5.

Original: If opposing agents are identiﬁable (i.e. each agent has a name-tag), the ﬁxed agent receives this information as a one-hot vector, which is then used to look up an embedding table with one embedding per opposing agent.
Revised: If opposing agents are identiﬁable (i.e. each agent has a name-tag), the ﬁxed agent receives this information as a one-hot vector, which is then used to look up an embedding table with one embedding per opposing agent.

Original: This opponent embedding is then concatenated with the encoded inputs, before being passed through the feedforward layer to get the agent’s hidden state.
Revised: This opponent embedding is then concatenated with the encoded inputs, before being passed through the feedforward layer to get the agent’s hidden state.

Original: Intuitively, the induced agent embedding should capture information about the agent’s negotiating behaviour.
Revised: Intuitively, the induced agent embedding should capture information about the agent’s negotiating behaviour.

Original: 6. A NALYSIS
Revised: 6. A NALYSIS

Original: Identifying agents Our results show that providing agent ID information does help the ﬁxed agent achieve its aims.
Revised: Identifying agents Our results show that providing agent ID information does help the ﬁxed agent achieve its aims.

Original: This effect is particularly pronounced for a selﬁsh ﬁxed agent; here, providing ID information uniformly improves performance.
Revised: This effect is particularly pronounced for a selﬁsh ﬁxed agent; here, providing ID information uniformly improves performance.

Original: For cooperative agents, the results mixed; for a ﬁxed agent A communicating via the linguistic channel, having agent ID information harms performance.
Revised: For cooperative agents, the results mixed; for a ﬁxed agent A communicating via the linguistic channel, having agent ID information harms performance.

Original: Indeed, the only prosocial agents who beat the no communication baseline (which achieved an average joint optimal reward of 0.95) was when agent A was ﬁxed and ID information was not provided.
Revised: Indeed, the only prosocial agents who beat the no communication baseline (which achieved an average joint optimal reward of 0.95) was when agent A was ﬁxed and ID information was not provided.

Original: We also computed the 2D PCA projections of the learnt opponent embeddings for a ﬁxed agent A under a variety of channels and prosocial levels (see Figure 4).
Revised: We also computed the 2D PCA projections of the learnt opponent embeddings for a ﬁxed agent A under a variety of channels and prosocial levels (see Figure 4).

Original: Even in cases where the agent ID does not aid negotiation, we ﬁnd that the embeddings cluster according to the reward scheme of the other agent.
Revised: Even in cases where the agent ID does not aid negotiation, we ﬁnd that the embeddings cluster according to the reward scheme of the other agent.

Original: This shows that the agents can distinguish different agents by their reward schemes from purely observing negotiating behaviour.
Revised: This shows that the agents can distinguish different agents by their reward schemes from purely observing negotiating behaviour.

Original: Community linguistic phenomena In one of our experimental settings, a community of prosocial agents developed a language and were able to use this to achieve better negotiation success.This
Revised: Community linguistic phenomena In one of our experimental settings, a community of prosocial agents developed a language and were able to use this to achieve better negotiation success.

Original: [Table]

Original: prosocial ﬁxed agent, the numbers report joint reward.

Original: The starred result indicates when a community has made use of the linguistic channel to transmit information.

Original:  is the outcome in the starred cell in Table 5.
Revised: This is the outcome in the starred cell in Table 5.

Original: For a visualisation of the bigram usage statistics in each interaction pair in this community, see Figure 6.
Revised: For a visualisation of the bigram usage statistics in each interaction pair in this community, see Figure 6.

Original: Interestingly, the only situation when agents developed a language was when ID information was not provided.
Revised: Interestingly, the only situation when agents developed a language was when ID information was not provided.

Revised: We suspect this result is due to a

Revised: [Table]

Revised: [Figure]

Original: We suspect this result is due to a poor local optimum; since the prosocial agents can learn to use IDs to distinguish prosocial agents from the selﬁsh agents they can ignore the linguistic utterance.
Revised: [Table] [Figure] [Table] poor local optimum; since the prosocial agents can learn to use IDs to distinguish prosocial agents from the selﬁsh agents they can ignore the linguistic utterance.

Original: Moreover, we ﬁnd that when prosocial agents make use of the linguistic channel, the communication protocol differs within the community.
Revised: Moreover, we ﬁnd that when prosocial agents make use of the linguistic channel, the communication protocol differs within the community.

Original: We calculate the Spearman correlation ρ between bigram ranks of different pairs of agents, all of them with the same ﬁxed agent A, and show the results in
Revised: We calculate the Spearman correlation ρ between bigram ranks of different pairs of agents, all of them with the same ﬁxed agent A, and show the results in

Original: Table 6.
Revised: Table 6.

Original: Even though all pairs of agents share the same ﬁxed agent A, different pairs of agents learn to use bigrams in a different way, resulting in relatively low correlations between -0.22 and 0.27.
Revised: Even though all pairs of agents share the same ﬁxed agent A, different pairs of agents learn to use bigrams in a different way, resulting in relatively low correlations between -0.22 and 0.27.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  D ISCUSSION

Original: We showed that by communicating through a veriﬁable and binding communication channel, selfinterested agents can learn to negotiate fairly by reinforcement learning, using only task success as the reward signal.
Revised: We showed that by communicating through a veriﬁable and binding communication channel, selfinterested agents can learn to negotiate fairly by reinforcement learning, using only task success as the reward signal.

Original: Moreover, cheap talk facilitated negotiation in prosocial but not in self-interested agents, corroborating theoretical results from the game theory literature (Crawford & Sobel, 1982).
Revised: Moreover, cheap talk facilitated negotiation in prosocial but not in self-interested agents, corroborating theoretical results from the game theory literature (Crawford & Sobel, 1982).

Original: An interesting future direction of research would be to investigate whether self-interested agents can learn to make use of cheap talk, such as modifying the learning algorithm
Revised: An interesting future direction of research would be to investigate whether cheap talk can be made to emerge out of self-interested agents interacting. Recent encouraging results by Crandall et al.

Original: (Foerster et al., 2017).
Revised: (2018) show that communication can help agents cooperate. However, their signalling mechanism is heavily engineered: the speech acts are predeﬁned and the consequences of the speech acts on the observed behaviour are deterministically hard-coded. It would be interesting to see whether a learning algorithm, such as Foerster et al. (2017), can discover the same result.

Original: A related paper from Lewis et al.
Revised: A related paper from Lewis et al.

Original: (2017) takes a top-down approach to learning to negotiate by leveraging dialogue data.
Revised: (2017) takes a top-down approach to learning to negotiate by leveraging dialogue data.

Original: We demonstrated a bottom up alternative towards learning communicative behaviours directly from interaction with peers.
Revised: We demonstrated a bottom up alternative towards learning communicative behaviours directly from interaction with peers.

Original: This opens up the exciting possibility of learning domain-speciﬁc reasoning capabilities from interaction, while having a general-purpose language layer at the top producing natural language.
Revised: This opens up the exciting possibility of learning domain-speciﬁc reasoning capabilities from interaction, while having a general-purpose language layer at the top producing natural language.

Revised: A CKNOWLEDGEMENTS

Revised: We would like to thank Mike Johanson for his insightful comments on an earlier version of this paper, as well as Karl Moritz Hermann and the rest of the DeepMind language team for many fruitful discussions over the course of the project

Original: A A DDITIONAL FIGURES AND TABLES
Revised: A A DDITIONAL FIGURES AND TABLES

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B H YPERPARAMETER DETAILS
Revised: B H YPERPARAMETER DETAILS

Original: Embedding sizes, and all neural network hidden states, had dimension 100.
Revised: Embedding sizes, and all neural network hidden states, had dimension 100.

Original: We used the ADAM optimizer (Kingma & Ba, 2014), with default parameter settings, to optimize the parameters of each agent.
Revised: We used the ADAM optimizer (Kingma & Ba, 2014), with default parameter settings, to optimize the parameters of each agent.

Original: Each agent had a separate optimizer.
Revised: Each agent had a separate optimizer.

Original: We used a separate value of λ , the entropy regularisation weight hyperparameter, for each policy.
Revised: We used a separate value of λ , the entropy regularisation weight hyperparameter, for each policy.

Original: For π term and π prop , λ = 0 . 05 ; for π utt , λ = 0 .
Revised: For π term and π prop , λ = 0 . 05 ; for π utt , λ = 0 .

Original: The symbol vocabulary size was 11, and the agents were allowed to generate utterances of up to length 6.
Revised: The symbol vocabulary size was 11, and the agents were allowed to generate utterances of up to length 6.

Original: The smoothing constant for the exponential moving average baseline was 0.7 (i.e. if the old baseline value was b old , and the current reward is R , then the new estimate of the baseline is b new = 0 .
Revised: The smoothing constant for the exponential moving average baseline was 0.7 (i.e. if the old baseline value was b old , and the current reward is R , then the new estimate of the baseline is b new = 0 .

Original: 7 b old + 0 .
Revised: 7 b old + 0 .

Original: 3 R ).
Revised: 3 R ).