Original: R EVISITING P OPULATIONS IN M ULTI -A GENT C OMMUNICATION
Revised: R EVISITING P OPULATIONS IN M ULTI -A GENT C OMMUNICATION

Original: Anonymous authors
Revised: Angeliki Lazaridou

Original: Paper under double-blind review
Revised: DeepMind

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite evidence from sociolinguistics that larger groups of speakers tend to develop more structured languages, the use of populations has failed to yield significant benefits in emergent multi-agent communication.
Revised: Despite evidence from sociolinguistics that larger groups of speakers tend to develop more structured languages, the use of populations has failed to yield significant benefits in emergent multi-agent communication.

Original: In this paper we reassess the validity of the standard training protocol and illustrate its limitations.
Revised: In this paper we reassess the validity of the standard training protocol and illustrate its limitations.

Original: Specifically, we analyze population-level communication at the equilibrium in sender-receiver Lewis games.
Revised: Specifically, we analyze population-level communication at the equilibrium in sender-receiver Lewis games.

Original: We find that receivers co-adapt to senders they are interacting with, which limits the effect of the population.
Revised: We find that receivers co-adapt to senders they are interacting with, which limits the effect of the population.

Original: Informed by this analysis, we propose an alternative training protocol based on “partitioning” agents.
Revised: Informed by this analysis, we propose an alternative training protocol based on “partitioning” agents.

Original: Partitioning isolates sender-receiver pairs, limits co-adaptation, and results in a new global optimization objective where agents maximize (1) their respective ”internal” communication accuracy and (2) their alignment with other agents.
Revised: Partitioning isolates sender-receiver pairs, limits co-adaptation, and results in a new global optimization objective where agents maximize (1) their respective ”internal” communication accuracy and (2) their alignment with other agents.

Original: In experiments, we find that agents trained in partitioned populations are able to communicate successfully with new agents which they have never interacted with and tend to develop a shared language.
Revised: In experiments, we find that agents trained in partitioned populations are able to communicate successfully with new agents which they have never interacted with and tend to develop a shared language.

Original: Moreover, we observe that larger populations develop languages that are more compositional.
Revised: Moreover, we observe that larger populations develop languages that are more compositional.

Original: Our findings suggest that scaling up to populations in multi-agent communication can be beneficial, but that it matters how we scale up.
Revised: Our findings suggest that scaling up to populations in multi-agent communication can be beneficial, but that it matters how we scale up.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Uncovering the mechanisms that underlie our ability to communicate using language is an important stepping stone towards developing machine learning models that are capable of coordinating and interacting via natural language.
Revised: Uncovering the mechanisms that underlie our ability to communicate using language is an important stepping stone towards developing machine learning models that are capable of coordinating and interacting via natural language.

Original: Over the last few years, there has been increasing interest in simulating the emergence of language using artificial agents trained with reinforcement learning to communicate to achieve a cooperative task (Lazaridou & Baroni, 2020).
Revised: Over the last few years, there has been increasing interest in simulating the emergence of language using artificial agents trained with reinforcement learning to communicate to achieve a cooperative task (Lazaridou & Baroni, 2020).

Original: Typically, agents are trained to perform a variant of the Lewis signaling game (Lewis, 1969; Skyrms, 2010) wherein a sender emits a message describing an object and a receiver attempts to reconstruct the object based on the description.
Revised: Typically, agents are trained to perform a variant of the Lewis signaling game (Lewis, 1969; Skyrms, 2010) wherein a sender emits a message describing an object and a receiver attempts to reconstruct the object based on the description.

Original: This line of work has applications to semi-supervised learning.
Revised: This line of work has applications to semi-supervised learning.

Original: For example, agents that develop languages exhibiting universal properties of natural languages may be used as useful initialization for downstream tasks such as image captioning (Lazaridou et al., 2020) or representation learning (Dess`ı et al., 2021).
Revised: For example, agents that develop languages exhibiting universal properties of natural languages may be used as useful initialization for downstream tasks such as image captioning (Lazaridou et al., 2020) or representation learning (Dess`ı et al., 2021).

Original: Most previous research has focused on communication between a single pair of agents.
Revised: Most previous research has focused on communication between a single pair of agents.

Original: However, there is mounting evidence that the communication protocols developed in this restricted setting become highly specialized and exhibit properties that are at odds with those found in human languages (Bouchacourt & Baroni, 2018; Chaabouni et al., 2019): for example agents are able to solve the task successfully while using languages that are not compositional (Kottur et al., 2017; Chaabouni et al., 2020).
Revised: However, there is mounting evidence that the communication protocols developed in this restricted setting become highly specialized and exhibit properties that are at odds with those found in human languages (Bouchacourt & Baroni, 2018; Chaabouni et al., 2019): for example agents are able to solve the task successfully while using languages that are not compositional (Kottur et al., 2017; Chaabouni et al., 2020).

Original: These idiosyncrasies of the emergent languages can preclude their use in practical applications (Lazaridou et al., 2020).
Revised: These idiosyncrasies of the emergent languages can preclude their use in practical applications (Lazaridou et al., 2020).

Original: As a possible solution, a growing body of work is advocating for scaling up the emergent communication literature to populations of more than two agents communicating simultaneously (Harding Graesser et al., 2019;
Revised: As a possible solution, a growing body of work is advocating for scaling up the emergent communication literature to populations of more than two agents communicating simultaneously (Harding Graesser et al., 2019;

Original: Kim & Oh, 2021; Rita et al., 2022a; Chaabouni et al., 2022).
Revised: Kim & Oh, 2021; Rita et al., 2022a; Chaabouni et al., 2022).

Original: Indeed, there is substantial evidence within the language sciences that population dynamics shape the language structure Raviv et al.
Revised: Indeed, there is substantial evidence within the language sciences that population dynamics shape the language structure Raviv et al.

Original: ; N¨olle et al.
Revised: ; N¨olle et al.

Original: In spite of this fact, several negative results have been obtained, showing that training agents in population yield marginal benefits without explicit pressure towards e.g. population diversity (Rita et al., 2022a) or emulation mechanisms (Chaabouni et al., 2022).
Revised: In spite of this fact, several negative results have been obtained, showing that training agents in population yield marginal benefits without explicit pressure towards e.g. population diversity ( Rita et al., 2022a or emulation mechanisms (Chaabouni et al.

Original: In this paper, we call into question the way such populations are trained.
Revised: In this paper, we call into question the way such populations are trained.

Original: By studying a simple referential game, we evaluate populations on two desirable features observed in natural language:
Revised: By studying a simple referential game, we evaluate populations on two desirable features observed in natural language:

Original: • Agents are able to communicate with new partners within the same population ( Gupta
Revised: • Agents are able to communicate with new partners within the same population (Gupta et al., 2021)

Original: et al. • Larger populations tend to develop more structured languages (N¨olle et al., 2020).
Revised: • Larger populations tend to develop more structured languages (N¨olle et al., 2020).

Original: We provide evidence that populations of artificial agents do not always possess these features (as also attested by previous work, e.g. Kim & Oh (2021); Chaabouni et al.
Revised: We provide evidence that populations of artificial agents do not always possess these features (as also attested by previous work, e.g. Kim & Oh (2021); Chaabouni et al.

Original: To shed light on this phenomenon, we analyze the behaviour of agents in a population at the equilibrium (§2).
Revised: To shed light on this phenomenon, we analyze the behaviour of agents in a population at the equilibrium (§2).

Original: We find that with the standard training procedure, the functional form of the objective is the same as that of a single pair of agents, due to receivers co-adapting to their training partners.
Revised: We find that with the standard training procedure, the functional form of the objective is the same as that of a single pair of agents, due to receivers co-adapting to their training partners.

Original: As our main contribution, we propose an alternative training procedure which partitions sender-receiver pairs and limits co-adaptation of receiver agents (§3).
Revised: As our main contribution, we propose an alternative training procedure which partitions sender-receiver pairs and limits co-adaptation of receiver agents (§3).

Original: We show that this new training paradigm maximizes a different objective at the population level.
Revised: We show that this new training paradigm maximizes a different objective at the population level.

Original: In particular, it explicitly promotes mutual-intelligibility across different agents.
Revised: In particular, it explicitly promotes mutual-intelligibility across different agents.

Original: In experiments, we find that agents trained in partitioned populations are able to communicate successfully with new communication partners with which they have never interacted during training, and that languages spoken by various agents tend to be similar to one another (§5).
Revised: In experiments, we find that agents trained in partitioned populations are able to communicate successfully with new communication partners with which they have never interacted during training, and that languages spoken by various agents tend to be similar to one another (§5).

Original: In addition, we observe that (1) languages developed in partitioned populations tend to be more compositional and (2) there is a population size effect whereby larger populations develop more structured languages (§6).
Revised: In addition, we observe that (1) languages developed in partitioned populations tend to be more compositional and (2) there is a population size effect whereby larger populations develop more structured languages (§6).

Original: Our results show that there are multiple ways to generalize from single agent pairs to larger populations, and that these design choices matter when it comes to studying the emergent language.
Revised: Our results show that there are multiple ways to generalize from single agent pairs to larger populations, and that these design choices matter when it comes to studying the emergent language.

Original:  C OMMUNICATION G AME
Revised:  C OMMUNICATION G AME

Original: We study communication in referential games, a variant of the Lewis signaling game (Lewis, 1969) proposed by Lazaridou et al.
Revised: We study communication in referential games, a variant of the Lewis signaling game (Lewis, 1969) proposed by Lazaridou et al.

Original: The game proceeds as follows: during each round, a sender agent π observes an object x ∈ X (e.g., an arbitrary categorical entity, or a natural images) sampled from input space X according to distribution p and generates a message m ∼ π ( · | x ) .
Revised: The game proceeds as follows: during each round, a sender agent π observes an object x ∈ X (e.g., an arbitrary categorical entity, or a natural images) sampled from input space X according to distribution p and generates a message m ∼ π ( · | x ) .

Original: Messages consist of variable length sequences of tokens picked from a discrete vocabulary V .
Revised: Messages consist of variable length sequences of tokens picked from a discrete vocabulary V .

Original: Note that the tokens themselves are arbitrary and meaningless (typically they are represented as numbers from 1 to | V | ).
Revised: Note that the tokens themselves are arbitrary and meaningless (typically they are represented as numbers from 1 to | V | ).

Original: A receiver agent ρ then observes message m and must predict the original object from among a set of candidates
Revised: A receiver agent ρ then observes message m and must predict the original object from among a set of candidates

Original: C = { x, y 1 , . . .
Revised: C = { x, y 1 , . . .

Original: y |C− 1 | } containing x and |C| − 1 distractors, where each distractor y is sampled uniformly without replacement from the input space excluding the original object, X \ { x } .
Revised: y |C− 1 | } containing x and |C| − 1 distractors, where each distractor y is sampled uniformly without replacement from the input space excluding the original object, X \ { x } .

Original: Concretely, this is implemented by calculating a score f
Revised: Concretely, this is implemented by calculating a score f

Original: ( y, m ) for each candidate y and defining the probability of a candidate conditioned on the message ρ ( · | m, C ) ase f ( x,m ) (cid:80) y ∈C f
Revised: ( y, m ) for each candidate y and defining the probability of a candidate conditioned on the message ρ ( · | m, C ) as

Original: ( y,m ) .
Revised: [Equation]

Original: Based on the receiver’s success, the sender agent receives a reward R ( x, ρ ( · | m, C )) .
Revised: Based on the receiver’s success, the sender agent receives a reward R ( x, ρ ( · | m, C )) .

Original: In practice, both senders and receivers are implemented as neural networks π θ and ρ ψ with parameters θ and ψ estimated by gradient descent.
Revised: In practice, both senders and receivers are implemented as neural networks π θ and ρ ψ with parameters θ and ψ estimated by gradient descent.

Original: The sender is trained to maximize its expected reward using the
Revised: The sender is trained to maximize its expected reward using the

Original: REINFORCE algorithm
Revised: REINFORCE algorithm

Original: (Williams, 1992), while the receiver maximizes the expected log-likelihood of identifying the original object, log ρ ψ ( x | m, C ) (also known as the InfoNCE objective; Oord et al.
Revised: (Williams, 1992), while the receiver maximizes the expected log-likelihood of identifying the original object, log ρ ψ ( x | m, C ) (also known as the InfoNCE objective; Oord et al.

Original: Denoting as E x ∼ p the expectation over x sampled from p , the corresponding training objectives are:
Revised: Denoting as E x ∼ p the expectation over x sampled from p , the corresponding training objectives are:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: 2. P OPULATION L EVEL T RAINING
Revised: 2. P OPULATION L EVEL T RAINING

Original: The two-player referential game can be generalized to larger populations of agents (Mordatch &
Revised: The two-player referential game can be generalized to larger populations of agents (Mordatch &

Original: Abbeel, 2018; Chaabouni et al., 2022).
Revised: Abbeel, 2018; Chaabouni et al., 2022).

Original: In the most general case, we consider a population of N s senders and N r receivers that are linked by a bipartite communication graph G defining connections between senders and receiver ( π θ i , ρ ψ j ) (Harding Graesser et al., 2019;
Revised: In the most general case, we consider a population of N s senders and N r receivers that are linked by a bipartite communication graph G defining connections between senders and receiver ( π θ i , ρ ψ j ) (Harding Graesser et al., 2019;

Original: Kim & Oh, 2021).
Revised: Kim & Oh, 2021).

Original: At training time, sender-receiver pairs are repeatedly sampled and trained to perform a round of the game.
Revised: At training time, sender-receiver pairs are repeatedly sampled and trained to perform a round of the game.

Original: Importantly, only agent pairs that are connected in the communication graph are sampled.
Revised: Importantly, only agent pairs that are connected in the communication graph are sampled.

Original: Throughout this paper, we will refer to this type of training as Standard training.
Revised: Throughout this paper, we will refer to this type of training as Standard training.

Original: With this training procedure, agents are trained to maximize their communicative success with all their neighbors in the communication graph.
Revised: With this training procedure, agents are trained to maximize their communicative success with all their neighbors in the communication graph.

Original: Let N G ( i ) refer to the neighbors of the i -th node in the graph, and J s,i → j (respectively J r,i → j ) denote the objective of π θ i (respectively ρ ψ j ) ) in the pairwise communication from sender i to receiver j .
Revised: Let N G ( i ) refer to the neighbors of the i -th node in the graph, and J s,i → j (respectively J r,i → j ) denote the objective of π θ i (respectively ρ ψ j ) ) in the pairwise communication from sender i to receiver j .

Original: We can write the overall objective for sender i (and receiver j , respectively) as:
Revised: We can write the overall objective for sender i (and receiver j , respectively) as:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: At test time, the population is evaluated by averaging the referential accuracy across all possible sender-receiver pairings.
Revised: At test time, the population is evaluated by averaging the referential accuracy across all possible sender-receiver pairings.

Original: Following previous work, in this paper we focus on populations with an equal number N :=
Revised: Following previous work, in this paper we focus on populations with an equal number N :=

Original: N s = N r of senders and receivers, meaning that there are up to N 2 possible pairings.
Revised: N s = N r of senders and receivers, meaning that there are up to N 2 possible pairings.

Original: 2. W HAT DOES P OPULATION - LEVEL T RAINING O PTIMIZE ?
Revised: 2. W HAT DOES P OPULATION - LEVEL T RAINING O PTIMIZE ?

Original: To shed light on the differences between training a single agent pair and training a population of agents, we analyze the objective optimized by the population.
Revised: To shed light on the differences between training a single agent pair and training a population of agents, we analyze the objective optimized by the population.

Original: Inspired by (Rita et al., 2022b)’s analysis in the two-player case, we study the behaviour of the population at the optimum, that is when senders and receivers have reached a Nash equilibrium (Osborne & Rubinstein, 1994).
Revised: Inspired by (Rita et al., 2022b)’s analysis in the two-player case, we study the behaviour of the population at the optimum, that is when senders and receivers have reached a Nash equilibrium (Osborne & Rubinstein, 1994).

Original: In this section, we make the simplifying assumption that C = X .
Revised: In this section, we make the simplifying assumption that C = X .

Original: In other words, receiver agents must pick the correct candidate out of all possible objects in X .
Revised: In other words, receiver agents must pick the correct candidate out of all possible objects in X .

Original: This allows us to remove the conditioning on C and write ρ ψ ( x | m, C ) =
Revised: This allows us to remove the conditioning on C and write ρ ψ ( x | m, C ) =

Original: ρ ψ ( x | m ) .
Revised: ρ ψ ( x | m ) .

Original: We make this simplification to reduce clutter in notations.
Revised: We make this simplification to reduce clutter in notations.

Original: Nevertheless, our key observations still hold for C ̸
Revised: Nevertheless, our key observations still hold for C ̸

Original: = X (see Appendix C for a detailed discussion).
Revised: = X (see Appendix C for a detailed discussion).

Original: At a Nash equilibrium, the optimal receiver parameters ψ ∗ j satisfy
Revised: At a Nash equilibrium, the optimal receiver parameters ψ ∗ j satisfy

Original: [Equation]
Revised: [Equation]

Original: Assuming that receiver ρ ψ j has high enough capacity, and training is able to reach the global optimum, the solution of the optimization problem in Equation 4 has an analytical solution ρ ψ ∗ j which can be written as a function of π ∗N G ( j ) ( m | x ) :
Revised: Assuming that receiver ρ ψ j has high enough capacity, and training is able to reach the global optimum, the solution of the optimization problem in Equation 4 has an analytical solution ρ ψ ∗ j which can be written as a function of π ∗N G ( j ) ( m | x ) :

Original: = 1 |N G ( j ) | (cid:80) i ∈N G ( j ) π θ ∗ i ( m | x ) , the mixture of all senders communicating with receiver j :
Revised: = 1 |N G ( j ) | (cid:80) i ∈N G ( j ) π θ ∗ i ( m | x ) , the mixture of all senders communicating with receiver j :

Original: [Equation]
Revised: [Equation]

Original: In other words, ρ ψ ∗ j is the posterior associated with π ∗N G ( j ) (full derivation in appendix B).
Revised: In other words, ρ ψ ∗ j is the posterior associated with π ∗N G ( j ) (full derivation in appendix B).

Original: An important implication of this result is that when the population graph is fully connected (all senders are connected to all receivers), each receiver converges to the same optimum π ∗ ( x | m )
Revised: An important implication of this result is that when the population graph is fully connected (all senders are connected to all receivers), each receiver converges to the same optimum π ∗ ( x | m )

Original: = (cid:80)ni
Revised: = (cid:80)

Original: [Equation]

Original: E y ∼

Original: [Equation]
Revised: [Equation]

Original: m | x ) , the posterior of the mixture of all senders in the population.
Revised: m | x ) , the posterior of the mixture of all senders in the population.

Original: Plugging this back into each sender’s objective, we have
Revised: Plugging this back into each sender’s objective, we have

Original: [Equation]
Revised: [Equation]

Original: Summing across all senders, we can rewrite the global objective optimized by the senders as
Revised: Summing across all senders, we can rewrite the global objective optimized by the senders as

Original: [Equation]
Revised: [Equation]

Original: In other words, at the equilibrium, the population maximizes the expected reward of the “sender ensemble” π ∗ , rather than that of individual agents π θ ∗ i : the objective of a population N agents is functionally the same irrespective of N .
Revised: In other words, at the equilibrium, the population maximizes the expected reward of the “sender ensemble” π ∗ , rather than that of individual agents π θ ∗ i : the objective of a population N agents is functionally the same irrespective of N .

Original: We postulate that this indifference to the population size may account for the surprising lack of effect of larger populations observed in some previous work (Rita et al., 2022a; Chaabouni et al., 2022).
Revised: We postulate that this indifference to the population size may account for the surprising lack of effect of larger populations observed in some previous work (Rita et al., 2022a; Chaabouni et al., 2022).

Original: Differences in behaviour must be attributed to effects stemming from training dynamics ( e.g. it becomes more difficult for receivers to learn the posterior π ∗ ( x | m ) ), or be imposed through extraneous modifications of the population objective (for example explicit imitation components; Chaabouni et al.
Revised: Differences in behaviour must be attributed to effects stemming from training dynamics ( e.g. it becomes more difficult for receivers to learn the posterior π ∗ ( x | m ) ), or be imposed through extraneous modifications of the population objective (for example explicit imitation components; Chaabouni et al.

Original: A second observation is that there is no direct pressure for agents that communicate at training time to develop the same language.
Revised: A second observation is that there is no direct pressure for agents that communicate at training time to develop the same language.

Original: Indeed, it is entirely possible that all senders develop different but non-overlapping languages: it suffices that no two senders communicating with a shared receiver use the same message m to describe a different object.
Revised: Indeed, it is entirely possible that all senders develop different but non-overlapping languages: it suffices that no two senders communicating with a shared receiver use the same message m to describe a different object.

Original: In this case receivers can simply learn their neighboring sender’s languages and there is no need for the senders to converge to a unified language.
Revised: In this case receivers can simply learn their neighboring sender’s languages and there is no need for the senders to converge to a unified language.

Original:  P ARTITIONING A GENTS
Revised:  P ARTITIONING A GENTS

Original: [Figure]
Revised: [Figure]

Original: A key difference between the usual population setting and populations of humans in laboratory experiments is that agents are not usually split into “senders” and “receivers”.
Revised: A key difference between the usual population setting and populations of humans in laboratory experiments is that agents are not usually split into “senders” and “receivers”.

Original: Rather, each participant in the experiment assumes both a sender and receiver role (Galke et al., 2022).
Revised: Rather, each participant in the experiment assumes both a sender and receiver role (Galke et al., 2022).

Original: Our hypothesis is that, counter to what is customary in the emergent communication literature, tying senders and receivers is key in surfacing useful population-level dynamics in multi-agent communication.
Revised: Our hypothesis is that, counter to what is customary in the emergent communication literature, tying senders and receivers is key in surfacing useful population-level dynamics in multi-agent communication.

Original: To operationalize this sender-receiver coupling, we identify an “agent” as a sender-receiver pair.
Revised: To operationalize this sender-receiver coupling, we identify an “agent” as a sender-receiver pair.

Original: During training, we only train receiver ρ ψ i with its associated sender π θ i .
Revised: During training, we only train receiver ρ ψ i with its associated sender π θ i .

Original: In other words,
Revised: In other words,

Original: J r,i ( ψ i ) :
Revised: J r,i ( ψ i ) :

Original: =
Revised: =

Original: J r,i → i ( ψ i ) .
Revised: J r,i → i ( ψ i ) .

Original: In doing so, we “partition” the agents by preventing receiver i from co-adapting to other senders j ̸ = i .
Revised: In doing so, we “partition” the agents by preventing receiver i from co-adapting to other senders j ̸ = i .

Original: This procedure is illustrated in Figure 1.
Revised: This procedure is illustrated in Figure 1.

Original: Note that senders can nevertheless still train with rewards from neighboring receivers, and so communication across agents can still emerge.
Revised: Note that senders can nevertheless still train with rewards from neighboring receivers, and so communication across agents can still emerge.

Original: Importantly, partitioning prevents receivers from learning to recognize multiple languages, as they are now only trained on messages emitted by a single sender.
Revised: Importantly, partitioning prevents receivers from learning to recognize multiple languages, as they are now only trained on messages emitted by a single sender.

Original: Following a similar analysis as Section 2.2, we derive that at the optimum, receiver ρ ψ ∗ i ( x | m ) now takes the form of the posterior associated with its respective sender, π θ ∗ i ( x | m ) =
Revised: Following a similar analysis as Section 2.2, we derive that at the optimum, receiver ρ ψ ∗ i ( x | m ) now takes the form of the posterior associated with its respective sender, π θ ∗ i ( x | m ) =

Original: π θ
Revised: π θ

Original: [Equation]
Revised: [Equation]

Original: i ( m | x ) p ( x ) E
Revised: i ( m | x [Equation] p ( x ) E

Original: [Equation]
Revised: [Equation]

Original: ∼
Revised: ∼

Original: [Equation]
Revised: [Equation]

Original: | y (derivation in Appendix B).
Revised: y (derivation in Appendix B).

Original: We can thus write the population-level objective at the equilibrium as
Revised: We can thus write the population-level objective at the equilibrium as

Original: [Equation]
Revised: [Equation]

Original: Note that the functional form of the objective can now be decomposed into two parts: an “internal communication” objective which takes the same form as that of a single pair of agents, and a “mutual intelligibility” objective which enforces that neighboring agents are able to communicate successfully.
Revised: Note that the functional form of the objective can now be decomposed into two parts: an “internal communication” objective which takes the same form as that of a single pair of agents, and a “mutual intelligibility” objective which enforces that neighboring agents are able to communicate successfully.

Original: In experiments, we show that this explicit pressure towards mutual intelligibility promotes the emergence of a single language within the population, which in turn enables agents to communicate with new partners outside of their training neighborhood.
Revised: In experiments, we show that this explicit pressure towards mutual intelligibility promotes the emergence of a single language within the population, which in turn enables agents to communicate with new partners outside of their training neighborhood.

Original:  E XPERIMENTAL S ETTING 4. D ATASETS
Revised:  E XPERIMENTAL S ETTING 4. D ATASETS

Original: We perform experiments on two datasets: a simple, synthetic “attribute/values” dataset and a more realistic image dataset.
Revised: We perform experiments on two datasets: a simple, synthetic “attribute/values” dataset and a more realistic image dataset.

Original: Attribute/Values In this dataset, each object is represent by a collection of abstract “attributes”.
Revised: Attribute/Values In this dataset, each object is represent by a collection of abstract “attributes”.

Original: Specifically, each input x is a vector of 4 attributes, each of which can take 10 total values.
Revised: Specifically, each input x is a vector of 4 attributes, each of which can take 10 total values.

Original: This results in 10 4 total attribute/value combinations (Kottur et al., 2017; Chaabouni et al., 2020).
Revised: This results in 10 4 total attribute/value combinations (Kottur et al., 2017; Chaabouni et al., 2020).

Original: In each setting we hold out 1 , 000 combinations to be used as a validation, and 1 , 000 more for use as a test set.
Revised: In each setting we hold out 1 , 000 combinations to be used as a validation, and 1 , 000 more for use as a test set.

Original: We can thus ensure that we are evaluating the agents’ ability to generalize to unseen combinations of attributes.
Revised: We can thus ensure that we are evaluating the agents’ ability to generalize to unseen combinations of attributes.

Original: ImageNet In addition to toy objects, we perform experiments with referential games based on more realistic objects.
Revised: ImageNet In addition to toy objects, we perform experiments with referential games based on more realistic objects.

Original: Following Chaabouni et al.
Revised: Following Chaabouni et al.

Original: (2022), we use the ImageNet (Deng et al., 2009) dataset of natural images.
Revised: (2022), we use the ImageNet (Deng et al., 2009) dataset of natural images.

Original: The dataset consists of about 1.4M training images collected on the internet and annotated for 1,000 labels from the WordNet database (Miller, 1995).
Revised: The dataset consists of about 1.4M training images collected on the internet and annotated for 1,000 labels from the WordNet database (Miller, 1995).

Original: Images are first encoded as 2048-sized real-valued vectors with a (frozen) ResNet pre-trained with BYOL (Grill et al., 2020) before being passed to sender and receivers.
Revised: Images are first encoded as 2048-sized real-valued vectors with a (frozen) ResNet pre-trained with BYOL (Grill et al., 2020) before being passed to sender and receivers.

Original: 4. G AME A RCHITECTURE
Revised: 4. G AME A RCHITECTURE

Original: Both sender and receiver agents are based on 1 layer LSTMs (Hochreiter & Schmidhuber, 1997) with embedding and hidden dimensions of size 256 .
Revised: Both sender and receiver agents are based on 1 layer LSTMs (Hochreiter & Schmidhuber, 1997) with embedding and hidden dimensions of size 256 .

Original: Specifically, the sender first encodes the object x into a vector of size 256 , which is concatenated to the input of the LSTM.
Revised: Specifically, the sender first encodes the object x into a vector of size 256 , which is concatenated to the input of the LSTM.

Original: At each step, the output of the LSTM cell is passed through a fully connected layer to produce logits of size | V | .
Revised: At each step, the output of the LSTM cell is passed through a fully connected layer to produce logits of size | V | .

Original: A softmax function is then applied to obtain normalized probabilities over the vocabulary.
Revised: A softmax function is then applied to obtain normalized probabilities over the vocabulary.

Original: During training, messages are generated by sampling from the distribution whereas at test time we generate messages deterministically via greedy decoding.
Revised: During training, messages are generated by sampling from the distribution whereas at test time we generate messages deterministically via greedy decoding.

Original: In both cases, generation stops whenever a special “ <EOS> ” is generated, or when the number of tokens reaches a fixed limit L .
Revised: In both cases, generation stops whenever a special “ <EOS> ” is generated, or when the number of tokens reaches a fixed limit L .

Original: The receiver encodes the message with an LSTM encoder, the output of which is the fed into a fully connected layer to yield a vector of size 512.
Revised: The receiver encodes the message with an LSTM encoder, the output of which is the fed into a fully connected layer to yield a vector of size 512.

Original: The candidate objects C are then scored by computing the dot product of this vector with a 512-dimensional encoding of each candidate.
Revised: The candidate objects C are then scored by computing the dot product of this vector with a 512-dimensional encoding of each candidate.

Original: The conditional distribution over candidates is then obtained by taking a softmax.
Revised: The conditional distribution over candidates is then obtained by taking a softmax.

Original: We set the reward function for the sender to the log-likelihood assigned by the receiver to the correct candidate,
Revised: We set the reward function for the sender to the log-likelihood assigned by the receiver to the correct candidate,

Original: R ( x, ρ ψ ( · | m ))
Revised: R ( x, ρ ψ ( · | m ))

Original: = log ρ ψ ( x | m ) .
Revised: = log ρ ψ ( x | m ) .

Original: Throughout all experiments, we set the vocabulary size | V | to 20 and the maximum length of the messages, L , to 10 .
Revised: Throughout all experiments, we set the vocabulary size | V | to 20 and the maximum length of the messages, L , to 10 .

Original: This means that the communication channel used by the agents has a capacity of about 20 10 which ensures that there is no communication bottleneck (the size of the channel is several orders of magnitude larger than the size of our datasets).
Revised: This means that the communication channel used by the agents has a capacity of about 20 10 which ensures that there is no communication bottleneck (the size of the channel is several orders of magnitude larger than the size of our datasets).

Original: Our implementation, based on the
Revised: Our implementation, based on the

Original: EGG toolkit (Kharitonov et al., 2021), will be open-sourced upon de-anonymization.
Revised: EGG toolkit (Kharitonov et al., 2021), will be open-sourced upon de-anonymization.

Original: 4. P OPULATION TRAINING
Revised: 4. P OPULATION TRAINING

Original: [Figure]
Revised: [Figure]

Original: We train populations following the procedure outlined by
Revised: We train populations following the procedure outlined by

Original: Chaabouni et al.
Revised: Chaabouni et al.

Original: (2022): for each minibatch of data, we sample K pairs from the population (uniformly among the pairs linked in the communication graph).
Revised: (2022): for each minibatch of data, we sample K pairs from the population (uniformly among the pairs linked in the communication graph).

Original: Each pair plays an episode of the game, and the agents are updated simultaneously following the gradients of their respective objectives.
Revised: Each pair plays an episode of the game, and the agents are updated simultaneously following the gradients of their respective objectives.

Original: We take K = max(10 , N ) to ensure that each agent plays the game at least once at every step on average.
Revised: We take K = max(10 , N ) to ensure that each agent plays the game at least once at every step on average.

Original: This procedure needs to be modified for partitioned populations: since receiver j is only with its respective sender instead of with all of its neighbors, there is now only a 1 | N G ( j )
Revised: This procedure needs to be modified for partitioned populations: since receiver j is only with its respective sender instead of with all of its neighbors, there is now only a 1 | N G ( j )

Original: | chance that receiver j will be updated every step (the probability that the pair ( j, j ) is sampled).
Revised: | chance that receiver j will be updated every step (the probability that the pair ( j, j ) is sampled).

Original: For larger populations, especially those that are fully-connected, this dramatically slows down training as receivers are updated very infrequently.
Revised: For larger populations, especially those that are fully-connected, this dramatically slows down training as receivers are updated very infrequently.

Original: To address this issue, we modify the procedure as follows: for every sampled agent pair ( π θ i , ρ ψ j ) , we calculate both J s,i → j and J r,i → i and update both π θ i and ρ ψ i .
Revised: To address this issue, we modify the procedure as follows: for every sampled agent pair ( π θ i , ρ ψ j ) , we calculate both J s,i → j and J r,i → i and update both π θ i and ρ ψ i .

Original: Note that this necessitates calculating both ρ ψ j ( x | m, C ) and ρ ψ i ( x | m, C ) and therefore we incur a small computational overhead.
Revised: Note that this necessitates calculating both ρ ψ j ( x | m, C ) and ρ ψ i ( x | m, C ) and therefore we incur a small computational overhead.

Original: However we only observe a ∼ 5% increase in training time due to the fact that we are back-propagating through only one of the two receivers, ρ ψ i ( x | m, C ) .
Revised: However we only observe a ∼ 5% increase in training time due to the fact that we are back-propagating through only one of the two receivers, ρ ψ i ( x | m, C ) .

Original: With this modification, we recover the property that each agent (sender or receiver) is updated once every step on average.
Revised: With this modification, we recover the property that each agent (sender or receiver) is updated once every step on average.

Original: In all experiments we train with a batch size of 1024 with the Adam optimizer (Kingma & Ba, 2014) using a learning rate of 0 .
Revised: In all experiments we train with a batch size of 1024 with the Adam optimizer (Kingma & Ba, 2014) using a learning rate of 0 .

Original: 001 for the attribute/value dataset and 0 .
Revised: 001 for the attribute/value dataset and 0 .

Original: 0001 for Imagenet.
Revised: 0001 for Imagenet.

Original: The other parameters are set to β 1 = 0 .
Revised: The other parameters are set to β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 and ε = 10 − 8 .
Revised: 999 and ε = 10 − 8 .

Original: We apply ℓ 2 regularization with a coefficient of 10 − 5 .
Revised: We apply ℓ 2 regularization with a coefficient of 10 − 5 .

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: We systematically augment the sender objectives with an entropy maximizing term, which has been found to encourage exploration (Williams & Peng, 1991).
Revised: We systematically augment the sender objectives with an entropy maximizing term, which has been found to encourage exploration (Williams & Peng, 1991).

Original: The coefficient for this entropy term is set to 0 .
Revised: The coefficient for this entropy term is set to 0 .

Original: 1 in all experiments.
Revised: 1 in all experiments.

Original: To reduce the variance of the policy gradient in REINFORCE, we substract a baseline computed by taking the average reward within a mini-batch for each pair (Sutton et al., 1999).
Revised: To reduce the variance of the policy gradient in REINFORCE, we substract a baseline computed by taking the average reward within a mini-batch for each pair (Sutton et al., 1999).

Original: We evaluate the population every epoch (every 5 epochs for the Attribute/Value dataset) on the validation set.
Revised: We evaluate the population every epoch (every 5 epochs for the Attribute/Value dataset) on the validation set.

Original: We only evaluate on up to 100 unique pairs sampled uniformly within the population, this time without consideration for the communication graph.
Revised: We only evaluate on up to 100 unique pairs sampled uniformly within the population, this time without consideration for the communication graph.

Original: We train for a fixed number of epochs, selecting the best model based on the average validation accuracy across all evaluation pairs.
Revised: We train for a fixed number of epochs, selecting the best model based on the average validation accuracy across all evaluation pairs.

Original: Although convergence is not guaranteed in this general-sum communication game, in all reported experiments we find that agents converge reliably with our choice of hyper-parameters.
Revised: Although convergence is not guaranteed in this general-sum communication game, in all reported experiments we find that agents converge reliably with our choice of hyper-parameters.

Original:  C OMMUNICATION WITH N EW P ARTNERS
Revised:  C OMMUNICATION WITH N EW P ARTNERS

Original: In our first set of experiments, we evaluate the ability of agents trained in populations to communicate with partners they haven’t interacted with during training.
Revised: In our first set of experiments [Table] we evaluate the ability of agents trained in populations to communicate with partners they haven’t interacted with during training.

Original: 5. C IRCULAR P OPULATIONS
Revised: 5. C IRCULAR P OPULATIONS

Original: Specifically, we study “circular” populations of agents arranged on a ring lattice.
Revised: Specifically, we study “circular” populations of agents arranged on a ring lattice.

Original: Each agent (senderreceiver pair)
Revised: Each agent (senderreceiver pair)

Original: i is only trained with neighboring agents i − 1 , . . .
Revised: i is only trained with neighboring agents i − 1 , . . .

Original: , i + 

Revised: , i +

Original: and the graph is cyclical (see
Revised: and the graph is cyclical (see

Original: Figure 6a).
Revised: Figure 6a).

Original: We choose this type of population because it is an extreme case of a population where each agent has the same, minimal amount of neighbors (two), yet there is still a path between any two agents.
Revised: We choose this type of population because it is an extreme case of a population where each agent has the same, minimal amount of neighbors (two), yet there is still a path between any two agents.

Original: In this context, training partners are sender-receiver pairs that are connected in the graph and have interacted during the training phase whereas new partners refers to pairs that have not interacted during training.
Revised: In this context, training partners are sender-receiver pairs that are connected in the graph and have interacted during the training phase whereas new partners refers to pairs that have not interacted during training.

Original: Experiments with other population types are reported in Appendix D.
Revised: Experiments with other population types are reported in Appendix D.

Original: We report results along two metrics:
Revised: We report results along two metrics:

Original: • Communication Accuracy of sender/receiver pairs on an evaluation set.
Revised: • Communication Accuracy of sender/receiver pairs on an evaluation set.

Original: This measures how successful the pair is in communicating.
Revised: This measures how successful the pair is in communicating.

Original: • Language Similarity between senders.
Revised: • Language Similarity between senders.

Original: This metric (also called synchronization in Rita et al.
Revised: This metric (also called synchronization in Rita et al.

Original: (2022a)) is calculated as 1 − δ i,j , where δ i,j is the normalized edit distance between messages output by two senders, averaged across all objects in our evaluation set.
Revised: (2022a)) is calculated as 1 − δ i,j , where δ i,j is the normalized edit distance between messages output by two senders, averaged across all objects in our evaluation set.

Original: We report these metrics for both training partners and new partners.
Revised: We report these metrics for both training partners and new partners.

Original: Note that high communication accuracy does not always entail similar languages: it is possible for the receivers to achieve high accuracy despite all senders sending different messages for any given object (it is only necessary for a given message to unambiguously refer to one object across senders).
Revised: Note that high communication accuracy does not always entail similar languages: it is possible for the receivers to achieve high accuracy despite all senders sending different messages for any given object (it is only necessary for a given message to unambiguously refer to one object across senders).

Original: 5. P ARTITIONING E NABLES S UCCESSFUL Z ERO -S HOT C OMMUNICATION
Revised: 5. P ARTITIONING E NABLES S UCCESSFUL Z ERO -S HOT C OMMUNICATION

Original: In Table 1 and 2, we report accuracies and similarities for circular populations of 20 sender-receiver pairs trained on ImageNet and the Attribute/Values dataset.
Revised: In Table 1 and 2, we report accuracies and similarities for circular populations of 20 sender-receiver pairs trained on ImageNet and the Attribute/Values dataset.

Original: All metrics are calculated on the test set and averaged across 3 independent experiments.
Revised: All metrics are calculated on the test set and averaged across 3 independent experiments.

Original:  We observe that in populations following the standard training paradigm ( Standard ), there is a stark discrepancy between training and new partners.
Revised: [Figure] We observe that in populations following the standard training paradigm ( Standard ), there is a stark discrepancy between training and new partners.

Original: Indeed, on both datasets the accuracy with training partners reaches a very high value, above 95% .
Revised: Indeed, on both datasets the accuracy with training partners reaches a very high value, above 95% .

Original: Yet, the accuracy when agents communicate with new partners drops down to less than 10% .
Revised: Yet, the accuracy when agents communicate with new partners drops down to less than 10% .

Original: On the other hand, in Partitioned populations, agents

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Table] reach a much higher accuracy with non-neighbors, up to 96% on ImageNet and 40% .
Revised: On the other hand, in Partitioned populations, agents reach a much higher accuracy with non-neighbors, up to 96% on ImageNet and 40% .

Original: A similar trend holds for language similarity.
Revised: A similar trend holds for language similarity.

Original: Note that all metrics on new partners exhibit high standard deviation.
Revised: Note that all metrics on new partners exhibit high standard deviation.

Original: An explanation is that among non-neighboring pairs there is a different behaviour depending on how far the two agents are in the population.
Revised: An explanation is that among non-neighboring pairs there is a different behaviour depending on how far the two agents are in the population.

Original: This is verified in Figure 3, which displays a breakdown as a function of the distance between two agents in the communication graph (on ImageNet).
Revised: This is verified in Figure 3, which displays a breakdown as a function of the distance between two agents in the communication graph (on ImageNet).

Original: We find that without partitioning, accuracy drops off sharply to close to 0 for agents at a distance ≥ 2 , whereas it decreases almost linearly with the distance in the partitioned case, down to about 95% for the most distant agents.
Revised: We find that without partitioning, accuracy drops off sharply to close to 0 for agents at a distance ≥ 2 , whereas it decreases almost linearly with the distance in the partitioned case, down to about 95% for the most distant agents.

Original: 5. T RAINING DYNAMICS
Revised: 5. T RAINING DYNAMICS

Original: We further investigate the evolution of accuracies during training.
Revised: We further investigate the evolution of accuracies during training.

Original: In Figure 4, we plot the evaluation accuracies of both standard and partitioned populations broken down by distance between pairs, focusing on the ImageNet dataset.
Revised: In Figure 4, we plot the evaluation accuracies of both standard and partitioned populations broken down by distance between pairs, focusing on the ImageNet dataset.

Original: Note that there are two training phases in the standard case.
Revised: Note that there are two training phases in the standard case.

Original: Up to epoch ≈ 10 , the accuracy for all training pairs increases, after which agents over-fit to their training partners (distances 0 and 1) and the accuracy on other pairs decreases to a plateau.
Revised: Up to epoch ≈ 10 , the accuracy for all training pairs increases, after which agents over-fit to their training partners (distances 0 and 1) and the accuracy on other pairs decreases to a plateau.

Original: On the other hand, Figure 4b illustrates the pressure for mutual-intelligibility in partitioned populations: as accuracy between training pairs reaches close to 99% accuracy (around epoch 20 ), accuracies across distant pairs increases rapidly before plateauing above 90% .
Revised: On the other hand, Figure 4b illustrates the pressure for mutual-intelligibility in partitioned populations: as accuracy between training pairs reaches close to 99% accuracy (around epoch 20 ), accuracies across distant pairs increases rapidly before plateauing above 90% .

Original: In fact, our results show that the most distant accuracies are still increasing after 150 epochs, albeit very slowly.
Revised: In fact, our results show that the most distant accuracies are still increasing after 150 epochs, albeit very slowly.

Original:  P ARTITIONED P OPULATION D EVELOP M ORE C OMPOSITIONAL L ANGUAGES
Revised:  P ARTITIONED P OPULATION D EVELOP M ORE C OMPOSITIONAL L ANGUAGES

Original: In this section, we investigate the effect of partitioning on the structure of the language, with a focus on compositionality .
Revised: In this section, we investigate the effect of partitioning on the structure of the language, with a focus on compositionality .

Original: 6. M EASURING C OMPOSITIONALITY
Revised: 6. M EASURING C OMPOSITIONALITY

Original: A language is said to be compositional (Szab´o, 2020) when the meaning of a whole utterance can be systematically deduced from the meaning of its components ( i.e. words).
Revised: A language is said to be compositional (Szab´o, 2020) when the meaning of a whole utterance can be systematically deduced from the meaning of its components ( i.e. words).

Original: It has been argued that compositionality makes languages easier to learn (Davidson, 1965).
Revised: It has been argued that compositionality makes languages easier to learn (Davidson, 1965).

Original: Consequently, emergent communication protocols that are compositional may ultimately be easier to understand by humans.
Revised: Consequently, emergent communication protocols that are compositional may ultimately be easier to understand by humans.

Original: A common metric for measuring compositionality in emergent languages is the topographic similarity (Brighton & Kirby, 2006; Lazaridou et al., 2018).
Revised: A common metric for measuring compositionality in emergent languages is the topographic similarity (Brighton & Kirby, 2006; Lazaridou et al., 2018).

Original: Topographic similarity captures the intuition that a[Figure] compositional language will map similar “meanings” to similar messages: the phrase “a red bird” is more similar to the phrase “a blue bird” than to “a powerful computer”.
Revised: Topographic similarity captures the intuition that a compositional language will map similar “meanings” to similar messages: the phrase “a red bird” is more similar to the phrase “a blue bird” than to “a powerful computer”.

Original: In practice, the topographic similarity is computed by measuring the Spearman rank correlation coefficient (Spearman, 1904) between (1) the pairwise distances across all objects and (2) the pairwise distance across all messages. 
Revised: In practice, the topographic similarity is computed by measuring the Spearman rank correlation coefficient (Spearman, 1904) between (1) the pairwise distances across all objects and (2) the pairwise distance across all messages. [Figure]

Original: 6. E FFECT OF P OPULATION S IZE ON C OMPOSITIONALITY
Revised: 6. E FFECT OF P OPULATION S IZE ON C OMPOSITIONALITY

Original: We run experiments on our Attribute/Values dataset, with both standard and partitioned populations that are fully-connected (see Figure 6d).
Revised: We run experiments on our Attribute/Values dataset, with both standard and partitioned populations that are fully-connected (see Figure 6d).

Original: Population sizes range from 2 to 25 sender-receiver pairs.
Revised: Population sizes range from 2 to 25 sender-receiver pairs.

Original: We compute topographic similarity using the Hamming distance in the object space (the distance between two objects is the number of attributes in which they differ) and the normalized edit distance between messages.
Revised: We compute topographic similarity using the Hamming distance in the object space (the distance between two objects is the number of attributes in which they differ) and the normalized edit distance between messages.

Original: In Figure 5a, we observe that while standard population-level training does increase the topographic similarity of the language overall, population size has very little effect: populations of sizes 3 and 20 both reach about the same value of 30 on average.
Revised: In Figure 5a, we observe that while standard population-level training does increase the topographic similarity of the language overall, population size has very little effect: populations of sizes 3 and 20 both reach about the same value of 30 on average.

Original: On the other hand, partitioning greatly increases the effect of population size on compositionality: populations of size 20 have a significantly higher topographic similarity than populations of size 5, with a ≈ 10 points difference.
Revised: On the other hand, partitioning greatly increases the effect of population size on compositionality: populations of size 20 have a significantly higher topographic similarity than populations of size 5, with a ≈ 10 points difference.

Original: 6. C O - ADAPTATION IS R ESPONSIBLE FOR THE D ECREASE IN C OMPOSITIONALITY
Revised: 6. C O - ADAPTATION IS R ESPONSIBLE FOR THE D ECREASE IN C OMPOSITIONALITY

Original: Up until this point, we have described partitioning (or lack thereof) as a binary choice.
Revised: Up until this point, we have described partitioning (or lack thereof) as a binary choice.

Original: However, it is possible to partition a population only partially, by allowing receiver j to train with senders i ̸
Revised: However, it is possible to partition a population only partially, by allowing receiver j to train with senders i ̸

Original: =
Revised: =

Original: j occasionally with probability α > 0 .
Revised: j occasionally with probability α > 0 .

Original: In doing so, the optimal receiver now becomes the posterior associated with a mixture between π θ ∗ i ( m | x ) and π ∗ ( m | x ) (see Appendix B for the derivation).
Revised: In doing so, the optimal receiver now becomes the posterior associated with a mixture between π θ ∗ i ( m | x ) and π ∗ ( m | x ) (see Appendix B for the derivation).

Original: If 0 < α < 1 , receivers are now optimizing for a different objective (as in partitioned populations), but some amount of co-adaptation is still allowed.
Revised: If 0 < α < 1 , receivers are now optimizing for a different objective (as in partitioned populations), but some amount of co-adaptation is still allowed.

Original: We perform this experiment on the Attribute/Values dataset with a fully connected population of size 10 , varying the degree of co-adaptation α ranging in { 0 , 0 . 1 , 0 . 5 , 0 . 9 , 1 } .
Revised: We perform this experiment on the Attribute/Values dataset with a fully connected population of size 10 , varying the degree of co-adaptation α ranging in { 0 , 0 . 1 , 0 . 5 , 0 . 9 , 1 } .

Original: α = 0 corresponds to partitioned training
Revised: α = 0 corresponds to partitioned training

Original: whereas α = 1 is equivalent to standard training.
Revised: whereas α = 1 is equivalent to standard training.

Original: All populations converge to > 99% accuracy.
Revised: All populations converge to > 99% accuracy.

Original: However, in Figure 5b we find that topographic similarity drops as soon as we introduce minimal amounts of co-adaptation ( α = 0 .
Revised: However, in Figure 5b we find that topographic similarity drops as soon as we introduce minimal amounts of co-adaptation ( α = 0 .

Original: and decreases steadily to the level of standard populations as α grows to 1 .
Revised: and decreases steadily to the level of standard populations as α grows to 1 .

Original: This further corroborates our hypothesis that reducing co-adaptation promotes the emergence of a more structured language, and that eliminating it altogether (in a partitioned population) yields the best results.
Revised: This further corroborates our hypothesis that reducing co-adaptation promotes the emergence of a more structured language, and that eliminating it altogether (in a partitioned population) yields the best results.

Original: 6. I MPORTANCE OF M UTUAL I NTELLIGIBILITY
Revised: 6. I MPORTANCE OF M UTUAL I NTELLIGIBILITY

Original: Recall that the objective of a partitioned population at the equilibrium (Equation 6) can be decomposed in two terms: an “internal communication” corresponding to the single agent pair objective and a “mutual intelligibility” term which encourages senders to align their languages.
Revised: Recall that the objective of a partitioned population at the equilibrium (Equation 6) can be decomposed in two terms: an “internal communication” corresponding to the single agent pair objective and a “mutual intelligibility” term which encourages senders to align their languages.

Original: Importantly, the latter is the only element that separates a partitioned population from a collection of isolated agents.
Revised: Importantly, the latter is the only element that separates a partitioned population from a collection of isolated agents.

Original: To measure its effect on the compositionality of the emergent language, we train fully connected populations of size 10 and decrease the relative weight of the mutual intelligibility term.
Revised: To measure its effect on the compositionality of the emergent language, we train fully connected populations of size 10 and decrease the relative weight of the mutual intelligibility term.

Original: This is implemented by making the pair ( π θ i , ρ θ i ) more likely to be sampled than other pairs ( π θ i , ρ θ j ) , j ̸
Revised: This is implemented by making the pair ( π θ i , ρ θ i ) more likely to be sampled than other pairs ( π θ i , ρ θ j ) , j ̸

Original: = i by a factor × 1 − ββ .
Revised: = i by a factor × 1 − ββ .

Original: We let β range from 0 .
Revised: We let β range from 0 .

Original: 5 (partitioned population) to 0 .
Revised: 5 (partitioned population) to 0 .

Original: (collection of isolated sender-receiver pairs).
Revised: (collection of isolated sender-receiver pairs).

Original: In Figure 5c, we find that emergent languages retain high topographic similarity even at small β , and the sharp drop-off occurs only when β is very close to 0.This confirms [Figure] i c
Revised: In Figure 5c, we find that emergent languages retain high topographic similarity even at small β , and the sharp drop-off occurs only when β is very close to 0.

Original: [Figure] that the mutual intelligibility term exerts a strong pressure towards compositionality.
Revised: This confirms that the mutual intelligibility term exerts a strong pressure towards compositionality.

Original: We investigate the evolution of the two terms during training in Appendix E.
Revised: We investigate the evolution of the two terms during training in Appendix E.

Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There is a rich history of modeling the emergence of language as the solution to a cooperative game that can be traced back to functional theories of language (Wittgenstein, 1953; Austin, 1962; Clark, 1996).
Revised: There is a rich history of modeling the emergence of language as the solution to a cooperative game that can be traced back to functional theories of language (Wittgenstein, 1953; Austin, 1962; Clark, 1996).

Original: With a regain of interest for the study of language evolution (Crawford & Sobel, 1982;
Revised: With a regain of interest for the study of language evolution (Crawford & Sobel, 1982;

Original: Christiansen & Kirby, 2003), a rich literature has developed around computational simulations of the emergence of language based on simple language games (Lewis, 1969; Skyrms, 2010; Batali, 1998; Cangelosi & Parisi, 2002).
Revised: Christiansen & Kirby, 2003), a rich literature has developed around computational simulations of the emergence of language based on simple language games (Lewis, 1969; Skyrms, 2010; Batali, 1998; Cangelosi & Parisi, 2002).

Original: Examples include studying evolutionary models of the emergence of grammar (Nowak & Komarova, 2001), the influence of cultural transmission (Brighton & Kirby, 2006), game theoretical considerations (Huttegger et al., 2014) or linguistic diversity (Livingstone &
Revised: Examples include studying evolutionary models of the emergence of grammar (Nowak & Komarova, 2001), the influence of cultural transmission (Brighton & Kirby, 2006), game theoretical considerations (Huttegger et al., 2014) or linguistic diversity (Livingstone &

Original: Fyfe, 1999) among others.
Revised: Fyfe, 1999) among others.

Original: The recent success of deep learning in natural language processing has spurred interest in studying signaling games between deep neural network trained with reinforcement learning to solve a signaling game (Lazaridou et al., 2017; Foerster et al., 2016).
Revised: The recent success of deep learning in natural language processing has spurred interest in studying signaling games between deep neural network trained with reinforcement learning to solve a signaling game (Lazaridou et al., 2017; Foerster et al., 2016).

Original: Several follow-ups have taken this idea further by extending it to more complex games or environment (Sukhbaatar et al., 2016; Havrylov & Titov, 2017; Jaques et al., 2019; Das et al., 2019) or by adding an element of competition (Singh et al., 2018;
Revised: Several follow-ups have taken this idea further by extending it to more complex games or environment (Sukhbaatar et al., 2016; Havrylov & Titov, 2017; Jaques et al., 2019; Das et al., 2019) or by adding an element of competition (Singh et al., 2018;

Original: Noukhovitch et al., 2021) or negotiation (Cao et al., 2018) or even explicit pressure towards certain desirable properties (Kottur et al., 2017; Choi et al., 2018; Li & Bowling, 2019; Ren et al., 2019).
Revised: Noukhovitch et al., 2021) or negotiation (Cao et al., 2018) or even explicit pressure towards certain desirable properties (Kottur et al., 2017; Choi et al., 2018; Li & Bowling, 2019; Ren et al., 2019).

Original: In parallel, several efforts have been made to understand the properties of the emergent languages (Bouchacourt & Baroni, 2018; Chaabouni et al., 2019; 2020).
Revised: In parallel, several efforts have been made to understand the properties of the emergent languages (Bouchacourt & Baroni, 2018; Chaabouni et al., 2019; 2020).

Original: Within this growing literature, many authors have explicitly studied the use of populations of more than two agents.
Revised: Within this growing literature, many authors have explicitly studied the use of populations of more than two agents.

Original: Various works have argued for augmenting populations with an explicit pressure towards more structure languages, via e.g. generational transmission (Cogswell et al., 2019), adversarial regularization (Tieleman et al., 2019), varying learning speeds (Rita et al., 2022a) or imitation learning and voting (Chaabouni et al., 2022).
Revised: Various works have argued for augmenting populations with an explicit pressure towards more structure languages, via e.g. generational transmission (Cogswell et al., 2019), adversarial regularization (Tieleman et al., 2019), varying learning speeds (Rita et al., 2022a) or imitation learning and voting (Chaabouni et al., 2022).

Original: Although the focus is often on fully-connected populations, some authors have explored more complex communication graphs, for the purpose of modeling contact linguistics (Harding Graesser et al., 2019) or the effect of social network structure on the language (Dubova et al., 2020).
Revised: Although the focus is often on fully-connected populations, some authors have explored more complex communication graphs, for the purpose of modeling contact linguistics (Harding Graesser et al., 2019) or the effect of social network structure on the language (Dubova et al., 2020).

Original: Recent work from Kim & Oh (2021) is perhaps closest to our own: the authors study the effect of population size and connectivity in the standard training paradigm.
Revised: Recent work from Kim & Oh (2021) is perhaps closest to our own: the authors study the effect of population size and connectivity in the standard training paradigm.

Original: The purpose of this paper is to highlight the impact of the training procedure on these very effects.
Revised: The purpose of this paper is to highlight the impact of the training procedure on these very effects.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Empirical findings in socio-linguistics suggest that population dynamics should help in simple sender-receiver communication games.
Revised: Empirical findings in socio-linguistics suggest that population dynamics should help in simple sender-receiver communication games.

Original: In this paper, we observed that populations trained by naively extending the simple 1-1 protocol to N × N agent pairs fail to exhibit some of the properties that are observed in human populations.
Revised: In this paper, we observed that populations trained by naively extending the simple 1-1 protocol to N × N agent pairs fail to exhibit some of the properties that are observed in human populations.

Original: Motivated by an analysis of populations at the equilibrium, we described an alternative training paradigm, based on agents partitioning to reduce co-adaptation.
Revised: Motivated by an analysis of populations at the equilibrium, we described an alternative training paradigm, based on agents partitioning to reduce co-adaptation.

Original: Empirically, we find that partitioning enables us to recover some of the aforementioned properties.
Revised: Empirically, we find that partitioning enables us to recover some of the aforementioned properties.

Original: Our findings call attention to the fact that there is more than one way to generalize two single to many agents, and simple design choices can have a great impact on the training dynamics and ultimately the effect of population on the emergent language.
Revised: Our findings call attention to the fact that there is more than one way to generalize two single to many agents, and simple design choices can have a great impact on the training dynamics and ultimately the effect of population on the emergent language.

Original: Beyond emergent communication, we hope that this observation can inspire similar work in other cooperative multi-agent problems where co-adaptation between agents may counteract population effects.
Revised: Beyond emergent communication, we hope that this observation can inspire similar work in other cooperative multi-agent problems where co-adaptation between agents may counteract population effects.

Original: A A PPENDIX B D ERIVATION OF THE O PTIMAL R ECEIVER
Revised: A A PPENDIX B D ERIVATION OF THE O PTIMAL R ECEIVER

Original: We first prove a more general result from which the optimal receiver both in the standard and partitioned can be derived.
Revised: We first prove a more general result from which the optimal receiver both in the standard and partitioned can be derived

Original:  B. G ENERAL C ASE
Revised: . B. G ENERAL C ASE

Original: Consider a receiver j trained to maximize
Revised: Consider a receiver j trained to maximize

Original: [Equation]
Revised: [Equation]

Original: (7) where α i =1 ...n are arbitrary weights for the senders (we assume that the α i are positive and sum to one).
Revised: (7) where α i =1 ...n are arbitrary weights for the senders (we assume that the α i are positive and sum to one).

Original: We can rewrite the objective as:
Revised: We can rewrite the objective as:

Original: [Equation]
Revised: [Equation]

Original: Note that by linearity of expectation we can pass the α i weighted average over the senders inside of the expectation and rewrite the second expectation in terms of the mixture π ∗ α ( m | x ) :
Revised: Note that by linearity of expectation we can pass the α i weighted average over the senders inside of the expectation and rewrite the second expectation in terms of the mixture π ∗ α ( m | x ) :

Original: = (cid:80) 
Revised: = (cid:80) i ∈

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: α

Original: [Equation] i
Revised: i π θ ∗ i

Original: [Equation]
Revised: [Equation]

Original: m 
Revised: m | x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: With slight abuse of notation, let us now denote by π ∗ α ( m ) :
Revised: With slight abuse of notation, let us now denote by π ∗ α ( m ) :

Original: = E x ∼ p π ∗ α ( m | x ) the marginal distribution over messages and π ∗ α ( x | m ) :
Revised: = E x ∼ p π ∗ α ( m | x ) the marginal distribution over messages and π ∗ α ( x | m ) :

Original: = π ∗ α ( m | x ) p ( x ) π ∗ α ( m ) the associated posterior.
Revised: = π ∗ α ( m | x ) p ( x ) π ∗ α ( m ) the associated posterior.

Original: Notice that since by definition π ∗ α ( m | x ) p ( x ) = π ∗ α ( x | m ) π ∗ α ( m ) , we can rewrite the double expectation E x ∼ p E m ∼ π ∗ α ( ·| x ) as E m ∼ π ∗ α ( · ) E x ∼ π ∗ α ( ·| m ) by inverting the order of summation.
Revised: Notice that since by definition π ∗ α ( m | x ) p ( x ) = π ∗ α ( x | m ) π ∗ α ( m ) , we can rewrite the double expectation E x ∼ p E m ∼ π ∗ α ( ·| x ) as E m ∼ π ∗ α ( · ) E x ∼ π ∗ α ( ·| m ) by inverting the order of summation.

Original: We can therefore rewrite
Revised: We can therefore rewrite

Original: [Equation]
Revised: [Equation]

Original: where H ( p, q ) denotes the cross-entropy E q [ − log p ] of two distributions p and q .
Revised: where H ( p, q ) denotes the cross-entropy E q [ − log p ] of two distributions p and q .

Original: Importantly the cross-entropy is non-negative and H ( p, q )
Revised: Importantly the cross-entropy is non-negative and H ( p, q )

Original: = 

Revised: =

Original: if and only if p = q .
Revised: if and only if p = q .

Original: Consequently, the receiver ρ ψ will be optimal ( J r,j ( ψ j ) = 0 )
Revised: Consequently, the receiver ρ ψ will be optimal ( J r,j ( ψ j ) = 0 )

Original: if and only if for all m :
Revised: if and only if for all m :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: □ B. O PTIMAL R ECEIVER IN S TANDARD P OPULATIONS
Revised: □ B. O PTIMAL R ECEIVER IN S TANDARD P OPULATIONS

Original: Recall that in standard populations, the training objective for receiver j is:
Revised: Recall that in standard populations, the training objective for receiver j is:

Original: J r,j ( ψ
Revised: [Equation] [Equation] ( j

Original: [Equation]
Revised: [Equation]

Original: ) = | N G ( j ) | (cid:88) i ∈N G ( j )
Revised: | (cid:88) i ∈N G ( j )

Original: J r,i → j ( ψ j ) .
Revised: J r,i → j ( ψ j ) .

Original: Note that this is a special case of Equation 7 with α i = (cid:40) 1 |N G
Revised: Note that this is a special case of Equation 7 with α i = (cid:40) 1 |N G

Original: ( j ) | if i ∈ N G ( j ) otherwise
Revised: ( j ) | if i ∈ N G ( j ) otherwise

Original: Consequently, the derivation in Section B.1 tells us that the optimal receiver is
Revised: Consequently, the derivation in Section B.1 tells us that the optimal receiver is

Original: [Equation]
Revised: [Equation]

Original: Where π ∗N G ( j ) ( m | x ) :
Revised: Where π ∗N G ( j ) ( m | x ) :

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: B. O PTIMAL R ECEIVER IN P ARTITIONED P OPULATIONS
Revised: B. O PTIMAL R ECEIVER IN P ARTITIONED P OPULATIONS

Original: In partitioned populations
Revised: In partitioned populations

Original: [Equation]
Revised: [Equation]

Original: the training objective for receiver j is:
Revised: the training objective for receiver j is:

Original: [Equation]
Revised: [Equation]

Original: This is also a special case of Equation 7 with
Revised: This is also a special case of Equation 7 with

Original: [Equation]
Revised: [Equation]

Revised: otherwise

Original: The derivation in Section B.1 thus yields the optimal receiver
Revised: The derivation in Section B.1 thus yields the optimal receiver

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: B. O PTIMAL R ECEIVER IN P ARTIALLY P ARTITIONED P OPULATIONS
Revised: B. O PTIMAL R ECEIVER IN P ARTIALLY P ARTITIONED P OPULATIONS

Original: In the partially partitioned populations used in Section 6.3, each receiver’s objective is a mixture between the standard and partitioned objective.
Revised: In the partially partitioned populations used in Section 6.3, each receiver’s objective is a mixture between the standard and partitioned objective.

Original: This can also be rewritten as a special case of
Revised: This can also be rewritten as a special case of

Original: Equation 7 with
Revised: Equation 7 with

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: otherwise

Original: The optimal receiver can then be rewritten as the posterior distribution associated with the mixture sender
Revised: The optimal receiver can then be rewritten as the posterior distribution associated with the mixture sender

Original: [Equation]
Revised: [Equation]

Original: C T HE C ASE OF R EFERENTIAL G AMES
Revised: C T HE C ASE OF R EFERENTIAL G AMES

Original: In the analysis from Section 2.2 onward, we assumed C
Revised: In the analysis from Section 2.2 onward [Equation] we assumed C

Original: =
Revised: =

Original: X to simplify notation.
Revised: X to simplify notation.

Original: We can relax this assumption without changing our key observation that all receivers are the same at the optimum.
Revised: We can relax this assumption without changing our key observation that all receivers are the same at the optimum.

Original: Indeed, in this case the receiver’s objective in a standard population is:
Revised: Indeed, in this case the receiver’s objective in a standard population is:

Original: [Equation]
Revised: [Equation]

Original: This objective, called InfoNCE (Oord et al., 2018) also has an analytical solution that can be expressed as a function of π ∗N G ( j ) , of the form:
Revised: This objective, called InfoNCE (Oord et al., 2018) also has an analytical solution that can be expressed as a function of π ∗N G ( j ) , of the form:

Original: [Equation]
Revised: [Equation]

Original: Despite the more complicated form of the optimal receiver, the key ingredients to our analysis in Sections 2.2 and 3 are preserved: at the optimum, each receiver is a function of the posterior π N G ( j ) ( x | m ) associated with the communication partners to which it co-adapts.
Revised: Despite the more complicated form of the optimal receiver, the key ingredients to our analysis in Sections 2.2 and 3 are preserved: at the optimum, each receiver is a function of the posterior π N G ( j ) ( x | m ) associated with the communication partners to which it co-adapts.

Original: A similar analysis in partitioned populations shows that the optimum for receiver j then only depends on the posterior associated with its respective sender π θ ∗ j instead.
Revised: A similar analysis in partitioned populations shows that the optimum for receiver j then only depends on the posterior associated with its respective sender π θ ∗ j instead.

Original: D E XPERIMENTS WITH R ING P OPULATIONS WITH GREATER CONNECTIVITY
Revised: D E XPERIMENTS WITH R ING P OPULATIONS WITH GREATER CONNECTIVITY

Original: The circular population described in Section 5 is an extreme case of a connected population where each agent has the minimal amount of neighbors (2).
Revised: The circular population described in Section 5 is an extreme case of a connected population where each agent has the minimal amount of neighbors (2).

Original: To verify that our findings apply to more general
Revised: To verify that our findings apply to more general

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: (non fully-connected) populations, we perform the same experiment on ring populations with higher connectivities.
Revised: (non fully-connected) populations, we perform the same experiment on ring populations with higher connectivities.

Original: We define a ring graph with connectivity k as a circular graph where each vertex is connected to its k -nearest neighbors on each side (ie agent i is connected to agents i − k, . .
Revised: We define a ring graph with connectivity k as a circular graph where each vertex is connected to its k -nearest neighbors on each side (ie agent i is connected to agents i − k, . .

Original: , i + k ).
Revised: , i + k ).

Original: Examples are shown in Figure 

Revised: Examples are shown in Figure

Original: This allows us to interpolate between circular populations ( k = 1 ) and fully connected populations ( k = n/ 2 ) while jointly (1) increasing the number of communication partner of a given agent and (2) decreasing the path length between any two agents, two properties of the graph that are arguably most impactful on the mutual intelligibility between agents.
Revised: This allows us to interpolate between circular populations ( k = 1 ) and fully connected populations ( k = n/ 2 ) while jointly (1) increasing the number of communication partner of a given agent and (2) decreasing the path length between any two agents, two properties of the graph that are arguably most impactful on the mutual intelligibility between agents.

Original: We perform the same experiment as Section 5 with graphs of connectivities k = 2 and k = 3 (resp.and 6 training partners for each agent) on the Attribute/values dataset.
Revised: We perform the same experiment as Section 5 with graphs of connectivities k = 2 and k = 3 (resp.and 6 training partners for each agent) on the Attribute/values dataset.

Original: We report accuracues in Table 3.
Revised: We report accuracues in Table 3.

Original: We observe that across the board, populations with greater connectivity exhibit higher accuracy in both conditions.
Revised: We observe that across the board, populations with greater connectivity exhibit higher accuracy in both conditions.

Original: However, we still find that partitioned populations still perform best.
Revised: However, we still find that partitioned populations still perform best.

Original: E F URTHER A NALYSIS OF THE E FFECT OF M UTUAL I NTELLIGIBILITY
Revised: E F URTHER A NALYSIS OF THE E FFECT OF M UTUAL I NTELLIGIBILITY

Original: In Section 6.4, we find that languages stay highly compositional until the mutual intelligibility weight β is decreased to almost 0 .
Revised: In Section 6.4, we find that languages stay highly compositional until the mutual intelligibility weight β is decreased to almost 0 .

Original: Our hypothesis is that even with small amounts of mutual intelligibility, agents will eventually have to optimize this part of the objective after they have maximized their respective internal communication to the point where the main contributor to the training gradient is the mutual intelligibility term.
Revised: Our hypothesis is that even with small amounts of mutual intelligibility, agents will eventually have to optimize this part of the objective after they have maximized their respective internal communication to the point where the main contributor to the training gradient is the mutual intelligibility term.

Original: To verify this hypothesis, in Figure 7 we report the evolution of both internal communication and mutual intelligibility losses during training for various values of the mutual intelligibility weight β .
Revised: To verify this hypothesis, in Figure 7 we report the evolution of both internal communication and mutual intelligibility losses during training for various values of the mutual intelligibility weight β .

Original: As expected, we observe that for all but very small values of β , the mutual intelligibility loss eventually decreases (although it decreases faster for high β ).
Revised: As expected, we observe that for all but very small values of β , the mutual intelligibility loss eventually decreases (although it decreases faster for high β ).