Original: Emergence of Multilingualism in Population based Referential Games
Revised: Emergence of Multilingualism in Population based Referential Games

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: The ability of agents to learn to communicate by interaction has been studied through emergent communication tasks.
Revised: The ability of agents to learn to communicate by interaction has been studied through emergent communication tasks.

Original: Inspired by previous work in this domain, we extend the referential game setup to a population of spatially distributed agents.
Revised: Inspired by previous work in this domain, we extend the referential game setup to a population of spatially distributed agents.

Original: In such a setting, our experiments reveal that multiple languages can emerge in the population and some agents develop multilingual traits.
Revised: In such a setting, our experiments reveal that multiple languages can emerge in the population and some agents develop multilingual traits.

Original: Further, an action-advising framework is proposed for improving sample efﬁciency in the learning process.
Revised: Further, an action-advising framework is proposed for improving sample efﬁciency in the learning process.

Original: 1. Introduction
Revised: 1. Introduction

Original: One of the long-standing challenges of AI is to develop agents that are capable of communication.
Revised: One of the long-standing challenges of AI is to develop agents that are capable of coordination.

Original: However,  such a coordination requires communication.
Revised: However, communication is essential for achieving such coordination.

Original: Furthermore, if the communication is to include humans, the channel must be that of natural language.
Revised: Further, if the communication is to include humans, the medium must be that of natural language.

Original: Recent approaches involving deep learning  have shown tremendous success in  languagerelated tasks such as machine translation, sentiment analysis, image captioning (Xu et al., 2015; Sutskever et al., 2014; Zhang et al., 2018) etc.
Revised: In recent years, deep learning based approaches have shown tremendous success in natural language tasks such as machine translation, sentiment analysis, image captioning (Xu et al., 2015; Sutskever et al., 2014; Zhang et al., 2018) etc.

Original: While these approaches are good at capturing the underlying statistical patterns in language, they are severely limited in understanding the interactive nature of language.
Revised: While these approaches are good at capturing the underlying statistical patterns in language, they are severely limited in understanding the interactive nature of language.

Original: Recently, there has been a widespread adoption of using referential games for facilitating emergence of natural language in multi-agent setting (Lazaridou et al., 2017;Havrylov &
Revised: More recently, there has been a widespread adoption of using referential games for facilitating emergence of natural language in multi-agent setting (Lazaridou et al., 2017;

Original:  Titov, 2017; Lazaridou et al., 2018; Evtimova et al., 2018).
Revised: Havrylov & Titov, 2017; Lazaridou et al., 2018; Evtimova et al., 2018; Graesser et al., 2019).

Original: However, these works lack several aspects related to language learning and evolution in humans.
Revised: However, these works are often limited in some aspects related to language learning and evolution in humans.

Original: We argue that language in humans emerges through interaction between population of human agents and their collective behaviour.
Revised: We argue that language in humans emerges through interaction between population of human agents and their collective behaviour.

Original: This human population can exist in the form of smaller, society based community groups, which are spatially distributed and develop their own distinctive linguistic characteristics (Lupyan & Dale, 2010; Wray & Grace, 2007).
Revised: This human population can exist in the form of smaller, society based community groups, which are spatially distributed and de- velop their own distinctive linguistic characteristics (Lupyan & Dale, 2010; Wray & Grace, 2007).

Original: Furthermore, interaction across these communities can result in intermixing of languages or  multilingual humans (Trudgill, 2011).
Revised: Furthermore, interaction across these communities can result in intermixing of languages or result in multilingual speakers and there may also be within group learning pressure (Trudgill, 2011).

Original: Thus, in order to simulate such emergence and evolution of languages in human populations, we extend the regular twoplayer setting of referential game into a Population based
Revised: Thus, in order to simulate such emergence and evolution of languages in human populations, we extend the regular twoplayer setting of referential game into a Population based

Original: Referential Game (PopRG).
Revised: Referential Game (PopRG).

Original: We introduce the notion of spatially spread out community groups in referential games where the interaction between agents can happen only within the group.
Revised: We introduce the notion of spatially spread out community groups in referential games where the interaction between agents can happen only within the group.

Original: We also consider travelling agents in our game formulation that are allowed to interact across communities.
Revised: We also consider travelling agents in our game formulation that are allowed to interact across communities.

Original: This setup helps us answer the following questions.
Revised: This setup helps us answer the following questions.

Original: Do multiple agents in a community specialize in communicating different entities during initial learning phase?
Revised: Do multiple agents in a community specialize in communicating different entities during initial learning phase?

Original: Can diverse languages emerge in different community groups through a Population based Referential Game? 3.
Revised: Can diverse languages emerge in different community groups through a Population based Referential Game? 3.

Original: If some agents are allowed to interact across communities, will it result inconvergence to a single global language or multilingual agents?
Revised: If some agents are allowed to interact across communities, will it result in multilingual agents?

Original: Drawing inspiration from successes in agent teaching agent (ATA) frameworks (da Silva et al., 2020b), we also demonstrate how action advising in PopRG can help improve sample efﬁciency by alleviating issues pertaining to both sparse rewards and non-stationarity.
Revised: Drawing inspiration from successes in agent teaching agent (ATA) frameworks (da Silva et al., 2020b), we also demonstrate how action advising in PopRG can help improve sample efﬁciency by alleviating issues pertaining to both sparse rewards and non-stationarity.

Original: 2. Related Work
Revised: 2. Related Work

Original: Recent years have seen a substantial surge in works on multi agent systems based on neural networks that communicate in order to solve a problem.
Revised: Recent years have seen a substantial surge in works on multi-agent systems based on neural networks that communicate in order to solve a problem.

Original: (Foerster et al., 2016;
Revised: (Foerster et al., 2016;

Original: Sukhbaatar & Fergus, 2016; Tampuu et al., 2017; Lazaridou et al., 2018; Evtimova et al., 2018)
Revised: Sukhbaatar & Fergus, 2016; Tampuu et al., 2017; Lazaridou et al., 2018; Evtimova et al., 2018)

Original: Amongst these works, those that are most closely aligned with our goal are based on the idea of investigating emergence of communication in referential games.
Revised: Amongst these works, those that are most closely aligned with our goal are based on the idea of investigating emergence of communication in referential games.

Original: Referential games are a form of Lewis Signalling games (Lewis, 1969), in which one agent (speaker) sees some entity and must communicate about it using a discrete message token.
Revised: Referential games are a form of Lewis Signalling games (Lewis, 1969), in which one agent (speaker) sees some entity and must communicate about it using a discrete message token.

Original: Given this message token, the original entity and some distracting entities, another agent (listener), must ﬁgure out what the ﬁrst agent was referring to.
Revised: Given this message token, the original entity and some distracting entities, another agent (listener), must ﬁgure out what the ﬁrst agent was referring to.

Original: In (Lazaridou et al., 2017), ﬁrst line of work in solving referential games through reinforcement learning is proposed. The authors restrict communication message to be a single token chosen from a ﬁxed vocabulary.
Revised: Several variations of this game have been proposed to study how the language can emerge.

Original: (Havrylov & Titov, 2017) extended this framework to pose messages as sequence of symbols by modelling speaker and listener as recurrent networks and used gumbelsoftmax estimator between speaker and listener to make the game end-to-end differentiable. (Lazaridou et al., 2018) also demonstrate that communication protocolscan also emerge inenvironments with structured symbolic or unstructured raw pixel inputs.
Revised: This includes referential games with image as input and single message token for communication (Lazaridou et al., 2017), sequence of message tokens for communication and end-to-end differentiability (Havrylov & Titov, 2017), emergent communication protocols in structured symbolic or unstructured raw pixel inputs (Lazaridou et al., 2018), and multi-step communication with different modalities (Evtimova et al., 2018).

Original: A variation of referential game was proposed by (Evtimova et al., 2018), where the speaker and listener see different modalities of the same entity. The authors also incorporate multi-step communication wherein both agents can exchange messages back and forth until either the listener ends communication or a maximum communication length is exceeded.
Revised: Another set of work explores different properties of the emergent language such as compositionality and generalization (Chaabouni et al., 2020), entropy (Kharitonov et al., 2019), word-order biases (Chaabouni et al., 2019) and visual representations (Bouchacourt & Baroni, 2018).

Original: While all of these works cover different functional aspects of natural language communication, one aspect that is missing in them is interaction between alarge population of agents.
Revised: While all of these works cover different functional aspects of natural language communication, one phenomenon that is often not studied extensively is interaction in a population of agents.

Original: In this regard, (Fitzgerald, 2019; Tieleman et al.,  2019) are some notable works that have explored emergent communication in population of agents.
Revised: In this regard, (Fitzgerald, 2019; Tieleman et al., 2019; Lowe* et al., 2020; Graesser et al., 2019) are some notable works that have explored emergent communication in population of agents.

Original: (Tieleman et al., 2019) suggests the use of community-based learning and explores how size of community affects complexity of emergent language.
Revised: Tieleman et al. (2019) suggest the use of community-based learning and explore how size of community affects complexity of emergent language.

Original: However, they learn emergent communication in an autoencoder setting rather then referential game framework, thus letting messagesto pass through a differentiable channel.
Revised: However, they learn emergent communication in an autoencoder setting rather than referential game framework, thus, letting messages pass through a differentiable channel.

Original: In (Fitzgerald, 2019), a population based referential game is proposed for the purpose of robust representations of unstructured data.
Revised: In the work by Fitzgerald (2019), a population based referential game is proposed for the purpose of robust representations of unstructured data.

Revised: Lowe* et al.

Revised: (2020) combine self-play and supervised learning and also introduce the notion of population of agents.

Original: Both of these works, however, do not consider any interaction between agents based on spatial arrangement.
Revised: But the languages learned in the population are ﬁnally distilled back to a single agent. Finally, the work by Graesser et al. (2019) is most closely related to our work as they consider spatial topology in population of agents and interaction between multiple languages through contact. However, sample efﬁciency is not taken into consideration.

Original: Apart from referential games, a multitude of literature has been proposed in action advising frameworks in multi agent tasks.
Revised: Apart from referential games, a multitude of literature has been proposed in action advising frameworks in multi agent tasks.

Original: These works are based on the idea that in scenarios with sparse rewards or partial observability, an expert agent (teacher) can guide a learner (student) to take optimal actions through demonstrations.
Revised: These works are based on the idea that in scenarios with sparse rewards or partial observability, an expert agent (teacher) can guide a learner (student) to take optimal actions through demonstrations.

Original: This has been shown to make several complex tasks tractable and improved sample efﬁciency.
Revised: This has been shown to make several complex tasks tractable and improved sample efﬁciency.

Original: We refer the reader to a survey paper on Agent teaching agents (da Silva et al., 2020b).
Revised: We refer the reader to a survey paper on agent teaching agents (da Silva et al., 2020b).

Original: Recently, simultaneously acting and learning (da Silva et al., 2017) and uncertainty aware advising (da Silva et al., 2020a) have been proposed to facilitate action advising when expert is not available or advisor is sub-optimal.
Revised: Recently, simultaneously acting and learning (da Silva et al., 2017) and uncertainty aware advising (da Silva et al., 2020a) have been proposed to facilitate action advising when expert is not available or advisor is sub-optimal.

Original: In our work, we show how a simple action advising scheme can be used in Population based Referntial Games for mitigating the effects of non-stationarity and sparse rewards.
Revised: In our work, we show how a simple action advising scheme can be used in Population based Referential Games for mitigating the effects of non-stationarity and sparse rewards.

Original: 3. Games and Terminology 3.1. Paired Referential Game
Revised: 3. Games and Terminology 3.1. Paired Referential Game

Original: In our work, we adapt the referential game formulation given in (Lazaridou et al., 2018).
Revised: In our work, we adapt the referential game formulation given by Lazaridou et al.

Original: This game is a variation of
Revised: This game is a variation of

Original: Lewis Signalling game (Lewis, 1969), and has been widely used in language evolution literature.
Revised: Lewis Signalling game (Lewis, 1969), and has been widely used in language evolution literature.

Original: The game consists of two players, a Speaker and a Listener.
Revised: The game consists of two players, a Speaker and a Listener.

Original: From a given set of entities E , we sample a target entity t ∈ E and K −distracting entities D
Revised: From a given set of entities E , we sample a target entity t ∈ E and K −distracting entities D

Original: = { d 1 , d 2 , ..., d k − 1 } s.t. ∀
Revised: = { d 1 , d 2 , ..., d k − 1 } s.t. ∀

Original: j t j (cid:54) =
Revised: j t j (cid:54) =

Original: d j , d j ⊂ E .
Revised: d j , d j ⊂ E .

Original: The speaker is shown the target entity t and must come up with a message token m chosen from a ﬁxed vocabulary V .
Revised: The speaker is shown the target entity t and must come up with a message token m chosen from a ﬁxed vocabulary V .

Original: The candidate set C = t ∪ D contains both the target and distracting entities.
Revised: The candidate set C = t ∪ D contains both the target and distracting entities.

Original: The listener is then shown
Revised: The listener is then shown

Original: U , a random permutation of C and the message token m .
Revised: U , a random permutation of C and the message token m .

Original: Communication success is deﬁned when the listener can correctly identify the target and a payoff of 1 is given to both the players.
Revised: Communication success is deﬁned when the listener can correctly identify the target and a payoff of 1 is given to both the players.

Original: In all other cases, payoff is 0.
Revised: In all other cases, payoff is 0.

Original: 3.2. Population based Referential Game
Revised: 3.2. Population based Referential Game

Original: In this section, we formulate an extension of two player referential game setup for a population of agents while also considering inter-agent interaction based on their spatial arrangement.
Revised: In this section, we formulate an extension of two player referential game setup for a population of agents while also considering inter-agent interaction based on their spatial arrangement.

Original: First, we propose the concept of community which is a closed group of agents that is geographically separated from other communities.
Revised: First, we deﬁne community as a closed group of agents that is geographically separated from other communities.

Original: A population of agents can have multiple coexisting communities and each agent is assigned to one of them.
Revised: A population of agents can have multiple coexisting communities and each agent is assigned to one of them.

Original: We then deﬁne three kinds of agents, a local speaker, a travelling speaker and a listener.
Revised: We then deﬁne three kinds of agents, a local speaker, a travelling speaker and a listener.

Original: Local speakers can only interact with agents within their community while travelling speakers can interact with agents across all the communities.
Revised: Local speakers can only interact with agents within their community while travelling speakers can interact with agents across all the communities.

Original: Listener agents are same as those described in paired referential game setup and we restrict them from moving out of their community.
Revised: Listener agents are same as those described in paired referential game setup and we restrict them from moving out of their community.

Original: The Population based Referential Game (PopRG) can then be represented using the tuple < c, N s , N l , T, F > , where c is the number of regionally separated communities, N s = { N 1 s , N 2 s , ..., N cs } represents the set of number of speakers N is in the i th community.
Revised: The Population based Referential Game (PopRG) can then be represented using the tuple < c, N s , N l , T, F > , where c is the number of regionally separated communities, N s = { N 1 s , N 2 s , ..., N cs } represents the set of number of speakers N is in the i th community.

Original: Similarly, N l = { N 1 l , N 2 l , ..., N cl } represents the set of number of listeners
Revised: Similarly, N l = { N 1 l , N 2 l , ..., N cl } represents the set of number of listeners

Original: N il in the i th community.
Revised: N il in the i th community.

Original: T represents number of travelling speakers that can interact across communities and
Revised: T represents number of travelling speakers that can interact across communities and

Original: F : a ⇒ W ,  a mapping from agent a to community index.
Revised: F : a ⇒ W , is a mapping from agent a to community index.

Original: We model the game play in PopRG as follows.
Revised: We model the game play in PopRG as follows.

Original: At every game-step, from the population of speaker agents (both local and travelling), we randomly sample a speaker agent s .
Revised: At every game-step, from the population of speaker agents (both local and travelling), we randomly sample a speaker agent s .

Original: If s is a local speaker, it is paired with a random listener l from the community F ( s ) .
Revised: If s is a local speaker, it is paired with a random listener l from the community F ( s ) .

Original: Else, if s is a travelling speaker, it is paired with a listener l , randomly sampled from the population of all listener agents.
Revised: Else, if s is a travelling speaker, it is paired with a listener l , randomly sampled from the population of all listener agents.

Original: At this point, a regular paired referential game, described in previous section, is played between agents s and l .
Revised: At this point, a regular paired referential game, described in previous section, is played between agents s and l .

Original: 4. Implementation Details 4.1. Agents
Revised: 4. Implementation Details 4.1. Agents

Original: The local speaker, travelling speaker and listener agents are all modelled as reinforcement learning policies.
Revised: The local speaker, travelling speaker and listener agents are all modelled as reinforcement learning policies.

Original: These policies are represented by neural network with separate parameters for every agent.
Revised: These policies are represented by neural network with separate parameters for every agent.

Original: During gameplay, the speaker agents observe a vector representation of target entity t along with a one hot encoded representation of the community in which the paired listener agent resides.
Revised: During gameplay, the speaker agents observe a vector representation of target entity t along with a one hot encoded representation of the community in which the paired listener agent resides.

Original: Community information is included soso that travelling speakers are aware of the community in which they are playing the referential game.
Revised: Community information is included so that travelling speakers are aware of the community in which they are playing the referential game.

Original: For the listener agent, the observation space contains vector representations of all the entities in the permuted candidate set U , concatenated with message token m .
Revised: For the listener agent, the observation space contains vector representations of all the entities in the permuted candidate set U , concatenated with message token m .

Original: The output of the speaker agents is anan integer between 0 to V − 1 which represents discrete message.
Revised: The output of the speaker agents is an integer between 0 to V − 1 which represents discrete message.

Original: The listener agent returns a value between 0 to K − 1 that points to target t (cid:48) among U .
Revised: The listener agent returns a value between 0 to K − 1 that points to target t (cid:48) among U .

Original: In all our experiments we take K as 3 and V as 5.
Revised: In all our experiments we take K as 3 and V as 5.

Original: 4.2. Learning and Action Advising
Revised: 4.2. Learning and Action Advising

Original: The learning goal in Population based Referential Game is maximization of sum of rewards for all the agents.
Revised: The learning goal in Population based Referential Game is maximization of sum of rewards for all the agents.

Original: While most of  previous works on referential games have used REINFORCE update rule for optimizing policies (Williams, 1992), we found that it is hard to converge and isvery sensitive to hyperparametrs in PopRG.
Revised: While most of the previous works on referential games have used REINFORCE update rule for optimizing policies (Williams, 1992), we found that it is hard to converge and is sensitive to hyperparameters in PopRG.

Original: Thus, in our work, we have used Proximal Policy Optimization (PPO)
Revised: Thus, in our work, we have used Proximal Policy Optimization (PPO)

Original: (Schulman et al., 2017), for updating weights of speaker and listener agents.
Revised: (Schulman et al., 2017), for updating weights of speaker and listener agents.

Original: Additionally, to improve sample efﬁciency and counter extreme non-stationarity and sparse rewards in referential game setup, we make use of action advising.
Revised: Additionally, to improve sample efﬁciency and counter extreme non-stationarity and sparse rewards in PopRG, we make use of action advising.

Original: Whenever a paired referential game results in communication success, we allow a speaker agent to advice to all  fellow speakers in the community.
Revised: Whenever a paired referential game results in communication success, we allow a speaker agent to advise all the fellow speakers in the community.

Original: Similarly, listener agent is allowed to advice to all fellow listeners in the community regarding the communication success.
Revised: Similarly, listener agent is allowed to advise all fellow listeners in the community regarding the communication success.

Original: Speciﬁcally, this advice is implemented as a broadcast of experience trajectory containing state transition, action and reward to fellow (similar) agents in the community.
Revised: Speciﬁcally, this advice is implemented as a broadcast of experience trajectory containing state transition, action and reward to fellow (similar) agents in the community.

Original: Note that a travelling speaker canadvice only to the speakers of the community in which communication success was achieved.
Revised: Note that a travelling speaker can only advise the speakers of the community in which communication success was achieved.

Original: However, it can receive advice from all  speakers (as their experience trajectory will contain community information).
Revised: However, it can receive advice from all the speakers (as their experience trajectory will contain community information).

Original: 4.3. Data
Revised: 4.3. Data

Original: In all our experiments, we call entities as classes of images in the CIFAR-10 dataset (Krizhevsky et al., 2009). 500 images are randomly samples for each class from the training set and 100 images are sampled for test set.
Revised: In all our experiments, we deﬁne entities as classes of images in the CIFAR-10 dataset (Krizhevsky et al., 2009).images are randomly samples for each class from the training set and 100 images are sampled for test set.

Original: Each image is represented as output from relu 7 layer of a pretrained
Revised: Each image is represented as output from relu 7 layer of a pretrained

Original: VGG16 network (Simonyan & Zisserman, 2014), resulting in a vector of length 512.
Revised: VGG16 network (Simonyan & Zisserman, 2014), resulting in a vector of length 512.

Original: Further, since population based referential games are computationally expensive to run, we have restricted the number of entities to 5 (out ofavailable in CIFAR-10 dataset).
Revised: Further, since population based referential games are computationally expensive to run, we have restricted the number of entities to 5 (out ofavailable in CIFAR-10 dataset).

Original: 5. Evaluation Criterion Communication Success:
Revised: 5. Evaluation Criterion Communication Success:

Original: For measuring the overall performance of the population, mean communication success for all paired referential games can be considered.
Revised: For measuring the overall performance of the population, mean communication success for all paired referential games can be considered.

Original: Language as probability distribution:
Revised: Language as probability distribution:

Original: We represent the language spoken by any speaker using a bivariate discrete probability distribution over entities and vocabulary.
Revised: We represent the language used by any speaker as a bivariate discrete probability distribution over entities and vocabulary.

Original: This can be approximated at test time by pairing the speaker with valid listeners in a large number of paired referential games and recording the frequency of message tokens for different target entities.
Revised: This can be approximated at test time by pairing the speaker with valid listeners in a large number of paired referential games and recording the frequency of message tokens for different target entities.

Original: The frequency distribution is then normalized to obtained discrete probability distribution.
Revised: The frequency distribution is then normalized to obtained discrete probability distribution.

Original: Divergence between languages
Revised: languages

Original: The divergence between two language distributions p and q is then reported as
Revised: The divergence between two language distributions p and q is then reported as

Original: [Equation]
Revised: [Equation]

Original: where D K L ( p || q ) is the Kullback–Leibler divergence between p and q .
Revised: where D KL ( p || q ) is the Kullback–Leibler divergence between p and q .

Original: This measure quantiﬁes how distinct two languages are from each other.
Revised: This measure quantiﬁes how different two languages are from each other.

Original: Using this measure, we deﬁne four metrics. Mean Intra-Community
Revised: Using this measure, we deﬁne four metrics as follows

Original:  Local Language Divergence ( Intra Local): The divergence between languages spoken by local speakers from same community, averaged over all communities.
Revised: Mean Intra-Community Local Language Divergence (Intra Local): The divergence between languages spoken by local speakers from same community, averaged over all communities.

Original: Mean Inter-Community Local Language Divergence (Inter Local) : The
Revised: Mean Inter-Community Local Language Divergence (Inter Local):

Original:  divergence between languages spoken by local speakers from different communities
Revised: The divergence between languages spoken by local speakers from different communities, averaged over all community pairs.

Original: , averaged

Original: over all community pairs. Mean Intra Community Travelling Language Divergence (Intra Travel) The divergence between languages spoken by traveller speakers when communicating with listeners from same community, averaged 
Revised: Mean Intra-Community Travelling Language Divergence (Intra Travel): The divergence between languages spoken by travelling speakers when communicating with listeners from same community, averaged over all communities.

Original: over all commu

Original: - nities

Original: Mean

Original: Inter

Original: -Community Travelling Language Divergence (Inter Travel : The divergence between languages spoken by traveller speakers when communicating with listeners 
Revised: Mean Inter-Community Travelling Language Divergence (Inter Travel): The divergence between languages spoken by travelling speakers when communicating with listeners from different communities, averaged over all community pairs.

Original: from

Original: different communities,

Original: averaged over all

Original: community pairs

Original: 6. Experiments and Results 6.1. Emergence of Specialization
Revised: 6. Experiments and Results 6.1. Emergence of Specialization

Original: We consider two PopRG, both having a single community with 3 local speakers and 3 listeners and no travelling speakers.
Revised: We consider two PopRG, both having a single community with 3 local speakers and 3 listeners and no travelling speakers.

Original: We use action advising in  one of them.
Revised: We use action advising in only one of them.

Original: Figure 1 shows average communication success of individual agents when communicating the entity ’Cat’.
Revised: Figureshows average communication success of individual agents when communicating the entity ’Cat’.

Original: It can be observed that after the initial phase of random guessing, in the game without action advising, some agents are better at communicating the speciﬁed entity than other agents (even though it is not incentivised explicitly through rewards and all speakers within a community are symmetric).
Revised: It can be observed that after the initial phase of random guessing, in the game without action advising, some agents are better at communicating the speciﬁed entity than other agents (even though it is not incentivized explicitly through rewards and all speakers within a community are symmetric).

Original: This hints that a specialization emerges among agents.
Revised: This hints that a specialization emerges among agents.

Original: As learning reaches convergence, the specialization disappears and all agents have almost similar communication success.
Revised: As learning reaches convergence, the specialization disappears and all agents have almost similar communication success.

Original: In the game with action advising, however, no such specialization is seen and all agents have very similar communication success for the speciﬁed entity.
Revised: In the game with action advising, however, no such specialization is seen and all agents have very similar communication success for the speciﬁed entity.

Original: Moreover, convergence is reached in fewer training iterations.
Revised: Moreover, convergence is reached in fewer training iterations.

Original: This suggests that in action advising, specialist agents teach fellow agents during learning, resulting in smaller success gap and faster learning.
Revised: This suggests that in action advising, specialist agents teach fellow agents during learning, resulting in smaller success gap and faster learning.

Original: 6.2. Emergence of Multilingualism
Revised: 6.2. Emergence of Multilingualism

Original: In the second experiment, we consider two different PopRG scenarios.
Revised: In the second experiment, we consider two different PopRG scenarios.

Original: Scenario one has 1 travelling speaker and 2 communities, each having 2 local speakers and 2 listeners.
Revised: Scenario one has 1 travelling speaker and 2 communities, each having 2 local speakers and 2 listeners (scenario 2-2-1).

Original: The other scenario has 1 travelling speaker and 3 communities, each having 3 local speakers and 3 listeners.
Revised: The other scenario has 1 travelling speaker and 3 communities, each having 3 local speakers and 3 listeners (scenario 3-3-1).

Original: Figure 2a and 2b show communication success over training iterations for both the scenarios, with and without action advising.
Revised: Figure 2a and 2b show communication success over training iterations for both the scenarios, with and without action advising.

Original: It can clearly be seen that using action advising results in faster convergence.
Revised: It can clearly be seen that using action advising results in faster convergence.

Original: Table 1 reports divergence metrics for these scenarios.
Revised: Tablereports divergence metrics for these scenarios.

Original: A low Intra-Local metric suggests that language within a community is consistent while high InterLocal metric shows that distinct languages have developed in different communities.
Revised: A low IntraLocal metric suggests that language within a community is consistent while high Inter-Local metric shows that distinct languages have developed in different communities.

Original: It canalso be seen that in all the
Revised: It can

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Table]
Revised: [Table]

Original:  scenarios, Intra-Travel metric is low and Inter-Travel metric is high.
Revised: also be seen that in all the scenarios, Intra-Travel metric is low and Inter-Travel metric is high.

Original: This suggests that the travelling speaker uses consistent language within a community but uses different languages when travelling across communities.
Revised: This suggests that the travelling speaker uses consistent language within a community but uses different languages when travelling across communities.

Original: Thus, the travelling agent demonstrates multilingual traits.
Revised: Thus, the travelling agent demonstrates multilingual traits that is community conditioned.

Original: However, we observe that from the given metrics, it is hard to establish how population size and advising affects multilingualism.
Revised: However, we observe that from the given metrics, it is hard to establish how population size and advising affects multilingualism.

Original: Thus, we leave it for future work.
Revised: Thus, we leave it for future work.

Original: 6.3. Conclusion
Revised: 6.3. Conclusion

Original: In this work [Table] we show that multlingualism can emerge through a population based referential game setup.
Revised: In this work, we show that multilingualism can emerge through a population based referential game setup.

Original: It is further demonstrated that action advising can improve sample efﬁciency by non-trivial factors.
Revised: It is further demonstrated that action advising can improve sample efﬁciency by non-trivial factors.

Original: Although the results reported are preliminary [Table] they are promising to motivate further exploration in poulation based referential games [Table]
Revised: Although the results reported are preliminary, they are promising to motivate further exploration in population based referential games.

Original: Hyperparameter details

Original: Both speaker and listener agents have 3-layer feed-forward policy network containinghidden units in each layer, The baseline function approximator is also a feed-forward policy network with 3 layers having 100 hidden units in each layer.

Original: ReLU function is used as activation between layers.

Original: For optimizing through PPO, the likelihood ratio clipping is kept at 0.2 .

Original: For optimizing through PPO, the likelihood ratio clipping is kept at 0.2.

Original: Moreover, we add an entropy regularization of 0.01 to encourage exploration.

Original: Adam optimizer is used with learning rate 1e-4 that is adjusted dynamically during the training.

Original: The vocabulary size V is ﬁxed at 5 and the candidate set length K is taken as 3.