Original: R EWARD S HAPING B IASES E MERGENT L ANGUAGE
Revised: S HAPED R EWARDS B IAS E MERGENT L ANGUAGE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: One of the primary characteristics of emergent phenomena is that they are determined by the basic properties of the system whence they emerge as opposed to explicitly designed constraints.
Revised: One of the primary characteristics of emergent phenomena is that they are determined by the basic properties of the system whence they emerge as opposed to explicitly designed constraints.

Original: Reinforcement learning is often used to elicit such phenomena which speciﬁcally arise from the pressure to maximize reward.
Revised: Reinforcement learning is often used to elicit such phenomena which speciﬁcally arise from the pressure to maximize reward.

Original: We distinguish two types of rewards.
Revised: We distinguish two types of rewards.

Original: The ﬁrst is the base reward which is motivated directly by the task being solved.
Revised: The ﬁrst is the base reward which is motivated directly by the task being solved.

Original: The second is shaped rewards which are designed speciﬁcally to make the task easier to learn by introducing biases in the learning process.
Revised: The second is shaped rewards which are designed speciﬁcally to make the task easier to learn by introducing biases in the learning process.

Original: The inductive bias which reward shaping introduces is problematic for emergent language experimentation because it biases the object of study: the emergent language.
Revised: The inductive bias which shaped rewards introduce is problematic for emergent language experimentation because it biases the object of study: the emergent language.

Original: The fact that shaped rewards are intentionally designed conﬂicts with the basic premise of emergent phenomena arising from basic principles.
Revised: The fact that shaped rewards are intentionally designed conﬂicts with the basic premise of emergent phenomena arising from basic principles.

Original: In this paper, we use a simple sender-receiver navigation game to demonstrate how reward shaping can 1) explicitly bias the semantics of the learned language, 2) signiﬁcantly change the entropy of the learned communication, and 3) mask the potential effects of other environmental variables of interest.
Revised: In this paper, we use a simple sender-receiver navigation game to demonstrate how shaped rewards can 1) explicitly bias the semantics of the learned language, 2) signiﬁcantly change the entropy of the learned language, and 3) mask the potential effects of other environmental variables of interest.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In emergent language research, the goal is to study language as it emergesnaturally from the  properties of the environment, communication, and agents, but there is a risk that the language simply mirrors the design choices of the environment.
Revised: In emergent language research, the goal is to study language as it emerges from the inherent properties of the environment, language, and agents. One pitfall for such experiments, though, is that the language simply mirrors some of design choices of the environment or experimental setting more generally. there is a risk that the language simply mirrors the design choices of the environment.

Revised: For example, Bullard et al.

Revised: (2021) introduce a method for discovering optimal languages for communication between independently trained agents, yet rather than emerging from basic principles, the learned language is the result of an intentionally designed search algorithm.

Original: Reinforcement learning is a common tool in this ﬁeld for observing the emergence of language out of a reward maximization pressure.
Revised: Reinforcement learning is a common tool in this ﬁeld for observing the emergence of language out of a reward maximization pressure.

Original: One such design choice which can obscure these emergent properties is adding shaped rewards on top of the base reward of the reinforcement learning environment (Wiewiora, 2010).
Revised: One such design choice which can obscure these emergent properties is adding shaped rewards on top of the base reward of the reinforcement learning environment (Wiewiora, 2010).

Original: The base reward of the environment corresponds to achieving ultimate goal of the task in question.
Revised: The base reward of the environment derives directly from succeeding at the task in question.

Original: The difﬁculty with relying solely on the base rewards is that if the task is especially long or complicated, the agent may only receive a reward infrequently which makes for a difﬁcult learning problem.
Revised: The difﬁculty with relying solely on the base rewards is that if the task is especially long or complicated, the agent may only receive a reward infrequently which makes for a difﬁcult learning problem.

Original: In such a case, base reward is considered sparse .
Revised: In such a case, base reward is considered sparse .

Original: This motivates shaped rewards which are inserted at intermediate steps based on domain knowledge in order to introduce an inductive bias towards good solutions.
Revised: This motivates shaped rewards which are inserted at intermediate steps based on domain knowledge in order to introduce an inductive bias towards good solutions.

Original: For example, the base reward in chess would simply be winning or losing the game.
Revised: For example, the base reward in chess would simply be winning or losing the game.

Original: A shaped reward could then be taking the opponents material while not losing your own.
Revised: A shaped reward could then be taking the opponents material while not losing your own.

Original: While this shaped reward is often a good heuristic, it can lead to local optima; for example, it discourages strategies which would sacriﬁce individual pieces in order to win the whole game.
Revised: While this shaped reward is often a good heuristic, it can lead to local optima; for example, it discourages strategies which would sacriﬁce individual pieces in order to win the whole game.

Original: While local optima present a problem for maximizing reward, the biases introduced by reward shaping present a unique problem for emergent language which we will highlight in this paper.
Revised: While local optima present a problem for maximizing reward, the biases introduced by shaped rewards present a unique problem for emergent language which we will highlight in this paper.

Original: For emergent language research, the inductive bias which reward shaping introduces is especially problematic because it exerts a signiﬁcant inﬂuence on the learned language whose emergent properties are the object of study.
Revised: For emergent language research, the inductive bias which shaped rewards introduce is especially problematic because it exerts a signiﬁcant inﬂuence on the learned language whose emergent properties are the object of study.

Original: This inﬂuence can comprise 1) biasing the semantics of the language, 2) changing a property of the whole language (e.g., communication entropy), or 3) masking the inﬂuence of some other environmental parameter on the language.
Revised: This inﬂuence can comprise 1) biasing the semantics of the language, 2) changing a property of the whole language (e.g., language entropy), or 3) masking the inﬂuence of some other environmental parameter on the language.

Revised: For example, some emergent language work incorporates shaped rewards into their environment without accounting for the the biases it may introduce (Mordatch & Abbeel, 2018;

Revised: Brandizzi et al., 2021; Harding Graesser et al., 2019).

Revised: From an engineering and design perspective, tweaking the system to achieve a desired result is standard

Revised: [Figure]

Revised: practice, but from a scientiﬁc and experimental perspective, these additional shaped rewards serve as potential confounding factors and hinder accurate observations of the emergent phenomena.

Original: We study this by introducing a simple navigation game with continuous states and actions where a sender passes a one-word message to the receiver (illustrated in Figure 1b and 1a).
Revised: We study this by introducing a simple navigation game with continuous states and actions where a sender passes a one-word message to the receiver (illustrated in Figure 1b and 1a).

Original: Within this environment, our experiments look at the entropy and semantics of the emergent communication in the absence and presence of shaped rewards.
Revised: Within this environment, our experiments look at the entropy and semantics of the emergent language in the absence and presence of shaped rewards.

Original: In the course of these experiments, we ﬁnd that reinforcement learning algorithm’s rollout buffer size has a signiﬁcant impact on the entropy of the learned communication and potentially explains our experimental ﬁndings.To this end we introduce
Revised: In the course of these experiments, we ﬁnd that reinforcement learning algorithm’s experience buffer size has a signiﬁcant impact on the entropy of the learned language and potentially explains our experimental ﬁndings.

Original: [Figure] Figure a mathematical model based on the Chinese restaurant process for understanding the effect of rollout buffer size on emergent communication more generally.
Revised: To this end we introduce a mathematical model based on the Chinese restaurant process for understanding the effect of experience buffer size on emergent language more generally.

Original: We highlight the following contributions in our paper:
Revised: We highlight the following contributions in our paper:

Original: • Demonstrating basic ways in which reward shaping can undesirably inﬂuence emergent language experiments • Presenting a mathematical model for understanding the role of rollout buffer size in the entropy of emergent language
Revised: • Demonstrating basic ways in which shaped rewards can undesirably inﬂuence emergent language experiments • Presenting a mathematical model for understanding the role of experience buffer size in the entropy of emergent language

Original:  R ELATED W ORK 2. T AXONOMY
Revised:  R ELATED W ORK 2. T AXONOMY

Original: We intentionally design our environment to study shaped rewards and the entropy of communication, which requires it to differ from prior art in speciﬁc ways.
Revised: We intentionally design our environment to study shaped rewards and the entropy of language, which requires it to differ from prior art in speciﬁc ways.

Original: To elucidate this, we create a taxonomy of emergent language environments based onthe presence of a trivially optimal language and whether or not the environment has multi-step episodes.
Revised: To elucidate this, we create a taxonomy of emergent language environments based on whether or not the environment has multi-step episodes and the presence of a trivially optimal language (deﬁned below).

Original: The taxonomy is given in Table 1 and a brief description of each environment is given in Appendix A.
Revised: The taxonomy is given in Tableand a brief description of each environment is given in Appendix A.

Original: Generally speaking, the motivation for reward shaping in a given environment is sparsity of the base reward which requires a multi-step, multi-utterance environment.
Revised: Generally speaking, the motivation for shaped rewards in a given environment is sparsity of the base reward which requires a multi-step, multi-utterance environment.

Original: Thus, our experiments naturally require a multi-step environment.
Revised: Thus, our experiments naturally require a multi-step environment.

Original: We consider an environment to have a trivially optimal language if the information which needs to be communicated from sender to receiver can be perfectly represented in the emergent communication.
Revised: We consider an environment to have a trivially optimal language if the information which needs to be communicated from sender to receiver can be perfectly represented in the emergent language.

Original: Such a protocol most frequently arises when the communicated information is derived from a small number of categorical variables encoded in the observation.
Revised: Such a language most frequently arises when the communicated information is derived from a small number of categorical variables encoded in the observation.

Original: For example, in an environment where the sender must specify an element of the set { red , green , blue } × { square , circle } using messages from the set { r, g, b } × { s, c } , a trivial protocol is where color maps to a unique letter in the ﬁrst position and shape maps to a unique letter in the second position.
Revised: For example, in an environment where the sender must specify an element of the set { red , green , blue } × { square , circle } using messages from the set { r, g, b } × { s, c } , a trivial language is where color maps to a unique letter in the ﬁrst position and shape maps to a unique letter in the second position.

Original: Other environments have no trivially optimal communication protocols.
Revised: Other environments have no trivially optimal languages.

Original: For example, if the sender must communicate an element of { 1 , 2 , ..., 100 } using messages from the set { a, b, c } , there is no trivially optimally protocol since the sender can at best partition the set of integers to reach an optimal but imperfect solution.
Revised: For example, if the sender must communicate an element of { 1 , 2 , ..., 100 } using messages from the set { a, b, c } , there is no trivially optimally language since the sender can at best partition the set of integers to reach an optimal but imperfect solution.

Original: Kharitonov et al.
Revised: Kharitonov et al.

Original: (2020) gives evidence that there is an entropy minimization in pressure inherent in emergent language settings.
Revised: (2020) gives evidence that there is an entropy minimization in pressure inherent in emergent language settings.

Original: Building on this, Chaabouni et al.
Revised: Building on this, Chaabouni et al.

Original: (2021) explicitly look at a thetradeoff between entropy and reward—higher entropy languages have the potential to convey more information at the expense of being more difﬁcult to learn. This tradeoff disappears if a trivially optimal language is learned since there is no further reward maximization pressure. Language entropy greater than this minimum, then, does not emerge from the the reward maximization pressure. Although such an environment does not preclude studying entropy we choose to use an environment where the information being communicated is fully continuous so an increase in entropy can always translate to a increase in reward.
Revised: (2021) explicitly look at a the

Original: This leads to a smooth tradeoff between entropy and reward which is illustrated in Figure 2.

Original: Entropy

Original: [Figure]
Revised: [Figure]

Original: Paper
Revised: Paper

Original: Task
Revised: Task

Original: Trivial?
Revised: Trivial?

Original: Multi-step?
Revised: Multi-step?

Original: Havrylov & Titov (2017) image signalling no no
Revised: Havrylov & Titov (2017) image signalling no no

Original: Kottur et al.
Revised: Kottur et al.

Original: (2017) dialog Q&A yes
Revised: (2017) dialog Q&A yes

Original: yes
Revised: yes

Original: Mordatch & Abbeel (2018) goal speciﬁcation yes yes
Revised: Mordatch & Abbeel (2018) goal speciﬁcation yes yes

Original: Lazaridou et al.
Revised: Lazaridou et al.

Original: (2018) image signalling yes
Revised: (2018) image signalling yes

Original: no
Revised: no

Original: Kharitonov et al.
Revised: Kharitonov et al.

Original: (2020) vector signalling
Revised: (2020) vector signalling

Original: yes no
Revised: yes no

Original: Chaabouni et al.
Revised: Chaabouni et al.

Original: color signalling no
Revised: color signalling no

Original: no
Revised: no

Original: This paper navigation no yes 
Revised: This paper navigation no yes tradeoff between entropy and reward—higher entropy languages have the potential to convey more information at the expense of being more difﬁcult to learn. This tradeoff disappears if a trivially optimal language is learned since there is no further reward maximization pressure. Language entropy greater than this minimum, then, does not emerge from the the reward maximization pressure. Although such an environment does not preclude studying entropy we choose to use an environment where the information being communicated is fully continuous so an increase in entropy can always translate to a increase in reward.

Revised: This leads to a smooth tradeoff between entropy and reward which is illustrated in Figure 2.

Original: 2. S PECIFIC A PPROACHES
Revised: 2. S PECIFIC A PPROACHES

Original: The environment and agent conﬁguration of this paperpaper are most closely related to Chaabouni et al.
Revised: The environment and agent conﬁguration of this paper are most closely related to Chaabouni et al.

Original: (2021) who test the balance between entropy and semantic precision in a two-agent color discrimination task.
Revised: (2021) who test the balance between entropy and semantic precision in a two-agent color discrimination task.

Original: Although the color space comprises 330 distinct colors, the environment facilitates languages which cover a (learned) region of nearby colors.
Revised: Although the color space comprises 330 distinct colors, the environment facilitates languages which cover a (learned) region of nearby colors.

Original: In this way, there is no trivially optimal, one-to-one communication protocol as the task is inherently fuzzy.
Revised: In this way, there is no trivially optimal, one-to-one language as the task is inherently fuzzy.

Original: Havrylov & Titov (2017) use a signalling game with natural images which lacks a trivially optimal language, but using natural images results in many uncontrolled variables, lessening the ability of the experiments to make basic, ﬁrst-principles claims.
Revised: Havrylov & Titov (2017) use a signalling game with natural images which lacks a trivially optimal language, but using natural images results in many uncontrolled variables, lessening the ability of the experiments to make basic, ﬁrst-principles claims.

Revised: Studying shaped rewards has been explored previously in emergent language primarily as inductive bias for encouraging some property, such as compositionality, to emerge (Hazra et al., 2021; Jaques et al., 2019; Eccles et al., 2019).

Revised: This paper, instead, focuses on the negative aspects of inductive biases introduced by shaped rewards regarding how they can hinder empirical investigation of emergent languages.

Original: Superﬁcially, the environment used in this paper bears resemblance to Mordatch & Abbeel (2018) and Kaji ´ c et al.
Revised: Superﬁcially, the environment used in this paper bears resemblance to Mordatch & Abbeel (2018) and Kaji´c et al.

Original: (2020) as these both deal with navigation.
Revised: (2020) as these both deal with navigation.

Original: In both of these environments, the agents communicate about a discrete set of actions or goal locations, whereas the agents in this paper communicate about a continuous action.
Revised: In both of these environments, the agents communicate about a discrete set of actions or goal locations, whereas the agents in this paper communicate about a continuous action.

Original: Neither of the papers speciﬁcally investigates the effects of reward shaping on the languages learned.
Revised: Neither of the papers speciﬁcally investigates the effects of shaped rewards on the languages learned.

Original: While reward shaping is commonplace in reinforcement learning more generally, it has not been investigated in the context of emergent language, though other forms of inductive biases have been used in emergent language.

Original: For example, Havrylov & Titov (2017) use an ungrounded language model to generate the messages to approximate natural language lexicon and syntax, and Mordatch & Abbeel (2018) bias their language toward having lexical frequencies consistent with a Dirichlet process.

Original: Since the environments and tasks of emergent language have been simple heretofore, there has been no need to introduce reward shaping.

Original: This paper, then, anticipates the presence of reward shaping in the future of emergent language research as the environments and tasks studied become more sophisticated.

Original:  M ETHODS 3. E NVIRONMENT
Revised:  M ETHODS 3. E NVIRONMENT

Original: In this paper we use a simple 2 -dimensional navigation environment with two closely related tasks.
Revised: In this paper we use a simple 2 -dimensional navigation environment with two closely related tasks.

Original: A sender agent observes the position of a receiver agent, sends a message to the receiver, and the receiver takes an action.
Revised: A sender agent observes the position of a receiver agent, sends a message to the receiver, and the receiver takes an action.

Original: In the centerward task (illustrated in Figure 1a), the receiver is initialized uniformly at random within a circle and must navigate towards a circular goal region at the center.
Revised: In the centerward task (illustrated in Figure 1a), the receiver is initialized uniformly at random within a circle and must navigate towards a circular goal region at the center.

Original: In the edgeward task, the receiver is initialized uniformly at random within a circle and must navigate to a goal region comprising the entire area outside of the circle.
Revised: In the edgeward task, the receiver is initialized uniformly at random within a circle and must navigate to a goal region comprising the entire area outside of the circle.

Original: The centerward environment is a more realistic environment while the edgeward environment admits a greater variety of languages, since essentially any combination of learned directions can allow the agents to solve the task.
Revised: The centerward environment is a more realistic environment. The edgeward environment, on the other hand, admits a greater variety of languages, since moving consistently in any one direction eventually gets you to the edge; therefore, learning to move in a variety of evenly spaced direction solves the task more quickly but is not strictly necessary.

Original: There are no obstacles or walls in the environment.
Revised: There are no obstacles or walls in the environment.

Original: The receiver’s location and action are continuous variables stored as ﬂoating-point values.
Revised: The receiver’s location and action are continuous variables stored as ﬂoating-point values.

Original: If the receiver does not reach the goal region within a certain number of steps, the episode ends with no reward given.
Revised: If the receiver does not reach the goal region within a certain number of steps, the episode ends with no reward given.

Original: 3. A GENT A RCHITECTURE
Revised: 3. A GENT A RCHITECTURE

Original: Conceptually, our conﬁguration comprises two agents which areoptimized as a single neural network.
Revised: Our architecture comprises two agents, conceptually speaking, but in practice, they are a single neural network.

Original: The sender is the agent which observes the location of the receiver and passes a message in order to guide it towards the goal.
Revised: The sender is a disembodied agent which observes the location of the receiver and passes a message in order to guide it towards the goal.

Original: The receiver is the agent which receives the message  and takes an action solely based on the message, since it cannot observe anything else.
Revised: The receiver is an agent which receives the message as its only input and takes an action solely based on that message (i.e., it is “blind”).

Original: The sender and receiver are randomly initialized at the start of training, trained together, and tested together.
Revised: The sender and receiver are randomly initialized at the start of training, trained together, and tested together.

Original: The architecture of the agents is illustrated in Figure 1b.
Revised: The architecture of the agents is illustrated in Figure 1b.

Original: The observation of the sender is a pair of ﬂoating-point values representing the receiver’s location.
Revised: The observation of the sender is a pair of ﬂoating-point values representing the receiver’s location.

Original: The sender itself is a 2 -layer perceptron with tanh activations.
Revised: The sender itself is a 2 -layer perceptron with tanh activations.

Original: The output of the second layer is passed to a Gumbel-Softmax bottleneck layer (Maddison et al., 2017; Jang et al., 2017) which enables learning a discrete, one-hot representation, as an information bottleneck (Tishby et al., 2000).
Revised: The output of the second layer is passed to a Gumbel-Softmax bottleneck layer (Maddison et al., 2017; Jang et al., 2017) which enables learning a discrete, one-hot representation, as an information bottleneck (Tishby et al., 2000).

Original: The activations of this layer can be thought of as the “words” or “lexicon” of the emergent language.
Revised: The activations of this layer can be thought of as the “words” or “lexicon” of the emergent language.

Original: At evaluation time, the bottleneck layer functions deterministically as an argmax layer, emitting onehot vectors.
Revised: At evaluation time, the bottleneck layer functions deterministically as an argmax layer, emitting onehot vectors.

Original: The receiver is a 1 -layer perceptron which receives the output of the Gumbel-Softmax layer as input.
Revised: The receiver is a 1 -layer perceptron which receives the output of the Gumbel-Softmax layer as input.

Original: The output is a pair of ﬂoating-point values which determine the action of the agent.
Revised: The output is a pair of ﬂoating-point values which determine the action of the agent.

Original: The action is clamped to a maximum step size.
Revised: The action is clamped to a maximum step size.

Original: Speciﬁc hyperparameters are given in Appendix B.

Original: 3. O PTIMIZATION
Revised: 3. O PTIMIZATION

Original: Optimization is performed using stochastic gradient descent as a part of proximal policy optimization (PPO) (Schulman et al., 2017).
Revised: Optimization is performed using stochastic gradient descent as a part of proximal policy optimization (PPO) (Schulman et al., 2017).

Original: Speciﬁcally, we use the implementation of PPO provided by
Revised: Speciﬁcally, we use the implementation of PPO provided by

Original: Stable Baselines 3 with our neural networks implemented in PyTorch (Rafﬁn et al., 2019; Paszke et al., 2019).
Revised: Stable Baselines 3 with our neural networks implemented in PyTorch (Rafﬁn et al., 2019; Paszke et al., 2019).

Original: Using a Gumbel-Softmax bottleneck layer allows for end-to-end backpropagation, making optimization faster and more consistent than using a backpropagation-free method like REINFORCE (Kharitonov et al., 2020; Williams, 1992).
Revised: Using a Gumbel-Softmax bottleneck layer allows for end-to-end backpropagation, making optimization faster and more consistent than using a backpropagation-free method like REINFORCE (Kharitonov et al., 2020; Williams, 1992).

Original: We will give a basic explanation of how PPO (and related algorithms) works as it is necessary to connect it to the mathematical model presented in Section 4.
Revised: We will give a basic explanation of how PPO (and related algorithms) works as it is necessary to connect it to the mathematical model presented in Section 4.

Original: PPO has two stages sampling and optimizing .
Revised: PPO has two stages sampling and optimizing .

Original: During the sampling stage, the algorithm runs the agents in the environment and stores the states, actions, and rewards in a “rollout buffer”.
Revised: During the sampling stage, the algorithm runs the agents in the environment and stores the states, actions, and rewards in a experience buffer (or rollout buffer).

Original: The optimization stage then comprises performing gradient descent on the agents (i.e., neural network policy) using the data from rollout buffer.
Revised: The optimization stage then comprises performing gradient descent on the agents (i.e., neural network policy) using the data from experience buffer.

Original: The next iteration starts with the updated agents and an empty rollout buffer.
Revised: The next iteration starts with the updated agents and an empty experience buffer.

Original: 3. R EWARDS
Revised: 3. R EWARDS

Original: We make use of two different rewards in our conﬁguration, a base reward and an shaped reward.
Revised: We make use of two different rewards in our conﬁguration, a base reward and an shaped reward.

Original: The base reward is simply a positive reward  given if the receiver reaches to the goal region before the episode ends and no reward otherwise.
Revised: The base reward is simply a positive reward of 1 given if the receiver reaches to the goal region before the episode ends and no reward otherwise.

Original: The shaped reward, given at every timestep, is the decrease in distance to the goal.
Revised: The shaped reward, given at every timestep, is the decrease in distance to the goal.

Revised: If the goal region is centered at (0 , 0) , the standard shaped reward for the centerward environment is given by Equation 1; we also use a trivially biased version of the reward which only takes into account horizontal distance speciﬁed in Equation 2.

Revised: [Equation]

Revised: For the edgeward environment, we use the opposite of r t as the goal is to move away from the center.

Original: The interplay between base and shaped rewards is important to understand in the larger context of how reinforcement learning problems are structured.
Revised: The interplay between base and shaped rewards is important to understand in the larger context of how reinforcement learning problems are structured.

Original: The base rewards are well-motivated and directly correspond to the ultimate aim of the task, but their sparsity can make it difﬁcult for the agents to learn to succeed.
Revised: The base rewards are well-motivated and directly correspond to the ultimate aim of the task, but their sparsity can make it difﬁcult for the agents to learn to succeed.

Original: Shaped rewards facilitate learning by using expert knowledge to form an inductive bias, yet they present a drawback for traditional reinforcement learning and emergent language.
Revised: Shaped rewards facilitate learning by using expert knowledge to form an inductive bias, yet they present a drawback for traditional reinforcement learning and emergent language.

Original: Within reinforcement learning where the goal is to train the best performing agent, shaped rewards can lead to the agent ﬁnding local optima if better solutions are excluded by the inductive bias.
Revised: Within reinforcement learning where the goal is to train the best performing agent, shaped rewards can lead to the agent ﬁnding local optima if better solutions are excluded by the inductive bias.

Original: Within emergent language, the problem is more nuanced as the goal is primarily to study a wide range of emergent properties of the language learned within the environment.
Revised: Within emergent language, the problem is more nuanced as the goal is primarily to study a wide range of emergent properties of the language learned within the environment.

Original: While base rewards have a natural connection to the environment and task, the shaped rewards introduce a reward signal which is not intrinsically connected with the task and environment, even if it is a good heuristic.3. E XPERIMENTS
Revised: While base rewards have a natural connection to the environment and task, the shaped rewards introduce a reward signal which is not intrinsically connected with the task and environment, even if it is a good heuristic.

Original: Biased Representations

Original: In our ﬁrst experiment we demonstrate how shaped rewards which are trivially biased directly distort the semantics of the language, that is, the action associated with each bottleneck unit.

Original: The speciﬁc instance of trivial bias we use is a shaped reward which only takes into account horizontal distance to the goal in our edgeward environment.

Original: We compare this against an environment with no reward shaping and an environment with the normal shaped rewards.

Original: We visualize the semantics of the language with so-called “sea star plots” (see Figure 3) where each bottleneck unit is represented by the receiver’s action in response to it, with opacity representing the frequency of use.

Original: For example, the sea star plot of a receiver which knows only North, South, East, and West would look like a “+”.

Original: Changing the Distribution of Entropy Naturally, a shaped reward which favors certain actions over others will bias the semantics of the language.

Original: Thus, our second experiment investigates more closely the effect that reward shaping without this explicit bias can have.

Original: Speciﬁcally, we investigate the distribution of communication entropies in the two environments.

Original: By entropy we are speciﬁcally referring to the Shannon entropy (measured in bits) of the bottleneck units as used in the trained agents’ communication protocol (as averaged over 3000 quasirandomly initialized episodes).

Original: Entropy is an important aspect of language as it represents the upper bound on the information that the language can convey.

Original: On the one hand, languages with higher entropy can yield more precise meaning in their utterances while at the cost of being more difﬁcult to learn or acquire as there is a greater variety of training examples needed to learn them.

Original: Masking Environmental Parameters In our ﬁnal primary experiment, we demonstrate how reward shaping can mitigate the inﬂuence of environmental parameters on the entropy of the learned communication.

Original: This is an issue insofar as the presence of reward shaping makes it difﬁcult to observe an emergent property of interest.

Original: Speciﬁcally, we look at how the standard reward shaping hides the effect of world radius on entropy in our centerward environment.

Original: Rollout Buffer Size

Original: As an explanatory experiment, we demonstrate how changing the size of the

Original: PPO rollout buffer has a signiﬁcant impact of the entropy of the emergent communication.

Original: We compare this with the effects we would expect to see according to the model presented in the next section, i.e., the expectation Chinese Restaurant process.

Original: In turn we use this to explain one mechanism by which reward shaping can have the observed effects on entropy shown by the previous experiments.

Original:  E XPLANATORY M ODEL
Revised:  E XPLANATORY M ODEL

Original: We argue that the results of experiments on changing the distribution of entropy and masking environmental parameters can be explained by an effective change in the size of PPO’s rollout buffer.
Revised: We argue that the results presented in Sections 5.2 and 5.3 can be explained by an effective change in the size of PPO’s experience buffer.

Original: In order illustrate this, we introduce a simple mathematical model based on the Chinese restaurant process (Blei, 2007; Aldous, 1985).
Revised: In order illustrate this, we introduce a simple mathematical model based on the Chinese restaurant process (Blei, 2007; Aldous, 1985).

Original: While the model does not exactly match our experimental setup, the key similarities allow us to reason about our results as well as potential future experiments.
Revised: While the model does not exactly match our experimental setup, the key similarities allow us to reason about our results as well as potential future experiments.

Original: The Chinese restaurant process is an iterative stochastic process which yields a probability distribution over the positive integers.
Revised: The Chinese restaurant process is an iterative stochastic process which yields a probability distribution over the positive integers.

Original: The key similarity between the Chinese restaurant process and our learning setup is that it they are self-reinforcing processes, that is, when a given value is selected in one iteration of the process, the probability that that value is chosen again in subsequent iterations increases.
Revised: The key similarity between the Chinese restaurant process and our learning setup is that it they are self-reinforcing processes, that is, when a given value is selected in one iteration of the process, the probability that that value is chosen again in subsequent iterations increases.

Original: We generalize one aspect of the Chinese restaurant process in Section 4.2 to better match the sampling-optimization procedure of PPO.
Revised: We generalize one aspect of the Chinese restaurant process in Section 4.2 to better match the sampling-optimization procedure of PPO.

Original: The primary simpliﬁcation which this model makes is that it does not take into account the “meaning” of actions and the effects they have within the environment.
Revised: The primary simpliﬁcation which this model makes is that it does not take into account the “meaning” of actions and the effects they have within the environment.

Original: For example, every successful agent in the centerward environment will use at least three distinct nearly equiprobably so as to span the 2 -dimensional space whereas as no such lower bound exists in the stochastic process.1011 weights = array 12 for _ range : addend array
Revised: For example, every successful agent in the centerward environment will use at least three distinct nearly equiprobably so as to span the 2 -dimensional space whereas as no such lower bound exists in the stochastic process.

Original: for

Original: addend i beta

Original: 18 return

Original: 4. C HINESE R ESTAURANT P ROCESS
Revised: 4. C HINESE R ESTAURANT P ROCESS

Original: As the name suggests, a useful analogy for the Chinese restaurant process starts with a restaurant with inﬁnitely many empty tables, indexed by the positive integers, which can hold an unbounded number of customers.
Revised: As the name suggests, a useful analogy for the Chinese restaurant process starts with a restaurant with inﬁnitely many empty tables, indexed by the positive integers, which can hold an unbounded number of customers.

Original: As each customer walks in, they sit at a populated table with a probability proportional to the number of people already at that table.
Revised: As each customer walks in, they sit at a populated table with a probability proportional to the number of people already at that table.

Original: The customer will sit at a new table proportional to a hyperparameter α which modulates the concentration of the ﬁnal distribution.
Revised: The customer will sit at a new table proportional to a hyperparameter α which modulates the concentration of the ﬁnal distribution.

Original: The decision the customer makes is equivalent to sampling from a categorical distribution where the unnormalized weights are the customer counts along with the weight, α , for the new table.
Revised: The decision the customer makes is equivalent to sampling from a categorical distribution where the unnormalized weights are the customer counts along with the weight, α , for the new table.

Original: The pseudocode for the Chinese restaurant process is given in Algorithm 1 for β = 1 .
Revised: The pseudocode for the Chinese restaurant process is given in Algorithm 1 for β = 1 .

Original: By analogy to the neural networks representing our agents, we can view the tables as bottleneck units and the customers choosing a table as parameter updates which reinforce the use of that unit in accordance with the reward.
Revised: By analogy to the neural networks representing our agents, we can view the tables as bottleneck units and the customers choosing a table as parameter updates which reinforce the use of that unit in accordance with the reward.

Original: Mordatch & Abbeel (2018) implicitly assume this when they introduce a reward corresponding to the probability that the emergent lexicon is generated by a Chinese restaurant process (a Dirichlet process, in their words).
Revised: Mordatch & Abbeel (2018) implicitly assume this when they introduce a reward corresponding to the probability that the emergent lexicon is generated by a Chinese restaurant process (a Dirichlet process, in their words).

Original: The self-reinforcing property can be expressed informally as: more popular tables get more new customers, keeping them popular.
Revised: The self-reinforcing property can be expressed informally as: more popular tables get more new customers, keeping them popular.

Original: A higher α means that customers are more likely to sit at a new table, so the distribution over tables will be more spread out in expectation.
Revised: A higher α means that customers are more likely to sit at a new table, so the distribution over tables will be more spread out in expectation.

Original: The distribution stabilizes as the number of iterations goes to inﬁnity as an individual new customers has a diminishing effect the relative size of the weights. 
Revised: The distribution stabilizes as the number of iterations goes to inﬁnity as an individual new customers has a diminishing effect the relative size of the weights. 1011 weights addend addend

Original: 4. E XPECTATION C HINESE R ESTAURANT P ROCESS
Revised: 4. E XPECTATION C HINESE R ESTAURANT P ROCESS

Original: The key difference between how the Chinese restaurant process and PPO works is the relationship between sampling (i.e., simulating episodes) and updating the weights/parameters.
Revised: The key difference between how the Chinese restaurant process and PPO works is the relationship between sampling (i.e., simulating episodes) and updating the weights/parameters.

Original: In each iteration, the regular Chinese restaurant process draws a sample based on its weights and updates those weight immediately.
Revised: In each iteration, the regular Chinese restaurant process draws a sample based on its weights and updates those weight immediately.

Original: In PPO, the agent will populate the rollout buffer with a number of steps (on the order of 100 to 1000 ) in the environment before performing gradient descent with that buffer to update the parameters.
Revised: In PPO, the agent will populate the experience buffer with a number of steps (on the order of 100 to 1000 ) in the environment before performing gradient descent with that buffer to update the parameters.

Original: As a result, the parameter update is performed based on a weighting across multiple bottleneck units based on how often they were used in the episodes recorded in the rollout buffer.
Revised: As a result, the parameter update is performed based on a weighting across multiple bottleneck units based on how often they were used in the episodes recorded in the experience buffer.

Original: Thus, to the appropriately generalize the Chinese restaurant process, we introduce the expectation
Revised: Thus, to the appropriately generalize the Chinese restaurant process, we introduce the expectation

Original: Chinese restaurant process .
Revised: Chinese restaurant process .

Original: In this process, we add a hyperparameter β which is a positive integer describing how many samples we take from the distribution before updating the weights; the updates are normalized by β so the sum of all weights still only increases by 1 per iteration.
Revised: In this process, we add a hyperparameter β which is a positive integer describing how many samples we take from the distribution before updating the weights; the updates are normalized by β so the sum of all weights still only increases by 1 per iteration.

Original: The restaurant analogy breaks down here as we would have to say that in each iteration, β customers simultaneously and independently make a decision, get shrunk to 1 β th their size, and then sit at their table of choice.
Revised: The restaurant analogy breaks down here as we would have to say that in each iteration, β customers simultaneously and independently make a decision, get shrunk to 1 β th their size, and then sit at their table of choice.

Original: The pseudocode for the expectation Chinese restaurant process is given in Algorithm 1.[Figure] [Figure]
Revised: The pseudocode for the expectation Chinese restaurant process is given in Algorithm 1.

Original:  R ESULTS
Revised:  E XPERIMENTS

Original: Each run of experiment starts by training a sender and receiver for a ﬁxed number of timesteps for a range of independent variable values.
Revised: Each run of experiment starts by training a sender and receiver for a ﬁxed number of timesteps for a range of independent variable values.

Original: The trained models are then evaluated by initializingepisodes at evenly distributed locations using Vogel’s method (Vogel, 1979).
Revised: The trained models are then evaluated by initializingepisodes at evenly distributed locations using Vogel’s method (Vogel, 1979).

Original: In most settings model are able to achieve a 100% success rate during training and evaluation, though remove any models which do not from consideration.
Revised: In most settings, the agents are able to achieve a 100% success rate during training and evaluation; we remove any models which do not from consideration.

Original: All model for our experiments use 2 6 = 64 bottleneck units which translates to a maximum entropy of 6 bits.
Revised: All model for our experiments use 2 6 = 64 bottleneck units which translates to a maximum entropy of 6 bits.

Revised: Hyperparameters are given in Appendix B.

Original: 5. B IASED R EPRESENTATIONS
Revised: 5. B IASED S EMANTICS

Original: The results of the experiment  are visualized in Figure 3. 
Revised: In our ﬁrst experiment we demonstrate how shaped rewards which are trivially biased directly distort the semantics of the language, that is, the action associated with each bottleneck unit. We compare three environments, no shaped rewards, the standard shaped reward, and the trivally biased shaped reward. We visualize the semantics of the language with so-called “sea star plots” in Figure 3. Each arm of the sea star is the action taken by the receiver in response to a single bottleneck unit with opacity representing the frequency of use.

Original:  In the setting with no reward shaping, we see representations learned featuring 2 to 4 arms pointing in a variety of directions.
Revised: [Figure] [Figure] In the setting with no shaped rewards, we see actions (i.e., the meanings of the messages) learned featuring 2 to 4 arms pointing in a variety of directions.

Original: Adding the normal reward shaping appears to bias the representations toward having 4 arms but does not  biasthe arms inany particular direction, as the shaped reward themselves do not favor a given direction.
Revised: Since the standard shaped reward takes both dimensions into account, we do not see any bias in the direction of the learned actions.

Original: Finally, with the  biased reward (i.e., reward shaping only in the horizontal dimension), we see the the learned languages exclusively favor actions near to the horizontal axis.
Revised: With the trivially biased reward, though, we see the the learned languages exclusively favor actions near to the horizontal axis.

Original: In this setting, nothing explicitly prevents the agents from learning vertical actions, but the fact that horizontal actions receive the shaped reward makes them easier to learn.
Revised: In this setting, nothing explicitly prevents the agents from learning vertical actions, but the fact that horizontal dimensions receive the shaped reward makes those actions easier to learn.

Original: 5. C HANGING THE D ISTRIBUTION OF E NTROPY
Revised: 5. C HANGING THE D ISTRIBUTION OF E NTROPY

Original: To investigate the distribution of communication entropies, we look at a histogram recording the 
Revised: Naturally, a shaped reward which favors certain actions over others will bias the semantics of the language. Thus, our second experiment investigates more closely the effect that shaped rewards without this explicit bias can have. Speciﬁcally, we investigate the distribution of language entropies in the two environments. By entropy we are speciﬁcally referring to the Shannon entropy (measured in bits) of the bottleneck units as used in the trained agents’ language (as averaged over 3000 quasirandomly initialized episodes).

Original:  Shannon entropy of communication over 2000 runs for an environment with and without  reward shaping.
Revised: Entropy is an important aspect of language as it represents the upper bound on the information that the language can convey. Languages with higher entropy can yield more precise meaning in their utterances, yet this comes at the cost of being more difﬁcult to learn or acquire as they need a greater variety of training example to be learned. To investigate the distribution of language entropies, we look at a histogram showing the Shannon entropy of languages belonging to environments with and without shaped rewards. The distributions is computed from 2000 independent runs for each reward setting.

Original: This is shown in Figure 4a.
Revised: This is shown in Figure 4a.

Original: The presence of reward shaping shifts the distribution upwards, demonstrating that even reward shaping which is free of a trivial bias can still bias the emergent communication.
Revised: The presence of shaped rewards shifts the distribution upwards, demonstrating that even shaped rewards which is free of a trivial bias can still bias the emergent language.

Original: A potential explanation of these results is discussed and illustrated in Section 5.4.
Revised: A potential explanation of these results is discussed and illustrated in Section 5.4.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: E n t r o p y
Revised: E n t r o p y

Original: ( b i t s )
Revised: ( b i t s )

Original: [Figure]
Revised: [Figure]

Original: M ASKING E NVIRONMENTAL P ARAMETERS
Revised: M ASKING E NVIRONMENTAL P ARAMETERS

Revised: In our ﬁnal primary experiment, we demonstrate how shaped rewards can mitigate the inﬂuence of environmental parameters on the entropy of the learned language.

Revised: This is an issue insofar as the presence of shaped rewards make it difﬁcult to observe an emergent property of interest.

Revised: Speciﬁcally, we look at how the standard shaped rewards hide the effect of world radius on entropy in our centerward environment.

Original: In Figures 4b and 4c, we plot the communication entropies against different world radii.
Revised: In Figures 4b and 4c, we plot the language entropies against different world radii.

Original: In both settings, we observe that entropy decreases as the world radius increases, but the setting with no reward shaping shows a much more rapid decrease in entropy.
Revised: In both settings, we observe that entropy decreases as the world radius increases, but the setting with no shaped rewards shows a much more rapid decrease in entropy.

Original: We offer one possible explanation for this effect in Section 5.4.
Revised: We offer one possible explanation for this effect in Section 5.4.

Original: When the only reward signal agents have access to is the base reward, they can only accomplish the task by ﬁnding the goal randomly at ﬁrst; as the size of the environment increases, the chance of ﬁnding the goal with random movements decreases and the agent pair often fails to learn a fully successful language at the highest world radii.
Revised: When the only reward signal agents have access to is the base reward, they can only accomplish the task by ﬁnding the goal randomly at ﬁrst; as the size of the environment increases, the chance of ﬁnding the goal with random movements decreases and the agent pair often fails to learn a fully successful language at the highest world radii.

Original: 5. R OLLOUT B UFFER S IZE
Revised: 5. E XPERIENCE B UFFER S IZE

Revised: As an explanatory experiment, we demonstrate how changing the size of the PPO experience buffer has a signiﬁcant impact of the entropy of the emergent language.

Revised: We compare this with the effects we would expect to see according to the model presented in the next section, i.e., the expectation

Revised: Chinese Restaurant process.

Revised: In turn we use this to explain one mechanism by which shaped rewards can have the observed effects on entropy shown by the previous experiments.

Original: In Figure 5a we show the effect of a logarithmic sweep of β on entropy of the expectation Chinese restaurant process.
Revised: In Figure 5a we show the effect of a logarithmic sweep of β on entropy of the expectation Chinese restaurant process.

Original: We ﬁrst observe that increasing β reduces the variance between distributions yielded by the process since, as β increases, the individual updates made in each iterations are also reduced in variance.
Revised: We ﬁrst observe that increasing β reduces the variance between distributions yielded by the process since, as β increases, the individual updates made in each iterations are also reduced in variance.

Original: In fact, in the limiting case as β → ∞ , the process will always yield the same distribution as the update will just be the expectation of sampling from the categorical distribution described by the (normalized) weights (plus α ).
Revised: In fact, in the limiting case as β → ∞ , the process will always yield the same distribution as the update will just be the expectation of sampling from the categorical distribution described by the (normalized) weights (plus α ).

Original: The second effect is that increasing β will decrease the concentration (i.e., increase the entropy), on average, of the distribution yielded from the process.
Revised: The second effect is that increasing β will decrease the concentration (i.e., increase the entropy), on average, of the distribution yielded from the process.

Original: The intuition behind this is that the since each update is less concentrated, the distribution as a whole will be less concentrated as the probability mass will be spread out.
Revised: The intuition behind this is that the since each update is less concentrated, the distribution as a whole will be less concentrated as the probability mass will be spread out.

Original: These results can be used to explain, in part, both the effect of shaped rewards and world radius on entropy.
Revised: These results can be used to explain, in part, both the effect of shaped rewards and world radius on entropy.

Original: First, though, we must establish an analogous correspondence between the expectation
Revised: First, though, we must establish an analogous correspondence between the expectation

Original: Chinese restaurant process and the PPO-learning process.
Revised: Chinese restaurant process and the PPO-learning process.

Original: An iteration of the process described in
Revised: An iteration of the process described in

Original: Algorithm 1 consists of sampling from the modiﬁed categorical distribution (Line 15) and incrementing the weights (Line 17).
Revised: Algorithm 1 consists of sampling from the modiﬁed categorical distribution (Line 15) and incrementing the weights (Line 17).

Original: In PPO, the sampling corresponds to populating the rollout buffer with steps associated with a reward, and the increment operation is analogous to PPO performing gradient descent on the agents using the buffer.
Revised: In PPO, the sampling corresponds to populating the experience buffer with steps associated with a reward, and the increment operation is analogous to PPO performing gradient descent on the agents using the buffer.

Original: Thus, β is analogously increased for PPO when the number of successful episodes per iteration which is dependent both on the size of the rollout buffer as well as the environmental factors affecting frequency of success.
Revised: Thus, β is analogously increased for PPO when the number of successful episodes per iteration which is dependent both on the size of the experience buffer as well as the environmental factors affecting frequency of success.

Original: In Figures 5b and 5c, we directly vary the size of the rollout buffer in our environments with and without reward shaping.
Revised: In Figures 5b and 5c, we directly vary the size of the experience buffer in our environments with and without shaped rewards.

Original: Both environments replicate the correlation between β /buffer size and entropy, though the decrease in variance is less distinct as buffer size increases.
Revised: Both environments replicate the correlation between β /buffer size and entropy, though the decrease in variance is less distinct as buffer size increases.

Original: Having established this correlation, we can offer a potential explanation to the experiments involving world radius as well as the distribution of entropies between the environments with and without reward shaping.
Revised: Having established this correlation, we can offer a potential explanation to the experiments involving world radius as well as the distribution of entropies between the environments with and without shaped rewards.

Original: Reward shaping effectively increases β since it assigns a reward signal to every step whereas the base reward-only environment requires successful episode.
Revised: Shaped rewards effectively increase β since it assigns a reward signal to every step whereas the base reward-only environment requires successful episode.

Original: This effect is exacerbated when the world radius is increased, the base reward-only environment yields rewards less frequently in the beginning because randomly ﬁnding the goal is less likely.
Revised: This effect is exacerbated when the world radius is increased, the base reward-only environment yields rewards less frequently in the beginning because randomly ﬁnding the goal is less likely.

Original: This effectively decreases β which corresponds to a lower entropy and higher variance.
Revised: This effectively decreases β which corresponds to a lower entropy and higher variance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The primary limitation of this work is thatonly a single environment was studied with a limited variety of experiments investigating shaped rewards. Furthermore, the environment used is relatively simple meaning 1) the scenarios proposed for the experiments are somewhat artiﬁcial and 2) the number of  emergent phenomenon to observe the ﬁrst place is limited.
Revised: We have, then, demonstrated the pitfalls that shaped rewards present for emergent language research: directly biasing the learned semantics of the language, changing the distribution of an emergent property of the language (i.e., entropy), and masking the emergent effects of other environmental variables. These experiments were performed with with a novel navigation-based emergent language environment.

Original: Hence, future work can extend similar experiments to more environments with more complexity and  a variety of solution strategies; the bias introduced by reward shaping could then be manifested as a bias towards certain strategies. For example, a navigation environment which features multiple paths with varying difﬁculties (i.e., requiring varyingly precise languages) could be one approach.
Revised: This environment allows allows for shaped rewards through multi-step episodes and avoids a trivially optimal language by employing a continuous state and action space. In addition to this, we introduced the expectation Chinese restaurant process both to explain our own experimental results and to provide a foundation for future models of emergent language. The limitations of this work can be illustrated through the future directions that could be taken.

Original: We have provided an exploratory and anticipatory study on the possible pitfalls that reward shaping presents for emergent language research: namely introducing bias intothe learned semantics, directly changing an emergent property of  the language (i.e., entropy), and masking effects from the environment on the language. We did this with a novel environment employing continuous states and actions and a discrete communication channel.
Revised: First, studying a variety of environments would further characterize what biases shaped rewards introduce into emergent languages. Second, increasing the complexity of environments would greatly expand the range of both emergent properties and types of shaped rewards which could be studied. For example, a rich environment like chess presents many emergent strategies such as the valuation of positions, balance between offense and defense, and favoring certain pieces; shaped rewards could then take the form of rewarding stochastic strategies, evaluating positions with traditional chess engines, or assigning pieces explicit point values.

Original: Heretofore, this  research has dealt with environments that are simple enough tonot require reward shaping, but we expect the ﬁeld to followthe direction of reinforcement learning in  progressively addressing harder problems which might require reward shaping.
Revised: Furthermore, while we only use the expectation Chinese restaurant process as an explanatory model, further work could design experiments to demonstrate its predictive power. The studies presented in this paper are both exploratory and anticipatory in nature since emergent language research has yet to tackle environments difﬁcult enough to require shaped rewards. Nevertheless, the ﬁeld will follow reinforcement learning in tackling progressively more difﬁcult tasks which will present further opportunities for or even require shaped rewards.

Original: This work, then, can prepare researchers to avoid and address the issues with reward shaping when they occur. Additionally, we have introduced an example of how the Chinese restaurant process can be used as a simple model to reason about the distribution of emergent lexicons.
Revised: When this occurs, accounting for the biases which are inherent to shaped rewards is imperative to preserving the integrity of emergent language experiments. This work, then, prepares researchers in the ﬁeld for these future challenges.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The code used in association with this paper is located at https://example.com/ anonymous-repo (in a ZIP ﬁle for review).
Revised: The code used in association with this paper is located at https://example.com/ repo-name (in a ZIP ﬁle for review).

Original: Reproduction instructions are located in the README.md ﬁle.
Revised: Reproduction instructions are located in the README.md ﬁle.

Original: Experiments were performed on a 20 -thread Intel Core i9-9900X server on which they take less than 24 hours to run.
Revised: Experiments were performed on a 20 -thread Intel Core i9-9900X server on which they take less than 24 hours to run.

Original: No random seeds are recorded or provided as the training process is stable and similar results should be observed for any random seed.
Revised: No random seeds are recorded or provided as the training process is stable and similar results should be observed for any random seed.

Original: A E NVIRONMENTS OF R ELATED W ORK
Revised: A E NVIRONMENTS OF R ELATED W ORK

Original: Havrylov & Titov (2017)
Revised: Havrylov & Titov (2017)

Original: Standard signalling game where the observations are natural images from Microsoft COCO.
Revised: Standard signalling game where the observations are natural images from Microsoft COCO.

Original: There is no trivially optimal protocol since the information being communicated is simply which image is being shown.
Revised: There is no trivially optimal language since the information being communicated is simply which image is being shown.

Original: There are natural image classes (e.g., cat vs. dog), but they are not necessarily the features of the images which the agents need to communicate.
Revised: There are natural image classes (e.g., cat vs. dog), but they are not necessarily the features of the images which the agents need to communicate.

Original: The standard signalling game is single-step.
Revised: The standard signalling game is single-step.

Original: Kottur et al.
Revised: Kottur et al.

Original: (2017) “Task and talk” dialog game where one agent must ask questions of the other agent to learn the attributes of an object.
Revised: (2017) “Task and talk” dialog game where one agent must ask questions of the other agent to learn the attributes of an object.

Original: Speciﬁcally, the questioner has a set of attributes it must learn (the objective) about the object that only the answerer can see.
Revised: Speciﬁcally, the questioner has a set of attributes it must learn (the objective) about the object that only the answerer can see.

Original: The desired communication protocol has the questioner using a unique message to specify a property and the answerer responding with a unique message to specify the value for that property such that messages are always in one-to-one correspondence what they are communicating.
Revised: The desired language has the questioner using a unique message to specify a property and the answerer responding with a unique message to specify the value for that property such that messages are always in one-to-one correspondence what they are communicating.

Original: Such communication protocols is trivially optimal.
Revised: Such a language is trivially optimal.

Original: Multi-round dialog environments are inherently multi-step.
Revised: Multi-round dialog environments are inherently multi-step.

Original: Mordatch & Abbeel (2018)
Revised: Mordatch & Abbeel (2018)

Original: Collaborative navigation game where multiple agents tell each other whither to move in a 2D environment.
Revised: Collaborative navigation game where multiple agents tell each other whither to move in a 2D environment.

Original: The environment consists of agents and landmarks distinguisted by color.
Revised: The environment consists of agents and landmarks distinguisted by color.

Original: Each agent is given a “goal”; for example, the blue agent might receive “red agent go to green landmark” (represented as a categorical feature vector) which the blue agent then has to communicate to the red agent.
Revised: Each agent is given a “goal”; for example, the blue agent might receive “red agent go to green landmark” (represented as a categorical feature vector) which the blue agent then has to communicate to the red agent.

Original: The agents have a vocabulary large enough to assign a unique “word” to each concept being expressed in one-to-one correspondence, yielding a trivially optimal protocol.
Revised: The agents have a vocabulary large enough to assign a unique “word” to each concept being expressed in one-to-one correspondence, yielding a trivially optimal language.

Original: Each episodes consists of multiple timesteps which can each have their own utterance.
Revised: Each episodes consists of multiple timesteps which can each have their own utterance.

Original: Lazaridou et al.
Revised: Lazaridou et al.

Original: An image-based signalling game using rendered images from MuJoCo.
Revised: An image-based signalling game using rendered images from MuJoCo.

Original: The information being communicated is the shape and color of an object where set of shapes and colors are both small.
Revised: The information being communicated is the shape and color of an object where set of shapes and colors are both small.

Original: Since the number of words available is at least as big as the cardinalities of the sets and the sender is able to use multiple words per utterance, it is possible to construct a trivially optimal protocol.
Revised: Since the number of words available is at least as big as the cardinalities of the sets and the sender is able to use multiple words per utterance, it is possible to construct a trivially optimal langauge.

Original: The standard signalling game is single-step.
Revised: The standard signalling game is single-step.

Original: Kharitonov et al.
Revised: Kharitonov et al.

Original: A binary vector-based signalling game with shared context.
Revised: A binary vector-based signalling game with shared context.

Original: The goal of the signalling game is to communicate the bits of a binary vector which are not shared by the sender and receiver.
Revised: The goal of the signalling game is to communicate the bits of a binary vector which are not shared by the sender and receiver.

Original: The messages consist of a single symbol where the number of unique symbols is greater than the combination of bits to communicated, thus there is a trivially optimal protocol where one unique symbol is assigned to each possible combination of unshared bits.
Revised: The messages consist of a single symbol where the number of unique symbols is greater than the combination of bits to communicated, thus there is a trivially optimal langauge where one unique symbol is assigned to each possible combination of unshared bits.

Original: The standard signalling game is single-step.
Revised: The standard signalling game is single-step.

Original: Chaabouni et al.
Revised: Chaabouni et al.

Original: A signalling game with 330 different colors represented as a real vector in CIELAB color space.
Revised: A signalling game with 330 different colors represented as a real vector in CIELAB color space.

Original: The game is set up so that colors which are nearby never appear in the same distractor set which encourages solutions which can cover some arbitrary region of colors.
Revised: The game is set up so that colors which are nearby never appear in the same distractor set which encourages solutions which can cover some arbitrary region of colors.

Original: Due to this “fuzzy” concept of solution (i.e., not using 330 distinct words for each color), we consider this environment not to have a trivially optimal solution.
Revised: Due to this “fuzzy” concept of solution (i.e., not using 330 distinct words for each color), we consider this environment not to have a trivially optimal solution.

Original: The standard signalling game is single-step.
Revised: The standard signalling game is single-step.

Original: B H YPERPARAMETERS 
Revised: B H YPERPARAMETERS B. D EFAULT C ONFIGURATION Environment

Original: Unless otherwise speciﬁed, the following hyperparameters are used for all experiments. Environment

Original: • Goal radius: 1 (centerward), 8 (edgeward) • World radius: 9 (centerward), 8 (edgeward) • Max steps per episode: 3 × world radius
Revised: • Type: centerward • World radius:• Goal radius:• Max steps per episode: 3 × world radius

Original: Agent Architecture
Revised: Agent Architecture

Original: • Bottleneck size:• Architecture; sender is 1-3 and receiver is 5; bottleneck size is N 1.
Revised: • Bottleneck size:• Architecture; sender is 1-3 and receiver is 5; bottleneck size is N 1.

Original: Linear w/ bias: 2 in, 32 out 2.
Revised: Linear w/ bias: 2 in, 32 out 2.

Original: Tanh activation 3.
Revised: Tanh activation 3.

Original: Linear w/ bias: 32 in, N out 4.
Revised: Linear w/ bias: 32 in, N out 4.

Original: Gumbel-Softmax: N in, N out 5.
Revised: Gumbel-Softmax: N in, N out 5.

Original: Linear w/ bias: N in, 2 out (action) and 1 out (value) • Bottleneck (Gumbel-Softmax) temperature: 1 .
Revised: Linear w/ bias: N in, 2 out (action) and 1 out (value) • Bottleneck (Gumbel-Softmax) temperature: 1 .•

Original: • Weight initialization: U (cid:16) − (cid:113) 1 n , (cid:113) 1 n (cid:17) , where n is the input size of the layer (PyTorch 1. default)
Revised: Weight initialization: U (cid:16) − (cid:113) 1 n , (cid:113) 1 n (cid:17) , where n is the input size of the layer (PyTorch 1. default)

Original: Optimization
Revised: Optimization

Original: • Reinforcement learning algorithm: proximal policy optimization – Default hyperparameters used unless otherwise noted: https:// stable-baselines3.readthedocs.io/en/v1.0/modules/ppo.
Revised: • Reinforcement learning algorithm: proximal policy optimization – Default hyperparameters used unless otherwise noted: https:// stable-baselines3.readthedocs.io/en/v1.0/modules/ppo.

Original: html
Revised: html

Original: • Training steps: 1 × 10• Evaluation episodes: 3 × 10• Learning rate: 3 × 10 −• Number of environment steps per update: 1024 ; 256 (for biased reward shaping experiment) • Batch size:• Temporal discount factor ( γ ): 0 . 
Revised: • Training steps: 1 × 10• Evaluation episodes: 3 × 10• Learning rate: 3 × 10 −• Experience buffer size: • Batch size:• Temporal discount factor ( γ ): 0 . B. E XPERIMENT -S PECIFIC C ONFIGURATIONS Note that we deﬁne a logarithmic sweep from x to y (inclusive) with n steps to be deﬁned by

Revised: Equation 3.

Revised: [Equation]

Revised: Biased Semantics

Revised: • Type: edgeward • World radius:• Goal Radius:• Experience buffer size:• Batch size:

Revised: Changing the Distribution of Entropy

Revised: • Number of independent runs per conﬁguration:• Experience buffer size:• Batch size:

Revised: Masking Environmental Parameters

Original: {+• World radius: logarithmic sweep from 2 to 40 with 300 steps Experience Buffer Size • Experience Buffer Size: logarithmic sweep from 32 to 4096 with 200 steps (ﬂoor function applied as experience buffer size is integer-valued) • Training steps: 2 × 
+}
Revised: {+• World radius: logarithmic sweep from 2 to 40 with 300 steps Experience Buffer Size • Experience Buffer Size: logarithmic sweep from 32 to 4096 with 200 steps (ﬂoor function applied as experience buffer size is integer-valued) • Training steps: 2 × 
+}

Revised: Expectation Chinese Restaurant Process

Revised: This is not an emergent language experiment and just consists of running the mathematical model.

Revised: • α :• β : logarithmic sweep from 1 to 1000 with 10 000 steps (ﬂoor function applied, though plotted without ﬂoor function) • Number of iterations per run: