Original: Emergent Graphical Conventions in a Visual Communication Game
Revised: Emergent Graphical Conventions in a Visual Communication Game

Original: https://sites.google.com/view/emergent-graphical-conventions
Revised: https://sites.google.com/view/emergent-graphical-conventions

Original: Abstract
Revised: Abstract

Original: Humans communicate with graphical sketches apart from symbolic languages (Fay et al., 2014).
Revised: Humans communicate with graphical sketches apart from symbolic languages (Fay et al., 2014).

Original: Primarily focusing on the latter, recent studies of emergent communication (Lazaridou and Baroni, 2020) overlook the sketches; they do not account for the evolution process through which symbolic sign systems emerge in the trade-off between iconicity and symbolicity.
Revised: Primarily focusing on the latter, recent studies of emergent communication (Lazaridou and Baroni, 2020) overlook the sketches; they do not account for the evolution process through which symbolic sign systems emerge in the trade-off between iconicity and symbolicity.

Original: In this work, we take the very first step to model and simulate this process via two neural agents playing a visual communication game; the sender communicates with the receiver by sketching on a canvas.
Revised: In this work, we take the very first step to model and simulate this process via two neural agents playing a visual communication game; the sender communicates with the receiver by sketching on a canvas.

Original: We devise a novel reinforcement learning method such that agents are evolved jointly towards successful communication and abstract graphical conventions.
Revised: We devise a novel reinforcement learning method such that agents are evolved jointly towards successful communication and abstract graphical conventions.

Original: To inspect the emerged conventions, we define three fundamental properties—iconicity, symbolicity, and semanticity—and design evaluation methods accordingly.
Revised: To inspect the emerged conventions, we define three fundamental properties—iconicity, symbolicity, and semanticity—and design evaluation methods accordingly.

Original: Our experimental results under different controls are consistent with the observation in studies of human graphical conventions (Hawkins et al., 2019; Fay et al., 2010).
Revised: Our experimental results under different controls are consistent with the observation in studies of human graphical conventions (Hawkins et al., 2019; Fay et al., 2010).

Original: Of note, we find that evolved sketches can preserve the continuum of semantics (Mikolov et al., 2013) under proper environmental pressures.
Revised: Of note, we find that evolved sketches can preserve the continuum of semantics (Mikolov et al., 2013) under proper environmental pressures.

Original: More interestingly, co-evolved agents can switch between conventionalized and iconic communication based on their familiarity with referents.
Revised: More interestingly, co-evolved agents can switch between conventionalized and iconic communication based on their familiarity with referents.

Original: We hope the present research can pave the path for studying emergent communication with the modality of sketches.
Revised: We hope the present research can pave the path for studying emergent communication with the modality of sketches.

Original:  Introduction
Revised:  Introduction

Original: Communication problem naturally arises when traveling in a foreign country where you do not speak the native language, which necessitates exploring non-linguistic means of communication, such as drawings.
Revised: Communication problem naturally arises when traveling in a foreign country where you do not speak the native language, which necessitates exploring non-linguistic means of communication, such as drawings.

Original: Due to its iconic nature ( i e ., perceptual resemblance to or natural association with the referent), drawings serve as a powerful tool to communicate concepts transcending language barriers (Fay et al., 2014).
Revised: Due to its iconic nature ( i e ., perceptual resemblance to or natural association with the referent), drawings serve as a powerful tool to communicate concepts transcending language barriers (Fay et al., 2014).

Original: In fact, we humans started to use drawings to convey messages dating back to 40,000–60,000 years ago (Hoffmann et al., 2018;
Revised: In fact, we humans started to use drawings to convey messages dating back to 40,000–60,000 years ago (Hoffmann et al., 2018;

Original: Hawkins et al., 2019).
Revised: Hawkins et al., 2019).

Original: Some cognitive science studies hypothesize a transition from sketch-based communication before the formation of sign systems and provide evidence that iconic signs can gradually become symbolic through repeated communication (Fay et al., 2014).
Revised: Some cognitive science studies hypothesize a transition from sketch-based communication before the formation of sign systems and provide evidence that iconic signs can gradually become symbolic through repeated communication (Fay et al., 2014).

Original: In contrast to icons , symbols are special forms bearing arbitrary relations to the referents.
Revised: In contrast to icons , symbols are special forms bearing arbitrary relations to the referents.

Original: Fig.
Revised: Fig.

Original: 1 describes a typical scenario of such phenomena: Alice (in green) uses a sketch to communicate the concept “rooster” to Bob (in yellow).
Revised: 1 describes a typical scenario of such phenomena: Alice (in green) uses a sketch to communicate the concept “rooster” to Bob (in yellow).

Original: Initially, they need to ground the sketch to the referent.
Revised: Initially, they need to ground the sketch to the referent.

Original: Later, details of the visual concept, such as strokes of the head and body, are gradually abstracted away, leaving only the most salient part, the crown.
Revised: Later, details of the visual concept, such as strokes of the head and body, are gradually abstracted away, leaving only the most salient part, the crown.

Original: The iconicity in the communicated sketch drops while the symbolicity rises.
Revised: The iconicity in the communicated sketch drops while the symbolicity rises.

Original: [Figure]
Revised: [Figure]

Original: While models of emerging communication protocols has attracted attention (Lazaridou et al., 2017;
Revised: While models of emerging communication protocols has attracted attention (Lazaridou et al., 2017;

Original: Cao et al., 2018; Evtimova et al., 2018; Havrylov and Titov, 2017; Lazaridou et al., 2018;
Revised: Cao et al., 2018; Evtimova et al., 2018; Havrylov and Titov, 2017; Lazaridou et al., 2018;

Original: Lazaridou and Baroni, 2020; Mordatch and Abbeel, 2018; Ren et al., 2020; Eccles et al., 2019; Graesser et al., 2019), the initial and primary communication medium is presumed and limited to be symbolic rather than iconic.
Revised: Lazaridou and Baroni, 2020; Mordatch and Abbeel, 2018; Ren et al., 2020; Eccles et al., 2019; Graesser et al., 2019), the initial and primary communication medium is presumed and limited to be symbolic rather than iconic.

Original: By simulating a multi-agent referential game , prior work seeks for the environmental driving forces behind the emergence of effective communications.
Revised: By simulating a multi-agent referential game , prior work seeks for the environmental driving forces behind the emergence of effective communications.

Original: In a typical setup of referential games, two agents play similar roles as in the above Alice-Bob example but share a primitive set of arbitrary tokens ( i.e ., the vocabulary).
Revised: In a typical setup of referential games, two agents play similar roles as in the above Alice-Bob example but share a primitive set of arbitrary tokens ( i.e ., the vocabulary).

Original: Using these tokens, an agent (the sender) attempts to communicate a message to another agent (the receiver).
Revised: Using these tokens, an agent (the sender) attempts to communicate a message to another agent (the receiver).

Original: A communication convention emerges when two agents successfully communicate by associating visual concepts in the images with tokens in the pre-selected vocabulary.
Revised: A communication convention emerges when two agents successfully communicate by associating visual concepts in the images with tokens in the pre-selected vocabulary.

Original: Though this line of work has probed into some linguistic properties of the established communication conventions (Lazaridou et al., 2017; 2018; Ren et al., 2020), some intriguing questions remain open: How do agents make a trade-off between iconicity and symbolicity to emerge symbolic sign systems?
Revised: Though this line of work has probed into some linguistic properties of the established communication conventions (Lazaridou et al., 2017; 2018; Ren et al., 2020), some intriguing questions remain open: How do agents make a trade-off between iconicity and symbolicity to emerge symbolic sign systems?

Original: In this work, we present the very first step of modeling and simulating the evolution process of graphical conventions (Hawkins et al., 2019), a two-participant communication convention whose medium is drawings in an abstract form.
Revised: In this work, we present the very first step of modeling and simulating the evolution process of graphical conventions (Hawkins et al., 2019), a two-participant communication convention whose medium is drawings in an abstract form.

Original: Specifically, we consider our contributions in three folds:
Revised: Specifically, we consider our contributions in three folds:

Original: First, we model a multi-agent visual communication game and propose a learning framework, wherein the sender and the receiver evolve jointly.
Revised: First, we model a multi-agent visual communication game and propose a learning framework, wherein the sender and the receiver evolve jointly.

Original: This visual communication game is an alternating sequential decision-making process, in which the sender generates a sequence of strokes step by step, terminated by the receiver.
Revised: This visual communication game is an alternating sequential decision-making process, in which the sender generates a sequence of strokes step by step, terminated by the receiver.

Original: In contrast to discretized tokens in prior work, strokes can be continuously parametrized (Ha and Eck, 2018; Huang et al., 2019)
Revised: In contrast to discretized tokens in prior work, strokes can be continuously parametrized (Ha and Eck, 2018; Huang et al., 2019)

Original: such that the derivatives of learning objectives can be more effectively back-propagated through communication channels (Foerster et al., 2016).
Revised: such that the derivatives of learning objectives can be more effectively back-propagated through communication channels (Foerster et al., 2016).

Original: We further derive a novel training surrogate for multi-agent reinforcement learning based on a joint value function and the eligibility trace method (Sutton and Barto, 2018).
Revised: We further derive a novel training surrogate for multi-agent reinforcement learning based on a joint value function and the eligibility trace method (Sutton and Barto, 2018).

Original: This differs from the REINFORCE-based surrogate in prior works on early-terminable sequential communication (Cao et al., 2018; Evtimova et al., 2018) and the TD learning method in the literature of multi-agent turn-taking games, wherein each agent has its own value function and has to model the value of others (Wen et al., 2019).
Revised: This differs from the REINFORCE-based surrogate in prior works on early-terminable sequential communication (Cao et al., 2018; Evtimova et al., 2018) and the TD learning method in the literature of multi-agent turn-taking games, wherein each agent has its own value function and has to model the value of others (Wen et al., 2019).

Original: In experiments, we empirically demonstrate that our integration of function approximation and Monte Carlo sampling facilitates the agents’ awareness of the correlation between complex and simple sketches, thereby enabling a smooth abstraction process.
Revised: In experiments, we empirically demonstrate that our integration of function approximation and Monte Carlo sampling facilitates the agents’ awareness of the correlation between complex and simple sketches, thereby enabling a smooth abstraction process.

Original: Second, we define essential properties in studying evolved sketches.
Revised: Second, we define essential properties in studying evolved sketches.

Original: Specifically, we define iconicity (Fay et al., 2014) as the drawings exhibiting high visual resemblance to the corresponding images, such that they are proximal to the latter when measured on the high-level embedding of a generalpurpose visual system; we define symbolicity (Fay et al., 2018) as these drawings being consistently separable in the high-level visual embedding, which facilitates new communication participants to easily distinguish between categories without grounding them to referents; and we define semanticity (Harispe et al., 2015) as the topography of the high-level embedding space of the drawings being strongly correlated to that of images, such that semantically similar instances and categories lie close to each other in the embedding space.
Revised: Specifically, we define iconicity (Fay et al., 2014) as the drawings exhibiting high visual resemblance to the corresponding images, such that they are proximal to the latter when measured on the high-level embedding of a generalpurpose visual system; we define symbolicity (Fay et al., 2018) as these drawings being consistently separable in the high-level visual embedding, which facilitates new communication participants to easily distinguish between categories without grounding them to referents; and we define semanticity (Harispe et al., 2015) as the topography of the high-level embedding space of the drawings being strongly correlated to that of images, such that semantically similar instances and categories lie close to each other in the embedding space.

Original: Of note, this is not the only way to define these cognitive concepts; we intend to align readers on critical concepts in our work.
Revised: Of note, this is not the only way to define these cognitive concepts; we intend to align readers on critical concepts in our work.

Original: Third, we present a suite of quantitative and qualitative methods to evaluate the emergent graphical conventions based on the above carefully defined iconicity , symbolicity , and semanticity .
Revised: Third, we present a suite of quantitative and qualitative methods to evaluate the emergent graphical conventions based on the above carefully defined iconicity , symbolicity , and semanticity .

Original: This is necessary because a high communication rate does not imply good representations (Bouchacourt and Baroni, 2018).
Revised: This is necessary because a high communication rate does not imply good representations (Bouchacourt and Baroni, 2018).

Original: The graphical nature of the communication medium mandates us to repurpose representation learning metrics rather than adopt linguistic metrics in emergent symbolic communication.
Revised: The graphical nature of the communication medium mandates us to repurpose representation learning metrics rather than adopt linguistic metrics in emergent symbolic communication.

Original: We evaluate the contribution of different environmental drivers, early decision, sender’s update, and sequential communication, to the three properties of the emergent conventions.
Revised: We evaluate the contribution of different environmental drivers, early decision, sender’s update, and sequential communication, to the three properties of the emergent conventions.

Original: Critically, the empirical results assessed on our metrics align well with our prediction based on the findings of human graphical conventions (Fay et al., 2010; Hawkins et al., 2019), justifying our environment, model, and evaluation.
Revised: Critically, the empirical results assessed on our metrics align well with our prediction based on the findings of human graphical conventions (Fay et al., 2010; Hawkins et al., 2019), justifying our environment, model, and evaluation.

Original: One of these setups emerges conventions where the three properties are consistent with our expectation of a sign system.
Revised: One of these setups emerges conventions where the three properties are consistent with our expectation of a sign system.

Original: Particularly, we find two inspiring phenomena: (i) Evolved sketches from semantically similar classes are perceptually more similar to each other than those falling into different superclasses.
Revised: Particularly, we find two inspiring phenomena: (i) Evolved sketches from semantically similar classes are perceptually more similar to each other than those falling into different superclasses.

Original: (ii) To communicate concepts not included in their established conventions, evolved agents can return to more iconic communications than humans.
Revised: (ii) To communicate concepts not included in their established conventions, evolved agents can return to more iconic communications than humans.

Original: We hope our work sheds light on emergent communication in the modality of sketches and facilitates the study of cognitive evolutionary theories of pictographic sign systems.
Revised: We hope our work sheds light on emergent communication in the modality of sketches and facilitates the study of cognitive evolutionary theories of pictographic sign systems.

Original:  Related work
Revised:  Related work

Original: Learning to sketch Ha and Eck (2018) begin the endeavor of teaching modern neural models to sketch stroke by stroke.
Revised: Learning to sketch Ha and Eck (2018) begin the endeavor of teaching modern neural models to sketch stroke by stroke.

Original: However, generating meaningful stroke sequences directly from various categories of natural images is still in the early phase (Song et al., 2018;
Revised: However, generating meaningful stroke sequences directly from various categories of natural images is still in the early phase (Song et al., 2018;

Original: Wang et al., 2021; Zou et al., 2018).
Revised: Wang et al., 2021; Zou et al., 2018).

Original: To assure the interestingness of the category-level sketch communication, we design a stagewise agent that transfers a natural image into a pixel-level sketch (Kampelmuhler and Pinz, 2020) and draws the sketch on the canvas stroke by stroke with a policy (Huang et al., 2019).
Revised: To assure the interestingness of the category-level sketch communication, we design a stagewise agent that transfers a natural image into a pixel-level sketch (Kampelmuhler and Pinz, 2020) and draws the sketch on the canvas stroke by stroke with a policy (Huang et al., 2019).

Original: To pre-train our neural agents to sketch, we select Sketchy (Sangkloy et al., 2016) from many datasets (Yu et al., 2016;
Revised: To pre-train our neural agents to sketch, we select Sketchy (Sangkloy et al., 2016) from many datasets (Yu et al., 2016;

Original: Ha and Eck, 2018; Eitz et al., 2012; Sangkloy et al., 2016) for its fine-grained photo-sketch correspondence, rich stroke-level annotations, and well-organized categorization structures.
Revised: Ha and Eck, 2018; Eitz et al., 2012; Sangkloy et al., 2016) for its fine-grained photo-sketch correspondence, rich stroke-level annotations, and well-organized categorization structures.

Original: Communication games While learning to sketch is formulated as a single-agent task with explicit supervision, our focus is on how sketches would evolve when utilized as the communication medium between two cooperative agents .
Revised: Communication games While learning to sketch is formulated as a single-agent task with explicit supervision, our focus is on how sketches would evolve when utilized as the communication medium between two cooperative agents .

Original: Their cooperation is always formulated as a communication game, recently adopted to study phenomena in human-robot teaming (Yuan et al., 2022) and natural languages, such as symbolic language acquisition (Graesser et al., 2019) and the emergence of compositionality (Ren et al., 2020).
Revised: Their cooperation is always formulated as a communication game, recently adopted to study phenomena in human-robot teaming (Yuan et al., 2022) and natural languages, such as symbolic language acquisition (Graesser et al., 2019) and the emergence of compositionality (Ren et al., 2020).

Original: Some notable works (Lazaridou et al., 2017; 2018; Evtimova et al., 2018; Havrylov and Titov, 2017) have devised interesting metrics, such as purity (Lazaridou et al., 2017) and topographic similarity (Lazaridou et al., 2018).
Revised: Some notable works (Lazaridou et al., 2017; 2018; Evtimova et al., 2018; Havrylov and Titov, 2017) have devised interesting metrics, such as purity (Lazaridou et al., 2017) and topographic similarity (Lazaridou et al., 2018).

Original: In comparison, our work is unique due to the distinctive communication medium, continuously parametrized sketches.
Revised: In comparison, our work is unique due to the distinctive communication medium, continuously parametrized sketches.

Original: Although a concurrent work (Mihai and Hare, 2021) also enables the agents to sketch in a communication game, it focuses primarily on drawing interpretable sketches without abstracting them into graphical symbols along the communication.
Revised: Although a concurrent work (Mihai and Hare, 2021) also enables the agents to sketch in a communication game, it focuses primarily on drawing interpretable sketches without abstracting them into graphical symbols along the communication.

Original: We position our work as an alternative to emergent symbolic communication, since the emergent graphical symbols may better represent the continuum of semantics, as encoded in the vector representation of tokens (Mikolov et al., 2013).
Revised: We position our work as an alternative to emergent symbolic communication, since the emergent graphical symbols may better represent the continuum of semantics, as encoded in the vector representation of tokens (Mikolov et al., 2013).

Original: Methodologically, we devise new evaluation metrics for sketch modality, assessing iconicity , symbolicity , and semanticity in the evolved sketches.
Revised: Methodologically, we devise new evaluation metrics for sketch modality, assessing iconicity , symbolicity , and semanticity in the evolved sketches.

Original: Emergent graphical conventions Evolving from sketches to graphical conventions/symbols is an active field in cognitive science under the banner of “emergent sign systems.”
Revised: Emergent graphical conventions Evolving from sketches to graphical conventions/symbols is an active field in cognitive science under the banner of “emergent sign systems.”

Original: Fay et al.
Revised: Fay et al.

Original: (2010) show that pair interaction can form their local conventions when they play the Pictionary game.
Revised: (2010) show that pair interaction can form their local conventions when they play the Pictionary game.

Original: Using natural images instead of texts as the prompt, Hawkins et al.
Revised: Using natural images instead of texts as the prompt, Hawkins et al.

Original: (2019) show that visual properties of the images also influence the formed graphical conventions besides partners’ shared interaction history; i.e ., the evolved sketches highlight visually salient parts.
Revised: (2019) show that visual properties of the images also influence the formed graphical conventions besides partners’ shared interaction history; i.e ., the evolved sketches highlight visually salient parts.

Original: Nevertheless, only a few computational models exist apart from these human behavior studies.
Revised: Nevertheless, only a few computational models exist apart from these human behavior studies.

Original: Fan et al.
Revised: Fan et al.

Original: (2020) describe a model for selecting complex or simple sketches considering the communication context.
Revised: (2020) describe a model for selecting complex or simple sketches considering the communication context.

Original: Bhunia et al.
Revised: Bhunia et al.

Original: (2020) and Muhammad et al.
Revised: (2020) and Muhammad et al.

Original: (2018) consider stroke selection and reordering to simplify the sketch.
Revised: (2018) consider stroke selection and reordering to simplify the sketch.

Original: In contrast to sketch or stroke selection, we model embodied agents who can draw and recognize sketches.
Revised: In contrast to sketch or stroke selection, we model embodied agents who can draw and recognize sketches.

Original:  The visual communication game
Revised:  The visual communication game

Original: Our visual communication game is formulated as a tuple ( I , C, A S , A R , G, r, γ ) , where I is the image set to be presented to the sender S and the receiver R .
Revised: Our visual communication game is formulated as a tuple ( I , C, A S , A R , G, r, γ ) , where I is the image set to be presented to the sender S and the receiver R .

Original: These images contain a single object in the foreground, and hence the image space I can be partitioned into N classes
Revised: These images contain a single object in the foreground, and hence the image space I can be partitioned into N classes

Original: [Figure]
Revised: [Figure]

Original: according to the object categories.
Revised: according to the object categories.

Original: In each game round, the sender is presented with one image I S , and the receiver is presented with M images { I 1 R , ..., I M R } .
Revised: In each game round, the sender is presented with one image I S , and the receiver is presented with M images { I 1 R , ..., I M R } .

Original: Prior work shows that category-level context can force the agent to coordinate on more abstract properties and help draw sketches to be more recognizable as the type of object (Lazaridou et al., 2017; Mihai and Hare, 2021).
Revised: Prior work shows that category-level context can force the agent to coordinate on more abstract properties and help draw sketches to be more recognizable as the type of object (Lazaridou et al., 2017; Mihai and Hare, 2021).

Original: We make the observations of S and R disjoint ( i.e ., I S / ∈ { I 1 R , ..., I MR } ) but with a target image I ∗ R in the same class as I S .
Revised: We make the observations of S and R disjoint ( i.e ., I S / ∈ { I 1 R , ..., I MR } ) but with a target image I ∗ R in the same class as I S .

Original: We refer to the M images that the receiver can see as the context .
Revised: We refer to the M images that the receiver can see as the context .

Original: Neither the receiver or the sender would see the image(s) presented to their partner; they can only communicate this information by drawing sketches on the canvas C , observable to both players.
Revised: Neither the receiver or the sender would see the image(s) presented to their partner; they can only communicate this information by drawing sketches on the canvas C , observable to both players.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, Cis initialized to be blank at the beginning of each round.
Revised: 2, Cis initialized to be blank at the beginning of each round.

Original: Only the sender can draw on the canvas with actions chosen from A S .
Revised: Only the sender can draw on the canvas with actions chosen from A S .

Original: The action at each time step consists of 5 strokes, which are continuous vectors in R 6 .
Revised: The action at each time step consists of 5 strokes, which are continuous vectors in R 6 .

Original: We constrain each dimension to be in (0 , 1) due to the limited space of the canvas.
Revised: We constrain each dimension to be in (0 , 1) due to the limited space of the canvas.

Original: The canvas is updated from C t to C t +1 by the renderer G after each step of the sender’s sketching.
Revised: The canvas is updated from C t to C t +1 by the renderer G after each step of the sender’s sketching.

Original: The receiver, after observing the updated canvas, would choose among the M images or wait for the next step from the sender; these M + 1 possible actions constitute A R .
Revised: The receiver, after observing the updated canvas, would choose among the M images or wait for the next step from the sender; these M + 1 possible actions constitute A R .

Original: A game round terminates when the receiver gives up waiting and chooses one from the images.
Revised: A game round terminates when the receiver gives up waiting and chooses one from the images.

Original: After the termination, the sender and the receiver will receive a shared reward or penalty, depending on if the receiver makes the right choice:
Revised: After the termination, the sender and the receiver will receive a shared reward or penalty, depending on if the receiver makes the right choice:

Original: [Equation]
Revised: [Equation]

Original: This reward/penalty is temporally decayed with a decay factor γ .
Revised: This reward/penalty is temporally decayed with a decay factor γ .

Original: That is, if the receiver decides to choose from the images at step t , this cooperating pair will receive either γ t or − γ t .
Revised: That is, if the receiver decides to choose from the images at step t , this cooperating pair will receive either γ t or − γ t .

Original: Hence, even though the players do not receive an explicit penalty for long conversations, there is an implicit penalty/reward for delayed positive/negative answers.
Revised: Hence, even though the players do not receive an explicit penalty for long conversations, there is an implicit penalty/reward for delayed positive/negative answers.

Original: No reward will be assigned if the receiver chooses to wait.
Revised: No reward will be assigned if the receiver chooses to wait.

Original: The next round starts after the reward/penalty assignment.
Revised: The next round starts after the reward/penalty assignment.

Original:  Agents
Revised:  Agents

Original: The two agents involved in the visual communication game are modeled with two decision policies, π S and π R , for the sender and the receiver, respectively.
Revised: The two agents involved in the visual communication game are modeled with two decision policies, π S and π R , for the sender and the receiver, respectively.

Original: These policies are stochastic mapping from the agents’ observation space to the action space:
Revised: These policies are stochastic mapping from the agents’ observation space to the action space:

Original: [Equation]
Revised: [Equation]

Original: where P ( A ) is a distribution over the support set A .
Revised: where P ( A ) is a distribution over the support set A .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, at each time step t ∈ { 0 , ...T } , the sender first emits the stroke parameters for the next five strokes a St ∼ π S ( I S , C t ) .
Revised: 2, at each time step t ∈ { 0 , ...T } , the sender first emits the stroke parameters for the next five strokes a St ∼ π S ( I S , C t ) .

Original: These strokes are applied to the canvas by a differentiable renderer, C t +1 =
Revised: These strokes are applied to the canvas by a differentiable renderer, C t +1 =

Original: G ( C t , a St ) .
Revised: G ( C t , a St ) .

Original: Next, the updated canvas is transmitted to the receiver.
Revised: Next, the updated canvas is transmitted to the receiver.

Original: The receiver decides whether to terminate the game by making its prediction ( i.e ., a Rt ∈ { 1 , ..., M } ) or wait for another round ( i.e ., a Rt = M + 1 ); its decision is sampled from π R .
Revised: The receiver decides whether to terminate the game by making its prediction ( i.e ., a Rt ∈ { 1 , ..., M } ) or wait for another round ( i.e ., a Rt = M + 1 ); its decision is sampled from π R .

Original: If a prediction is made, it is used to select the image I a R R from { I 1 R , ...
Revised: If a prediction is made, it is used to select the image I a R R from { I 1 R , ...

Original: I MR } and score this game with r ( I S , I a R R ) .
Revised: I MR } and score this game with r ( I S , I a R R ) .

Original: Otherwise, this routine repeats in the next step t ← t + 1 .
Revised: Otherwise, this routine repeats in the next step t ← t + 1 .

Original: 4.1 Sender
Revised: 4.1 Sender

Original: Prior to playing the visual communication game, the sender should be able to (i) extract edges from natural images (Xie and Tu, 2015) and (ii) draw sketches that closely resemble the configurations of salient edges (Li et al., 2019), just as humans do (Sayim and Cavanagh, 2011).
Revised: Prior to playing the visual communication game, the sender should be able to (i) extract edges from natural images (Xie and Tu, 2015) and (ii) draw sketches that closely resemble the configurations of salient edges (Li et al., 2019), just as humans do (Sayim and Cavanagh, 2011).

Original: To endow the sender with these capabilities, we design a stage-wise architecture h S
Revised: To endow the sender with these capabilities, we design a stage-wise architecture h S

Original: = g S ◦ f S .
Revised: = g S ◦ f S .

Original: Specifically, I S is first converted to a target sketch
Revised: Specifically, I S is first converted to a target sketch

Original: ˆ I S using a visual module f S (Kampelmuhler and Pinz, 2020), capturing the salient edge information in the natural image; we directly adopt the pre-trained model from the referred work.
Revised: ˆ I S using a visual module f S (Kampelmuhler and Pinz, 2020), capturing the salient edge information in the natural image; we directly adopt the pre-trained model from the referred work.

Original: Next, ˆ I S is concatenated with the current canvas C t and fed to the sketching module g S , whose architecture is built upon Huang et al.
Revised: Next, ˆ I S is concatenated with the current canvas C t and fed to the sketching module g S , whose architecture is built upon Huang et al.

Original: This sketching module outputs five vectors in the form ( x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ) , which parametrizes the curve of one stroke.
Revised: This sketching module outputs five vectors in the form ( x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ) , which parametrizes the curve of one stroke.

Original: The policy is parametrized as a Gaussian distribution during training,
Revised: The policy is parametrized as a Gaussian distribution during training,

Original: [Equation]
Revised: [Equation]

Original: where I is the identity matrix, and c is a constant hyperparameter.
Revised: where I is the identity matrix, and c is a constant hyperparameter.

Original: During the testing, we set c = 0 .
Revised: During the testing, we set c = 0 .

Original: These stroke parameters a St are fed into a pre-trained renderer G (Huang et al., 2019) to update the canvas, C t +1 =
Revised: These stroke parameters a St are fed into a pre-trained renderer G (Huang et al., 2019) to update the canvas, C t +1 =

Original: G ( C t , a St ) .
Revised: G ( C t , a St ) .

Original: This renderer is fully differentiable, enabling end-to-end model-based training (Hafner et al., 2019) of the sketching module g S .
Revised: This renderer is fully differentiable, enabling end-to-end model-based training (Hafner et al., 2019) of the sketching module g S .

Original: We pre-train g S on Sketchy (Sangkloy et al., 2016); please refer to the supplementary video for results.
Revised: We pre-train g S on Sketchy (Sangkloy et al., 2016); please refer to the supplementary video for results.

Original: 4.2 Receiver
Revised: 4.2 Receiver

Original: The receiver, similar to the sender, should also carry some rudimentary visual capability to this game.
Revised: The receiver, similar to the sender, should also carry some rudimentary visual capability to this game.

Original: Unlike the low-level vision needed for the sender, the requirement for the receiver is high-level visual recognition.
Revised: Unlike the low-level vision needed for the sender, the requirement for the receiver is high-level visual recognition.

Original: Therefore, we adopt a pre-trained VGG16 (Simonyan and Zisserman, 2015) as the visual module f R : I → R 4096 of the receiver, following a similar practice in recent literature (Lazaridou et al., 2017; Havrylov and Titov, 2017).
Revised: Therefore, we adopt a pre-trained VGG16 (Simonyan and Zisserman, 2015) as the visual module f R : I → R 4096 of the receiver, following a similar practice in recent literature (Lazaridou et al., 2017; Havrylov and Titov, 2017).

Original: The output of this visual module is a vector, and further transformed by two separate linear layers, g KR and g QR , into visual embeddings, h KR ( I ) and h QR ( I ) .
Revised: The output of this visual module is a vector, and further transformed by two separate linear layers, g KR and g QR , into visual embeddings, h KR ( I ) and h QR ( I ) .

Original: That is, h KR =
Revised: That is, h KR =

Original: [Equation]
Revised: [Equation]

Original: When observing both the context { I 1 R , ..., I MR } and the canvas C t , the receiver first embeds each of them with h R .
Revised: When observing both the context { I 1 R , ..., I MR } and the canvas C t , the receiver first embeds each of them with h R .

Original: Next, it makes the decision based on the similarity between the current canvas and each option in the context.
Revised: Next, it makes the decision based on the similarity between the current canvas and each option in the context.

Original: The decision module is thus realized by a Boltzmann distribution based on resemblance:
Revised: The decision module is thus realized by a Boltzmann distribution based on resemblance:

Original: [Equation]
Revised: [Equation]

Original: where I M +1 R =
Revised: where I M +1 R =

Original: C t − 1 .
Revised: C t − 1 .

Original: Of note, although a similar policy was proposed before (Lazaridou et al., 2018;
Revised: Of note, although a similar policy was proposed before (Lazaridou et al., 2018;

Original: Havrylov and Titov, 2017), our π R is distinct as it is endowed with an external memory of C t − 1 .
Revised: Havrylov and Titov, 2017), our π R is distinct as it is endowed with an external memory of C t − 1 .

Original: Intuitively, if the receiver finds the current canvas C t closer to the last canvas C t − 1 in the embedding space than all M options in the context, it will choose to emit a Rt = M + 1 and delay the decision to the next step; a prediction can only be made when the receiver finds the current canvas is informative enough.
Revised: Intuitively, if the receiver finds the current canvas C t closer to the last canvas C t − 1 in the embedding space than all M options in the context, it will choose to emit a Rt = M + 1 and delay the decision to the next step; a prediction can only be made when the receiver finds the current canvas is informative enough.

Original: As a result, the sender would draw informative strokes as early as possible to avoid the implicit penalty in the decayed reward.
Revised: As a result, the sender would draw informative strokes as early as possible to avoid the implicit penalty in the decayed reward.

Original: 4.3 Learning to communicate
Revised: 4.3 Learning to communicate

Original: Policies of the sender and the receiver are trained jointly to maximize the objective:
Revised: Policies of the sender and the receiver are trained jointly to maximize the objective:

Original: [Equation]
Revised: [Equation]

Original: where τ = { C 0 , a S 0 , C 1 , a R 1 , a S 1 , ... } is the simulated episodic trajectory.
Revised: where τ = { C 0 , a S 0 , C 1 , a R 1 , a S 1 , ... } is the simulated episodic trajectory.

Original: As well known in reinforcement learning, the analytical expectation in Eq.
Revised: As well known in reinforcement learning, the analytical expectation in Eq.

Original: (4) is intractable to calculate along the trajectory τ .
Revised: (4) is intractable to calculate along the trajectory τ .

Original: We devise value functions V ( X t ) and V λ ( X t ) for an optimization surrogate:
Revised: We devise value functions V ( X t ) and V λ ( X t ) for an optimization surrogate:

Original: [Equation]
Revised: [Equation]

Original: where δ ( a Rt ) is the Dirac delta function that returns 1 when the action is wait and 0 otherwise.
Revised: where δ ( a Rt ) is the Dirac delta function that returns 1 when the action is wait and 0 otherwise.

Original: X t = cat ([ I S ] , ˆ X t ) , where ˆ X t =
Revised: X t = cat ([ I S ] , ˆ X t ) , where ˆ X t =

Original: [ I 1 R , ...I MR , C t − 1 , C t ] .
Revised: [ I 1 R , ...I MR , C t − 1 , C t ] .

Original: The expectation E π S ( a St | I S ,C t − 1 ) [ · ] is approximated with the point estimate, as in the reparametrization in VAE (Kingma and Welling, 2014).
Revised: The expectation E π S ( a St | I S ,C t − 1 ) [ · ] is approximated with the point estimate, as in the reparametrization in VAE (Kingma and Welling, 2014).

Original: The expectation E π R ( a Rt | ˆ X t ) [ · ] can be calculated analytically because π Rt is a categorical distribution.
Revised: The expectation E π R ( a Rt | ˆ X t ) [ · ] can be calculated analytically because π Rt is a categorical distribution.

Original: The connection between E π S and E π R is one of our contributions.
Revised: The connection between E π S and E π R is one of our contributions.

Original: Specifically, C t in ˆ X t in π Rt ( a Rt | ˆ X t ) is generated by the differentiable renderer G with the actions a St from the sender policy π S ( a St | I S , C t − 1 ) .
Revised: Specifically, C t in ˆ X t in π Rt ( a Rt | ˆ X t ) is generated by the differentiable renderer G with the actions a St from the sender policy π S ( a St | I S , C t − 1 ) .

Original: Hence, we can have both ∂ V /∂π Rt and ∂ V /∂π St based on Eq.
Revised: Hence, we can have both ∂ V /∂π Rt and ∂ V /∂π St based on Eq.

Original: This results in a novel multi-agent variant of the general policy gradient (Sutton et al., 2000; Silver et al., 2014).
Revised: This results in a novel multi-agent variant of the general policy gradient (Sutton et al., 2000; Silver et al., 2014).

Original: V λ ( X t ) in Eq.
Revised: V λ ( X t ) in Eq.

Original: (5) is an eligibility trace approximation (Sutton and Barto, 2018) of the ground-truth value function.
Revised: (5) is an eligibility trace approximation (Sutton and Barto, 2018) of the ground-truth value function.

Original: Intuitively, a value estimate with eligibility trace V λ mixes the bootstrapped Monte
Revised: Intuitively, a value estimate with eligibility trace V λ mixes the bootstrapped Monte

Original: Carlo estimate V kN ( X t ) =
Revised: Carlo estimate V kN ( X t ) =

Original: E π S ,π R [ (cid:80) h − 1 n = t γ n − t r n + γ h − t
Revised: E π S ,π R [ (cid:80) h − 1 n = t γ n − t r n + γ h − t

Original: δ
Revised: δ

Original: ( a Rh ) υ ϕ ( X h )] at different roll-out lengths k , with h = min ( t + k, T choice ) being the maximal timestep.
Revised: ( a Rh ) υ ϕ ( X h )] at different roll-out lengths k , with h = min ( t + k, T choice ) being the maximal timestep.

Original: T choice is the timestamp when the receiver stops waiting.
Revised: T choice is the timestamp when the receiver stops waiting.

Original: The articulation of such termination also makes our eligibility trace deviate from the general derivation with infinite horizon.
Revised: The articulation of such termination also makes our eligibility trace deviate from the general derivation with infinite horizon.

Original: We derive an episodic version as
Revised: We derive an episodic version as

Original: [Equation]
Revised: [Equation]

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: v ϕ
Revised: v ϕ

Original: ( X
Revised: ( X

Original: [Equation]
Revised: [Equation]

Original: ) otherwise, (6) where
Revised: ) otherwise, (6) where

Original: H =
Revised: H =

Original: T choice − t
Revised: T choice − t

Original: + 1 .
Revised: + 1 .

Original: Please refer to Appendix C for a detailed derivation.
Revised: Please refer to Appendix C for a detailed derivation.

Original: Finally, v ϕ ( X t ) is trained by regressing the value returns:
Revised: Finally, v ϕ ( X t ) is trained by regressing the value returns:

Original: [Equation]
Revised: [Equation]

Original: The training algorithm is summarized in Alg. 1.
Revised: The training algorithm is summarized in Alg. 1.

Original:  Experiments 5.1 Settings
Revised:  Experiments 5.1 Settings

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Images We used the Sketchy dataset (Sangkloy et al., 2016) as the image source.
Revised: Images We used the Sketchy dataset (Sangkloy et al., 2016) as the image source.

Original: Due to the limited performance of the sketching module on open-domain image-to-sketch sequential generation, we select 40 categories (10 images per category, see Appendix A) with satisfactory sketching behaviors.
Revised: Due to the limited performance of the sketching module on open-domain image-to-sketch sequential generation, we select 40 categories (10 images per category, see Appendix A) with satisfactory sketching behaviors.

Original: Environmental drivers With the visual communication game and the learning agents at hand, we investigate the factors in emergent graphical conventions with controlled experiments.
Revised: Environmental drivers With the visual communication game and the learning agents at hand, we investigate the factors in emergent graphical conventions with controlled experiments.

Original: Tab.
Revised: Tab.

Original: 1 lists all designed settings.
Revised: 1 lists all designed settings.

Original: Specifically, we consider the following:
Revised: Specifically, we consider the following:

Original: [Table]
Revised: [Table]

Original: • Can receiver make early decisions?
Revised: • Can receiver make early decisions?

Original: The hypothesis is that the receiver’s decision before exhausting the allowed communication steps may inform the sender of the marginal benefit of each stroke and incentivize it to prioritize the most informative strokes.
Revised: The hypothesis is that the receiver’s decision before exhausting the allowed communication steps may inform the sender of the marginal benefit of each stroke and incentivize it to prioritize the most informative strokes.

Original: The corresponding control setting is max-step , wherein the receiver can only make the choice after the sender finishes its drawing at the maximum step.
Revised: The corresponding control setting is max-step , wherein the receiver can only make the choice after the sender finishes its drawing at the maximum step.

Original: This factor is on in other settings.
Revised: This factor is on in other settings.

Original: • Can the sender change its way of drawing?
Revised: • Can the sender change its way of drawing?

Original: The hypothesis is that the mutual adaptation of the sender and the receiver may lead to better abstraction in the evolved sketches.
Revised: The hypothesis is that the mutual adaptation of the sender and the receiver may lead to better abstraction in the evolved sketches.

Original: Particularly, the sender may develop new ways of drawing in the evolution.
Revised: Particularly, the sender may develop new ways of drawing in the evolution.

Original: The corresponding control setting is sender-fixed , wherein we freeze the parameters of the sender such that the receiver has to adapt to its partner.
Revised: The corresponding control setting is sender-fixed , wherein we freeze the parameters of the sender such that the receiver has to adapt to its partner.

Original: This factor is on in other settings.
Revised: This factor is on in other settings.

Original: • Is the game sequential, and can the receiver observe more complex drawings?
Revised: • Is the game sequential, and can the receiver observe more complex drawings?

Original: The hypothesis is that the modeling of a sequential decision-making game and the evolution from more complex sketches may regularize the arbitrariness, which is unique for graphical conventions.
Revised: The hypothesis is that the modeling of a sequential decision-making game and the evolution from more complex sketches may regularize the arbitrariness, which is unique for graphical conventions.

Original: The corresponding control setting is one-step : There only exists one step of sketching, thus no sequential decisionmaking at all.
Revised: The corresponding control setting is one-step : There only exists one step of sketching, thus no sequential decisionmaking at all.

Original: This factor is on in other settings.
Revised: This factor is on in other settings.

Original: Training, validation, and generalization We train the sender/receiver with batched forward and back-propagation, with a batch size of 64 and maximum roll-out step T = 7 .
Revised: Training, validation, and generalization We train the sender/receiver with batched forward and back-propagation, with a batch size of 64 and maximum roll-out step T = 7 .

Original: We update using Adam (Kingma and Ba, 2015) with the learning rate 0.0001 for a total of 30k iterations.
Revised: We update using Adam (Kingma and Ba, 2015) with the learning rate 0.0001 for a total of 30k iterations.

Original: We train each model on a single Nvidia RTX A6000; one experiment takes 20 hours.
Revised: We train each model on a single Nvidia RTX A6000; one experiment takes 20 hours.

Original: Except for the sender-fixed setting, there is a warm-up phase in the first 2000 iterations for the sender where we linearly increase the learning rate to 0.0001.
Revised: Except for the sender-fixed setting, there is a warm-up phase in the first 2000 iterations for the sender where we linearly increase the learning rate to 0.0001.

Original: After the warm-up phase, the learning rate of both agents will be decreased exponentially by 0 .
Revised: After the warm-up phase, the learning rate of both agents will be decreased exponentially by 0 .

Original: 99 i − 20001000 , where i is the number of training iterations.
Revised: 99 i − 20001000 , where i is the number of training iterations.

Original: In all settings, we set
Revised: In all settings, we set

Original: M = 3 , γ = 0 .
Revised: M = 3 , γ = 0 .

Original: Between iterations, we randomly sample another 10 batches for validation.
Revised: Between iterations, we randomly sample another 10 batches for validation.

Original: We use 30 categories (10 images per category) for training and held out 10 images per category for the unseen-instance test; another 10 categories are for the unseen-class test.
Revised: We use 30 categories (10 images per category) for training and held out 10 images per category for the unseen-instance test; another 10 categories are for the unseen-class test.

Original: Each image is communicated as the target, resulting in 300 + 100 pairs in the test set.
Revised: Each image is communicated as the target, resulting in 300 + 100 pairs in the test set.

Original: At each iteration, the categories and instances are sampled randomly to constitute the context.
Revised: At each iteration, the categories and instances are sampled randomly to constitute the context.

Original: Results henceforth are statistics of 5 random seeds.
Revised: Results henceforth are statistics of 5 random seeds.

Original: 5.2 Results 5.2.1 Communication efficacy and sketch abstraction
Revised: 5.2 Results 5.2.1 Communication efficacy and sketch abstraction

Original: We record both the success rate and the communication steps over the training iterations; see Fig. 3.
Revised: We record both the success rate and the communication steps over the training iterations; see Fig. 3.

Original: The communication is considered successful when the receiver makes the correct prediction at the end of the game.
Revised: The communication is considered successful when the receiver makes the correct prediction at the end of the game.

Original: In Fig.
Revised: In Fig.

Original: 3a, agents in all settings except one-step converge to a success rate above 80% .
Revised: 3a, agents in all settings except one-step converge to a success rate above 80% .

Original: Among them, the communicating pairs in the complete setting and the sender-fixed setting evolve to achieve a comparable success rate with the retrieve baseline.
Revised: Among them, the communicating pairs in the complete setting and the sender-fixed setting evolve to achieve a comparable success rate with the retrieve baseline.

Original: Interestingly, these two pairs also emerge a phenomenon resembling the natural observation in human studies, named systematic reduction (Lewis, 1969):
Revised: Interestingly, these two pairs also emerge a phenomenon resembling the natural observation in human studies, named systematic reduction (Lewis, 1969):

Original: The average steps first increase and then gradually decrease as in Fig.
Revised: The average steps first increase and then gradually decrease as in Fig.

Original: 3b.
Revised: 3b.

Original: Contrasting complete and sender-fixed , we can see: (i) The emergent conventions in the former is much simpler than the latter (less steps in Fig.
Revised: Contrasting complete and sender-fixed , we can see: (i) The emergent conventions in the former is much simpler than the latter (less steps in Fig.

Original: 3b), which implies the contribution of mutual adaptation in sketch abstraction.
Revised: 3b), which implies the contribution of mutual adaptation in sketch abstraction.

Original: (ii)
Revised: (ii)

Original: The success rate in Fig.
Revised: The success rate in Fig.

Original: 3a in the former converges
Revised: 3a in the former converges

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: bit more slowly, which is reasonable since the senders can explore and change the way of drawing.
Revised: bit more slowly, which is reasonable since the senders can explore and change the way of drawing.

Original: In comparison, if the receiver cannot make early decisions, it has no intention to relate sketches ( i.e ., C t − 1 and C t ) at consecutive timesteps.
Revised: In comparison, if the receiver cannot make early decisions, it has no intention to relate sketches ( i.e ., C t − 1 and C t ) at consecutive timesteps.

Original: The sender is thus unaware of each stroke’s marginal information gain, which in return makes their learning harder.
Revised: The sender is thus unaware of each stroke’s marginal information gain, which in return makes their learning harder.

Original: This might explain the relatively low success rate in the max-step setting.
Revised: This might explain the relatively low success rate in the max-step setting.

Original: The failure of the one-step pairs reveals the irreplaceable roles of sequential decision-making and observing complex sketches in emergent graphical communication.
Revised: The failure of the one-step pairs reveals the irreplaceable roles of sequential decision-making and observing complex sketches in emergent graphical communication.

Original: To further inspect how our proposed modeling and learning on sequential decision-making facilitate the desired evolution in the sketches, we conduct an ablation study by comparing our proposed learning surrogate (Eq.
Revised: To further inspect how our proposed modeling and learning on sequential decision-making facilitate the desired evolution in the sketches, we conduct an ablation study by comparing our proposed learning surrogate (Eq.

Original: (5)) and a vanilla policy gradient baseline, REINFORCE (Williams, 1992) with Monte Carlo cumulative rewards
Revised: (5)) and a vanilla policy gradient baseline, REINFORCE (Williams, 1992) with Monte Carlo cumulative rewards

Original: [Equation]
Revised: [Equation]

Original: Our comparison spans three axes.
Revised: Our comparison spans three axes.

Original: First, the REINFORCE baseline converges much more slowly than the proposed surrogate; see cumulative vs complete in Fig. 3a.
Revised: First, the REINFORCE baseline converges much more slowly than the proposed surrogate; see cumulative vs complete in Fig. 3a.

Original: Second, we check the robustness under the variation of decay factor γ .
Revised: Second, we check the robustness under the variation of decay factor γ .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3b, while the proposed method shows stable convergence in the communication steps under γ = 0 .
Revised: 3b, while the proposed method shows stable convergence in the communication steps under γ = 0 .

Original: 85 and 0 .
Revised: 85 and 0 .

Original: 95 , the REINFORCE baseline exhibits high-variance behavior under certain setup ( γ = 0 .
Revised: 95 , the REINFORCE baseline exhibits high-variance behavior under certain setup ( γ = 0 .

Original: Third, we check if agents’ early terminations are caused by their awareness of the indistinguishable performance in longer and shorter episodes.
Revised: Third, we check if agents’ early terminations are caused by their awareness of the indistinguishable performance in longer and shorter episodes.

Original: Given a precondition that the longer the episodes are, the earlier the success rate increases, it should be the increase in the average performance of shorter episodes that causes the average timesteps to decrease.
Revised: Given a precondition that the longer the episodes are, the earlier the success rate increases, it should be the increase in the average performance of shorter episodes that causes the average timesteps to decrease.

Original: Taking 1-step and 3-step communication for example, in the complete setting, we shall see the success rate of the 3-step to achieve high earlier, which is then caught up but not exceeded by the 1-step.
Revised: Taking 1-step and 3-step communication for example, in the complete setting, we shall see the success rate of the 3-step to achieve high earlier, which is then caught up but not exceeded by the 1-step.

Original: The not exceeding condition is a crucial cue to validate that the communicating partners were actively pursuing the Pareto front (Kim and De Weck, 2005) / efficiency bound (Zaslavsky et al., 2018) of accuracy and complexity.
Revised: The not exceeding condition is a crucial cue to validate that the communicating partners were actively pursuing the Pareto front (Kim and De Weck, 2005) / efficiency bound (Zaslavsky et al., 2018) of accuracy and complexity.

Original: This is exactly what our proposed method emerges as shown in
Revised: This is exactly what our proposed method emerges as shown in

Original: Fig.
Revised: Fig.

Original: 3c.
Revised: 3c.

Original: In contrast, in the REINFORCE baseline, under the same decay factor, the performance of 1-step surpasses 3-step communication.
Revised: In contrast, in the REINFORCE baseline, under the same decay factor, the performance of 1-step surpasses 3-step communication.

Original: It seems as if the incapability of learning long episodes caused the agents to avoid them.
Revised: It seems as if the incapability of learning long episodes caused the agents to avoid them.

Original: Together, all our results on success rate and communication steps are consistent with predictions based on our hypotheses, which justifies our environments and models.
Revised: Together, all our results on success rate and communication steps are consistent with predictions based on our hypotheses, which justifies our environments and models.

Original: However, a high success rate does not necessarily imply high convention quality.
Revised: However, a high success rate does not necessarily imply high convention quality.

Original: Another essential property for conventions is stability (Lewis, 1969): There should exist common patterns in the repeated usage of conventions to convey similar concepts.
Revised: Another essential property for conventions is stability (Lewis, 1969): There should exist common patterns in the repeated usage of conventions to convey similar concepts.

Original: We take the viewpoint of representation learning and concretize the vague stability with three formally defined properties: iconicity , symbolicity , and semanticity .
Revised: We take the viewpoint of representation learning and concretize the vague stability with three formally defined properties: iconicity , symbolicity , and semanticity .

Original: Below, we introduce our experiments to measure these properties systematically.
Revised: Below, we introduce our experiments to measure these properties systematically.

Original: 5.2.2 Iconicity: generalizing to unseen images
Revised: 5.2.2 Iconicity: generalizing to unseen images

Original: We define iconicity as the drawings exhibiting a high visual resemblance with the corresponding images, such that they are proximal to the latter when measured on the high-level embedding of a general-purpose visual system.
Revised: We define iconicity as the drawings exhibiting a high visual resemblance with the corresponding images, such that they are proximal to the latter when measured on the high-level embedding of a general-purpose visual system.

Original: To quantitatively measure the visual similarities, we need to compute the distance between the sketch and image in the visual embedding space ( e.g ., with cosine similarity).
Revised: To quantitatively measure the visual similarities, we need to compute the distance between the sketch and image in the visual embedding space ( e.g ., with cosine similarity).

Original: However, since we do not know how the embeddings distribute in their space, this unnormalized measure is prone to model-specific biases.
Revised: However, since we do not know how the embeddings distribute in their space, this unnormalized measure is prone to model-specific biases.

Original: Fortunately, the receiver’s policy takes the form of a softmax of the cosine similarity between the embedding of the sketch and a context set with a target image and some randomly sampled distractors, which naturally approximates the normalization we want.
Revised: Fortunately, the receiver’s policy takes the form of a softmax of the cosine similarity between the embedding of the sketch and a context set with a target image and some randomly sampled distractors, which naturally approximates the normalization we want.

Original: Therefore, communication accuracy can serve as a visual similarity measure, and its generalizability to unseen images can pinpoint the emergent preservation of iconicity .
Revised: Therefore, communication accuracy can serve as a visual similarity measure, and its generalizability to unseen images can pinpoint the emergent preservation of iconicity .

Original: Intuitively, in open-domain communication, agents would see novel scenes with known or unknown concepts— unseen instances of seen classes and unseen classes, respectively.
Revised: Intuitively, in open-domain communication, agents would see novel scenes with known or unknown concepts— unseen instances of seen classes and unseen classes, respectively.

Original: They should still be able to communicate with established conventions or with unconventionalized iconic drawings.
Revised: They should still be able to communicate with established conventions or with unconventionalized iconic drawings.

Original: A successful generalization to unseen classes ( i.e ., zero-shot generalization) is more difficult than unseen instances; partners may increase the conversation length and communicate with more complex sketches.
Revised: A successful generalization to unseen classes ( i.e ., zero-shot generalization) is more difficult than unseen instances; partners may increase the conversation length and communicate with more complex sketches.

Original: Tab. 1 reports the success rates and average timesteps in our generalization tests.
Revised: Tab. 1 reports the success rates and average timesteps in our generalization tests.

Original: The retrieve setting is the baseline since there is no evolution at all and the sketches should resemble the original images the most ( i.e ., possessing the highest iconicity ).
Revised: The retrieve setting is the baseline since there is no evolution at all and the sketches should resemble the original images the most ( i.e ., possessing the highest iconicity ).

Original: Unsurprisingly, its generalization performance upperbounds all other settings.
Revised: Unsurprisingly, its generalization performance upperbounds all other settings.

Original: Among the experimental and controlled settings, the complete , the max-step , and the sender-fixed agents generalize relatively well in unseen instances ( 70 .
Revised: Among the experimental and controlled settings, the complete , the max-step , and the sender-fixed agents generalize relatively well in unseen instances ( 70 .

Original: 37 ± 0 .
Revised: 37 ± 0 .

Original: 93 ± 0 .
Revised: 93 ± 0 .

Original: 02 , 71 . 80 ± 0 . 02 )
Revised: 02 , 71 . 80 ± 0 . 02 )

Original: and generalize above chance ( 39 .
Revised: and generalize above chance ( 39 .

Original: 40 ± 0 . 40 ± 0 .
Revised: 40 ± 0 . 40 ± 0 .

Original: 40 ± 0 .
Revised: 40 ± 0 .

Original: 02 > 25.00) in unseen classes.
Revised: 02 > 25.00) in unseen classes.

Original: Interestingly, complete and sender-fixed communicating partners intelligently turn to longer episodes for better generalization, better than the max-step agents.
Revised: Interestingly, complete and sender-fixed communicating partners intelligently turn to longer episodes for better generalization, better than the max-step agents.

Original: This finding implies the partners may turn to more iconic communication when there is no established conventions/symbols, just as we humans do.
Revised: This finding implies the partners may turn to more iconic communication when there is no established conventions/symbols, just as we humans do.

Original: Strikingly, the max-step conventions seem to loose more iconicity , possibly due to confusion on marginal information gains.
Revised: Strikingly, the max-step conventions seem to loose more iconicity , possibly due to confusion on marginal information gains.

Original: The one-step drawings seem to lack iconicity .
Revised: The one-step drawings seem to lack iconicity .

Original: 5.2.3 Symbolicity: separating evolved sketches
Revised: 5.2.3 Symbolicity: separating evolved sketches

Original: Next, we measure symbolicity to evaluate the graphical conventionalization.
Revised: Next, we measure symbolicity to evaluate the graphical conventionalization.

Original: We define symbolicity as the drawings being consistently separable in the high-level visual embedding, which facilitates new communication partners to easily distinguish between categories without grounding them to referents.
Revised: We define symbolicity as the drawings being consistently separable in the high-level visual embedding, which facilitates new communication partners to easily distinguish between categories without grounding them to referents.

Original: Based on this definition, a more symbolic drawing should be more easily separable into their corresponding categories.
Revised: Based on this definition, a more symbolic drawing should be more easily separable into their corresponding categories.

Original: To measure such separability , we use a pre-trained VGG16 as the new learner and finetune the last fully connected layer to classify evolved sketches into the 30 categories.
Revised: To measure such separability , we use a pre-trained VGG16 as the new learner and finetune the last fully connected layer to classify evolved sketches into the 30 categories.

Original: Technically, we first get the 300 final canvases from the communication game, 10 for each category.
Revised: Technically, we first get the 300 final canvases from the communication game, 10 for each category.

Original: Among them, we use 70% for training and 30% for testing.
Revised: Among them, we use 70% for training and 30% for testing.

Original: The bar plot in Fig.
Revised: The bar plot in Fig.

Original: 4 shows the classification results.
Revised: 4 shows the classification results.

Original: Since agents in the one-step setting do not play the game successfully, they may not form a consistent way to communicate.
Revised: Since agents in the one-step setting do not play the game successfully, they may not form a consistent way to communicate.

Original: Agents in the complete setting achieve the highest accuracy, higher even compared with the result of the original images.
Revised: Agents in the complete setting achieve the highest accuracy, higher even compared with the result of the original images.

Original: This finding indicates that agents in the complete setting agree on a graphical convention that consistently highlights some features across all training instances in each category, which are also distinguishable between categories.
Revised: This finding indicates that agents in the complete setting agree on a graphical convention that consistently highlights some features across all training instances in each category, which are also distinguishable between categories.

Original: Comparing the max-step with the complete setting, we know that early decision is a crucial factor for more symbolic conventions.
Revised: Comparing the max-step with the complete setting, we know that early decision is a crucial factor for more symbolic conventions.

Original: Comparison between the sender-fixed setting and the complete setting suggests that the sender’s evolution also contributes to high symbolicity .
Revised: Comparison between the sender-fixed setting and the complete setting suggests that the sender’s evolution also contributes to high symbolicity .

Original: [Figure]
Revised: [Table]

Original: 5.2.4 Semanticity: correlating category embedding
Revised: 5.2.4 Semanticity: correlating category embedding

Original: The last desired property is that the evolved sketches can preserve the perceptual metric in images (Zhang et al., 2018).
Revised: The last desired property is that the evolved sketches can preserve the perceptual metric in images (Zhang et al., 2018).

Original: We define semanticity as the topography of the high-level visual embedding space of drawings being strongly correlated to that of images, such that semantically similar instances and categories lie close to each other in the embedding space.
Revised: We define semanticity as the topography of the high-level visual embedding space of drawings being strongly correlated to that of images, such that semantically similar instances and categories lie close to each other in the embedding space.

Original: To examine such topographic correlation , we project category names to the feature space using word2vec (Mikolov et al., 2013) as featureA, and project the evolved sketches to the feature space using the trained VGG in Sec.
Revised: To examine such topographic correlation , we project category names to the feature space using word2vec (Mikolov et al., 2013) as featureA, and project the evolved sketches to the feature space using the trained VGG in Sec.

Original: 5.2.3 as featureB.
Revised: 5.2.3 as featureB.

Original: We compute the correlation of the distances between all the possible pairs of featureB and the corresponding pairs of featureA as the semanticity measure.
Revised: We compute the correlation of the distances between all the possible pairs of featureB and the corresponding pairs of featureA as the semanticity measure.

Original: The results in Tab. 2 show that semanticity can be better retained in the complete setting compared with the retrieve baseline.
Revised: The results in Tab. 2 show that semanticity can be better retained in the complete setting compared with the retrieve baseline.

Original: We further visualize the embeddings of the sketches in a 2D space via t-SNE (Van der Maaten and Hinton, 2008).
Revised: We further visualize the embeddings of the sketches in a 2D space via t-SNE (Van der Maaten and Hinton, 2008).

Original: Fig.show the visualization of the original images and the sketches in the retrieve and complete settings; please refer to Appendix B for results of other settings.
Revised: Fig.show the visualization of the original images and the sketches in the retrieve and complete settings; please refer to Appendix B for results of other settings.

Original: Images/drawings from the same category are marked in the same color.
Revised: Images/drawings from the same category are marked in the same color.

Original: As shown, boundaries between categories are clearer in the evolved drawings in the complete setting than in retrieve or original images; but semantically similar categories are still close to each other.
Revised: As shown, boundaries between categories are clearer in the evolved drawings in the complete setting than in retrieve or original images; but semantically similar categories are still close to each other.

Original: For example, cow, deer, horse, and camel are proximal to each other, while burger and apple are far from them.
Revised: For example, cow, deer, horse, and camel are proximal to each other, while burger and apple are far from them.

Original: These results highlight another uniqueness of visual communication over its symbolic counterpart: The similarity in the visual cues in the conventions may hint the semantic correlation between the referents.
Revised: These results highlight another uniqueness of visual communication over its symbolic counterpart: The similarity in the visual cues in the conventions may hint the semantic correlation between the referents.

Original: 5.2.5 Visualizing sketch evolution
Revised: 5.2.5 Visualizing sketch evolution

Original: To better understand the nature of the emerged conventions, we inspect the intermediate sketches in the evolution processes.
Revised: To better understand the nature of the emerged conventions, we inspect the intermediate sketches in the evolution processes.

Original: Specifically, we visualize the process under the complete setting.
Revised: Specifically, we visualize the process under the complete setting.

Original: Fig.shows three instances in two categories.
Revised: Fig.shows three instances in two categories.

Original: For each example, drawings from the left to the right show the change of the final-step canvas.
Revised: For each example, drawings from the left to the right show the change of the final-step canvas.

Original: Sketches’ complexity gradually decreases after an initial increase, echoing the trend of reduction described in Sec.
Revised: Sketches’ complexity gradually decreases after an initial increase, echoing the trend of reduction described in Sec.

Original: For rabbits, in the beginning, the strokes may depict instances from different perspectives; through iterations, they converge to highlight the rabbit’s long ear.
Revised: For rabbits, in the beginning, the strokes may depict instances from different perspectives; through iterations, they converge to highlight the rabbit’s long ear.

Original: As for the giraffe, the agents gradually learn to emphasize the long neck.
Revised: As for the giraffe, the agents gradually learn to emphasize the long neck.

Original: In the third example, although the giraffe lowers its head, we can still see an exaggerated vertical stroke for the neck, similar to the first example where the giraffe’s head is raised.
Revised: In the third example, although the giraffe lowers its head, we can still see an exaggerated vertical stroke for the neck, similar to the first example where the giraffe’s head is raised.

Original: These examples show how the sender gradually unifies drawings of the same category: After evolution, the sender is inclined to use the first several strokes to depict the most salient parts of visual concepts.
Revised: These examples show how the sender gradually unifies drawings of the same category: After evolution, the sender is inclined to use the first several strokes to depict the most salient parts of visual concepts.

Original:  Conclusion
Revised:  Conclusion

Original: We present the first step of modeling and simulating the evolution of graphical conventions between two agents in a visual communication game.
Revised: We present the first step of modeling and simulating the evolution of graphical conventions between two agents in a visual communication game.

Original: Agents modeled in our framework can successfully communicate visual concepts using sketches as the medium.
Revised: Agents modeled in our framework can successfully communicate visual concepts using sketches as the medium.

Original: We measure the emergent graphical conventions over three carefully defined properties, iconicity , symbolicty , and semanticity .
Revised: We measure the emergent graphical conventions over three carefully defined properties, iconicity , symbolicty , and semanticity .

Original: The experimental results under different controls suggest that early decision, mutual adaptation, and sequential decision-making can encourage symbolicity while preserving iconicity and semanticity .
Revised: The experimental results under different controls suggest that early decision, mutual adaptation, and sequential decision-making can encourage symbolicity while preserving iconicity and semanticity .

Original: However, two-stage pre-trained senders possess limitations; an ideal sender would not need to convert the images to pixel-level sketches before it starts sketching.
Revised: However, two-stage pre-trained senders possess limitations; an ideal sender would not need to convert the images to pixel-level sketches before it starts sketching.

Original: The limitations in the pre-trained sketching module also constrain the discriminative need among the selected classes.
Revised: The limitations in the pre-trained sketching module also constrain the discriminative need among the selected classes.

Original: We hope to investigate these limitations in future research.
Revised: We hope to investigate these limitations in future research.

Original: With the uniqueness of visual conventions demonstrated, we hope our work can invoke the study of emergent communication in the modality of sketches.
Revised: With the uniqueness of visual conventions demonstrated, we hope our work can invoke the study of emergent communication in the modality of sketches.

Original: Acknowledgments J. Joo was supported in part by the NSF SMA/SBE #1831848.
Revised: Acknowledgments J. Joo was supported in part by the NSF SMA/SBE #1831848.

Original: We would like to thank Prof. Marco Baroni (Pompeu Fabra Univ.)
Revised: We would like to thank Prof. Marco Baroni (Pompeu Fabra Univ.)

Original: and Prof. Yaodong Yang (Peking Univ.)
Revised: and Prof. Yaodong Yang (Peking Univ.)

Original: for helpful comments on the draft, Miss Chen Zhen (BIGAI) for making the nice figures, all anonymous reviewers for their constructive comments on prior submissions, and Prof. Ying Nian Wu (UCLA),
Revised: for helpful comments on the draft, Miss Chen Zhen (BIGAI) for making the nice figures, all anonymous reviewers for their constructive comments on prior submissions, and Prof. Ying Nian Wu (UCLA),

Original: Prof. Hongjing Lu (UCLA), Prof. Federico Rossano (UCSD), Dr. Zilong Zheng (BIGAI),
Revised: Prof. Hongjing Lu (UCLA), Prof. Federico Rossano (UCSD), Dr. Zilong Zheng (BIGAI),

Original: Dr. Chi
Revised: Dr. Chi

Original: Zhang (BIGAI), Baoxiong Jia (UCLA), Dr. Qing Li (BIGAI), and Dr. Bo Pang (UCLA) for discussion on the algorithm and experimental designs.B Category embedding for other game settings Fig.
Revised: Zhang (BIGAI), Baoxiong Jia (UCLA), Dr. Qing Li (BIGAI), and Dr. Bo Pang (UCLA) for discussion on the algorithm and experimental designs.

Original: A1 shows the t-SNE visualization for other game settings.

Original: Agents under max-step , sender-fixed , and one-step settings fail to form clear boundaries between different categories, which makes it hard to observe semantic relations.

Original: C Learning objectives and training algorithm

Original: Agents are trained jointly to maximize the objective:

Original: [Equation]

Original: (A1) where τ = { C 0 , a S 0 , C 1 , a R 1 , a S 1 , ... } is the simulated episodic trajectory.

Original: To further expand the objective,

Original: [Equation]

Original: We calculate E I S ,I 1 R ,...,I MR ,C 0 [ · ] by sampling I S , I R , and initializing C 0 to blank at each round.

Original: We represent the E ( π S ,π R ) [ · ] as V ( X 0 ) and use V λ ( X 1 ) to estimate the reward expectation E (

Original: [Equation]

Original: S ,

Original: [Equation]

Original: [Equation]

Original: V

Original: [Equation]

Original: where X t =

Original: [ I S , I 1 R , ..., I MR , C t , C t +1 ] , t = 0 , 1 ... , δ ( · ) is the Dirac delta function that returns 1 when the action is wait and 0 otherwise.

Original: The sender policy is parametrized as a Gaussian distribution,

Original: [Equation]

Original: such that a S 0 can be written as

Original: [Equation]

Original: Therefore, we can expand V ( X 0 ) as,

Original: [Equation]

Original: [Equation]

Original: E ϵ [ · ] is approximated with a point estimate.

Original: Since π R is a categorical distribution, we expand E π R as

Original: [Equation]

Original: V λ ( X t ) in Eq.

Original: (A3) is an eligibility trace approximation of the ground-truth value function (Sutton and Barto, 2018).

Original: Considering the early termination in our setting, we set the time step when the receiver makes the prediction as T choice .

Original: When t is the time step less or equal than T choice , V λ mixes

Original: Monte Carlo estimate at different roll-out lengths.

Original: Otherwise, we only have an estimated value v ϕ ( X t ) .

Original: [Equation]

Original: otherwise

Original: [Equation]

Original: where H =

Original: T choice − t + 1 , and V kN ( X t ) is the Monte Carlo estimate at k roll-out lengths.

Original: V kN ( X t ) =

Original: E π S ,π R [ (cid:80)

Original: [Equation]

Original: n =

Original: [Equation]

Original: n − t r n + γ h − t δ ( a Rh ) υ ϕ ( X h )] , with h = min( t + k, T choice ) being the maximal timestep.

Original: Due to the error reduction property Sutton and Barto (2018), the eligibility trace estimation

Original: V λ ( · ) is less biased than v ϕ ( · ) .

Original: When regressing v ϕ ( X t ) towards the bootstrapped V λ ( X t ) ,

Original: [Equation]

Original: v ϕ

Original: ( X t ) will be improved towards the fixed point.

Original: D Visualizing sketch evolution

Original: Visualizing the evolution process helps us understand what the agents have learned through communication regarding different categories.

Original: By comparing the evolved sketches with the intermediate results, we can know (i) how the agents abstract the sketch, (ii) which parts of the visual concept they highlight, and (iii) which parts are de-emphasized.

Original: Fig.

Original: A2 to A4 show some evolution examples under different settings.

Original: Agents under max-step seem to abstract their drawings by repeatedly placing new strokes near old strokes, resulting in bold drawings.

Original: The number of strokes under sender-fixed gradually decreases, but the way of drawing will not change.

Original: Senders under one-step change more wildly, but cannot form a consistent drawing behavior.

Original: Overall, compared with the complete setting, agents under the control settings do not form patterns to draw sketches, which echoes their relatively low classification results.

Original: [Figure]

Original: [Figure]

Original: [Figure]