Original: Emergent Graphical Conventions in a Visual Communication Game Anonymous Author(s) AffiliationAddress email
Revised: Emergent Graphical Conventions in a Visual Communication Game https://sites.google.com/view/emergent-graphical-conventions

Original: Abstract
Revised: Abstract

Original: Humans communicate with graphical sketches apart from symbolic languages [8].
Revised: Humans communicate with graphical sketches apart from symbolic languages (Fay et al., 2014).

Original: Primarily focusing on the latter, recent studies of emergent communication [24]overlook the sketches; they do not account for the evolution process through whichsymbolic sign systems emerge in the trade-off between iconicity and symbolicity.
Revised: Primarily focusing on the latter, recent studies of emergent communication (Lazaridou and Baroni, 2020) overlook the sketches; they do not account for the evolution process through which symbolic sign systems emerge in the trade-off between iconicity and symbolicity.

Original: In this work, we take the very first step to model and simulate this process via twoneural agents playing a visual communication game; the sender communicates withthe receiver by sketching on a canvas.
Revised: In this work, we take the very first step to model and simulate this process via two neural agents playing a visual communication game; the sender communicates with the receiver by sketching on a canvas.

Original: We devise a novel reinforcement learningmethod such that agents are evolved jointly towards successful communication andabstract graphical conventions.
Revised: We devise a novel reinforcement learning method such that agents are evolved jointly towards successful communication and abstract graphical conventions.

Original: To inspect the emerged conventions, we define threekey properties – iconicity, symbolicity, and semanticity – and design evaluationmethods accordingly.
Revised: To inspect the emerged conventions, we define three fundamental properties—iconicity, symbolicity, and semanticity—and design evaluation methods accordingly.

Original: Our experimental results under different controls are consistent with the observation in studies of human graphical conventions [17, 9].
Revised: Our experimental results under different controls are consistent with the observation in studies of human graphical conventions (Hawkins et al., 2019; Fay et al., 2010).

Original: Ofnote, we find that evolved sketches can preserve the continuum of semantics [30]under proper environmental pressures.
Revised: Of note, we find that evolved sketches can preserve the continuum of semantics (Mikolov et al., 2013) under proper environmental pressures.

Original: More interestingly, co-evolved agents canswitch between conventionalized and iconic communication based on their familiarity with referents.
Revised: More interestingly, co-evolved agents can switch between conventionalized and iconic communication based on their familiarity with referents.

Original: We hope the present research can pave the path for studyingemergent communication with the modality of sketches.
Revised: We hope the present research can pave the path for studying emergent communication with the modality of sketches.

Original:  Introduction
Revised:  Introduction

Original: Communication problem naturally arises when traveling in a foreign country where you do not speakthe native language, which necessitates exploring non-linguistic means of communication, suchas drawings.
Revised: Communication problem naturally arises when traveling in a foreign country where you do not speak the native language, which necessitates exploring non-linguistic means of communication, such as drawings.

Original: Due to its iconic nature ( i e ., perceptual resemblance to or natural association withthe referent), drawings serve as a powerful tool to communicate concepts transcending languagebarriers [8].
Revised: Due to its iconic nature ( i e ., perceptual resemblance to or natural association with the referent), drawings serve as a powerful tool to communicate concepts transcending language barriers (Fay et al., 2014).

Original: In fact, we humans started to use drawings to convey messages dating back to 40,000–60,000 years ago [18, 17].
Revised: In fact, we humans started to use drawings to convey messages dating back to 40,000–60,000 years ago (Hoffmann et al., 2018;

Original:  Somestudies from cognitive science  hypothesize a transition fromsketch-based communication before the formation of sign systems and provide evidence that iconicsigns can gradually become symbolic through repeated communication [8].
Revised: Hawkins et al., 2019). Some cognitive science studies hypothesize a transition from sketch-based communication before the formation of sign systems and provide evidence that iconic signs can gradually become symbolic through repeated communication (Fay et al., 2014).

Original: In contrast to icons ,symbols are special forms bearing arbitrary relations to the referents.
Revised: In contrast to icons , symbols are special forms bearing arbitrary relations to the referents.

Original: Fig.
Revised: Fig.

Original: 1 describes a typicalscenario of such phenomena: Alice (in green) uses a sketch to communicate the concept “rooster” to Bob (in yellow).
Revised: 1 describes a typical scenario of such phenomena: Alice (in green) uses a sketch to communicate the concept “rooster” to Bob (in yellow).

Original: Initially, they need to ground the sketch to the referent.
Revised: Initially, they need to ground the sketch to the referent.

Original: Later, details of the visualconcept, such as strokes of  head and body, are gradually abstracted away, leaving only the mostsalient part, the crown.
Revised: Later, details of the visual concept, such as strokes of the head and body, are gradually abstracted away, leaving only the most salient part, the crown.

Original: The iconicity in the communicated sketch drops while the symbolicity rises.
Revised: The iconicity in the communicated sketch drops while the symbolicity rises.

Original:  While models of emerging communication protocols has attracted attention [26, 3, 6, 16, 25, 24, 31,33 , 4, 12], the initial and primary communication medium is presumed and limited to be symbolicrather than iconic.
Revised: [Figure] While models of emerging communication protocols has attracted attention (Lazaridou et al., 2017; Cao et al., 2018; Evtimova et al., 2018; Havrylov and Titov, 2017; Lazaridou et al., 2018; Lazaridou and Baroni, 2020; Mordatch and Abbeel, 2018; Ren et al., 2020; Eccles et al., 2019; Graesser et al., 2019), the initial and primary communication medium is presumed and limited to be symbolic rather than iconic.

Original: By simulating a multi-agent referential game , prior work seeks for environmentaldriving forces behind the emergence of effective communications.
Revised: By simulating a multi-agent referential game , prior work seeks for the environmental driving forces behind the emergence of effective communications.

Original: In a typical setup of referentialgames, two agents play similar roles as in the above Alice-Bob example but share a primitive setof arbitrary tokens ( i.e ., the vocabulary).
Revised: In a typical setup of referential games, two agents play similar roles as in the above Alice-Bob example but share a primitive set of arbitrary tokens ( i.e ., the vocabulary).

Original: Using these tokens, an agent (the sender) attempts tocommunicate a message to another agent (the receiver).
Revised: Using these tokens, an agent (the sender) attempts to communicate a message to another agent (the receiver).

Original: A communication convention is emerged [Figure] when two agents successfully communicate by associating visual concepts in the images with tokensin the pre-selected vocabulary.
Revised: A communication convention emerges when two agents successfully communicate by associating visual concepts in the images with tokens in the pre-selected vocabulary.

Original: Though this line of work has probed into some linguistic propertiesof the established communication conventions [26, 25, 33], some intriguing questions remain open: How do agents make a trade-off between iconicity and symbolicity to emerge symbolic sign systems?
Revised: Though this line of work has probed into some linguistic properties of the established communication conventions (Lazaridou et al., 2017; 2018; Ren et al., 2020), some intriguing questions remain open: How do agents make a trade-off between iconicity and symbolicity to emerge symbolic sign systems?

Original: In this work, we present the very first step of modeling and simulating the evolution process ofgraphical conventions [17], a two-participant communication convention whose medium is drawingsin an abstract form.
Revised: In this work, we present the very first step of modeling and simulating the evolution process of graphical conventions (Hawkins et al., 2019), a two-participant communication convention whose medium is drawings in an abstract form.

Original: Specifically, we consider our contributions in three folds:
Revised: Specifically, we consider our contributions in three folds:

Original: First, we model a multi-agent visual communication game and propose a learning framework,wherein the sender and the receiver evolve jointly.
Revised: First, we model a multi-agent visual communication game and propose a learning framework, wherein the sender and the receiver evolve jointly.

Original: This visual communication game is an alternatingsequential decision-making process, in which the sender generates a sequence of strokes step by step,terminated by the receiver.
Revised: This visual communication game is an alternating sequential decision-making process, in which the sender generates a sequence of strokes step by step, terminated by the receiver.

Original: In contrast to discretized tokens in prior work, strokes can be continuouslyparametrized [13, 19] such that the derivatives of learning objectives can be more effectively backpropagated through communication channels [11].
Revised: In contrast to discretized tokens in prior work, strokes can be continuously parametrized (Ha and Eck, 2018; Huang et al., 2019) such that the derivatives of learning objectives can be more effectively back-propagated through communication channels (Foerster et al., 2016).

Original: We further derive a novel training surrogate formulti-agent reinforcement learning based on a joint value function and the eligibility trace method [39].
Revised: We further derive a novel training surrogate for multi-agent reinforcement learning based on a joint value function and the eligibility trace method (Sutton and Barto, 2018).

Original: This differs from the REINFORCE-based surrogate in prior works on early-terminable sequentialcommunication [3, 6] and the TD learning method in the literature of multi-agent turn-taking gameswhere each agent has its own value function and has to model the value of others [43].
Revised: This differs from the REINFORCE-based surrogate in prior works on early-terminable sequential communication (Cao et al., 2018; Evtimova et al., 2018) and the TD learning method in the literature of multi-agent turn-taking games, wherein each agent has its own value function and has to model the value of others (Wen et al., 2019).

Original: In experiments,we empirically demonstrate that our integration of function approximation and Monte Carlo samplingfacilitates the agents to be aware of the correlation between complex and simple sketches, herebyenabling a smooth abstraction process.
Revised: In experiments, we empirically demonstrate that our integration of function approximation and Monte Carlo sampling facilitates the agents’ awareness of the correlation between complex and simple sketches, thereby enabling a smooth abstraction process.

Original: Second, we define essential properties in studying evolved sketches.
Revised: Second, we define essential properties in studying evolved sketches.

Original: Specifically, we define iconicity [8] as the drawings exhibiting high visual resemblance to the corresponding images, such that theyare proximal to the latter when measured on the high-level embedding of a general-purpose visualsystem; we define symbolicity [10] as these drawings being consistently separable in the high-levelvisual embedding, which facilitates new communication participants to easily distinguish betweencategories without grounding them to referents;  we define semanticity [15] as the topography of thehigh-level embedding space of the drawings being strongly correlated to that of images, such thatsemantically similar instances and categories lie close to each other in the embedding space.
Revised: Specifically, we define iconicity (Fay et al., 2014) as the drawings exhibiting high visual resemblance to the corresponding images, such that they are proximal to the latter when measured on the high-level embedding of a generalpurpose visual system; we define symbolicity (Fay et al., 2018) as these drawings being consistently separable in the high-level visual embedding, which facilitates new communication participants to easily distinguish between categories without grounding them to referents; and we define semanticity (Harispe et al., 2015) as the topography of the high-level embedding space of the drawings being strongly correlated to that of images, such that semantically similar instances and categories lie close to each other in the embedding space.

Original: Of note,this is not the only way to define these cognitive concepts; our intention is to align readers on criticalconcepts in our work.
Revised: Of note, this is not the only way to define these cognitive concepts; we intend to align readers on critical concepts in our work.

Original: Third, we present a suite of quantitative and qualitative methods to evaluate the emergent graphicalconventions based on the  carefully defined iconicity , symbolicity , and semanticity .
Revised: Third, we present a suite of quantitative and qualitative methods to evaluate the emergent graphical conventions based on the above carefully defined iconicity , symbolicity , and semanticity .

Original: This is necessarybecause a high communication rate does not imply good representations [2].
Revised: This is necessary because a high communication rate does not imply good representations (Bouchacourt and Baroni, 2018).

Original: The graphical natureof the communication medium mandates us to repurpose representation learning metrics ratherthan adopt linguistic metrics in emergent symbolic communication.
Revised: The graphical nature of the communication medium mandates us to repurpose representation learning metrics rather than adopt linguistic metrics in emergent symbolic communication.

Original: We evaluate the contributionof different environmental drivers, early decision, sender’s update, and sequential communication,to the three properties of the emergent conventions.
Revised: We evaluate the contribution of different environmental drivers, early decision, sender’s update, and sequential communication, to the three properties of the emergent conventions.

Original: Critically, the empirical results assessed on ourmetrics well align  with our prediction based on the findings of human graphical conventions [9, 17],justifying our environment, model, and evaluation.
Revised: Critically, the empirical results assessed on our metrics align well with our prediction based on the findings of human graphical conventions (Fay et al., 2010; Hawkins et al., 2019), justifying our environment, model, and evaluation.

Original: One of these setups emerges conventions wherethe three properties are consistent with our expectation of a sign system.
Revised: One of these setups emerges conventions where the three properties are consistent with our expectation of a sign system.

Original: Particularly, we find twoinspiring phenomena: (i) Evolved sketches from semantically similar classes are perceptually moresimilar to each other than those falling into different superclasses.
Revised: Particularly, we find two inspiring phenomena: (i) Evolved sketches from semantically similar classes are perceptually more similar to each other than those falling into different superclasses.

Original: (ii) To communicate concepts notincluded in their established conventions, evolved agents can return to more iconic communicationsas humans do.
Revised: (ii) To communicate concepts not included in their established conventions, evolved agents can return to more iconic communications than humans.

Original: We hope our work sheds light on emergent communication in the modality of sketchesand facilitate the study of cognitive evolutionary theories of pictographic sign systems.
Revised: We hope our work sheds light on emergent communication in the modality of sketches and facilitates the study of cognitive evolutionary theories of pictographic sign systems.

Original:  Related work
Revised:  Related work

Original: Learning to sketch Ha and Eck [13] begin the endeavor of teaching modern neural models tosketch stroke by stroke.
Revised: Learning to sketch Ha and Eck (2018) begin the endeavor of teaching modern neural models to sketch stroke by stroke.

Original: However, generating meaningful stroke sequences directly from variouscategories of natural images is still in the early phase [38, 42, 49].
Revised: However, generating meaningful stroke sequences directly from various categories of natural images is still in the early phase (Song et al., 2018;

Original:  To assure the interestingness of thecategory-level sketch communication, we design a stage-wise agent thatfirst transfers a natural imageinto a pixel-level sketch through a CNN-based model [20] andthen draws the sketch on the canvasstroke by stroke with a policy [19].
Revised: Wang et al., 2021; Zou et al., 2018). To assure the interestingness of the category-level sketch communication, we design a stagewise agent that transfers a natural image into a pixel-level sketch (Kampelmuhler and Pinz, 2020) and draws the sketch on the canvas stroke by stroke with a policy (Huang et al., 2019).

Original: To pretrain our neural agents to sketch, we select Sketchy [34]from many datasets [46, 13, 5, 34] for its fine-grained photo-sketch correspondence, rich stroke-levelannotations, and well-organized categorization structure.
Revised: To pre-train our neural agents to sketch, we select Sketchy (Sangkloy et al., 2016) from many datasets (Yu et al., 2016; Ha and Eck, 2018; Eitz et al., 2012; Sangkloy et al., 2016) for its fine-grained photo-sketch correspondence, rich stroke-level annotations, and well-organized categorization structures.

Original: Communication games While learning to sketch is formulated as a single-agent task with explicitsupervision, our focus is on how sketches would evolve when utilized as the communication mediumbetween two cooperative agents.
Revised: Communication games While learning to sketch is formulated as a single-agent task with explicit supervision, our focus is on how sketches would evolve when utilized as the communication medium between two cooperative agents .

Original: Their cooperation is always formulated as a communication game,recently adopted to study phenomena in  natural languages, such as symbolic language acquisition [12]and the emergence of compositionality [33].
Revised: Their cooperation is always formulated as a communication game, recently adopted to study phenomena in human-robot teaming (Yuan et al., 2022) and natural languages, such as symbolic language acquisition (Graesser et al., 2019) and the emergence of compositionality (Ren et al., 2020).

Original: Some notable works [26, 25, 6, 16] have devisedinteresting metrics, such as purity [26] and topographic similarity [25].
Revised: Some notable works (Lazaridou et al., 2017; 2018; Evtimova et al., 2018; Havrylov and Titov, 2017) have devised interesting metrics, such as purity (Lazaridou et al., 2017) and topographic similarity (Lazaridou et al., 2018).

Original: In comparison, our work isunique due to the distinctive communication medium, continuously parametrized sketches.
Revised: In comparison, our work is unique due to the distinctive communication medium, continuously parametrized sketches.

Original: Althougha concurrent work [29] also enables the agents to sketch in a communication game, it focusesmainly on drawing interpretable sketches without abstracting  into graphical symbols along thecommunication.
Revised: Although a concurrent work (Mihai and Hare, 2021) also enables the agents to sketch in a communication game, it focuses primarily on drawing interpretable sketches without abstracting them into graphical symbols along the communication.

Original: We position our work as an alternative to emergent symbolic communication, sincethe emergent graphical symbols may better represent the continuum of semantics, as encoded in thevector representation of tokens [30].
Revised: We position our work as an alternative to emergent symbolic communication, since the emergent graphical symbols may better represent the continuum of semantics, as encoded in the vector representation of tokens (Mikolov et al., 2013).

Original: Methodologically, we devise new evaluation metrics for sketchmodality, assessingthe iconicity , symbolicity , and semanticity in the evolved sketches.
Revised: Methodologically, we devise new evaluation metrics for sketch modality, assessing iconicity , symbolicity , and semanticity in the evolved sketches.

Original: Emergent graphical conventions Evolving from sketches to graphical conventions/symbols is anactive field in cognitive science under the banner of “emergent sign systems.”
Revised: Emergent graphical conventions Evolving from sketches to graphical conventions/symbols is an active field in cognitive science under the banner of “emergent sign systems.”

Original: Fay et al .
Revised: Fay et al.

Original: [9] showthat pair interaction can form their local conventions when they play the Pictionary game.
Revised: (2010) show that pair interaction can form their local conventions when they play the Pictionary game.

Original: Usingnatural images instead of texts as the prompt, Hawkins et al .
Revised: Using natural images instead of texts as the prompt, Hawkins et al.

Original: [17] show that, besides partners’ sharedinteraction history, visual properties of the images also influence the formed graphical conventions. That is, the evolved sketches highlight visually salient parts.
Revised: (2019) show that visual properties of the images also influence the formed graphical conventions besides partners’ shared interaction history; i.e ., the evolved sketches highlight visually salient parts.

Original: Nevertheless, only a few computationalmodels exist apart from these human behavior studies.
Revised: Nevertheless, only a few computational models exist apart from these human behavior studies.

Original: Fan et al .
Revised: Fan et al.

Original: [7] describe a model for selectingcomplex or simple sketches considering the communication context.
Revised: (2020) describe a model for selecting complex or simple sketches considering the communication context.

Original: Bhunia et al .
Revised: Bhunia et al.

Original: [1] and Muhammadet al .
Revised: (2020) and Muhammad et al.

Original: [32] consider stroke selection and reordering to simplify the sketch.
Revised: (2018) consider stroke selection and reordering to simplify the sketch.

Original: In contrast to sketch orstroke selection, we model embodied agents who can draw and recognize sketches.
Revised: In contrast to sketch or stroke selection, we model embodied agents who can draw and recognize sketches.

Original:  The visual communication game
Revised:  The visual communication game

Original: Our visual communication game is formulated as a tuple( I , C, A S , A R , G, r, γ ) , where I is the image set to be presented to the sender S and the receiver R .
Revised: Our visual communication game is formulated as a tuple ( I , C, A S , A R , G, r, γ ) , where I is the image set to be presented to the sender S and the receiver R .

Original: These images containa single object in the foreground, and hence the image space I can be partitioned into N classesaccording to the category of the objects.
Revised: These images contain a single object in the foreground, and hence the image space I can be partitioned into N classes [Figure] according to the object categories.

Original: In each round of the game, the sender is presented withone image I S , and the receiver is presented with M images { I 1 R , ..., I MR } .
Revised: In each game round, the sender is presented with one image I S , and the receiver is presented with M images { I 1 R , ..., I M R } .

Original: Prior work shows thatcategory-level context can force the agent to coordinate on more abstract properties and help drawsketches to be more recognizable as the type of object [26, 29], we make the observations of S and Rdisjoint ( i.e .,
Revised: Prior work shows that category-level context can force the agent to coordinate on more abstract properties and help draw sketches to be more recognizable as the type of object (Lazaridou et al., 2017; Mihai and Hare, 2021).

Original:  I S / ∈ { I 1 R , ..., I MR } ), but with a target image I ∗ R in the same class as I S .
Revised: We make the observations of S and R disjoint ( i.e ., I S / ∈ { I 1 R , ..., I MR } ) but with a target image I ∗ R in the same class as I S .

Original: We refer to the M images that the receiver can see as the context .
Revised: We refer to the M images that the receiver can see as the context .

Original: Neither the receiver nor the sender would see theimage(s) presented to their partner; they can only communicate this information by drawing sketcheson the canvas C , observable to both players.
Revised: Neither the receiver or the sender would see the image(s) presented to their partner; they can only communicate this information by drawing sketches on the canvas C , observable to both players.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2,  at the beginning of each round, C 0is initialized to be blank.
Revised: 2, Cis initialized to be blank at the beginning of each round.

Original: Only the sender can draw on the canvas with actions chosen from A S .
Revised: Only the sender can draw on the canvas with actions chosen from A S .

Original: Theaction at each time step consists of 5 strokes, which are continuous vectors in R 6 .
Revised: The action at each time step consists of 5 strokes, which are continuous vectors in R 6 .

Original: We constrain eachdimension to be in (0 , 1) due to  limited space on the canvas.
Revised: We constrain each dimension to be in (0 , 1) due to the limited space of the canvas.

Original: The canvas is updated from C t to C t +1by the renderer G after each step of the sender’s sketching.
Revised: The canvas is updated from C t to C t +1 by the renderer G after each step of the sender’s sketching.

Original: The receiver, after observing the updatedcanvas, wouldhave to choose among the M images or wait for the next step from the sender; these M +1 possible actions constitute A R .
Revised: The receiver, after observing the updated canvas, would choose among the M images or wait for the next step from the sender; these M + 1 possible actions constitute A R .

Original: A game round terminates when the receiver gives up waitingand chooses one from the images.
Revised: A game round terminates when the receiver gives up waiting and chooses one from the images.

Original: After the termination, the sender and the receiver will receive a 
Revised: After the termination, the sender and the receiver will receive a shared reward or penalty, depending on if the receiver makes the right choice:

Original: [Figure]
Revised: [Equation]

Original: shared reward or penalty, depending on if the receiver makes the right choice: r : I ×I → {− 1 , 1 } .

Original: This reward/penalty is temporally decayed with a decay factor γ .
Revised: This reward/penalty is temporally decayed with a decay factor γ .

Original: That is, if the receiver decides tochoose from the images at step t , this cooperating pair will receive either γ t or − γ t .
Revised: That is, if the receiver decides to choose from the images at step t , this cooperating pair will receive either γ t or − γ t .

Original: Hence, eventhough the players do not receive an explicit penalty for long conversations, there is an implicitpenalty/reward for delayed positive/negative answers.
Revised: Hence, even though the players do not receive an explicit penalty for long conversations, there is an implicit penalty/reward for delayed positive/negative answers.

Original: No reward will be assigned if the receiverchooses to wait.
Revised: No reward will be assigned if the receiver chooses to wait.

Original: The next round starts after the reward/penalty assignment.
Revised: The next round starts after the reward/penalty assignment.

Original:  Agents
Revised:  Agents

Original: The two agents involved in the visual communication game are modeled with two decision policies,π S and π R , for the sender and the receiver, respectively.
Revised: The two agents involved in the visual communication game are modeled with two decision policies, π S and π R , for the sender and the receiver, respectively.

Original: These policies are stochastic mapping fromthe agents’ observation space to the action space:
Revised: These policies are stochastic mapping from the agents’ observation space to the action space:

Original: [Equation]
Revised: [Equation]

Original: where P ( A ) is a distribution over the support set A .
Revised: where P ( A ) is a distribution over the support set A .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, at each time step t ∈ { 0 , ...T } ,the sender first emits the stroke parameters for the next five strokes a St ∼ π S ( I S , C t ) .
Revised: 2, at each time step t ∈ { 0 , ...T } , the sender first emits the stroke parameters for the next five strokes a St ∼ π S ( I S , C t ) .

Original: These strokesare applied to the canvas by a differentiable renderer, C t +1 =
Revised: These strokes are applied to the canvas by a differentiable renderer, C t +1 =

Original: G ( C t , a St ) .
Revised: G ( C t , a St ) .

Original: Next, the updated canvas istransmitted to the receiver.
Revised: Next, the updated canvas is transmitted to the receiver.

Original: The receiver decides whether to terminate the game and make its prediction( i.e ., a Rt ∈ { 1 , ..., M } ) or wait for another round ( i.e ., a Rt = M +1 ); its decision is sampled fromπ R .
Revised: The receiver decides whether to terminate the game by making its prediction ( i.e ., a Rt ∈ { 1 , ..., M } ) or wait for another round ( i.e ., a Rt = M + 1 ); its decision is sampled from π R .

Original: If a prediction is made, it is used to select the image I a R R from { I 1 R , ...
Revised: If a prediction is made, it is used to select the image I a R R from { I 1 R , ...

Original: I MR } and score this gamewith r ( I S , I a R R ) .
Revised: I MR } and score this game with r ( I S , I a R R ) .

Original: Otherwise, this routine repeats in the next step t ← t +1 .
Revised: Otherwise, this routine repeats in the next step t ← t + 1 .

Original: 4.1 Sender
Revised: 4.1 Sender

Original: Prior to playing the visual communication game, the sender should be able to (i) extract edgesfrom natural images [45] and (ii) draw sketches that closely resemble the configurations of salientedges [28], just as humans [35] do.
Revised: Prior to playing the visual communication game, the sender should be able to (i) extract edges from natural images (Xie and Tu, 2015) and (ii) draw sketches that closely resemble the configurations of salient edges (Li et al., 2019), just as humans do (Sayim and Cavanagh, 2011).

Original: To endow the sender with these capabilities, we design a stagewise architecture h S
Revised: To endow the sender with these capabilities, we design a stage-wise architecture h S

Original: = g S ◦ f S .
Revised: = g S ◦ f S .

Original: Specifically, I S is first converted to a target sketch
Revised: Specifically, I S is first converted to a target sketch

Original: ˆ I S using a visualmodule f S [20], capturing the salient edge information in the natural image; we directly adopt thepretrained model from the referred work.
Revised: ˆ I S using a visual module f S (Kampelmuhler and Pinz, 2020), capturing the salient edge information in the natural image; we directly adopt the pre-trained model from the referred work.

Original: Next, ˆ I S is concatenated with the current canvas C t andfed to the sketching module g S , whose architecture is built upon Huang et al .
Revised: Next, ˆ I S is concatenated with the current canvas C t and fed to the sketching module g S , whose architecture is built upon Huang et al.

Original: This sketchingmodule outputs five vectors in the form ( x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ) , which parametrizes the curve of onestroke.
Revised: This sketching module outputs five vectors in the form ( x 0 , y 0 , x 1 , y 1 , x 2 , y 2 ) , which parametrizes the curve of one stroke.

Original: The policy is parametrized as a Gaussian distribution during training,
Revised: The policy is parametrized as a Gaussian distribution during training,

Original: [Equation]
Revised: [Equation]

Original: where I is the identity matrix, and c is a constant hyperparameter.
Revised: where I is the identity matrix, and c is a constant hyperparameter.

Original: During  testing, we set c = 0 .
Revised: During the testing, we set c = 0 .

Original: These stroke parameters a St are fed into a pretrained renderer G [19] to update the canvas, C t +1 =
Revised: These stroke parameters a St are fed into a pre-trained renderer G (Huang et al., 2019) to update the canvas, C t +1 =

Original: G ( C t , a St ) .
Revised: G ( C t , a St ) .

Original: This renderer is fully differentiable, enabling end-to-end model-based training [14] ofthe sketching module g S .
Revised: This renderer is fully differentiable, enabling end-to-end model-based training (Hafner et al., 2019) of the sketching module g S .

Original: We pretrain g S on Sketchy [34]; see Supp. for results.
Revised: We pre-train g S on Sketchy (Sangkloy et al., 2016); please refer to the supplementary video for results.

Original: 4.2 Receiver
Revised: 4.2 Receiver

Original: The receiver, similar to the sender, should also carry some rudimentary visual capability to this game.
Revised: The receiver, similar to the sender, should also carry some rudimentary visual capability to this game.

Original: Unlike the low-level vision needed for the sender, the requirement for the receiver is high-level visualrecognition.
Revised: Unlike the low-level vision needed for the sender, the requirement for the receiver is high-level visual recognition.

Original: Therefore, we adopt a pretrained VGG16 [37] as the visual module f R : I → R 4096 ofthe receiver, following a similar practice in recent literature [26, 16].
Revised: Therefore, we adopt a pre-trained VGG16 (Simonyan and Zisserman, 2015) as the visual module f R : I → R 4096 of the receiver, following a similar practice in recent literature (Lazaridou et al., 2017; Havrylov and Titov, 2017).

Original: The output of this visual moduleis a vector, and further transformed by two separate linear layers, g KR and g QR , into visual embeddings,h KR ( I ) and h QR ( I ) .
Revised: The output of this visual module is a vector, and further transformed by two separate linear layers, g KR and g QR , into visual embeddings, h KR ( I ) and h QR ( I ) .

Original: That is, h KR =
Revised: That is, h KR =

Original: g KR ◦ f R , h QR = g QR ◦ f R .
Revised: [Equation]

Original: When observing both the context { I 1 R , ..., I MR } and the canvas C t , the receiver first embeds each ofthem with h R .
Revised: When observing both the context { I 1 R , ..., I MR } and the canvas C t , the receiver first embeds each of them with h R .

Original: Next, it makes the decision based on the similarity between the current canvas andeach option in the context.
Revised: Next, it makes the decision based on the similarity between the current canvas and each option in the context.

Original: The decision module is thus realized by a Boltzmann distribution basedon resemblance:
Revised: The decision module is thus realized by a Boltzmann distribution based on resemblance:

Original: [Equation]
Revised: [Equation]

Original: where I M +1 R =
Revised: where I M +1 R =

Original: C t − 1 .
Revised: C t − 1 .

Original: Although a similar policy was proposed before [25, 16], our π R is distinct asit is endowed with an external memory of C t − 1 .
Revised: Of note, although a similar policy was proposed before (Lazaridou et al., 2018; Havrylov and Titov, 2017), our π R is distinct as it is endowed with an external memory of C t − 1 .

Original: Intuitively, if the receiver finds the current canvas C t closer to the last canvas C t − 1 in the embedding space than all M options in the context, it willchoose to emit a Rt = M +1 and delay the decision to the next step; a prediction can only be madewhen the receiver finds the current canvas is informative enough.
Revised: Intuitively, if the receiver finds the current canvas C t closer to the last canvas C t − 1 in the embedding space than all M options in the context, it will choose to emit a Rt = M + 1 and delay the decision to the next step; a prediction can only be made when the receiver finds the current canvas is informative enough.

Original: As a result, the sender would drawinformative strokes as early as possible to avoid the implicit penalty in the decayed reward.
Revised: As a result, the sender would draw informative strokes as early as possible to avoid the implicit penalty in the decayed reward.

Original: 4.3 Learning to communicate
Revised: 4.3 Learning to communicate

Original: Policies of the sender and the receiver are trained jointly to maximize the objective:
Revised: Policies of the sender and the receiver are trained jointly to maximize the objective:

Original: [Equation]
Revised: [Equation]

Original: where τ = { C 0 , a S 0 , C 1 , a R 1 , a S 1 , ... } is the simulated episodic trajectory.
Revised: where τ = { C 0 , a S 0 , C 1 , a R 1 , a S 1 , ... } is the simulated episodic trajectory.

Original: As well known in reinforcement learning, the analytical expectation in Eq.
Revised: As well known in reinforcement learning, the analytical expectation in Eq.

Original: (4) is intractable to calculate along the trajectoryτ .
Revised: (4) is intractable to calculate along the trajectory τ .

Original: We devise value functions V ( X t ) and V λ ( X t ) for an optimization surrogate:
Revised: We devise value functions V ( X t ) and V λ ( X t ) for an optimization surrogate:

Original: [Equation]
Revised: [Equation]

Original: where  ˆ X t = [ I 1 R , ...I MR , C t − 1 , C t ] , X t = cat ([ I S ] , ˆ X t ) , δ ( a Rt ) is the Dirac delta function that returns1 when the action is wait and 0 otherwise.
Revised: where δ ( a Rt ) is the Dirac delta function that returns 1 when the action is wait and 0 otherwise. X t = cat ([ I S ] , ˆ X t ) , where ˆ X t = [ I 1 R , ...I MR , C t − 1 , C t ] .

Original: The expectation E π S ( a St | I S ,C t − 1 ) [ · ] is approximated withthe point estimate, as in the reparametrization in VAE [23].
Revised: The expectation E π S ( a St | I S ,C t − 1 ) [ · ] is approximated with the point estimate, as in the reparametrization in VAE (Kingma and Welling, 2014).

Original: The expectation E π R ( a Rt | ˆ X t ) [ · ] canbe calculated analytically because π Rt is a categorical distribution.
Revised: The expectation E π R ( a Rt | ˆ X t ) [ · ] can be calculated analytically because π Rt is a categorical distribution.

Original: The connection between thesetwo expectation is one of our contributions.
Revised: The connection between E π S and E π R is one of our contributions.

Original: Of note, C t in ˆ X t in π Rt ( a Rt | ˆ X t ) is generated by thedifferentiable renderer G with the actions a St from the sender policy π S ( a St | I S , C t − 1 ) .
Revised: Specifically, C t in ˆ X t in π Rt ( a Rt | ˆ X t ) is generated by the differentiable renderer G with the actions a St from the sender policy π S ( a St | I S , C t − 1 ) .

Original: Hence, wecan have both ∂ V /∂π Rt and ∂ V /∂π St based on Eq.
Revised: Hence, we can have both ∂ V /∂π Rt and ∂ V /∂π St based on Eq.

Original: This results in a novel multi-agent variant ofthe general policy gradient [40, 36].
Revised: This results in a novel multi-agent variant of the general policy gradient (Sutton et al., 2000; Silver et al., 2014).

Original: V λ ( X t ) in Eq.
Revised: V λ ( X t ) in Eq.

Original: (5) is an eligibility trace approximation [39] of the ground-truth value function.
Revised: (5) is an eligibility trace approximation (Sutton and Barto, 2018) of the ground-truth value function.

Original: Intuitively, a value estimate with eligibility trace V λ mixes the bootstrapped MonteCarlo estimate
Revised: Intuitively, a value estimate with eligibility trace V λ mixes the bootstrapped Monte

Original:  V kN ( X t ) =
Revised: Carlo estimate V kN ( X t ) =

Original: E π S ,π R [ (cid:80) h − 1 n = t γ n − t r n + γ h − t
Revised: E π S ,π R [ (cid:80) h − 1 n = t γ n − t r n + γ h − t

Original: δ
Revised: δ

Original: ( a Rh ) υ ϕ ( X h )] at different roll-out lengths k , with h =min ( t + k, T choice ) being the maximal timestep.
Revised: ( a Rh ) υ ϕ ( X h )] at different roll-out lengths k , with h = min ( t + k, T choice ) being the maximal timestep.

Original: T choice is the timestamp when the receiver stopswaiting.
Revised: T choice is the timestamp when the receiver stops waiting.

Original: The articulation of such termination also makes our eligibility trace deviate from the generalderivation with infinite horizon.
Revised: The articulation of such termination also makes our eligibility trace deviate from the general derivation with infinite horizon.

Original: We derive an episodic version as
Revised: We derive an episodic version as

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: if

Original: [Equation]
Revised: [Equation]

Revised: v ϕ

Revised: ( X

Original: [Equation]
Revised: [Equation]

Original:  where H =
Revised: ) otherwise, (6) where H =

Original: T choice − t +1 .
Revised: T choice − t + 1 .

Original: Please refer to the supplementary material for a detailed derivation andthe algorithm.
Revised: Please refer to Appendix C for a detailed derivation.

Original: Finally, v ϕ ( X t ) is trained by regressing the value returns:
Revised: Finally, v ϕ ( X t ) is trained by regressing the value returns:

Original: [Equation]
Revised: [Equation]

Revised: The training algorithm is summarized in Alg. 1.

Original:  Experiments 
Revised:  Experiments 5.1 Settings

Original: 5.1 Settings

Original: Images We used the Sketchy dataset [34] as the source of images.
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: Images We used the Sketchy dataset (Sangkloy et al., 2016) as the image source.

Original: Due to the limited performance ofthe sketching module on open-domain image-to-sketch sequential generation, we select 40 categories(10 images per category) that enable satisfactory sketching behaviors.
Revised: Due to the limited performance of the sketching module on open-domain image-to-sketch sequential generation, we select 40 categories (10 images per category, see Appendix A) with satisfactory sketching behaviors.

Original: Environmental drivers With the visual communication game and the learning agents at hand, weinvestigate thecausal factors in emergent graphical conventions with controlled experiments.
Revised: Environmental drivers With the visual communication game and the learning agents at hand, we investigate the factors in emergent graphical conventions with controlled experiments.

Original: Tab.
Revised: Tab.

Original: 1lists all designed settings.
Revised: 1 lists all designed settings.

Original: Specifically, we consider the following factors:
Revised: Specifically, we consider the following: [Table]

Original: • Can receiver make early decisions?
Revised: • Can receiver make early decisions?

Original: The hypothesis is that the receiver’s decision before exhaustingthe allowed communication steps may inform the sender of the marginal benefit of each stroke and incentivize it to prioritize the most informative strokes.
Revised: The hypothesis is that the receiver’s decision before exhausting the allowed communication steps may inform the sender of the marginal benefit of each stroke and incentivize it to prioritize the most informative strokes.

Original: The corresponding control setting ismax-step , where the receiver can only make the choice after the sender finishes its drawing at the maximum step.
Revised: The corresponding control setting is max-step , wherein the receiver can only make the choice after the sender finishes its drawing at the maximum step.

Original: This factor is on in other settings.• Can  sender change its way of drawing?
Revised: This factor is on in other settings. • Can the sender change its way of drawing?

Original: The hypothesis is that the mutual adaptation of the sender and the receiver may lead to better abstraction in the evolved sketches.
Revised: The hypothesis is that the mutual adaptation of the sender and the receiver may lead to better abstraction in the evolved sketches.

Original: Particularly, the sender may develop new ways of drawing in the evolution.
Revised: Particularly, the sender may develop new ways of drawing in the evolution.

Original: The corresponding control setting is sender-fixed ,wherein we freeze the parameters of the sender, such that the receiver has to adapt to its partner.
Revised: The corresponding control setting is sender-fixed , wherein we freeze the parameters of the sender such that the receiver has to adapt to its partner.

Original: This factor is on in other settings.
Revised: This factor is on in other settings.

Original: • Is the game sequential, and can the receiver observe more complex drawings?
Revised: • Is the game sequential, and can the receiver observe more complex drawings?

Original: The hypothesis is that the modeling of a sequential decision-making game and the evolution from more complex sketches may regularize the arbitrariness, which is unique for graphical conventions.
Revised: The hypothesis is that the modeling of a sequential decision-making game and the evolution from more complex sketches may regularize the arbitrariness, which is unique for graphical conventions.

Original: The correspondingcontrol setting is one-step : There only exists one step of sketching, thus no sequential decision- making at all.
Revised: The corresponding control setting is one-step : There only exists one step of sketching, thus no sequential decisionmaking at all.

Original: This factor is on in other settings.
Revised: This factor is on in other settings.

Original: [Table] Training, validation and generalizationtest We train the sender and the receiver with batchedforward and back-propagation, with a batch size of 64 and maximum roll-out step T = 7 .
Revised: Training, validation, and generalization We train the sender/receiver with batched forward and back-propagation, with a batch size of 64 and maximum roll-out step T = 7 .

Original: We updateusing Adam [22] with the learning rate 0.0001 for a total of 30k iterations.
Revised: We update using Adam (Kingma and Ba, 2015) with the learning rate 0.0001 for a total of 30k iterations.

Original: We train each modelon a single GPU (Nvidia RTX A6000); a single experiment takes 20 hours.
Revised: We train each model on a single Nvidia RTX A6000; one experiment takes 20 hours.

Original: Except for the senderfixed setting, there is a warm-up phase in the first 2000 iterations for the sender where we linearlyincrease the learning rate to 0.0001.
Revised: Except for the sender-fixed setting, there is a warm-up phase in the first 2000 iterations for the sender where we linearly increase the learning rate to 0.0001.

Original: After the warm-up phase, the learning rate of both agents will bedecreased exponentially by 0 .
Revised: After the warm-up phase, the learning rate of both agents will be decreased exponentially by 0 .

Original: 99 i − 20001000 , where i is the number of training iterations.
Revised: 99 i − 20001000 , where i is the number of training iterations.

Original: In all settings,we set M = 3 , γ = 0 .85 .
Revised: In all settings, we set M = 3 , γ = 0 .

Original: Between iterations, we randomly sample another 10 batches for validation.
Revised: Between iterations, we randomly sample another 10 batches for validation.

Original: We use 30 categories (10 images per category) for training and held out 10 images per category forunseen-instance test. Another 10 categories are for  unseen-class test.
Revised: We use 30 categories (10 images per category) for training and held out 10 images per category for the unseen-instance test; another 10 categories are for the unseen-class test.

Original: Each image is communicated asthe target, resulting in 300 + 100 pairs in the test set.
Revised: Each image is communicated as the target, resulting in 300 + 100 pairs in the test set.

Original: At each iteration, the categories and instancesare sampled randomly to constitute the context.
Revised: At each iteration, the categories and instances are sampled randomly to constitute the context.

Original: Results below are statistics of 5 random seeds.
Revised: Results henceforth are statistics of 5 random seeds.

Original: 5.2 Results5.2.1 Communication efficacy and sketch abstraction
Revised: 5.2 Results 5.2.1 Communication efficacy and sketch abstraction

Original: We record both the success rate and the communication steps over the training iterations; see Fig. 3.
Revised: We record both the success rate and the communication steps over the training iterations; see Fig. 3.

Original: The communication is considered success when the receiver makes the corrent prediction at the endof the game.
Revised: The communication is considered successful when the receiver makes the correct prediction at the end of the game.

Original: In Fig.
Revised: In Fig.

Original: 3a, agents in all settings except one-step converge to a success rate above 80% .
Revised: 3a, agents in all settings except one-step converge to a success rate above 80% .

Original: Among them, the communicating pairs in the complete setting and the sender-fixed setting evolve to[Figure] achieve a comparable success rate with the retrieve baseline.
Revised: Among them, the communicating pairs in the complete setting and the sender-fixed setting evolve to achieve a comparable success rate with the retrieve baseline.

Original: Interestingly, these two pairs also emergea phenomenon resembling the natural observation in human studies, named systematic reduction [27]:
Revised: Interestingly, these two pairs also emerge a phenomenon resembling the natural observation in human studies, named systematic reduction (Lewis, 1969):

Original: The average steps first increase and then gradually decrease as in Fig.
Revised: The average steps first increase and then gradually decrease as in Fig.

Original: 3b.
Revised: 3b.

Original: Contrasting complete andsender-fixed , we can see: (i) The emergent conventions in the former is much simpler than the latter(less steps in Fig.
Revised: Contrasting complete and sender-fixed , we can see: (i) The emergent conventions in the former is much simpler than the latter (less steps in Fig.

Original: 3b), which implies the contribution of mutual adaptation in sketch abstraction.
Revised: 3b), which implies the contribution of mutual adaptation in sketch abstraction.

Original: (ii)
Revised: (ii)

Original: The success rate in Fig.
Revised: The success rate in Fig.

Original: 3a in the former converges a  bit more slowly, which is reasonable since thesenders can explore and change the way of drawing.
Revised: 3a in the former converges a [Figure] bit more slowly, which is reasonable since the senders can explore and change the way of drawing.

Original: In comparison, if the receiver cannot makeearly decisions, it has no intention to relate sketches ( i.e ., C t − 1 and C t ) at consecutive timesteps.
Revised: In comparison, if the receiver cannot make early decisions, it has no intention to relate sketches ( i.e ., C t − 1 and C t ) at consecutive timesteps.

Original: The sender is thus unaware of each stroke’s marginal information gain, which in return makes theirlearning harder.
Revised: The sender is thus unaware of each stroke’s marginal information gain, which in return makes their learning harder.

Original: This might explain the relatively low success rate in the max-step setting.
Revised: This might explain the relatively low success rate in the max-step setting.

Original: The failureof the one-step pairs reveals the irreplaceable roles of sequential decision-making and observingcomplex sketches in emergent graphical communication.
Revised: The failure of the one-step pairs reveals the irreplaceable roles of sequential decision-making and observing complex sketches in emergent graphical communication.

Original: To further inspect how our proposed modeling and learning on sequential decision-making facilitatethe desired evolution in the sketches, we conduct an ablation study by comparing our proposedlearning surrogate (Eq.
Revised: To further inspect how our proposed modeling and learning on sequential decision-making facilitate the desired evolution in the sketches, we conduct an ablation study by comparing our proposed learning surrogate (Eq.

Original: (5)) and a vanilla policy gradient baseline, REINFORCE [44] with Monte Carlo cumulative rewards
Revised: (5)) and a vanilla policy gradient baseline, REINFORCE (Williams, 1992) with Monte Carlo cumulative rewards

Original: [Equation]
Revised: [Equation]

Original: Tn =

Original: t γ n − t r n ]] .

Original: Our comparison spans three axes.
Revised: Our comparison spans three axes.

Original: First, the REINFORCE baseline converges much more slowlythan the proposed surrogate; see cumulative vs complete in Fig. 3a.
Revised: First, the REINFORCE baseline converges much more slowly than the proposed surrogate; see cumulative vs complete in Fig. 3a.

Original: Second, we check the robustnessunder variation of decay factor γ .
Revised: Second, we check the robustness under the variation of decay factor γ .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3b, while the proposed method shows stableconvergence in the communication steps under γ = 0 .
Revised: 3b, while the proposed method shows stable convergence in the communication steps under γ = 0 .

Original: 85 and 0 .
Revised: 85 and 0 .

Original: 95 , the REINFORCE baseline exhibitshigh-variance behavior under certain setup ( γ = 0 .
Revised: 95 , the REINFORCE baseline exhibits high-variance behavior under certain setup ( γ = 0 .

Original: Finally, we check if agents’ early terminationsare caused by their awareness of the indistinguishable performance in longer and shorter episodes.
Revised: Third, we check if agents’ early terminations are caused by their awareness of the indistinguishable performance in longer and shorter episodes.

Original: Given a precondition that the longer the episodes are, the earlier the success rate increases, it shouldbe the increase in the average performance of shorter episodes that causes the average timesteps todecrease.
Revised: Given a precondition that the longer the episodes are, the earlier the success rate increases, it should be the increase in the average performance of shorter episodes that causes the average timesteps to decrease.

Original: Taking 1-step and 3-step communication for example, in the complete setting, we shall seethe success rate of the 3-step to achieve high earlier, which is then caught up but not exceeded bythe 1-step.
Revised: Taking 1-step and 3-step communication for example, in the complete setting, we shall see the success rate of the 3-step to achieve high earlier, which is then caught up but not exceeded by the 1-step.

Original: The not exceeding condition is a crucial cue to validate that the communicating partnerswere actively pursuing the Pareto front [21] / efficiency bound [47] of accuracy and complexity.
Revised: The not exceeding condition is a crucial cue to validate that the communicating partners were actively pursuing the Pareto front (Kim and De Weck, 2005) / efficiency bound (Zaslavsky et al., 2018) of accuracy and complexity.

Original: Thisis exactly what our proposed method emerges as shown in Fig.
Revised: This is exactly what our proposed method emerges as shown in Fig.

Original: 3c.
Revised: 3c.

Original: In contrast, in the REINFORCEbaseline, under the same decay factor, the performance of 1-step surpasses 3-step communication.
Revised: In contrast, in the REINFORCE baseline, under the same decay factor, the performance of 1-step surpasses 3-step communication.

Original: Itseems as if the incapability of learning long episodes caused the agents to avoid them.
Revised: It seems as if the incapability of learning long episodes caused the agents to avoid them.

Original: All our results on success rate and communication steps are consistent with predictions based onour hypotheses, which justifies our environments and models.
Revised: Together, all our results on success rate and communication steps are consistent with predictions based on our hypotheses, which justifies our environments and models.

Original: However, a high success rate does notnecessarily imply high convention quality.
Revised: However, a high success rate does not necessarily imply high convention quality.

Original: Another essential property for conventions is stability [27]: There should exist common patterns in the repeated usage of conventions to convey similar concepts.
Revised: Another essential property for conventions is stability (Lewis, 1969): There should exist common patterns in the repeated usage of conventions to convey similar concepts.

Original: We take the viewpoint of representation learning and concretize the vague stability with threeformally defined properties: iconicity , symbolicity  and semanticity .
Revised: We take the viewpoint of representation learning and concretize the vague stability with three formally defined properties: iconicity , symbolicity , and semanticity .

Original: In the following, we introduce ourexperiments to measure these properties.
Revised: Below, we introduce our experiments to measure these properties systematically.

Original: 5.2.2 Iconicity: generalizing to unseen images
Revised: 5.2.2 Iconicity: generalizing to unseen images

Original: We define iconicity as the drawings exhibiting  high visual resemblance with the correspondingimages, such that they are proximal to the latter when measured on the high-level embedding of a general-purpose visual system.
Revised: We define iconicity as the drawings exhibiting a high visual resemblance with the corresponding images, such that they are proximal to the latter when measured on the high-level embedding of a general-purpose visual system.

Original: To quantitatively measure the visual similarities, we need to computethe distance between the sketch and image in the visual embedding space with e.g . cosine similarity.
Revised: To quantitatively measure the visual similarities, we need to compute the distance between the sketch and image in the visual embedding space ( e.g ., with cosine similarity).

Original: However, since we don’t know how the embeddings distribute in their space, this unnormalizedmeasure is prone to model-specific biases.
Revised: However, since we do not know how the embeddings distribute in their space, this unnormalized measure is prone to model-specific biases.

Original: Note that the  policyof the receiver takes the form ofa softmax of the cosine similarity between the embedding of the sketch and a context set with atarget image and some randomly sampled distractors, which naturally approximates the normalizationwe want.
Revised: Fortunately, the receiver’s policy takes the form of a softmax of the cosine similarity between the embedding of the sketch and a context set with a target image and some randomly sampled distractors, which naturally approximates the normalization we want.

Original: Therefore,the communication accuracy can serve as a visual similarity measure, and itsgeneralizability to unseen images can pinpoint the emergent preservation of iconicity .
Revised: Therefore, communication accuracy can serve as a visual similarity measure, and its generalizability to unseen images can pinpoint the emergent preservation of iconicity .

Original: Intuitively, inopen-domain communication, agents would see novel scenes containing known or unknown concepts,i.e . unseen instances of seen classes and unseen classe.
Revised: Intuitively, in open-domain communication, agents would see novel scenes with known or unknown concepts— unseen instances of seen classes and unseen classes, respectively.

Original: They should still be able to communicate withestablished conventions or with unconventionalized iconic drawings.
Revised: They should still be able to communicate with established conventions or with unconventionalized iconic drawings.

Original: A successful generalization tounseen classes ( i.e ., zero-shot generalization) is more difficult than unseen instances; partners mayincrease the conversation length and communicate with more complex sketches.
Revised: A successful generalization to unseen classes ( i.e ., zero-shot generalization) is more difficult than unseen instances; partners may increase the conversation length and communicate with more complex sketches.

Original: Tab. 1 reports the success rates and average timesteps in our generalization tests.
Revised: Tab. 1 reports the success rates and average timesteps in our generalization tests.

Original: The retrieve settingis the baseline, since there is no evolution at all and the sketches should resemble the original imagesthe most ( i.e ., possessing the highest iconicity ).
Revised: The retrieve setting is the baseline since there is no evolution at all and the sketches should resemble the original images the most ( i.e ., possessing the highest iconicity ).

Original: Unsurprisingly, its generalization performance upperbounds all other settings.
Revised: Unsurprisingly, its generalization performance upperbounds all other settings.

Original: Among the experimental and controlled settings, the complete , the max-step ,and the sender-fixed agents generalize relatively well in unseen instances ( 70 .
Revised: Among the experimental and controlled settings, the complete , the max-step , and the sender-fixed agents generalize relatively well in unseen instances ( 70 .

Original: 37 ± 0 .
Revised: 37 ± 0 .

Original: 93 ±0 .
Revised: 93 ± 0 .

Original: 02 , 71 . 80 ± 0 . 02 )
Revised: 02 , 71 . 80 ± 0 . 02 )

Original: and generalize above chance ( 39 .
Revised: and generalize above chance ( 39 .

Original: 40 ± 0 .
Revised: 40 ± 0 .

Original: 40 ± 0 . 40 ± 0 .
Revised: 40 ± 0 . 40 ± 0 .

Original: 02 > 25.00)in unseen classes.
Revised: 02 > 25.00) in unseen classes.

Original: Interestingly, complete and sender-fixed communicating partners intelligently turnto longer episodes for better generalization, better than the max-step agents.
Revised: Interestingly, complete and sender-fixed communicating partners intelligently turn to longer episodes for better generalization, better than the max-step agents.

Original: This finding implies thepartners may turn to more iconic communication when there is no established conventions/symbols,just as we humans do.
Revised: This finding implies the partners may turn to more iconic communication when there is no established conventions/symbols, just as we humans do.

Original: Strikingly, the max-step conventions seem to loose more iconicity , possiblydue to confusion on marginal information gains.
Revised: Strikingly, the max-step conventions seem to loose more iconicity , possibly due to confusion on marginal information gains.

Original: The one-step drawings seem to lack iconicity .
Revised: The one-step drawings seem to lack iconicity .

Original: 5.2.3 Symbolicity: separating evolved sketches
Revised: 5.2.3 Symbolicity: separating evolved sketches

Original: Next, we measure symbolicity to evaluate the graphical conventionalization.
Revised: Next, we measure symbolicity to evaluate the graphical conventionalization.

Original: We define symbolicityas the drawings being consistently separable in the high-level visual embedding, which facilitatesnew communication partners to easily distinguish between categories without grounding them toreferents.
Revised: We define symbolicity as the drawings being consistently separable in the high-level visual embedding, which facilitates new communication partners to easily distinguish between categories without grounding them to referents.

Original: Based on this definition, a more symbolic drawing should be more easily separable intotheir corresponding categories.
Revised: Based on this definition, a more symbolic drawing should be more easily separable into their corresponding categories.

Original: To measure such separability , we use a pretrained VGG16 as the newlearner and finetune the last fully connected layer to classify evolved sketches into the 30 categories.
Revised: To measure such separability , we use a pre-trained VGG16 as the new learner and finetune the last fully connected layer to classify evolved sketches into the 30 categories.

Original: Technically, we first get the 300 final canvases from the communication game, 10 for each category.
Revised: Technically, we first get the 300 final canvases from the communication game, 10 for each category.

Original: Among them, we use 70% for training and 30% for testing.
Revised: Among them, we use 70% for training and 30% for testing.

Original: The bar plot in Fig.
Revised: The bar plot in Fig.

Original: 4 shows the classificationresults.
Revised: 4 shows the classification results.

Original: Since agents in the one-step setting donot play the game successfully, they may notform a consistent way to communicate.
Revised: Since agents in the one-step setting do not play the game successfully, they may not form a consistent way to communicate.

Original: Agentsin the complete setting achieve the highest accuracy, higher even compared with the result ofthe original images.
Revised: Agents in the complete setting achieve the highest accuracy, higher even compared with the result of the original images.

Original: This finding indicates thatagents in the complete setting agree on a graphical convention that consistently highlights somefeatures across all training instances in each category, which are also distinguishable between categories.
Revised: This finding indicates that agents in the complete setting agree on a graphical convention that consistently highlights some features across all training instances in each category, which are also distinguishable between categories.

Original: Comparing the max-step with the complete setting, we know that early decision is an importantfactor for more symbolic conventions.
Revised: Comparing the max-step with the complete setting, we know that early decision is a crucial factor for more symbolic conventions.

Original: Comparison between the sender-fixed setting and the completesetting suggests that the sender’s evolution also contributes to high symbolicity . 
Revised: Comparison between the sender-fixed setting and the complete setting suggests that the sender’s evolution also contributes to high symbolicity . [Figure]

Original: 5.2.4 Semanticity: correlating category embedding
Revised: 5.2.4 Semanticity: correlating category embedding

Original: The last desired propertyof graphical conventions is that the evolved sketches can preserve theperceptual metric in images [48].
Revised: The last desired property is that the evolved sketches can preserve the perceptual metric in images (Zhang et al., 2018).

Original: We define semanticity as the topography of the high-level visualembedding space of drawings being strongly correlated to that of images, such that semanticallysimilar instances and categories lie close to each other in the embedding space.
Revised: We define semanticity as the topography of the high-level visual embedding space of drawings being strongly correlated to that of images, such that semantically similar instances and categories lie close to each other in the embedding space.

Original: To examine suchtopographic correlation , we project  the embeddings obtained in Sec. 
Revised: To examine such topographic correlation , we project category names to the feature space using word2vec (Mikolov et al., 2013) as featureA, and project the evolved sketches to the feature space using the trained VGG in Sec. 5.2.3 as featureB. We compute the correlation of the distances between all the possible pairs of featureB and the corresponding pairs of featureA as the semanticity measure. The results in Tab. 2 show that semanticity can be better retained in the complete setting compared with the retrieve baseline.

Original: 5.2.3 to a 2D space via tSNE [41].
Revised: We further visualize the embeddings of the sketches in a 2D space via t-SNE (Van der Maaten and Hinton, 2008).

Original: Fig. 5 show the visualization of the original images and the sketches in the retrieve andcomplete settings; please refer to supplementary for results of other settings.
Revised: Fig.show the visualization of the original images and the sketches in the retrieve and complete settings; please refer to Appendix B for results of other settings.

Original: Images/drawings from the same category are marked in the same color.
Revised: Images/drawings from the same category are marked in the same color.

Original: As shown, boundaries between categories are clearerin the evolved drawings in the complete setting than in retrieve or original images; but semanticallysimilar categories are still close to each other.
Revised: As shown, boundaries between categories are clearer in the evolved drawings in the complete setting than in retrieve or original images; but semantically similar categories are still close to each other.

Original: For example, cow, deer, horse, and camel are proximalto each other, while burger and apple are far from them.
Revised: For example, cow, deer, horse, and camel are proximal to each other, while burger and apple are far from them.

Original: These results highlight another uniquenessof visual communication over its symbolic counterpart: The similarity in the visual cues in theconventions may hint the semantic correlation between the referents.
Revised: These results highlight another uniqueness of visual communication over its symbolic counterpart: The similarity in the visual cues in the conventions may hint the semantic correlation between the referents.

Original: 5.2.5 Visualizing sketch evolution
Revised: 5.2.5 Visualizing sketch evolution

Original: To better understand the nature of the emerged conventions, we inspect the intermediate sketches inthe evolution processes.
Revised: To better understand the nature of the emerged conventions, we inspect the intermediate sketches in the evolution processes.

Original: Specifically, wechoose to visualize the process under the complete setting.
Revised: Specifically, we visualize the process under the complete setting.

Original: Fig. 6 shows three instances in two categories.
Revised: Fig.shows three instances in two categories.

Original: For each example, drawings from the left to the rightshow the change of the final-step canvas from iteration 0 to 30,000.
Revised: For each example, drawings from the left to the right show the change of the final-step canvas.

Original: Sketches’ complexity graduallydecreases after an initial increase, echoing the trend of reduction described in Sec.
Revised: Sketches’ complexity gradually decreases after an initial increase, echoing the trend of reduction described in Sec.

Original: For rabbit,in the beginning, the strokes may depict instances from different perspectives; through iterations, theyconverge to highlight the rabbit’s long ear.
Revised: For rabbits, in the beginning, the strokes may depict instances from different perspectives; through iterations, they converge to highlight the rabbit’s long ear.

Original: As for  giraffe, the agents gradually learn to emphasize thelong neck.
Revised: As for the giraffe, the agents gradually learn to emphasize the long neck.

Original: Particularly, in the third example, although the giraffe lowers its head, we can still see anexaggerated vertical stroke for the neck, similar to the first example where the giraffe’s head is raised.
Revised: In the third example, although the giraffe lowers its head, we can still see an exaggerated vertical stroke for the neck, similar to the first example where the giraffe’s head is raised.

Original: These examples show how the sender gradually unifies drawings of the same category: after evolution,the sender is inclined to use the first several strokes to depict the most salient parts of visual concepts.
Revised: These examples show how the sender gradually unifies drawings of the same category: After evolution, the sender is inclined to use the first several strokes to depict the most salient parts of visual concepts.

Original:  Conclusion
Revised:  Conclusion

Original: We present the first step of modeling and simulating the evolution of graphical conventions betweentwo agents in a visual communication game.
Revised: We present the first step of modeling and simulating the evolution of graphical conventions between two agents in a visual communication game.

Original: Agents modeled in our framework can successfullycommunicate visual concepts using sketches as the medium.
Revised: Agents modeled in our framework can successfully communicate visual concepts using sketches as the medium.

Original: We measure the emergent graphicalconventions over three carefully defined properties, iconicity , symbolicty , and semanticity .
Revised: We measure the emergent graphical conventions over three carefully defined properties, iconicity , symbolicty , and semanticity .

Original: Theexperimental results under different controls suggest that early decision, mutual adaptation, andsequential decision-making can encourage symbolicity while preserving iconicity and semanticity .
Revised: The experimental results under different controls suggest that early decision, mutual adaptation, and sequential decision-making can encourage symbolicity while preserving iconicity and semanticity .

Original: However, two-stage pretrained senders possess limitations; an ideal sender would not need to convertthe images to pixel-level sketches before it starts sketching.
Revised: However, two-stage pre-trained senders possess limitations; an ideal sender would not need to convert the images to pixel-level sketches before it starts sketching.

Original: The limitations in the pretrained sketchingmodule also constrain the discriminative need among the selected classes.
Revised: The limitations in the pre-trained sketching module also constrain the discriminative need among the selected classes.

Original: We hope to investigatethese limitations in future research.
Revised: We hope to investigate these limitations in future research.

Original: With the uniqueness of visual conventions demonstrated, we hopeour work can invoke the study of emergent communication in the modality of sketches.
Revised: With the uniqueness of visual conventions demonstrated, we hope our work can invoke the study of emergent communication in the modality of sketches.

Revised: Acknowledgments J. Joo was supported in part by the NSF SMA/SBE #1831848.

Revised: We would like to thank Prof. Marco Baroni (Pompeu Fabra Univ.)

Revised: and Prof. Yaodong Yang (Peking Univ.)

Original: Checklist
Revised: for helpful comments on the draft, Miss Chen Zhen (BIGAI) for making the nice figures, all anonymous reviewers for their constructive comments on prior submissions, and Prof. Ying Nian Wu (UCLA), Prof. Hongjing Lu (UCLA), Prof. Federico Rossano (UCSD), Dr. Zilong Zheng (BIGAI), Dr. Chi Zhang (BIGAI), Baoxiong Jia (UCLA), Dr. Qing Li (BIGAI), and Dr. Bo Pang (UCLA) for discussion on the algorithm and experimental designs.

Revised: B Category embedding for other game settings

Revised: Fig.

Revised: A1 shows the t-SNE visualization for other game settings.

Revised: Agents under max-step , sender-fixed , and one-step settings fail to form clear boundaries between different categories, which makes it hard to observe semantic relations.

Revised: C Learning objectives and training algorithm

Revised: Agents are trained jointly to maximize the objective:

Revised: [Equation]

Revised: (A1) where τ = { C 0 , a S 0 , C 1 , a R 1 , a S 1 , ... } is the simulated episodic trajectory.

Revised: To further expand the objective,

Revised: [Equation]

Revised: We calculate E I S ,I 1 R ,...,I MR ,C 0 [ · ] by sampling I S , I R , and initializing C 0 to blank at each round.

Revised: We represent the E ( π S ,π R ) [ · ] as V ( X 0 ) and use V λ ( X 1 ) to estimate the reward expectation E (

Revised: [Equation]

Revised: S ,

Revised: [Equation]

Original: For all authors...
Revised: [Equation] V [Equation] where X t =

Revised: [ I S , I 1 R , ..., I MR , C t , C t +1 ] , t = 0 , 1 ... , δ ( · ) is the Dirac delta function that returns 1 when the action is wait and 0 otherwise.

Revised: The sender policy is parametrized as a Gaussian distribution,

Revised: [Equation]

Revised: such that a S 0 can be written as

Revised: [Equation]

Revised: Therefore, we can expand V ( X 0 ) as,

Revised: [Equation]

Revised: [Equation]

Revised: E ϵ [ · ] is approximated with a point estimate.

Original: (a) Do the main claims made in  the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: Since π R is a categorical distribution, we expand E π R as [Equation] V λ ( X t ) in Eq. (A3) is an eligibility trace approximation of the ground-truth value function (Sutton and Barto, 2018).

Revised: Considering the early termination in our setting, we set the time step when the receiver makes the prediction as T choice .

Revised: When t is the time step less or equal than T choice , V λ mixes

Revised: Monte Carlo estimate at different roll-out lengths.

Revised: Otherwise, we only have an estimated value v ϕ ( X t ) .

Original: [Yes] See 3 and 5.2.
Revised: [Equation] otherwise [Equation] where H =

Revised: T choice − t + 1 , and V kN ( X t ) is the Monte Carlo estimate at k roll-out lengths.

Original: (b) Did you describe the limitations of your work?
Revised: V kN ( X t ) = E π S ,π R [ (cid:80) [Equation] n =

Revised: [Equation]

Revised: n − t r n + γ h − t δ ( a Rh ) υ ϕ ( X h )] , with h = min( t + k, T choice ) being the maximal timestep.

Original: [Yes] See 6.
Revised: Due to the error reduction property Sutton and Barto (2018), the eligibility trace estimation V λ ( · ) is less biased than v ϕ ( · ) . When regressing v ϕ ( X t ) towards the bootstrapped V λ ( X t ) , [Equation]

Revised: v ϕ

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: ( X t ) will be improved towards the fixed point. D Visualizing sketch evolution Visualizing the evolution process helps us understand what the agents have learned through communication regarding different categories. By comparing the evolved sketches with the intermediate results, we can know (i) how the agents abstract the sketch, (ii) which parts of the visual concept they highlight, and (iii) which parts are de-emphasized.

Original: [No]
Revised: Fig.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]
Revised: A2 to A4 show some evolution examples under different settings.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: Agents under max-step seem to abstract their drawings by repeatedly placing new strokes near old strokes, resulting in bold drawings.

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)? [No] We will make the code and data publicly available upon paper acceptance. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: The number of strokes under sender-fixed gradually decreases, but the way of drawing will not change.

Original: [Yes] See 5. (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
Revised: Senders under one-step change more wildly, but cannot form a consistent drawing behavior.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] See 5. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: Overall, compared with the complete setting, agents under the control settings do not form patterns to draw sketches, which echoes their relatively low classification results.

Original: [Yes]
Revised: [Figure]

Original: (b) Did you mention the license of the assets?

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [Yes]
Revised: [Figure]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Figure]

Original: spent on participant compensation?

Original: [N/A]