Original: C O D RAW : C OLLABORATIVE D RAWING AS A T ESTBED FOR G ROUNDED G OAL - DRIVEN C OMMUNICATION
Revised: C O D RAW : C OLLABORATIVE D RAWING AS A T ESTBED FOR G ROUNDED G OAL - DRIVEN C OMMUNICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this work we propose a goal-driven collaborative task that contains language, vision, and action in a virtual environment as its core components.
Revised: In this work, we propose a goal-driven collaborative task that contains language, vision, and action in a virtual environment as its core components.

Original: Speciﬁcally, we develop a Co llaborative image-
Revised: Speciﬁcally, we develop a Co llaborative image-

Original: Draw ing game between two agents, called
Revised: Draw ing game between two agents, called

Original: CoDraw .
Revised: CoDraw .

Original: Our game is grounded in a virtual world that contains movable clip art objects.
Revised: Our game is grounded in a virtual world that contains movable clip art objects.

Original: The game involves two players: a Teller and a Drawer .
Revised: The game involves two players: a Teller and a Drawer .

Original: The Teller sees an abstract scene containing multiple clip art pieces in a semantically meaningful conﬁguration, while the Drawer tries to reconstruct the scene on an empty canvas using available clip art pieces.
Revised: The Teller sees an abstract scene containing multiple clip art pieces in a semantically meaningful conﬁguration, while the Drawer tries to reconstruct the scene on an empty canvas using available clip art pieces.

Original: The two players communicate via twoway communication using natural language.
Revised: The two players communicate via two-way communication using natural language.

Original: We collect the CoDraw dataset of ∼ 10K dialogs consisting of ∼ 138K messages exchanged between human agents.
Revised: We collect the CoDraw dataset of ∼ 10K dialogs consisting of ∼ 138K messages exchanged between human agents.

Original: We deﬁne protocols and metrics to evaluate the effectiveness of learned agents on this testbed, highlighting the need for a novel crosstalk condition which pairs agents trained independently on disjoint subsets of the training data for evaluation.
Revised: We deﬁne protocols and metrics to evaluate the effectiveness of learned agents on this testbed, highlighting the need for a novel crosstalk condition which pairs agents trained independently on disjoint subsets of the training data for evaluation.

Original: We present models for our task, including simple but effective baselines and neural network approaches trained using a combination of imitation learning and goal-driven training.
Revised: We present models for our task, including simple but effective nearest-neighbor techniques and neural network approaches trained using a combination of imitation learning and goal-driven training.

Original: All models are benchmarked using both fully automated evaluation and by playing the game with live human agents.
Revised: All models are benchmarked using both fully automated evaluation and by playing the game with live human agents

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Building agents that can interact with humans in natural language while perceiving and taking actions in their environments is one of the fundamental goals in artiﬁcial intelligence.
Revised: Building agents that can interact with humans in natural language while perceiving and taking actions in their environments is one of the fundamental goals in artiﬁcial intelligence.

Original: One of the required components, language understanding, has traditionally been studied in isolation and with tasks aimed at imitating human behavior ( e.g . language modeling Bengio et al.
Revised: One of the required components, language understanding, has traditionally been studied in isolation and with tasks aimed at imitating human behavior ( e.g . language modeling Bengio et al.

Original: Mikolov et al.
Revised: Mikolov et al.

Original: (2010), machine translation Bahdanau et al.
Revised: (2010), machine translation Bahdanau et al.

Original: ; Sutskever et al.
Revised: ; Sutskever et al.

Original: (2014), etc .)
Revised: (2014), etc .)

Original: by learning from large text-only corpora.
Revised: by learning from large text-only corpora.

Original: To incorporate both vision and action, it is important to have the language grounded (Harnad, 1990; Barsalou, 1999), where words like cat are connected to visual percepts and words like move relate to actions taken in an environment.
Revised: To incorporate both vision and action, it is important to have the language grounded (Harnad, 1990; Barsalou, 1999), where words like cat are connected to visual percepts and words like move relate to actions taken in an environment.

Original: Additionally, judging language understanding purely based on the ability to mimic human utterances has limitations: there are many ways to express roughly the same meaning, and conveying the correct information is often more important than the particular choice of words.
Revised: Additionally, judging language understanding purely based on the ability to mimic human utterances has limitations: there are many ways to express roughly the same meaning, and conveying the correct information is often more important than the particular choice of words.

Original: An alternative approach, which has recently gained increased prominence, is to train and evaluate language generation capabilities in an interactive setting, where the focus is on successfully communicating information that an agent must share in order to achieve its goals.
Revised: An alternative approach, which has recently gained increased prominence, is to train and evaluate language generation capabilities in an interactive setting, where the focus is on successfully communicating information that an agent must share in order to achieve its goals.

Original: [Figure]
Revised: [Figure]

Original: In this paper, we propose the Collaborative Drawing (CoDraw) task, which combines grounded language understanding and learning effective goal-driven communication into a single, uniﬁed testbed.
Revised: In this paper, we propose the Collaborative Drawing (CoDraw) task, which combines grounded language understanding and learning effective goal-driven communication into a single, uniﬁed testbed.

Original: This task involves perception, communication, and actions in a partially observable virtual environment.
Revised: This task involves perception, communication, and actions in a partially observable virtual environment.

Original: As shown in Figure 1, our game is grounded in a virtual world constructed by clip art objects (Zitnick et al., 2013; Zitnick & Parikh, 2013).
Revised: As shown in Figure 1, our game is grounded in a virtual world constructed by clip art objects (Zitnick et al., 2013; Zitnick & Parikh, 2013).

Original: Two players, Teller and Drawer, play the game.
Revised: Two players, Teller and Drawer, play the game.

Original: The Teller sees an abstract scene made from clip art objects in a semantically meaningful conﬁguration, while the Drawer sees a drawing canvas that is initially empty.
Revised: The Teller sees an abstract scene made from clip art objects in a semantically meaningful conﬁguration, while the Drawer sees a drawing canvas that is initially empty.

Original: The goal of the game is to have both players communicate so that the Drawer can reconstruct the image of the Teller, without ever seeing it.
Revised: The goal of the game is to have both players communicate so that the Drawer can reconstruct the image of the Teller, without ever seeing it.

Original: Our task requires effective communication because the two players cannot see each other’s scenes.
Revised: Our task requires effective communication because the two players cannot see each other’s scenes.

Original: The Teller has to describe the scenes in sufﬁcient detail for the Drawer to reconstruct it, which will require rich grounded language.
Revised: The Teller has to describe the scene in sufﬁcient detail for the Drawer to reconstruct it, which will require rich grounded language.

Original: Moreover, the Drawer will need to carry out a series of actions from a rich action space to position, orient, and resize all of the clip art pieces required for the reconstruction.
Revised: Moreover, the Drawer will need to carry out a series of actions from a rich action space to position, orient, and resize all of the clip art pieces required for the reconstruction.

Original: Note that such actions are only made possible through clip art pieces: they can represent semantically meaningful conﬁgurations of a visual scene that are easy to manipulate, in contrast to low-level pixel-based image representations.
Revised: Note that such actions are only made possible through clip art pieces: they can represent semantically meaningful conﬁgurations of a visual scene that are easy to manipulate, in contrast to low-level pixel-based image representations.

Original: The performance of a pair of agents is judged based on the quality of reconstructed scenes.
Revised: The performance of a pair of agents is judged based on the quality of reconstructed scenes.

Original: We consider high-quality reconstructions as a signal that communication has been successful.
Revised: We consider high-quality reconstructions as a signal that communication has been successful.

Original: As we develop models and protocols for CoDraw, we found it critical to train the Teller and the
Revised: As we develop models and protocols for CoDraw, we found it critical to train the Teller and the

Original: Drawer separately on disjoint subsets of the training data.
Revised: Drawer separately on disjoint subsets of the training data.

Original: Otherwise, the two machine agents may conspire to successfully achieve the goal while communicating using a shared “codebook” that bears little resemblance to natural language.
Revised: Otherwise, the two machine agents may conspire to successfully achieve the goal while communicating using a shared “codebook” that bears little resemblance to natural language.

Original: We call this separate-training, joint-evaluation protocol crosstalk , which prevents learning of mutually agreed upon codebooks, while still checking for goal completion at test time.
Revised: We call this separate-training, joint-evaluation protocol crosstalk , which prevents learning of mutually agreed upon codebooks, while still checking for goal completion at test time.

Original: We highlight crosstalk as one of our contributions, and believe it can be generally applicable to other related tasks (Sukhbaatar et al., 2016;
Revised: We highlight crosstalk as one of our contributions, and believe it can be generally applicable to other related tasks (Sukhbaatar et al., 2016;

Original: Foerster et al., 2016; de Vries et al., 2016; Das et al., 2017b; Lewis et al., 2017).
Revised: Foerster et al., 2016; de Vries et al., 2016; Das et al., 2017b; Lewis et al., 2017).

Original: Summary of Contributions
Revised: Summary of Contributions

Original: • We propose a novel CoDraw task, which is a game designed to facilitate the learning and evaluation of effective natural language communication in a grounded context.
Revised: • We propose a novel CoDraw task, which is a game designed to facilitate the learning and evaluation of effective natural language communication in a grounded context.

Original: • We collect a CoDraw dataset of ∼ 10K variable-length dialogs consisting of ∼ 138K messages with the drawing history at each step of the dialog.
Revised: • We collect a CoDraw dataset of ∼ 10K variable-length dialogs consisting of ∼ 138K messages with the drawing history at each step of the dialog.

Original: • We deﬁne a scene similarity metric, which allows us to automatically evaluate the effectiveness of the communication at the end and at intermediate states.
Revised: • We deﬁne a scene similarity metric, which allows us to automatically evaluate the effectiveness of the communication at the end and at intermediate states.

Original: • We propose a cross-talk training and evaluation protocol that prevents agents from potentially learning joint uninterpretable codebooks, rendering them ineffective at communicating with humans.
Revised: • We propose a cross-talk training and evaluation protocol that prevents agents from potentially learning joint uninterpretable codebooks, rendering them ineffective at communicating with humans.

Original: • We present and benchmark several Drawer and Teller models, both with fully automatic evaluation as well as by pairing them with live human agents.
Revised: • We evaluate several Drawer and Teller models automatically as well as by pairing them with humans, and show that long-term planning and context reasoning in the conversation are key challenges of the CoDraw task.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Language and Vision.
Revised: Language and Vision.

Original: The proposed CoDraw game is related to several well-known language and vision tasks that study grounded language understanding (Karpathy & Fei-Fei, 2015; Donahue et al., 2015; de Vries et al., 2016).
Revised: The proposed CoDraw game is related to several well-known language and vision tasks that study grounded language understanding (Karpathy & Fei-Fei, 2015; Donahue et al., 2015; de Vries et al., 2016).

Original: For instance, compared to image captioning (Vinyals et al., 2017; Xu et al., 2015; Chen & Zitnick, 2015; Lu et al., 2017), visual question answering (Antol et al., 2015; Zhang et al., 2016; Goyal et al., 2016; Gao et al., 2015; Krishna et al., 2017; Malinowski & Fritz, 2014; Ren et al., 2015; Tapaswi et al., 2016;
Revised: For instance, compared to image captioning (Vinyals et al., 2017; Xu et al., 2015; Chen & Zitnick, 2015; Lu et al., 2017), visual question answering (Antol et al., 2015; Zhang et al., 2016; Goyal et al., 2016; Gao et al., 2015; Krishna et al., 2017; Malinowski & Fritz, 2014; Ren et al., 2015; Tapaswi et al., 2016;

Original: Yu et al., 2015; Zhu et al., 2016) and recent embodied extensions (Das et al., 2018), CoDraw involves multiple rounds of interactions between two agents.
Revised: Yu et al., 2015; Zhu et al., 2016) and recent embodied extensions (Das et al., 2018), CoDraw involves multiple rounds of interactions between two agents.

Original: Both agents hold their own partially observable states and need to build a mental model for each other to collaborate.
Revised: Both agents hold their own partially observable states and need to build a mental model for each other to collaborate.

Revised: Compared to work on navigation (Vogel & Jurafsky, 2010; Anderson et al., 2018; Fried et al., 2018b) where an agent must follow instructions to move itself in a static environment, CoDraw involves moving and manipulating multiple clip art pieces, which must jointly form a semantically meaningful scene.

Original: Compared to visual dialog (Das et al., 2017a;b; Strub et al., 2017; Mostafazadeh et al., 2017) tasks, agents need to additionally cooperate to change the environment with actions ( e.g ., move around pieces).
Revised: Compared to visual dialog (Das et al., 2017a;b; Strub et al., 2017; Mostafazadeh et al., 2017) tasks, agents need to additionally cooperate to change the environment with actions ( e.g ., move around pieces).

Original: Thus, the agents have to possess the ability to adapt and hold a dialog about novel scenes that will be constructed as a consequence of their dialog.
Revised: Thus, the agents have to possess the ability to adapt and hold a dialog about novel scenes that will be constructed as a consequence of their dialog.

Original: In addition, we also want to highlight that CoDraw has a well-deﬁned communication goal, which facilitates objective measurement of success and enables end-to-end goal-driven learning.
Revised: In addition, we also want to highlight that CoDraw has a well-deﬁned communication goal, which facilitates objective measurement of success and enables end-to-end goal-driven learning.

Original: End-to-end Goal-Driven Dialog.
Revised: End-to-end Goal-Driven Dialog.

Original: Traditional goal-driven agents are often based on ‘slot ﬁlling’ (Lemon et al., 2006; Wang & Lemon, 2013; Yu et al., 2015), in which the structure of the dialog is pre-speciﬁed but the individual slots are replaced by relevant information.
Revised: Traditional goal-driven agents are often based on ‘slot ﬁlling’ (Lemon et al., 2006; Wang & Lemon, 2013; Yu et al., 2015), in which the structure of the dialog is pre-speciﬁed but the individual slots are replaced by relevant information.

Original: Recently, end-to-end neural models are also proposed for goal-driven dialog (Bordes et al., 2017; 
Revised: Recently, endto-end neural models are also proposed for goal-driven dialog (Bordes et al., 2017; Li et al., 2017a;b;

Original: Li et al., 2017a;b), as well as goal-free dialog or ‘chit-chat’ (Shang et al., 2015; Sordoni et al., 2015;Vinyals & Le, 2015;
Revised: He et al., 2017), as well as goal-free dialog or ‘chit-chat’ (Shang et al., 2015; Sordoni et al., 2015;

Original:  Li et al., 2016; Dodge et al., 2016).
Revised: Vinyals & Le, 2015; Li et al., 2016; Dodge et al., 2016).

Original: Unlike CoDraw, in these approaches, symbols in the dialog are not grounded into visual objects.
Revised: Unlike CoDraw, in these approaches, symbols in the dialog are not grounded into visual objects.

Original: Language Grounded in Environments.
Revised: Language Grounded in Environments.

Original: Learning language games to change the environment has been studied recently (Wang et al., 2016; 2017).
Revised: Learning language games to change the environment has been studied recently (Wang et al., 2016; 2017).

Original: The agent can change the environment using the grounded natural language.
Revised: The agent can change the environment using the grounded natural language.

Original: However, agents do not have the need to cooperate.
Revised: However, agents do not have the need to cooperate.

Revised: Other work on grounded instruction following relies on datasets of pre-generated action sequences annotated with human descriptions, rather than using a single end goal (Long et al., 2016).

Revised: Speaker models for these tasks are only evaluated based on their ability to describe an action sequence that is given to them (Fried et al., 2018a), whereas Teller models for CoDraw also need to select a desired action sequence in a goal-driven manner.

Original: Language grounding has also been studied for robot navigation, manipulation, and environment mapping (Tellex et al., 2011; Mei et al., 2015; Daniele et al., 2016).
Revised: Language grounding has also been studied for robot navigation, manipulation, and environment mapping (Tellex et al., 2011; Mei et al., 2015; Daniele et al., 2016).

Original: However, these works manually pair each command with robot actions and lack end-to-end training (Tellex et al., 2011), dialog (Mei et al., 2015; Daniele et al., 2016), or both (Walter et al., 2014).
Revised: However, these works manually pair each command with robot actions and lack end-to-end training (Tellex et al., 2011), dialog (Mei et al., 2015; Daniele et al., 2016), or both (Walter et al., 2014).

Original: Emergent Communication.
Revised: Emergent Communication.

Original: Building on the seminal works by Lewis (1969; 1975), a number of recent works study cooperative games between agents where communication protocols emerge as a consequence of training the agents to accomplish shared goals (Sukhbaatar et al., 2016;
Revised: Building on the seminal works by Lewis (1969; 1975), a number of recent works study cooperative games between agents where communication protocols emerge as a consequence of training the agents to accomplish shared goals (Sukhbaatar et al., 2016;

Original: Foerster et al., 2016).
Revised: Foerster et al., 2016).

Original: These methods have typically been applied to learn to communicate small amounts of information, rather than the complete, semantically meaningful scenes used in the CoDraw task.
Revised: These methods have typically been applied to learn to communicate small amounts of information, rather than the complete, semantically meaningful scenes used in the CoDraw task.

Original: In addition, the learned communication protocols are usually not natural (Kottur et al., 2017) or interpretable.
Revised: In addition, the learned communication protocols are usually not natural (Kottur et al., 2017) or interpretable.

Original: On the other hand, since our goal is to develop agents that can assist and communicate with humans, we must pre-train our agents on human communication and use techniques that can cope with the greater linguistic variety and richness of meaning present in natural language.
Revised: On the other hand, since our goal is to develop agents that can assist and communicate with humans, we must pre-train our agents on human communication and use techniques that can cope with the greater linguistic variety and richness of meaning present in natural language.

Original:  C O D RAW TASK AND DATASET
Revised:  C O D RAW TASK AND DATASET

Original: In this section, we ﬁrst detail our task, then present the CoDraw dataset, and ﬁnally propose a Scene
Revised: In this section, we ﬁrst detail our task, then present the CoDraw dataset, and ﬁnally propose a Scene

Original: Similarity Metric which allows automatic evaluation of the reconstructed and original scene.
Revised: Similarity Metric which allows automatic evaluation of the reconstructed and original scene.

Original: 3. T ASK
Revised: 3. T ASK

Original: Abstract Scenes.
Revised: Abstract Scenes.

Original: To enable people to easily draw semantically rich scenes on a canvas, we leverage the Abstract Scenes dataset of (Zitnick et al., 2013; Zitnick & Parikh, 2013).
Revised: To enable people to easily draw semantically rich scenes on a canvas, we leverage the Abstract Scenes dataset of (Zitnick et al., 2013; Zitnick & Parikh, 2013).

Original: This dataset consists of 10,020 semantically consistent scenes created by human annotators.
Revised: This dataset consists of 10,020 semantically consistent scenes created by human annotators.

Original: An example scene is shown in the left portion of Figure 1.
Revised: An example scene is shown in the left portion of Figure 1.

Original: Most scenes contain 6 objects (min 6, max 17, mean 6.67).
Revised: Most scenes contain 6 objects (min 6, max 17, mean 6.67).

Original: These scenes depict children playing in a park, and are made from a library of 58 clip arts, including a boy (Mike) and a girl (Jenny) in one of 7 poses and 5 expressions, and various other objects including trees, toys, hats, animals, food, etc.
Revised: These scenes depict children playing in a park, and are made from a library of 58 clip arts, including a boy (Mike) and a girl (Jenny) in one of 7 poses and 5 expressions, and various other objects including trees, toys, hats, animals, food, etc.

Original: An abstract scene is created by dragging and dropping multiple clip art objects to any ( x, y ) position on the canvas.
Revised: An abstract scene is created by dragging and dropping multiple clip art objects to any ( x, y ) position on the canvas.

Original: Also, for each clip art, different spatial transformations can be applied, including sizes (Small, Normal, Large), and two orientations (facing left or right).
Revised: Also, for each clip art, different spatial transformations can be applied, including sizes (Small, Normal, Large), and two orientations (facing left or right).

Original: The clip art serve simultaneously as a high-level visual representation and as a mechanism by which rich drawing actions can be carried out.
Revised: The clip art serve simultaneously as a high-level visual representation and as a mechanism by which rich drawing actions can be carried out.

Original: Interface.
Revised: Interface.

Original: We built a drag-and-drop interface based on the Visual Dialog chat interface (Das et al., 2017a)
Revised: We built a drag-and-drop interface based on the Visual Dialog chat interface (Das et al., 2017a)

Original: (see Figures3 and 4  in Appendix A for screen shots of the interface).
Revised: (see Figures 4 and 5 in Appendix A for screen shots of the interface).

Original: The interface allows real-time interaction between two people.
Revised: The interface allows real-time interaction between two people.

Original: During the conversation, the Teller describes the scene and answers any questions from the Drawer on the chat interface, while Drawer “draws” or reconstructs the scene based on the Teller’s descriptions and instructions.
Revised: During the conversation, the Teller describes the scene and answers any questions from the Drawer on the chat interface, while Drawer “draws” or reconstructs the scene based on the Teller’s descriptions and instructions.

Original: Each side is only allowed to send one message at a time, and must wait for a reply before continuing.
Revised: Each side is only allowed to send one message at a time, and must wait for a reply before continuing.

Original: The maximum length of a single message is capped at 140 characters: this prevents excessively verbose descriptions and gives the Drawer more chances to participate in the dialog by encouraging the Teller to pause more frequently.
Revised: The maximum length of a single message is capped at 140 characters: this prevents excessively verbose descriptions and gives the Drawer more chances to participate in the dialog by encouraging the Teller to pause more frequently.

Original: Both participants were asked to submit the task when they are both conﬁdent that Drawer has accurately reconstructed the scene of Teller.
Revised: Both participants were asked to submit the task when they are both conﬁdent that Drawer has accurately reconstructed the scene of Teller.

Original: Our dataset, as well as this infrastructure for live chat with live drawing, will be made publicly available.
Revised: Our dataset, as well as this infrastructure for live chat with live drawing, will be made publicly available.

Original: Additional Interaction.
Revised: Additional Interaction.

Original: We did not allow Teller to continuously observe Drawer’s canvas to make sure that the natural language focused on the high-level semantics of the scene rather than instructions calling for the execution of low-level clip art manipulation actions, but we hypothesize that direct visual feedback may be necessary to get the all the details right.
Revised: We did not allow Teller to continuously observe Drawer’s canvas to make sure that the natural language focused on the high-level semantics of the scene rather than instructions calling for the execution of low-level clip art manipulation actions, but we hypothesize that direct visual feedback may be necessary to get the all the details right.

Original: For this, we give one chance for the Teller to look at the Drawer’s canvas using a ‘peek’ button in the interface.
Revised: For this, we give one chance for the Teller to look at the Drawer’s canvas using a ‘peek’ button in the interface.

Original: Communication is only allowed after the peek window is closed.
Revised: Communication is only allowed after the peek window is closed.

Original: 3. D ATASET
Revised: 3. D ATASET

Original: We collect 9,993 1 dialogs, one  per scene, consisting of a total of 138K utterances.
Revised: We collect 9,993 1 dialogs where pairs of people complete the CoDraw task, consisting of one dialog per scene in the Abstract Scenes dataset. The dialogs contain of a total of 138K utterances and include snapshots of the intermediate state of the Drawer’s canvas after each round of each conversation. We reserve 10% of the scenes (1,002) to form a test set and an additional 10% (1,002) to form a development set; the corresponding dialogs are used to evaluate human performance for the CoDraw task. The remaining dialogs are used for training (see Section 5 for details about our training and evaluation setup.)

Original: The message length distribution for the Drawer is skewed toward 1 with the passive replies like “ok” , “done” , etc.
Revised: The message length distribution for the Drawer is skewed toward 1 with the passive replies like “ok” , “done” , etc.

Original: There does exist a heavy tail, which shows that Drawers do ask clarifying questions about the scene like “where is trunk of second tree, low or high” .
Revised: There does exist a heavy tail, which shows that Drawers do ask clarifying questions about the scene like “where is trunk of second tree, low or high” .

Original: On the other hand, the distribution of number of tokens in Tellers’ utterances is relatively smooth with long tails.
Revised: On the other hand, the distribution of number of tokens in Tellers’ utterances is relatively smooth with long tails.

Original: The vocabulary size is 4,555.
Revised: The vocabulary size is 4,555.

Original: Since the subject of conversations is about abstract scenes with a limited number of clip arts, the vocabulary is relatively small compared to those on real images.
Revised: Since the subject of conversations is about abstract scenes with a limited number of clip arts, the vocabulary is relatively small compared to those on real images.

Original: See Appendix B for a more detailed analysis of our dataset, where we study the lengths of the conversations, the number of rounds, and the distributions of scene similarity scores when humans perform the task.
Revised: See Appendix B for a more detailed analysis of our dataset, where we study the lengths of the conversations, the number of rounds, and the distributions of scene similarity scores when humans perform the task.

Original: 3. S CENE S IMILARITY M ETRIC
Revised: 3. S CENE S IMILARITY M ETRIC

Original: The goal-driven nature of our task naturally lends itself to evaluation by measuring the similarity of the reconstructed scene to the original.
Revised: The goal-driven nature of our task naturally lends itself to evaluation by measuring the similarity of the reconstructed scene to the original.

Original: For this purpose we deﬁne a scene similarity metric, which allows us to automatically evaluate communication effectiveness both at the end of a dialog and at intermediate states.
Revised: For this purpose we deﬁne a scene similarity metric, which allows us to automatically evaluate communication effectiveness both at the end of a dialog and at intermediate states.

Original: We use the metric to compare how well different machine-machine, humanmachine, and human-human pairs can complete the CoDraw task.
Revised: We use the metric to compare how well different machine-machine, humanmachine, and human-human pairs can complete the CoDraw task.

Original: Let c i , c
Revised: Let c i , c

Original: j denote the identity, location, conﬁguration of two clipart pieces i and j .
Revised: j denote the identity, location, conﬁguration of two clipart pieces i and j .

Original: A clipart image
Revised: A clipart image

Original: C =
Revised: C =

Original: { c i } is then simply a set of clipart pieces.
Revised: { c i } is then simply a set of clipart pieces.

Original: Given two images C and ˆ C , we compute scene similarity by ﬁrst ﬁnding the common clipart
Revised: Given two images C and ˆ C , we compute scene similarity by ﬁrst ﬁnding the common clipart

Original: pieces C ∩ ˆ C and then computing unary f
Revised: pieces C ∩ ˆ C and then computing unary f

Original: ( c i ) and pairwise terms g
Revised: ( c i ) and pairwise terms g

Original: ( c i , c j ) on these pieces in common:
Revised: ( c i , c j ) on these pieces in common:

Original: [Equation]
Revised: [Equation]

Original: Using f ( c ) = 1 and g ( c i , c j ) = 0 would result in the standard intersection-over-union measure used for scoring set predictions.
Revised: Using f ( c ) = 1 and g ( c i , c j ) = 0 would result in the standard intersection-over-union measure used for scoring set predictions.

Original: The denominator terms normalize the metric to penalize missing or extra clip art, and we set f and g such that our metric is on a 0-5 scale.
Revised: The denominator terms normalize the metric to penalize missing or extra clip art, and we set f and g such that our metric is on a 0-5 scale.

Original: The exact terms f and g are described in Appendix C.
Revised: The exact terms f and g are described in Appendix C.

Original:  M ODELS
Revised:  M ODELS

Original: We model both the Teller and the Drawer, and evaluate the agents using the metrics described in the previous section.
Revised: We model both the Teller and the Drawer, and evaluate the agents using the metrics described in the previous section.

Original: Informed by our analysis of the collected dataset (see Appendix B), we make three modeling assumptions compared to the full generality of the setup that humans were presented with during data collection.
Revised: Informed by our analysis of the collected dataset (see Appendix B), we make three modeling assumptions compared to the full generality of the setup that humans were presented with during data collection.

Original: These assumptions hold for all models studied in this paper.
Revised: These assumptions hold for all models studied in this paper.

Original: Assumption 1: Silent Drawer.
Revised: Assumption 1: Silent Drawer.

Original: We choose to omit the Drawer’s ability to ask clariﬁcation questions; instead, our Drawer models will always answer “ok” and our Teller models will not condition on the text of the Drawer replies.
Revised: We choose to omit the Drawer’s ability to ask clariﬁcation questions; instead, our Drawer models will always answer “ok” and our Teller models will not condition on the text of the Drawer replies.

Original: This is consistent with typical human replies (around 62% of which only use a single token) and the fact that the Drawer talking is not strictly required to resolve the information asymmetry inherent in the task.
Revised: This is consistent with typical human replies (around 62% of which only use a single token) and the fact that the Drawer talking is not strictly required to resolve the information asymmetry inherent in the task.

Original: We note that this assumption does not reduce the number of modalities needed to solve the task: there is still language generation on the Teller side, in addition to language understanding, scene perception, and scene generation on the Drawer side.
Revised: We note that this assumption does not reduce the number of modalities needed to solve the task: there is still language generation on the Teller side, in addition to language understanding, scene perception, and scene generation on the Drawer side.

Original: Drawer models that can detect when a clariﬁcation is required, and then generate a natural language clariﬁcation question is interesting future work.
Revised: Drawer models that can detect when a clariﬁcation is required, and then generate a natural language clariﬁcation question is interesting future work.

Original: Assumption 2: No Peek Action.
Revised: Assumption 2: No Peek Action.

Original: The second difference is that the data collection process for humans gives the Teller a single chance to peek at the Drawer’s canvas, a behavior we omit from our models.
Revised: The second difference is that the data collection process for humans gives the Teller a single chance to peek at the Drawer’s canvas, a behavior we omit from our models.

Original: Rich communication is still required without this behavior, and omitting it also does not decrease the number of modalities needed to complete the task.
Revised: Rich communication is still required without this behavior, and omitting it also does not decrease the number of modalities needed to complete the task.

Original: We leave for future work the creation of models that can peek at the time that maximizes task effectiveness.
Revised: We leave for future work the creation of models that can peek at the time that maximizes task effectiveness.

Original: Assumption 3: Full Clip Art Library.
Revised: Assumption 3: Full Clip Art Library.

Original: The ﬁnal difference is that our drawer models can select from the full clip art library.
Revised: The ﬁnal difference is that our drawer models can select from the full clip art library.

Original: Humans are only given access to a smaller set so that it can easily ﬁt in the user interface, while ensuring that all pieces needed to reconstruct the target scene are available.
Revised: Humans are only given access to a smaller set so that it can easily ﬁt in the user interface, while ensuring that all pieces needed to reconstruct the target scene are available.

Original: We choose to adopt the full-library condition as the standard for models because it gives the models greater latitude to make mistakes (making the problem more challenging) and makes it easier to detect obviously incorrect groundings.
Revised: We choose to adopt the full-library condition as the standard for models because it gives the models greater latitude to make mistakes (making the problem more challenging) and makes it easier to detect obviously incorrect groundings.

Original: 4. B ASELINES
Revised: 4. R ULE -B ASED N EAREST -N EIGHBOR M ETHODS

Original: Simple methods can be quite effective even for what appear to be challenging tasks, so we begin by building models based on nearest-neighbors and rule-based approaches.
Revised: Simple methods can be quite effective even for what appear to be challenging tasks, so we begin by building models based on nearest-neighbors and rule-based approaches.

Original: We use the following models as our baselines:

Original: Scene Nearest-Neighbor Teller.

Original: As a simple baseline, when the Teller is presented with the novel scene it can use the scene similarity metric to retrieve a nearest-neighbor scene from the training data.

Original: The agent then copies verbatim the human utterances associated with that nearest-neighbor scene, making sure to pause between utterances to give the Drawer a chance to act.

Original: Following our assumptions above, Drawer replies are ignored.

Original: Rule-based Nearest-Neighbor Teller.
Revised: Rule-based Nearest-Neighbor Teller.

Original: For a more ﬁne-grained baseline, we consider a rule-based dialog policy where the Teller describes exactly one clip art each time it talks.
Revised: For our ﬁrst Teller model, We consider a rule-based dialog policy where the Teller describes exactly one clip art each time it talks.

Original: The rule-based system determines which clip art to describe during each round of conversation, following a ﬁxed order that roughly starts with objects in the sky (sun, clouds, airplanes), then objects in the scene (trees, Mike,
Revised: The rule-based system determines which clip art to describe during each round of conversation, following a ﬁxed order that roughly starts with objects in the sky (sun, clouds, airplanes), then objects in the scene (trees, Mike,

Original: Jenny), and ends with small objects (sunglasses, baseball bat).
Revised: Jenny), and ends with small objects (sunglasses, baseball bat).

Original: Having selected a clip art, the Teller then searches the training data for a Drawer action that most closely resembles the selected clip art, as measured by applying the scene similarity metric to individual clip art.
Revised: The Teller then produces an utterance by performing a nearest-neighbor lookup in a database containing ( Teller utterance , clip art object ) pairs, where the similarity between the selected clip art and each database element is measured by applying the scene similarity metric to individual clip art.

Original: Only Drawer actions that modify a single clip art  are considered.
Revised: The database is constructed by collecting all instances in the training data where the Teller sent a message and the Drawer responded by adding a single clip art piece to the canvas. Instances where the Drawer added multiple clip art pieces or made any changes to the position or other attributes of pieces already on the canvas are not used when constructing the nearest-neighbor database.

Original: Finally, the system assumes that the selected draw action was elicited by the Teller utterance immediately prior, so it copies that Teller utterance as its output.
Revised: This baseline approach is based on the assumptions that the Drawer’s action was elicited by the Teller utterance immediately prior, and that the Teller’s utterance will have a similar meaning when copied verbatim into a new conversation.

Original: Rule-based Nearest-Neighbor Drawer.
Revised: Rule-based Nearest-Neighbor Drawer.

Original: This Drawer model is the complement to the rule-based nearest-neighbor Teller; it likewise follows a hard-coded rule that the response to each Teller utterance should be the addition of a single clip art to the scene. The training data is ﬁrst split into (Teller utterance, Drawer action) tuples, where only Drawer actions that add a single clip art are retained.
Revised: This Drawer model is the complement to the rule-based nearest-neighbor Teller. It likewise follows a hard-coded rule that the response to each Teller utterance should be the addition of a single clip art to the scene, and makes use of the same database of ( Teller utterance , clip art object ) tuples collected from the training data.

Original: Each Teller utterance the agent receives is compared with the stored tuples using character-level string edit distance.
Revised: Each Teller utterance the agent receives is compared with the stored tuples using character-level string edit distance.

Original: The action from the most similar tuple is selected as the Drawer output.
Revised: The clip art object from the most similar tuple is selected and added to the canvas by the Drawer.

Original: 4. N EURAL D RAWER
Revised: 4. N EURAL D RAWER

Original: In this section, we describe a neural network approach to the Drawer.
Revised: In this section, we describe a neural network approach to the Drawer.

Original: Contextual reasoning is an important part of the CoDraw task: each message from the Teller can relate back to what the
Revised: Contextual reasoning is an important part of the CoDraw task: each message from the Teller can relate back to what the

Original: Drawer has previously heard or drawn, and the clip art pieces it places on the canvas must form a semantically coherent scene.
Revised: Drawer has previously heard or drawn, and the clip art pieces it places on the canvas must form a semantically coherent scene.

Original: To capture these effects, our model should condition on the past history of the conversation and use an action representation that is conducive to generating coherent scenes. 
Revised: To capture these effects, our model should condition on the past history of the conversation and use an action representation that is conducive to generating coherent scenes. [Figure]

Original: When considering past history, we make the Markovian assumption that the current state of the
Revised: When considering past history, we make the Markovian assumption that the current state of the

Original: Drawer’s canvas captures all information from the previous rounds of dialog.
Revised: Drawer’s canvas captures all information from the previous rounds of dialog.

Original: Thus, the Drawer need only consider the most recent utterance from the Teller and the current canvas to decide what to draw next.
Revised: Thus, the Drawer need only consider the most recent utterance from the Teller and the current canvas to decide what to draw next.

Original: We experimented with incorporating additional context – such as previous messages from the teller or the action sequence by which the Drawer arrived at its current canvas conﬁguration – but did not observe any gains in performance.
Revised: We experimented with incorporating additional context – such as previous messages from the teller or the action sequence by which the Drawer arrived at its current canvas conﬁguration – but did not observe any gains in performance.

Original: The current state of the canvas is represented using a collection of indicator features and real-valued features.
Revised: The current state of the canvas is represented using a collection of indicator features and real-valued features.

Original: For each of the n c = 58 clip art types, there is an indicator feature for its presence on the canvas, and an indicator feature for each discrete assignment of an attribute to the clip art (e.g.
Revised: For each of the n c = 58 clip art types, there is an indicator feature for its presence on the canvas, and an indicator feature for each discrete assignment of an attribute to the clip art (e.g.

Original: 1 size = small , 1 size = medium , etc.)
Revised: 1 size = small , 1 size = medium , etc.)

Original: for a total of n b = 41 binary features.
Revised: for a total of n b = 41 binary features.

Original: There are additionally two real-valued features that encode the x
Revised: There are additionally two real-valued features that encode the x

Original: and y position of the clip art on the canvas, normalized to the [Figure] Attend Attend
Revised: and y position of the clip art on the canvas, normalized to the 0-1 range.

Original: Attend [Figure] 0-1 range. The resulting canvas representation is a feature vector v canvas of size n c × ( n b +2) , where all features for absent clip art types are set to zero.
Revised: The resulting canvas representation is a feature vector v canvas of size n c × ( n b +2) , where all features for absent clip art types are set to zero.

Original: We run a bi-directional LSTM over the Teller’s most recent message and extract the ﬁnal hidden states for both directions, which we concatenate to form a vector v message .
Revised: We run a bi-directional LSTM over the Teller’s most recent message and extract the ﬁnal hidden states for both directions, which we concatenate to form a vector v message .

Original: The Drawer is then a feedforward neural network that takes as input v canvas and v message and produces an output vector v action .
Revised: The Drawer is then a feedforward neural network that takes as input v canvas and v message and produces an output vector v action .

Original: The action representation v action also consists of n c × ( n b + 2) elements and can be thought of as a continuous relaxation of the mostly-discrete canvas encoding.
Revised: The action representation v action also consists of n c × ( n b + 2) elements and can be thought of as a continuous relaxation of the mostly-discrete canvas encoding.

Original: For each clip art type, there is a realvalued score that determines whether a clip art piece of that type should be added to the canvas: a positive score indicates that it should be added as part of the action.
Revised: For each clip art type, there is a realvalued score that determines whether a clip art piece of that type should be added to the canvas: a positive score indicates that it should be added as part of the action.

Original: During training, a binary crossentropy loss compares these scores with the actions taken by human drawers.
Revised: During training, a binary crossentropy loss compares these scores with the actions taken by human drawers.

Original: v action also contains unnormalized log-probabilities for each attribute-value assignment (e.g. z size = small , z size = medium , etc.
Revised: v action also contains unnormalized log-probabilities for each attribute-value assignment (e.g. z size = small , z size = medium , etc.

Original: for each clip art type); when a clip art piece is added to the canvas, its attributes are assigned to their most-probable values.
Revised: for each clip art type); when a clip art piece is added to the canvas, its attributes are assigned to their most-probable values.

Original: The log-probabilities are trained using softmax losses.
Revised: The log-probabilities are trained using softmax losses.

Original: Finally, v action contains two entries for each clip art type that determine the clip art’s x, y position if added to the canvas; these elements are trained using an L 2 loss.
Revised: Finally, v action contains two entries for each clip art type that determine the clip art’s x, y position if added to the canvas; these elements are trained using an L 2 loss.

Original: 4. N EURAL T ELLER : SCENE 2 SEQ
Revised: 4. N EURAL T ELLER : SCENE 2 SEQ

Original: For our neural Teller models, we adopt an architecture that we call scene2seq .
Revised: For our neural Teller models, we adopt an architecture that we call scene2seq .

Original: This architecture is a conditional language model over the Teller’s side of the conversation with special next-utterance tokens to indicate when the Teller ends its current utterance and waits for a reply from the Drawer.
Revised: This architecture is a conditional language model over the Teller’s side of the conversation with special next-utterance tokens to indicate when the Teller ends its current utterance and waits for a reply from the Drawer.

Original: The language model is implemented using an LSTM, where information about the ground-truth scene is incorporated both before and after each LSTM cell through the use of an attention mechanism.
Revised: The language model is implemented using an LSTM, where information about the ground-truth scene is incorporated both before and after each LSTM cell through the use of an attention mechanism.

Original: Attention occurs over individual clip art pieces: each clip art object in the ground-truth scene is represented using a vector that is the sum of learned embeddings for different clip art attributes (e.g. e type=
Revised: Attention occurs over individual clip art pieces: each clip art object in the ground-truth scene is represented using a vector that is the sum of learned embeddings for different clip art attributes (e.g. e type=

Original: Mike , e size=small , etc.)
Revised: Mike , e size=small , etc.)

Original: At test time, the Teller’s messages are constructed by decoding from the language model using greedy word selection.
Revised: At test time, the Teller’s messages are constructed by decoding from the language model using greedy word selection.

Original: To communicate effectively, the Teller must keep track of which parts of the scene it has and has not described, and also generate language that is likely to accomplish the task objective when interpreted by the Drawer.
Revised: To communicate effectively, the Teller must keep track of which parts of the scene it has and has not described, and also generate language that is likely to accomplish the task objective when interpreted by the Drawer.

Original: We found that training the scene2seq model using a maximum likelihood objective did not result in long-term coherent dialogs for novel scenes.
Revised: We found that training the scene2seq model using a maximum likelihood objective did not result in long-term coherent dialogs for novel scenes.

Original: Rather than introducing a new architecture to address these deﬁciencies, we explore reducing them by using alternative training objectives.
Revised: Rather than introducing a new architecture to address these deﬁciencies, we explore reducing them by using alternative training objectives.

Original: To better ensure that the model keeps track of which pieces of information it has already communicated, we take advantage of the availability of drawings at each round of the recorded human dialogs and introduce an auxiliary loss based on predicting these drawings.
Revised: To better ensure that the model keeps track of which pieces of information it has already communicated, we take advantage of the availability of drawings at each round of the recorded human dialogs and introduce an auxiliary loss based on predicting these drawings.

Original: To select language that is more likely to lead to successful task completion, we further ﬁne-tune our Teller models to directly optimize the end-task goal using reinforcement learning.
Revised: To select language that is more likely to lead to successful task completion, we further ﬁne-tune our Teller models to directly optimize the end-task goal using reinforcement learning.

Original: 4.3. T RAINING WITH INTERMEDIATE SUPERVISION
Revised: 4.3. T RAINING WITH INTERMEDIATE SUPERVISION

Original: We incorporate state tracking into the scene2seq architecture through the use of an auxiliary loss.
Revised: We incorporate state tracking into the scene2seq architecture through the use of an auxiliary loss.

Original: This formulation maintains the end-to-end training procedure and keeps test-time decoding exactly the same.
Revised: This formulation maintains the end-to-end training procedure and keeps test-time decoding exactly the same.

Original: The only change is that, at each utterance separator token, the output from the LSTM is used to predict which clip art still need to be described.
Revised: The only change is that, at each utterance separator token, the output from the LSTM is used to predict which clip art still need to be described.

Original: More precisely, the network must classify whether each clip art in the ground truth has been drawn already or not.
Revised: More precisely, the network must classify whether each clip art in the ground truth has been drawn already or not.

Original: The supervisory signal makes use of the fact that the CoDraw dataset records human drawer actions at each round of the conversation, not just at the end.
Revised: The supervisory signal makes use of the fact that the CoDraw dataset records human drawer actions at each round of the conversation, not just at the end.

Original: The network outputs a score for each clip art ID, which is connected to a softmax loss for the clip art in the ground truth scene (the scores for absent clip arts do not contribute to the auxiliary loss).
Revised: The network outputs a score for each clip art ID, which is connected to a softmax loss for the clip art in the ground truth scene (the scores for absent clip arts do not contribute to the auxiliary loss).

Original: We ﬁnd that adding such a supervisory signal reduces the Teller’s propensity for repeating itself or omitting objects.
Revised: We ﬁnd that adding such a supervisory signal reduces the Teller’s propensity for repeating itself or omitting objects.

Original: 4.3. F INE - TUNING WITH REINFORCEMENT LEARNING
Revised: 4.3. F INE - TUNING WITH REINFORCEMENT LEARNING

Original: The auxiliary loss helps the agent be more coherent throughout the dialog, but it is still trained to imitate human behavior rather than to complete the downstream task.
Revised: The auxiliary loss helps the agent be more coherent throughout the dialog, but it is still trained to imitate human behavior rather than to complete the downstream task.

Original: By training the agents using reinforcement learning (RL), they can learn to use language that is more effective at achieving highﬁdelity scene reconstructions.
Revised: By training the agents using reinforcement learning (RL), they can learn to use language that is more effective at achieving highﬁdelity scene reconstructions.

Original: In this work we only train the Teller with RL, because the Teller has challenges maintaining a long-term strategy throughout a long dialog, whereas preliminary results showed that making local decisions is less detrimental for Drawers.
Revised: In this work we only train the Teller with RL, because the Teller has challenges maintaining a long-term strategy throughout a long dialog, whereas preliminary results showed that making local decisions is less detrimental for Drawers.

Original: The scene2seq Teller architecture remains unchanged, and each action from the agent is to output a word or one of two special tokens: a next-utterance token and a stop token.
Revised: The scene2seq Teller architecture remains unchanged, and each action from the agent is to output a word or one of two special tokens: a next-utterance token and a stop token.

Original: After each next-utterance token, our neural Drawer model is used to take an action in the scene and the resulting change in scene similarity metric is used as a reward.
Revised: After each next-utterance token, our neural Drawer model is used to take an action in the scene and the resulting change in scene similarity metric is used as a reward.

Original: However, this reward scheme alone has an issue: once all objects in the scene are described, any further messages will not result in a change in the scene and have a reward of zero.
Revised: However, this reward scheme alone has an issue: once all objects in the scene are described, any further messages will not result in a change in the scene and have a reward of zero.

Original: As a result, there is no incentive to end the conversation.
Revised: As a result, there is no incentive to end the conversation.

Original: We address this by applying a penalty of 0.3 to the reward whenever the Drawer makes no changes to the scene.
Revised: We address this by applying a penalty of 0.3 to the reward whenever the Drawer makes no changes to the scene.

Original: We train our model with REINFORCE (Williams, 1992).
Revised: We train our model with REINFORCE (Williams, 1992).

Original:  T RAINING PROTOCOL AND EVALUATION
Revised:  T RAINING PROTOCOL AND EVALUATION

Original: To evaluate our models, we pair our models with other models, as well as with a human.
Revised: To evaluate our models, we pair our models with other models, as well as with a human.

Original: Human-Machine Pairs.
Revised: Human-Machine Pairs.

Original: We modiﬁed the interface used for data collection to allow human-machine pairs to complete the tasks.
Revised: We modiﬁed the interface used for data collection to allow human-machine pairs to complete the tasks.

Original: Each model plays one game with a human per scene in the test set, and we compare the scene reconstruction quality between different models and with human-human pairs.
Revised: Each model plays one game with a human per scene in the test set, and we compare the scene reconstruction quality between different models and with human-human pairs.

Original: Script-based Drawer Evaluation.
Revised: Script-based Drawer Evaluation.

Original: In addition to human evaluation, we would like to have automated evaluation protocols that can quickly estimate the quality of different models.
Revised: In addition to human evaluation, we would like to have automated evaluation protocols that can quickly estimate the quality of different models.

Original: Drawer models can be evaluated by pairing them with a Teller that replays recorded human conversation from a script (a recorded dialog from the dataset) and measuring scene similarity at the end of the dialog.
Revised: Drawer models can be evaluated by pairing them with a Teller that replays recorded human conversation from a script (a recorded dialog from the dataset) and measuring scene similarity at the end of the dia- Ground Truth Scene similarity: 4. similarity: 4.

Original: While this setup does not capture the full interactive nature of the task, the Drawer model still receives human descriptions of the scene and should be able to reconstruct it.
Revised: Human Reconstruction Machine Reconstruction Teller: to the right of swing set is big table . girl in front with hands out , not smiling burger on ground in frontof her

Revised: Round 3 of Machine Conversation

Revised: Drawer:

Revised: Teller: crown is tilted down to right not straight on head .

Revised: and i am going to peek also

Revised: Round 8 of Machine Conversation

Revised: Drawer: log.

Revised: While this setup does not capture the full interactive nature of the task, the Drawer model still receives human descriptions of the scene and should be able to reconstruct it.

Original: Our modeling assumptions include not giving Drawer models the ability to ask clarifying questions, which further suggests that script-based evaluation can reasonably measure model quality.
Revised: Our modeling assumptions include not giving Drawer models the ability to ask clarifying questions, which further suggests that script-based evaluation can reasonably measure model quality.

Original: Machine-Machine Evaluation.
Revised: Machine-Machine Evaluation.

Original: Unlike Drawer models, Teller models cannot be evaluated using a “script” from the dataset.
Revised: Unlike Drawer models, Teller models cannot be evaluated using a “script” from the dataset.

Original: We instead consider an evaluation where a Teller model and Drawer model are paired, and their joint performance is evaluated using the scene similarity metric.[Table]
Revised: We instead consider an evaluation where a Teller model and Drawer model are paired, and their joint performance is evaluated using the scene similarity metric.

Original: 5. C ROSSTALK T RAINING P ROTOCOL
Revised: 5. C ROSSTALK T RAINING P ROTOCOL

Original: Automatically evaluating agents, especially in the machine-machine paired setting, requires some care because a pair of agents can achieve a perfect score while communicating in a shared code that bears no resemblance to natural language.
Revised: Automatically evaluating agents, especially in the machine-machine paired setting, requires some care because a pair of agents can achieve a perfect score while communicating in a shared code that bears no resemblance to natural language.

Original: There are several ways such co-adaptation can develop.
Revised: There are several ways such co-adaptation can develop.

Original: One is by overﬁtting to the training data to the extent that it’s used as a codebook – we see this with the rule-based nearest-neighbor agents described in Section 4. [Table] where a Drawer-Teller pair “trained” on the same data outperforms humans on the CoDraw task.
Revised: One is by overﬁtting to the training data to the extent that it’s used as a codebook – we see this with the rule-based nearest-neighbor agents described in Section 4.1, where a Drawer-Teller pair “trained” on the same data outperforms humans on the CoDraw task.

Original: An examination of the language, however, reveals that only limited generalization has taken place.
Revised: An examination of the language, however, reveals that only limited generalization has taken place (see Figure 3).

Original: Another way that agents can co-adapt is if they are trained jointly, for example using reinforcement learning.
Revised: Another way that agents can co-adapt is if they are trained jointly, for example using reinforcement learning.

Original: To limit these sources of co-adaptation, we propose a training protocol we call “crosstalk.”
Revised: To limit these sources of co-adaptation, we propose a training protocol we call “crosstalk.”

Original: In this setting, the training data is split in half [Table] and the Teller and Drawer are trained separately on disjoint halves of the training data.
Revised: In this setting, the training data is split in half, and the Teller and Drawer are trained separately on disjoint halves of the training data.

Original: When multiple agents are required during training (as with reinforcement learning), the joint training process is run separately for both halves of the training data, but evaluation pairs a Teller from the ﬁrst partition with a Drawer from the second.
Revised: When multiple agents are required during training (as with reinforcement learning), the joint training process is run separately for both halves of the training data, but evaluation pairs a Teller from the ﬁrst partition with a Drawer from the second.

Original: This ensures to some extent that the models can succeed only if they have learned generalizable language, and not via a highly specialized codebook speciﬁc to model instances.
Revised: This ensures to some extent that the models can succeed only if they have learned generalizable language, and not via a highly specialized codebook speciﬁc to model instances.

Original: We split the dataset into  40% teller-train (3,994), 40% drawer-train (3,995), 10% val (1,002) and 10% test (1,002).
Revised: Taking the crosstalk training protocol into account, the dataset split we use for all experiments is: 40% Teller training data (3,994 scenes/dialogs), 40% Drawer training data (3,995), 10% development data (1,002) and 10% testing data (1,002).

Original:  R ESULTS
Revised:  R ESULTS

Original: Results for our models are shown in Table 1.
Revised: Results for our models are shown in Table 1.

Original: All numbers are scene similarities, averaged across scenes in the test set.
Revised: All numbers are scene similarities, averaged across scenes in the test set.

Original: Neural Drawer Performs the Best.
Revised: Neural Drawer Performs the Best.

Original: In the script setting, our neural Drawer is able to outperform the rule-based nearest-neighbor baseline (3.34 vs. 0.96) and close most of the gap between baseline (0.96) and human performance (4.17).
Revised: In the script setting, our neural Drawer is able to outperform the rule-based nearest-neighbor baseline (3.39 vs. 0.94) and close most of the gap between baseline (0.94) and human performance (4.17).

Original: Validity of Script-Based Drawer Evaluation.
Revised: Validity of Script-Based Drawer Evaluation.

Original: To test the validity of script-based Drawer evaluation – where a Drawer is paired with a Teller that recites the human script from the dataset corresponding to the test scenes – we include results from interactively pairing human Drawers with a Teller that recites the scripted messages.
Revised: To test the validity of script-based Drawer evaluation – where a Drawer is paired with a Teller that recites the human script from the dataset corresponding to the test scenes – we include results from interactively pairing human Drawers with a Teller that recites the scripted messages.

Original: While average scene similarity is lower than when using live human
Revised: While average scene similarity is lower than when using live human

Original: Tellers (3.83 vs. 4.17), the scripts are sufﬁcient to achieve over 91% of the effectiveness of the  same Teller utterances when they were communicated live (according to our metric).
Revised: Tellers (3.83 vs. 4.17), the scripts are sufﬁcient to achieve over 91% of the effectiveness of the [Table] same Teller utterances when they were communicated live (according to our metric).

Original: The drop in similarity may be in part due to the inability of the Teller to peek at the Drawer’s canvas and suggest speciﬁc corrections, and in part because the Teller can’t answer clarifying questions speciﬁc to the
Revised: The drop in similarity may be in part due to the inability of the Teller to peek at the Drawer’s canvas and suggest speciﬁc corrections, and in part because the Teller can’t answer clarifying questions speciﬁc to the

Original: Drawer’s personal understanding of the instructions.
Revised: Drawer’s personal understanding of the instructions.

Original: Note that a human Drawer with a script-based
Revised: Note that a human Drawer with a script-based

Original: Teller still outperforms our best Drawer model paired with a script-based Teller.
Revised: Teller still outperforms our best Drawer model paired with a script-based Teller.

Original: Beneﬁts of Intermediate Supervision and Goal-Driven Training.
Revised: Beneﬁts of Intermediate Supervision and Goal-Driven Training.

Original: Pairing our models with humans shows that the scene2seq Teller model trained with imitation learning is worse than the rulebased nearest-neighbor baseline (2.69 vs. 3.21), but that the addition of an auxiliary loss followed by ﬁne-tuning with reinforcement learning allow it to outperform the baseline (3.65 vs. 3.21).
Revised: Pairing our models with humans shows that the scene2seq Teller model trained with imitation learning is worse than the rulebased nearest-neighbor baseline (2.69 vs. 3.21), but that the addition of an auxiliary loss followed by ﬁne-tuning with reinforcement learning allow it to outperform the baseline (3.65 vs. 3.21).

Original: However, there is still a gap between to human Tellers (3.65 vs. 4.17).
Revised: However, there is still a gap between to human Tellers (3.65 vs. 4.17).

Original: Many participants in our human study noted that they received unclear instructions from the models they were paired with, or expressed frustration that their partners could not answer clarifying questions as a way of resolving such situations.
Revised: Many participants in our human study noted that they received unclear instructions from the models they were paired with, or expressed frustration that their partners could not answer clarifying questions as a way of resolving such situations.

Original: Recall that our Teller models currently ignore any utterances from the Drawer.
Revised: Recall that our Teller models currently ignore any utterances from the Drawer.

Original: Correlation Between Fully-automated and Human-machine Evaluation.
Revised: Correlation Between Fully-automated and Human-machine Evaluation.

Original: We also report the result of paired evaluation for different Teller models and our best Drawer, showing that the relative rankings of the different Teller types match those we see when models are paired with humans.
Revised: We also report the result of paired evaluation for different Teller models and our best Drawer, showing that the relative rankings of the different Teller types match those we see when models are paired with humans.

Original: This shows that automated evaluation while following the crosstalk training protocol is a suitable automated proxy for human-evaluation. 
Revised: This shows that automated evaluation while following the crosstalk training protocol is a suitable automated proxy for human-evaluation. 6. T YPICAL E RRORS

Revised: The errors made by Teller reﬂect two key challenges posed by the CoDraw task: reasoning about the context of the conversation and drawing, and planning ahead to fully and effectively communicate the information required.

Revised: A common mistake the rule-based nearest-neighbor Teller makes is to reference objects that are not present in the current scene.

Original: C ONCLUSION
Revised: Figure 3 shows an example (second panel from the right) where the Teller has copied a message referencing a “swing” that does not exist in the current scene. In a sample of 5 scenes from the test set, the rule-based nearest-neighbor describes a non-existent object 11 times, compared to just 1 time for the scene2seq Teller trained with imitation learning. The scene2seq Teller, on the other hand, frequently describes clip art pieces multiple times or forgets to mention some of them: in the same sample of scenes, it re-describes an object 10 times (vs. 2 for the baseline) and fails to mention 11 objects (vs. 2.) The addition of an auxiliary loss and RL ﬁne-tuning reduces these classes of errors while avoiding frequent descriptions of irrelevant objects (0 references to non-existent objects, 3 instances of re-describing an object, and 4 objects omitted.)

Revised: On the Drawer side, the most salient class of mistakes made by the neural network model is semantically inconsistent placement of multiple clip art pieces.

Revised: Several instances of this can be seen in or where the clip pieces overlap in a visually unnatural

Revised: Qualitative examples of both human and model behavior are provided in Appendix D.

Original: In this paper, we introduce CoDraw: a collaborative task designed to facilitate learning of effective natural language communication in a grounded context.
Revised: In this paper we introduce CoDraw: a collaborative task designed to facilitate learning of effective natural language communication in a grounded context.

Original: The task combines language, perception, and actions while permitting automated goal-driven evaluation both at the end and as a measure of intermediate progress.
Revised: The task combines language, perception, and actions while permitting automated goal-driven evaluation both at the end and as a measure of intermediate progress.

Original: We introduce a dataset and models for this task, and propose a crosstalk training + evaluation protocol that is more generally applicable to studying emergent communication.
Revised: We introduce a dataset and models for this task and propose a crosstalk training + evaluation protocol that is more generally applicable to studying emergent communication.

Original: The models we present in this paper show levels of task performance that are still far from what humans can achieve.
Revised: The models we present in this paper show levels of task performance that are still far from what humans can achieve.

Original: Long-term planning and contextual reasoning as two key challenges for this task that our models only begin to address.
Revised: Long-term planning and contextual reasoning as two key challenges for this task that our models only begin to address.

Original: We hope that the grounded, goal-driven communication setting that CoDraw is a testbed for can lead to future progress in building agents that can speak more naturally and better maintain coherency over a long dialog, while being grounded in perception and actions.
Revised: We hope that the grounded, goal-driven communication setting that CoDraw is a testbed for can lead to future progress in building agents that can speak more naturally and better maintain coherency over a long dialog, while being grounded in perception and actions.

Original: A I NTERFACE AND DATA COLLECTION
Revised: A I NTERFACE AND DATA COLLECTION

Original: [Figure]
Revised: [Figure]

Original: I
Revised: I

Original: [Figure]
Revised: [Figure]

Original: Figure 3 shows the interface for the Teller
Revised: Figure 4 shows the interface for the Teller

Original: [Figure]
Revised: [Figure]

Original: and Figure 4 shows the interface for the Drawer.
Revised: and Figure 5 shows the interface for the Drawer.

Original: Following previous works (Zitnick et al., 2013; Zitnick & Parikh, 2013), Drawers are given 20 clip art objects selected randomly from the 58 clip art objects in the library, while ensuring that all objects required to reconstruct the scene are available.
Revised: Following previous works (Zitnick et al., 2013; Zitnick & Parikh, 2013), Drawers are given 20 clip art objects selected randomly from the 58 clip art objects in the library, while ensuring that all objects required to reconstruct the scene are available.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: button is disabled.
Revised: button is disabled.

Original: Onlythe Teller can use it.
Revised: Only Teller can use it.

Original: A. P ARTICIPANT STATISTICS
Revised: A. P ARTICIPANT STATISTICS

Original: We found that approximately 13.6% of human participants disconnect voluntarily in an early stage of the session.
Revised: We found that approximately 13.6% of human participants disconnect voluntarily in an early stage of the session.

Original: We paid participants who stayed in the conversation and had posted at least three messages.
Revised: We paid participants who stayed in the conversation and had posted at least three messages.

Original: However, we exclude those incomplete sessions in the dataset, and only use the completed sessions.
Revised: However, we exclude those incomplete sessions in the dataset, and only use the completed sessions.

Original: There are 616 unique participants represented in our collected data.
Revised: There are 616 unique participants represented in our collected data.

Original: Among these workers, themost active have done 26.63% of all ﬁnished tasks (1,419, 1,358, 1,112, 1,110, and 1,068 tasks).
Revised: Among these workers, themost active have done 26.63% of all ﬁnished tasks (1,419, 1,358, 1,112, 1,110, and 1,068 tasks).

Original: Across all workers, the maximum, median, and minimum numbers of tasks ﬁnished by a worker are 1,419, 3, and 1, respectively.
Revised: Across all workers, the maximum, median, and minimum numbers of tasks ﬁnished by a worker are 1,419, 3, and 1, respectively.

Original: *Collected 9,993 sessions as of Apr 

Revised: *Collected 9,993 sessions as of Apr

Original: T he nu m be r o f s e ss i on s .

Revised: T he nu m be r o f s e ss i on s .

Original: K .4K .

Revised: K .4K .

Original: K 1.2K 1.6K 2.0K
Revised: K 1.2K 1.6K 2.0K

Original: The number of rounds 1 4 7 10 13 16 19 22 25 28 

Revised: The number of rounds 1 4 7 10 13 16 19 22 25 28

Original: T he nu m be r
Revised: T he nu m be r

Original: o f m e ss age s 0K 2K 4K 6K 8K 10K
Revised: o f m e ss age s 0K 2K 4K 6K 8K 10K

Original: The number of tokens 1 4 7 10 13 16 19 22 25 28 31 

Revised: The number of tokens 1 4 7 10 13 16 19 22 25 28 31

Original: Drawer
Revised: Drawer

Original: Teller
Revised: Teller

Original: Median
Revised: Median

Original: Median 41,
Revised: Median 41,

Original: T he nu m be r o f s e ss i on
Revised: T he nu m be r o f s e ss i on

Original: [Figure]
Revised: [Figure]

Original: T he nu m be r o f s e ss i on s .

Revised: T he nu m be r o f s e ss i on s .

Original: K .4K .

Revised: K .4K .

Original: K 1.2K 1.6K 2.0K
Revised: K 1.2K 1.6K 2.0K

Original: Duration (minutes) 1 3 5 7 

Revised: Duration (minutes) 1 3 5 7

Original: Median
 Median
Revised: Median
 Median

Original: [Figure]
Revised: [Figure]

Original: B D ATASET STATISTICS
Revised: B D ATASET STATISTICS

Original: The CoDraw dataset consists of 9,993 dialogs consisting of a total of 138K utterances.
Revised: The CoDraw dataset consists of 9,993 dialogs consisting of a total of 138K utterances.

Original: Each dialog describes a distinct abstract scene.
Revised: Each dialog describes a distinct abstract scene.

Original: Messages.
Revised: Messages.

Original: Figure 5a shows the distribution of message lengths for both Drawers and Tellers.
Revised: Figure 6a shows the distribution of message lengths for both Drawers and Tellers.

Original: Drawer messages tend to be short (the median length is 1 accounts for 62% of messages), but there does exist a heavy tail where the Drawer asks clarifying questions about the scene.
Revised: Drawer messages tend to be short (the median length is 1 accounts for 62% of messages), but there does exist a heavy tail where the Drawer asks clarifying questions about the scene.

Original: Teller message length have a more smooth distribution with a median length of 16 tokens.
Revised: Teller message length have a more smooth distribution with a median length of 16 tokens.

Original: The size of vocabulary is 4,555: since conversations describe abstract scenes consisting of a limited number of clip art types, the vocabulary is relatively small compared to tasks involving real images.
Revised: The size of vocabulary is 4,555: since conversations describe abstract scenes consisting of a limited number of clip art types, the vocabulary is relatively small compared to tasks involving real images.

Original: Rounds.
Revised: Rounds.

Original: Figure 5b shows the distribution of the numbers of conversational rounds for dialog sessions.
Revised: Figure 6b shows the distribution of the numbers of conversational rounds for dialog sessions.

Original: Most interactions are shorter than 20 rounds, median being 7.
Revised: Most interactions are shorter than 20 rounds, median being 7.

Original: Durations.
Revised: Durations.

Original: In Figure 5c we see that the median session duration is 6 minutes.
Revised: In Figure 6c we see that the median session duration is 6 minutes.

Original: We had placed a 20-minute maximum limit on each session.
Revised: We had placed a 20-minute maximum limit on each session.

Original: Scores.
Revised: Scores.

Original: Figure 6 shows the distribution of scene similarity scores throughout the dataset.
Revised: Figure 7 shows the distribution of scene similarity scores throughout the dataset.

Original: Figureshows the progress of scene similarity scores over the rounds of a conversation.
Revised: Figureshows the progress of scene similarity scores over the rounds of a conversation.

Original: An average conversations is done improving the scene similarity after about 5 rounds, but for longer conversations that continue to 23 rounds, there is still room for improvement.
Revised: An average conversations is done improving the scene similarity after about 5 rounds, but for longer conversations that continue to 23 rounds, there is still room for improvement.

Original: C S CENE SIMILARITY METRIC
Revised: C S CENE SIMILARITY METRIC

Original: Given a ground-truth scene C and a predicted scene ˆ C (where the presence of a clip art type c in the scene C is indicated by c ∈ C ) scene similarity s is deﬁned as:
Revised: Given a ground-truth scene C and a predicted scene ˆ C (where the presence of a clip art type c in the scene C is indicated by c ∈ C ) scene similarity s is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here x c and y c refer to the position of the clip art piece in the ground-truth scene, ˆ x c and ˆ y c refer to its position in the predicted scene, and W, H are the width and height of the canvas, respectively.
Revised: Here x c and y c refer to the position of the clip art piece in the ground-truth scene, ˆ x c and ˆ y c refer to its position in the predicted scene, and W, H are the width and height of the canvas, respectively.

Original: We use parameters w =
Revised: We use parameters w =

Original: 5 , 1 , 1 , 1 , 1] , which provides a balance between the different components and ensures that scene similarities are constrained to be between 0 and 5.
Revised: 5 , 1 , 1 , 1 , 1] , which provides a balance between the different components and ensures that scene similarities are constrained to be between 0 and 5.

Original: D V ISUALIZATION
Revised: D Q UALITATIVE EXAMPLES

Original: Figure 8 shows some examples of scenes  from the CoDraw dataset.
Revised: Figure 9 shows some examples of scenes and dialogs from the CoDraw dataset.

Original: Figure 9 shows the behavior of ourtwo Drawer models when paired with a script-based Teller that recites human messages from the dataset.
Revised: The behavior of our

Original: Figure 10 shows the utterances generated by different Teller models for a randomly-sampled scene from the test set.
Revised: Drawer and Teller models on a few randomly-selected scenes is illustrated in Figures 10, 11, and 12. [Table] [Figure] [Table]

Revised: a s

Original: [Table]
Revised: [Table]

Revised: d

Revised: NN

Revised: [Table]

Revised: l

Revised: N

Revised: [Table]

Revised: k

Revised: [Table]

Revised: R u l e - B a s e d

Revised: NN

Revised: N e u r a l

Revised: N

Revised: [Figure]

Revised: t w o r k

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: R u l e - B a s e d NN N e u r a l N e t w o r k

Original: [Table]
Revised: [Table]

Original: Neural Network

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Similarity:

Original: [Table]
Revised: [Table]

Original: Similarity: 0.

Original: [Figure]
Revised: [Figure]

Original: Similarity: 1. Similarity: 1. [Figure] [Figure]
Revised: [Table]

Original: [Figure][Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Similarity: 1. Similarity: 2.
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Similarity: 0. Similarity: 2. Note: agentplacesthetoybehindthetree
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: she is surprised

Original: [Figure]
Revised: [Table]

Original: eyebrows

Original: [Figure]
Revised: [Figure]

Original: Similarity: 1.

Original: Similarity: 3.

Original: [Table] [Table]
Revised: [Table] [Figure]

Revised: communicating with

Original: [Figure]
Revised: Drawer). as in Figure 9, which were