This paper introduces the PhotoBook dataset , a large-scale collection of visually-grounded , task-oriented dialogues in English designed to investigate shared dialogue history accumulating during conversation . Taking inspiration from seminal work on dialogue analysis , we propose a data-collection task formulated as a collaborative game prompting two online participants to refer to images utilising both their visual context as well as previously established referring expressions . We provide a detailed description of the task setup and a thorough analysis of the 2,500 dialogues collected . To further illustrate the novel features of the dataset , we propose a baseline model for reference resolution which uses a simple method to take into account shared information accumulated in a reference chain . Our results show that this information is particularly important to resolve later descriptions and underline the need to develop more sophisticated models of common ground in dialogue interaction . 1 The past few years have seen an increasing interest in developing computational agents for visually grounded dialogue , the task of using natural language to communicate about visual content in a multi-agent setup . The models developed for this task often focus on specific aspects such as image labelling ( Mao et al . , 2016 ; Vedantam et al . , 2017 ) , object reference ( Kazemzadeh et al . , 2014 ; De Vries et al . , 2017a ) , visual question answering ( Antol et al . , 2015 ) , and first attempts of visual dialogue proper ( Das et al . , 2017 ) , but fail to produce consistent outputs over a conversation . We hypothesise that one of the main reasons for this shortcoming is the models ' inability to effectively utilise dialogue history . Human interlocutors are known to collaboratively establish a shared repository of mutual information during a conversation ( Clark and Wilkes-Gibbs , 1986 ; Clark , 1996 ; Brennan and Clark , 1996 ) . This common ground ( Stalnaker , 1978 ) then is used to optimise understanding and communication efficiency . Equipping artificial dialogue agents with a similar representation of dialogue context thus is a pivotal next step in improving the quality of their dialogue output . To facilitate progress towards more consistent and effective conversation models , we introduce the PhotoBook dataset : a large collection of 2,500 human-human goal-oriented English conversations between two participants , who are asked to identify shared images in their respective photo books by exchanging messages via written chat . This setup takes inspiration from experimental paradigms extensively used within the psycholinguistics literature to investigate partnerspecific common ground ( for an overview , see Brown-Schmidt et al . , 2015 ) , adapting them to the requirements imposed by online crowdsourcing methods . The task is formulated as a game consisting of five rounds . Figure 1 shows an example of a participant 's display . Over the five rounds of a game , a selection of previously displayed images will be visible again , prompting participants to re-refer to images utilising both their visual context as well as previously established referring expressions . The resulting dialogue data therefore allows for tracking the common ground developing between dialogue participants . We describe in detail the PhotoBook task and the data collection , and present a thorough analysis of the dialogues in the dataset . In addition , to showcase how the new dataset may be exploited for computational modelling , we propose a reference resolution baseline model trained to identify target images being discussed in a given dialogue segment . The model uses a simple method to take into account information accumulated in a reference chain . Our results show that this information is particularly important to resolve later descriptions and highlight the importance of developing more sophisticated models of common ground in dialogue interaction . The PhotoBook dataset , together with the data collection protocol , the automatically extracted reference chains , and the code used for our analyses and models are available at the following site : https://dmg-photobook.github.io . We have presented the first large-scale dataset of goal-oriented , visually grounded dialogues for investigating shared linguistic history . Through the data collection 's task setup , participants repeatedly refer to a controlled set of target images , which allows them to improve task efficiency if they utilise their developing common ground and establish conceptual pacts ( Brennan and Clark , 1996 ) on referring expressions . The collected dialogues exhibit a significant shortening of utterances throughout a game , with final referring expressions starkly differing from both standard image captions and initial descriptions . To illustrate the potential of the dataset , we trained a baseline reference resolution model and showed that information accumulated over a reference chain helps to resolve later descriptions . Our results suggest that more sophisticated models are needed to fully exploit shared linguistic history . The current paper showcases only some of the aspects of the PhotoBook dataset , which we hereby release to the public ( https:// dmg-photobook.github.io ) . In future work , the data can be used to further investigate common ground and conceptual pacts ; be extended through manual annotations for a more thorough linguistic analysis of co-reference chains ; exploit the combination of vision and language to develop computational models for referring expression generation ; or use the PhotoBook task in the ParlAI framework for Turing-Test-like evaluation of dialogue agents .

Challenge: Existing computational agents for visually grounded dialogue fail to produce consistent outputs because of models' inability to effectively utilise dialogue history.
Approach: They present a collection of visually-grounded, task-oriented dialogue in English from a collaborative game with two online participants and a baseline model for reference resolution.
Outcome: Their experiments on the proposed dataset and model show that shared information accumulated in a reference chain is important to resolve later descriptions.