R efer, R euse, R educe: G enerating S ubsequent R eferences in V isual and C onversational C ontexts (2020)

Abstract: Dialogue participants often refer to entities or situations repeatedly within a conversation, which contributes to its cohesiveness. Subsequent references exploit the common ground accumulated by the interlocutors and hence have several interesting properties, namely, they tend to be shorter and reuse expressions that were effective in previous mentions. In this paper, we tackle the generation of first and subsequent references in visually grounded dialogue. We propose a generation model that produces referring utterances grounded in both the visual and the conversational context. To assess the referring effectiveness of its output, we also implement a reference resolution system. Our experiments and analyses show that the model produces better, more effective referring utterances than a model not grounded in the dialogue context, and generates subsequent references that exhibit linguistic patterns akin to humans.

## introduction when speakers engage in conversation, they often refer to the same objects or situations more than once. subsequent references @xcite are dependent on the shared knowledge that speakers accumulate during dialogue. for example, dialogue participants may first mention "a white fuzzy dog with a wine glass up to his face" and later refer to it as "the wine glass dog", as shown in figure 1 , dialogue 1. speakers establish 'conceptual pacts', i.e., particular ways of conceptualising referents that condition what is perceived as coherent in a given dialogue @xcite . while "the wine glass dog" may be odd as a standalone description, it is an appropriate referring expression in the above conversational context. yet, uttering it in a different context (such as dialogue 2 in figure 1 , after the participants had successfully referred to the image figure 1 : two chains of referring utterances from two games with different participants, including the first description of the target image in that dialogue and two subsequent references (;). in the game, each participant sees 5 additional images besides the target shown here. the distractor images change at every round of the game, i.e., each co-referring utterance within a dialogue is produced in a slightly different visual context. as "the dog on the red chair") may disrupt the cohesion of the dialogue and lead to communication problems @xcite . in this paper, we tackle the generation of referring utterances-i.e., utterances that contain referring descriptions, as in figure 1 -grounded both in the visual environment and the dialogue context. these utterances have several interesting properties that make their automatic generation challenging. first, they are produced with the communicative goal of helping the addressee identify the intended referent. second, because humans operate under cognitive and time-bound constraints, dialogue participants will aim to fulfil this communicative goal while optimising the use of their limited cognitive resources. this results in two common fea-tures of subsequent mentions: (1) reduction: utterances tend to become shorter-a well attested phenomenon since the work of @xcite -as a result of interlocutors' reliance on their common ground @xcite : as more shared information is accumulated, it becomes predictable and can be left implicit @xcite @xcite . sentence compression also takes place in discourse, as predicted by the entropy rate principle @xcite . (2) lexical entrainment: speakers tend to reuse words that were effective in previous mentions @xcite possibly due to priming effects @xcite . thus, besides being a challenging problem intriguing from a linguistic and psycholinguistic point of view, computationally modelling the generation of subsequent references can contribute to better user adaptation in dialogue systems and to more natural humancomputer interaction. for our study, we use data from the photobook dataset @xcite , developed to elicit subsequent references to the same images within task-oriented dialogue. to isolate the issue we are interested in, we extract, from each dialogue, the utterances that refer to a given image. this results in a dataset of dialogue-specific chains of coreferring utterances: for example, figure 1 shows two chains of co-referring utterances from two different dialogues, both referring to the same image. figure 2 shows another example. we then formulate the problem as the generation of the next utterance in a chain given the current visual context and the common ground established in previous coreferring utterances (whenever these are available). to computationally model this problem, we propose three variants of a generation system based on the encoder-decoder architecture @xcite . we evaluate their output with metrics commonly used in the domain of natural language generation and with several linguistic measures. in addition, to assess the communicative effectiveness of the generated references, we implement a reference resolution agent in the role of addressee. we find that conditioning the generation of referring utterances on previous mentions leads to better, more effective descriptions than those generated by a model that does not exploit the conversational history. furthermore, our quantitative and qualitative analysis shows that the context-aware model generates subsequent references that exhibit linguistic patterns akin to humans' regarding markers of new vs. given information, reduction, and lexical entrainment, including novel noun-noun compounds. our data, code, and models are available at https://dmg-photobook.github.io . in addition, our setup has several novel aspects: the referring utterances are the result of interactive dialogue between two participants and include subsequent references. ## models with the new dataset of referring utterance chains in place, we operationalise the problem of generating a referring utterance taking into account the visual and conversational context as follows. the model aims to generate a referring utterance given (a) the visual context in the current game round made up of 6 images from the perspective of the player who produced the utterance, (b) the target among those images, and (c) the previous co-referring utterances in the chain (if any). besides being contextually appropriate, the generated utterance has to be informative and discriminative enough to allow an addressee to identify the target image. we thus also develop a reference resolution model that plays the role of addressee. the two models are trained independently. ## linguistic analysis we analyse the linguistic properties of the utterances generated by the best performing run of each of our models and compare them with patterns observed in the human data. extensive descriptive statistics are available in appendix g. ## conclusion we have addressed the generation of descriptions that are (1) discriminative with respect to the visual context and (2) grounded in the linguistic common ground established in previous mentions. to our knowledge, this is the first attempt at tackling this problem at the level of surface realisation within a multimodal dialogue context. we proposed an encoder-decoder model that is able to generate both first mentions and subsequent references by encoding the dialogue context in a multimodal fashion and dynamically attending over it. we showed that our best performing model is able to produce better, more effective referring utterances than a variant that is solely grounded in the visual context. our analysis revealed that the generated utterances exhibit linguistic properties that are similar to those observed in the human utterances regarding reuse of words and reduction. generating subsequent references with such properties has the potential to enhance user adaptation and successful communication in dialogue systems. yet, in our approach we abstracted away from important interactive aspects such as the collaborative nature of referring in dialogue @xcite ), which was considered by @xcite for the task of reference resolution. in the present work, we simplified the interactive aspects of reference by extracting referring utterances from the photobook dialogues and framing the problem as that of generating the next referring utterance given the previous mention. we believe that the resulting dataset of referring utterance chains can be a useful resource to analyse and model other dialogue phenomena, such as saliency or partner specificity, both on language alone or on the interaction of language and vision. pamela downing. 1977. on the creation and use of english compound nouns. language, 53(4):810-842. ondřej dušek and filip jurčíček. 2016. a contextaware natural language generator for dialogue systems. in proceedings of the 17th annual meeting of the special interest group on discourse and dialogue, pages 185-190, los angeles. association for computational linguistics. simon garrod and anthony anderson. 1987. saying what you mean in dialogue: a study in conceptual and semantic co-ordination. cognition, 27(2):181 -218. dmitriy genzel and eugene charniak. 2002. entropy rate constancy in text. in proceedings of the 40th annual meeting of the association for computational linguistics, pages 199-206, philadelphia, pennsylvania, usa. association for computational linguistics. conference on computer vision and pattern recognition (cvpr), pages 4566-4575. jette viethen, robert dale, and markus guhe. 2011. generating subsequent reference in shared visual scenes: computation vs re-use. in proceedings of the 2011 conference on empirical methods in natural language processing, pages 1158-1167, edinburgh, scotland, uk. association for computational linguistics. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, rémi louf, morgan funtowicz, and jamie brew. 2019. huggingface's transformers: state-of-the-art natural language processing. arxiv, abs/1910.03771. licheng yu, hao tan, mohit bansal, and tamara l. berg. 2017. a joint speaker-listener-reinforcer model for referring expressions. 2017 ieee conference on computer vision and pattern recognition (cvpr), pages 3521-3529. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. in international conference on learning representations.