Reference and coreference in situated dialogue (2021)

Abstract: In recent years several corpora have been developed for vision and language tasks. We argue that there is still significant room for corpora that increase the complexity of both visual and linguistic domains and which capture different varieties of perceptual and conversational contexts. Working with two corpora approaching this goal, we present a linguistic perspective on some of the challenges in creating and extending resources combining language and vision while preserving continuity with the existing best practices in the area of coreference annotation.

## introduction with the ease of combining representations from different modalities provided by neural networks, text and vision are coming together. there is a growing body of resources addressing a setting in which the visual context can be exploited to support a textual task, for example visual coreference resolution. several corpora have been developed in the domain of vision and language (v&l), for example corpora of image captions @xcite @xcite , images and paragraph descriptions @xcite , visual question answering @xcite , visual dialogue @xcite and embodied question answering @xcite . through these the v&l research has progressively moved from sentence descriptions to descriptions involving utterances and conversations, therefore adding complexity to their semantic representations. in parallel to the corpora, v&l systems have been developed but of course these are limited by the complexity of the task for which the dataset has been collected. the end goal of the current research is to move to a more complex linguistic setting involving multiparty dialogue and visual representations that go beyond individual images. situated reference resolution involves grounding linguistic expressions in perceptual representations @xcite . coreference resolution, traditionally a textual task, involves linking linguistic expressions referring to the same discourse entities @xcite . while challenging, the task is defined by the familiar nature of written texts: linear, planned and structured; defining thus the coreference mechanisms and devices found in them. in resources combining v&l, however, the textual part is often a dialogue or pairs of question-answers. as a result, the coreference devices differ considerably from those found in texts and are closer to actual conversations whereby people create reference to entities on the fly. this of course comes with its own challenges, but there are also some relations made easier since they can be grounded in the image. as v&l come together, there is therefore an increased need for extending resources for the task of visual coreference resolution. this means engaging with the challenges along two axes: the first corpus contains a conversation between two participants over an almost identical visual scene involving a table and cups where participants have different locations (figure 1 ). some cups have been removed from each participant's view and they are instructed to discuss over a computer terminal in order to find the cups that each does not see. the tell-me-more corpus consists of images accompanied with a small text of five complete sentences, collected by asking participants to describe the image to a friend, successively adding details. the genre of these texts is therefore mixed: in between standard text (as found in news text for example) and dialogue data which reflects the features found in conversations rather than written conventions. these corpora are complementary as cups gives us accurate visual ground truth information with free and unrestricted dialogue, while tell-me-more offers a richer unrestricted image with short and task constrained (pseudo-)dialogues. in this paper, we discuss a number of cases from these corpora that challenge both standard language grounding annotations as well as standard coreference annotation. this work points thus towards required future work in creating (co)reference annotation schemes that can handle situated dialogue. ## related work pointing to the inability of nlp tools to handle the textual part in situated dialogue, early works had described the need to ground the dialogue in the image in a manner informed by linguistics @xcite . as content develops in a text, entities are introduced and re-mentioned, establishing discourse referents. the context is provided by the document and no extra-linguistic reference is needed for resolving the reference to an entity @xcite . in situated dialogue, on the other hand, the visual modality brings the extra-linguistic context as a source of referents. here, resolving references to entities can be thus achieved by either looking at the picture or by reading the discourse. recording both strategies separately is crucial if we want to understand and model them soundly, in keeping with theories of cognitive processing (cf. @xcite ). extending the coreference annotation paradigm is thus the best bet although not a lot of work exists in this area. textual coreference annotated data for the coreference resolution task has mainly focused on news texts and concrete nouns, excluding reference to events and other coreferential relations such as bridging, deixis, and ambiguous items well documented in the linguistic literature but deemed infrequent or too difficult to process (poesio, 2016). in contrast, there is a growing body of literature interested in phenomena beyond the nominal case @xcite , resulting in new, although still small in size, annotated corpora @xcite @xcite . visual coreference coreference work based on the popular visdial dataset @xcite targets only a limited set of referential expressions, partly because it relies on automatic tools @xcite , which are known to be problematic with this genre. with a focus in grounded human interaction, there are corpora whose textual part comprises question answer pairs @xcite . those, however, are short in nature, with few opportunities for re-mention of the different objects in the image and hence coreference. last, corpora designed towards navigation and location @xcite focusing on different kind of task and descriptions might be good candidates that could be explored and extended in a similar fashion as our corpora. ## understanding reference in situated dialogue the notion of coreference chain-the sequence of mentions pointing to a same entity in a text-is central in coreference resolution. built on top of the document as a unit, this notion relies on and in turn informs theories about accessibility hierarchy and salience of entities @xcite @xcite . in dialogue, however, references crisscross between the speakers and, one step further, in situated dialogue references crisscross between the speakers and the objects in the image. in this section we revise the annotation challenges in the annotation of anaphoric phenomena in data of this genre. ## conclusions v&l resources provide a unique opportunity to explore the notion of discourse entity in grounded context. extending the coreference annotation to this domain is essential to understand the relationship between reference and coreference. the same mechanisms that humans adopt to solve coreference in the textual domain should underlay results in the v&l domain. indeed, reference is underspecified in both modalities; any kind of information extraction from these domains will benefit from mechanisms that resolve this underspecification: capturing coreference is a door to capturing coherence. furthermore, a rich annotation scheme leads to the development of corpora allowing the training of data driven systems for the v&l domain and social robotics.