Annotating anaphoric phenomena in situated dialogue (2021)

Abstract: In recent years several corpora have been developed for vision and language tasks. With this paper, we intend to start a discussion on the annotation of referential phenomena in situated dialogue. We argue that there is still significant room for corpora that increase the complexity of both visual and linguistic domains and which capture different varieties of perceptual and conversational contexts. In addition, a rich annotation scheme covering a broad range of referential phenomena and compatible with the textual task of coreference resolution is necessary in order to take the most advantage of these corpora. Consequently, there are several open questions regarding the semantics of reference and annotation, and the extent to which standard textual coreference accounts for the situated dialogue genre. Working with two corpora on situated dialogue, we present our extension to the ARRAU (Uryupina et al., 2020) annotation scheme in order to start this discussion.

## introduction with the ease of combining representations from different modalities provided by neural networks, text and vision are coming together. there is a growing body of resources addressing a setting in which the visual context can be exploited to support a textual task, for example visual anaphora resolution. 1 several corpora have been developed in the domain of vision and language (v&l), for example corpora of image captions @xcite @xcite , images and paragraph descriptions @xcite , visual question answering @xcite , visual dialogue @xcite and embodied question answering @xcite . through these the v&l research has progressively moved from sentence descriptions to descriptions involving utterances and conversations, therefore adding complexity to their semantic representations. in parallel to the corpora, v&l systems have been developed but of course these are limited by the complexity of the task for which the dataset has been collected. the end goal of the current research is to move to a more complex linguistic setting involving multiparty dialogue and visual representations that go beyond individual images. anaphora resolution has been studied both in the textual and situated dialogue domains (cf. @xcite for an extensive survey of anaphora and coreference; @xcite @xcite @xcite ). in the textual domain, this has been formulated as a standard task with several corpora annotated uniformly for the most part, while in situated dialogue each corpus presents its own individual solution (cf. @xcite @xcite ). with the increasing interest in the combination of v&l in deep learning applications, multimodal resources are increasingly used in the context of traditional textual natural language processing (nlp) tasks. as such, it makes sense to consider a common annotation strategy both for the textual and situated dialogue domains, basing it on the rich work of textual anaphora resolution standards. doing so, we also hope to get new insights about the semantics of reference in natural language. situated reference resolution involves grounding linguistic expressions in perceptual representations @xcite or representations of actions @xcite . anaphora resolution, traditionally a textual task, involves linking linguistic expressions referring to the same discourse entities @xcite . while challenging, the task is defined by the familiar nature of written texts: linear, planned and structured; defining thus the mechanisms and devices found in them. in resources combining v&l, however, the textual part is often a dialogue or pairs of question-answers. as a result, the coreference devices differ from those found in texts and are closer to actual conversations in which people create reference to entities on the fly. this of course comes with its own challenges, but there are also some relations made easier since they can be grounded in the image. as v&l come together, there is therefore an increased need for extending resources for the task of visual anaphora resolution. this means engaging with the challenges along two axes: we present our extension to the arrau @xcite @xcite annotation scheme by analysing two situated dialogue corpora: the cups corpus @xcite and the tell-me-more corpus @xcite , shown below in figures 1 and 2 respectively. this exercise proved useful to pinpoint in what ways the purely textual document scenario is different from the domain of embodied interaction both in terms of the semantics of interaction and annotation practices. the cups corpus contains a conversation between two participants over an (almost) identical visual scene involving a table and cups where participants have different locations. some cups have been removed from each participant's view and they are instructed to discuss over a computer terminal in order to find the cups that each does not see. the ground truth of the visual scene is known as it has been artificially generated. it may take over an hour for the participants to solve the task and their activity results in free dialogue close to spoken conversations including phenomena such as clarifications, repairs, restarts and variable grammar. (the conversations are logged at a key-press level.) the tell-me-more corpus consists of images accompanied with a short text of five complete sentences, collected by asking participants to describe the image to a friend, successively adding details in short constrained conversations. the genre of these texts is therefore mixed: in between standard text (as found in news text for example) and dialogue data which reflects the features found in conversations rather than written conventions. these corpora are complementary as cups gives us accurate visual ground truth information with free and unrestricted dialogue, while tell-me-more offers a richer unrestricted image with short and task-constrained (pseudo-)dialogues. in this paper, we discuss a number of cases from these corpora that challenge both standard language grounding annotations as well as standard anaphora annotation. this work points thus towards required future work in creating anaphora annotation schemes that can handle situated dialogue. ## related work pointing to the inability of nlp tools to handle the textual part in situated dialogue, early works had described the need to ground the dialogue in the image in a manner informed by linguistics @xcite . as content develops in a text, entities are introduced and re-mentioned, establishing discourse referents. the context is provided by the document and no extra-linguistic reference is needed for resolving the reference to an entity @xcite . in situated dialogue, on the other hand, the visual modality brings the extra-linguistic context as a source of referents. here, resolving references to entities can be thus achieved by either looking at the picture or relating to the information that has been said previously in the discourse. both of these processes happen simultaneously and therefore their interaction must be explained by theories of cognitive processing related to attention and memory @xcite . however, in order to understand both processes and their interaction we need to disentangle them. extending the anaphora annotation paradigm is thus the best bet although not a lot of work exists in this area. textual coreference annotated data for the coreference resolution task has mainly focused on news texts and concrete nouns, excluding reference to events and other coreferential relations such as bridging, deixis, and ambiguous items well documented in the linguistic literature but deemed infrequent or too difficult to process @xcite . in contrast, there is a growing body of literature interested in phenomena beyond the nominal case @xcite , resulting in new annotated corpora @xcite @xcite , although smaller in 1. it's a bedroom scene with the bed partially visible 2. the bed has a curved wooden headboard with slots like a fence 3. there is framed art hanging above the bed 4. to the left of the bed is a door, which is open 5. there is a small square nighstand next to the bed which has a lamp on top of it size than ontonotes @xcite , the largest and most used coreference corpus in the field. moreover, as a product of this year's edition of the crac foot_1 and codi foot_2 workshops, a shared task on anaphora resolution in dialogues has been proposed. this will undoubtedly result in additional corpora annotated with the standards used for the coreference resolution task. visual coreference coreference work based on the popular visdial dataset @xcite targets only a limited set of referential expressions, partly because it relies on automatic tools @xcite , which are known to be problematic with this genre. with a focus in grounded human interaction, there are corpora whose textual part comprises question answer pairs @xcite . those, however, are short in nature, with few opportunities for re-mention of the different objects in the image and hence coreference. last, corpora designed towards navigation and location involve considerable dialogue interaction between instruction giver and instruction follower which include examples of coreference. for example, the scare corpus @xcite provides natural interactions, it has been audio recorded and then transcribed, the conversations are long and there are frequent referring expressions (it is hard to understand transcribed dialogues on its own), but overall the size of the corpus is small. @xcite present a corpus of 2050 short human-human interactions in a virtual environment collected with crowd-sourcing. ## the arrau annotation scheme deeply rooted in linguistic theory, the arrau corpus annotation scheme is particularly well-suited for annotating situated dialogue. indeed, its annotation scheme was designed to accommodate different genres, including news, dialogue and narrative texts, and in consequence anaphoric phenomena beyond the nominal standard case typically found in other coreference corpora @xcite . the dialogue genre has its own idiosyncrasies not covered by annotation schemes designed for news text, for example collaborative completions giving way for discontinuous markables @xcite , and more pronouns including deictics @xcite . the annotation scheme also includes guidelines for bridging reference, a much less studied type of reference but very commonly used in the tell-me-more corpus discussed here. arrau is also known for containing annotations for both referring and non-referring expressions. most coreference corpora focus on identity anaphora, meaning that only multiple mentions of the same discourse entity are annotated, leaving out those mentioned only once, also known as singletons. the large ontonotes corpus, for instance, does not include annotations of singletons or expletives. in the next section, we describe the general ar-rau annotation scheme along with our proposed adaptations. with the goal of moving towards general guidelines for the situated dialogue genre, the extensions we present target the common challenges of our two corpora. ## annotating situated dialogue 4.1 mention identification and object detection the first step is identifying the referring expressions or mentions to annotate. in arrau, all noun phrases are considered, marking the complete phrase with all its modifiers and not just its head. this includes noun phrases which are nonreferring such as pleonastics and also noun phrases not re-mentioned later in the text. the mentions also include personal pronouns and demonstrative pronouns used as deictics (to refer back to nonnominal antecedents). we also consider all noun phrases, including pronouns and deictics as mentions. for cups, we created a simple np chunker based on the regular expression method @xcite with moderate success: a manual annotation of one of the documents showed an error rate of about 30% (295 errors out of 1030 identified chunks). in contrast, for tell-me-more we had annotators identify the nps completely by hand. compared to arrau, the noun phrases in these corpora are rather simple, without a lot of modifiers. however, this does not mean that mention identification is straightforward as complex noun phrases with embedded markables such as the blue cup with a white handle do arrive. consider also the blue cup to the left of the red cup, where a particular cup is referred to by taking another cup as a landmark: is it the left or the red cup or the left of the red cup which should be considered for re-mention? akin to the mention identification, the image in the multimodal corpora is processed in order to detect objects. in cups, we have the ground truth of the scenes from which participants views have been generated. all the objects and geometrically defined regions are assigned a predefined id as shown in figure 1 . in tell-me-more, the object labels are part of the underlying ade20k data @xcite , extracted using tools from @xcite . here, an automatic object classifier may not detect all the objects in the scene or assign them different labels than participants use when referring to them in the dialogue. ## conclusions different v&l resources provide with an opportunity to explore the notion of discourse entity and (co)reference in grounded context. since the nature of contexts defined by the tasks in which the corpora were collected varies considerably we get an opportunity to study the phenomena over these contexts and get a more complete picture of reference. extending the coreference annotation to the v&l domain is essential to understand the relationship between reference and coreference. work around textual coreference has defined the task with insufficient consideration of the semantic aspects involved in the interpretation of anaphoric phenomena; whereas work from the v&l community assumes that coreferential information can be inferred latently. by extending the coreference annotation scheme to rich situated dialogue corpora, we make explicit the relations at play between the text and the image. the same mechanisms that humans adopt to solve coreference in the textual domain should underlay results in the v&l domain. indeed, reference is underspecified in both modalities; any kind of information extraction from these domains will benefit from mechanisms that resolve this underspecification: capturing coreference is a door to capturing coherence. furthermore, a rich annotation scheme that is portable between tasks and contexts, leads to the development of corpora allowing the training of data driven systems for the v&l domain and social robotics.