Anaphoric Phenomena in Situated dialog: A First Round of Annotations (2022)

Abstract: We present a first release of 500 documents from the multimodal corpus Tell-me-more (Ilinykh et al., 2019) annotated with coreference information according to the ARRAU guidelines (Poesio et al., 2021). The corpus consists of images and short texts of five sentences. We describe the annotation process and present the adaptations to the original guidelines in order to account for the challenges of grounding the annotations to the image. 50 documents from the 500 available are annotated by two people and used to estimate inter-annotator agreement (IAA) relying on Krippendorff’s alpha.

## introduction coreference resolution-linking together all referring expressions that refer to the same discourse entity-has a long tradition in computational linguistics. the progress is undeniable as evidenced by recent systems @xcite , particularly in the text domain, rich in news data. coreference resolution work for dialog and spoken data in general, on the other hand, has been less predominant, as the phenomena in this genre are broader and harder to tackle @xcite . however, interest in tackling these challenges is possible thanks to the creation of new resources. situated dialog corpora-dialog about an image presented as common ground to the dialog participants-is part of these recent resources. dialog text approximates natural conversations, while the image constraints an otherwise unlimited choice of entities and events in the dialog. in this paper, we present a first release of a portion of the multimodal corpus tell-me-more @xcite annotated with coreference information according to the arrau guidelines @xcite . the tell-me-more corpus consists of images accompanied with a short english text of five complete sentences, collected by asking participants to describe the image to a friend. an example is presented in (1). the genre of these texts is therefore in between standard text (as found in news text for example) and dialog data which reflects the features found in conversations rather than written conventions. the simplicity of the text is essential for our purposes, as it allows us to test the limits of the guidelines to account for reference and grounding. in contrast, standard situated dialog is very rich in changes of point of reference, spacial references, and dynamic references depending on the participant's cognitive state that are very challenging to ground to the image. there exist other corpora whose textual part comprises question answer pairs @xcite . unlike dialog data, question answer pairs are short, with few opportunities for re-mention of the different objects in the image and hence coreference. there is also corpora designed towards navigation and location involving videos and long dialog interactions between an instruction giver and an instruction follower. examples include the scare corpus @xcite and the corpus by @xcite . on a similar venue, recent work has used minecraft 4 to collect dialog where an architect gives instructions to a builder about how to move and position some pieces in order to achieve a target structure @xcite . due to the multiple changes in reference perspective and very long dialog games, this type of corpora is more difficult to annotate than the corpus used in this paper. in this sense, we see our work as a stepping stone towards achieving the annotation of more complex data in the future. ## the annotation process the annotation was carried out by two annotators with a background in computational linguistics. the mmax annotation tool @xcite was chosen with the aim to replicate the arrau scheme easily. ## measuring agreement this release contains 500 annotated documents by one annotator and 50 annotated by two. in this section, we detail the computation of the interannotator agreement (iaa) using the 50 documents which have been doubled annotated. computing iaa for coreference resolution is non-trivial, as annotators need to decide on the mentions boundaries and also which ones belong together in a chain. following @xcite , we report krippendorff's α with weighted δ: where m is the number of annotators, and r is the number of coding units, i.e., mentions. for every pair of mentions b and c, δ bc is the distance between the sets formed by their tokens; n b i is the number of times the value b was assigned by each annotator to each mention i. the distance between the mentions' tokens is 0 when the mentions' tokens are identical, 0.33 when one set subsumes the other, 0.67 when one intersects the other, and 1 when they are disjoint. the iaa results obtained are very mixed. the low scores of some documents are partly explained by our choice to do the mention identification completely by hand. this means that the annotators had to decide the boundaries of each mention, yielding 7 as an illustration, consider the example in (4). here the tokens {mostly, is, is, has, a, and, and, , is, on, is} are left non-annotated by annotator a; while {mostly, is, is, has, and, and, is, on, is} by annotator b. this is expected as they do not form part of any of their identified mentions, but by scoring at the token level, each set would then be taken as forming a "mention" and hence compared. ## conclusion in this paper, we presented the first release of a portion of the tell-me-more corpus manually annotated with anaphora information according to standard guidelines used for the task of coreference resolution. we also presented iaa scores on 50 documents annotated by two people with training in computational linguistics. our resource is the first of its kind, although its size is small. however, we believe that it can support linguistic studies about the relationship between textual anaphora and reference to objects, and that it can contribute to research on bridging reference. in addition, it can be used as validation data for automatic methods developed for grounding the entities in the text to the image. this is still work in progress and we look forward to future cycles of revisions and updates of our guidelines in the near-future.