The CODI - CRAC 2022 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue (2022)

Abstract: The CODI-CRAC 2022 Shared Task on Anaphora Resolution in Dialogues is the second edition of an initiative focused on detecting different types of anaphoric relations in conversations of different kinds. Using five conversational datasets, four of which have been newly annotated with a wide range of anaphoric relations: identity, bridging references and discourse deixis, we defined multiple tasks focusing individually on these key relations. The second edition of the shared task maintained the focus on these relations and used the same datasets as in 2021, but new test data were annotated, the 2021 data were checked, and new subtasks were added. In this paper, we discuss the annotation schemes, the datasets, the evaluation scripts used to assess the system performance on these tasks, and provide a brief summary of the participating systems and the results obtained across 230 runs from three teams, with most submissions achieving significantly better results than our baseline methods.

## introduction the performance of models for single-antecedent anaphora resolution on the aspects of anaphoric interpretation annotated in the standard ontonotes dataset @xcite has greatly improved in recent years @xcite @xcite @xcite . so the attention of the community has started to turn to more complex cases of anaphora not found or not properly tested in ontonotes, and on genres other than news. well-known examples of this trend are work on the cases of anaphora whose interpretation requires some form of commonsense knowledge tested by benchmarks for the winograd schema challenge @xcite @xcite , or the pronominal anaphors that cannot be resolved purely using gender, for which * work was done prior to joining aws ai labs. there has also been interest in other genres apart from news. this includes substantial research on annotating and resolving coreference in biomedical and other scientific domains @xcite as well as in literary documents @xcite . there are, however, language genres still understudied in the literature on anaphoric reference. arguably the most important among these is conversational language in dialogue. anaphora resolution in dialogue requires systems to handle grammatically incorrect language suffering from disfluencies and mentions jointly created across utterances @xcite or whose function is to establish common ground rather than refer @xcite . dialogue involves much more deictic reference, vaguer anaphoric and discourse deictic reference, speaker grounding of pronouns and long-distance conversation structure. these are complexities that are normally absent from news or wikipedia articles, which constitute the bulk of current datasets for coreference resolution @xcite . there has been some research on coreference in dialogue @xcite @xcite , but very limited in scope (primarily related to pronominal interpretation), due to the lack of suitable corpora. the one language for which substantial corpora of coreference in dialogue exist is french: the an-cor corpus @xcite has enabled the development of an end-to-end neural model for coreference interpretation in dialogue by @xcite . for english, the one resource we are aware of fully annotated for anaphoric reference is the trains corpora included in the arrau corpus @xcite . the codi-crac 2021 shared task in anaphora resolution in dialogue @xcite was organized to address this need for datasets about anaphoric reference in dialogue by providing participants with the opportunity to develop automated approaches for anaphora resolution that tackle less studied forms of anaphora as well as coreference, and generalize to different types of conversational setups. a number of groups participated to this first edition, but we organizers also realised that the community could benefit from a second edition using more data and more cleaned-up, adding more tasks, and improving the evaluation. as a result, we organized this year's second edition. 1 like the first edition, codi-crac 2022 involved three tasks that individually tackle a particular anaphoric relation: identity, bridging, and discourse deixis, in four conversational datasets from different domains newly annotated with the above-mentioned relations. unlike the first edition, participants also had training data in those four domains, in addition to development and test sets. to accommodate for systems that use gold/predicted mentions for bridging and discourse deixis tasks, we set up separate leaderboards for the two settings. in this paper wepresent an overview of the codi-crac 2022 shared task. we begin by providing some background in section 2 and introducing the new codi-crac 2022 corpus in section 3. we then provide an extensive overview of the different codi-crac 2022 tasks, markable settings, and evaluation metrics in section 4, and submission details in section 5. this is followed by details of the baselines in section 6 and participating systems in section 7. we present a discussion of the performance of the systems on different tasks and sub-corpora in section 8, and finally conclude this paper in section 9. (1) [mary] i bought [a new dress] j but [it] j didn't fit [her] i . however, many other types of identity anaphora exist, as well as other types of anaphoric relations that are not annotated in ontonotes but are annotated in other corpora. the codi-crac 2021 and 2022 shared tasks covered the range of anaphoric relations included in the first universal anaphora survey of phenomena to be covered (see below) split-antecedent anaphora split-antecedent anaphors @xcite are cases of plural identity reference to sets composed of two or more entities introduced by separate noun phrases, as in (2). such references are annotated in, e.g., arrau @xcite , gum @xcite and phrase detectives @xcite . ## the codi-crac 2022 corpus one of the objectives of the codi-crac shared tasks was to annotate new data for studying anaphora in dialogue. the only existing dataset covering the full range of phenomena and with some coverage of dialogue, the arrau data used for the crac 2018 shared task, was made available as training material. in addition, new data from dialogue corpora were annotated for development and testing using the same annotation scheme used in arrau. ## submission details the shared task was hosted on a single codalab page, including evaluations and datasets distribution. the competition consists of three development phases and seven evaluation phases. in the development phases, a small in-domain training set for each domain alongside a large out-of-domain training set (i.e. the arrau corpus) is available. in addition, a validation set for each domain is also provided. the development phases are handy tools to get the systems prepared for the evaluation phases. apart from the development phases, the participants can also download the scoring script to evaluate their systems offline. during the evaluation phases, the different versions of the unseen test sets (pred, gold m, gold a) were released incrementally to accommodate the needs of the evaluation phases. the submissions were evaluated individually on each of the four domains, and then the macro-average of the four scores are used for the final ranking of individual tasks. apart from the corpora provided by us, additional resources were also permitted. ## baselines we used the same baseline systems from last year's shared task, and further, evaluate those baselines in the newly introduced phases. more precisely the baselines for identity anaphora and bridging reference resolution tasks are derived from state-ofthe-art neural models, whereas the discourse deixis baseline is a simple but effective system based on heuristic rules. for identity anaphora resolution (task 1), we used the coreference resolution model provided by the xu and choi (2020) 15 . more specifically, we use their spanbert setting without any higherorder inference (spanbert + no hoi), the model was trained with the ontonotes (english) dataset and then evaluated directly on codi-crac 2022 datasets without fine-tuning. for bridging reference resolution (task 2), we use the single-task variant of the yu and poesio (2020) system foot_14 . the system is trained on the bridging annotations of the rst sub-corpus of arrau. since the system do not predict the mentions itself, for the predicted mention setting (pred), we supply the system with mentions predicted by @xcite 's mention detector (biaffine md) 17 . the mention detector was also trained on the same rst sub-corpus of arrau. for gold m and gold a settings, we use the gold mentions and anaphoras provided respectively. the system is evaluated on codi-crac 2022 data without further training. for discourse deixis (task 3), the baseline for predicted mention setting (pred) uses two simple heuristics: first only considers demonstrative pronouns (this, that) as anaphors and then uses the immediately preceding clause/utterance in the conver-sation to be their antecedent. for the gold mention setting (gold m) we further restrict the anaphors to be the intersection of the demonstrative pronouns and the gold mentions and then apply the same rule for antecedent selection. for the gold anaphor setting (gold a), the baseline links the gold anaphors to their immediately preceding clause/utterance. the heuristic-based baselines are then evaluated on the codi-crac 2022 data of all four domains. the performance of our baselines on different sub-corpora is shown in tables 3 , 4 , and 5 alongside the participant systems. a helper script developed from last year's shared task is available to help participants convert the conll-ua format to and back from the various json format used by our baselines foot_16 . ## participating systems similar to last year, a total of 54 individual participants registered for the codi-crac 2022 shared task on codalab. among them, three teams submitted results for task 1, and two submitted results for task 2 and task 3. apart from emory_nlp, all the teams from last year participated in this year's shared task, but dfki and inria joined forces to participate as one team. all three teams (utd_nlp, ku_nlp, dfki-inria) submitted system description papers. we summarize their approaches below and in utd_nlp participated in all three tasks. for identity anaphora, the authors built a pipeline system consisting of three components: a mention detector, an entity coreference resolver and a nonreferring/entity classifier. all three components use the same underlining system they used in last year's shared task @xcite , a multi-task learning approach adapted from the xu and choi (2020) system for mention detector and coreference resolution. the training objectives and priorities, however, were configured differently to maximise the performance of the individual tasks. finally, those components were used in a pipeline fashion to deliver their final results. for discourse deixis, a system similar to xu and choi (2020)'s was used. they use both heuristics and a binary classifier to supply the anaphors. for each anaphor, antecedents were selected from up to 10 immediate previous utterances. the team based their bridging resolution system on the yu and poesio (2020)'s model, with additional dialogue-specific features included. the main focus of this year was on exploring the different pre-training and fine-tuning strategy. in total, four different training strategies were evaluated by them.. ku_nlp submitted results for identity anaphora resolution (task 1). the team proposed a pipeline system that resolves the mentions separately from the coreference resolution. the mention detection part solves the problem by classifying all possible mentions into mentions and non-mentions. the predicted mentions then feed into the coreference part of the system that solves the task in a mentionpair fashion. additional speaker features were used to leverage the mention representations. dfki-inria participated in all three tasks. for the identity anaphora task, they utilise the workspace coreference system (wcs) @xcite they introduced in last year's shared task with the xu and choi ( 2020 ) system. the singletons predicted by the wcs system are added to the xu and choi (2020) to create their final results. similar to the wcs system, the mentions are predicted separately using spacy. for bridging, they build their system on a simplified @xcite system with mention pruning and coarse-to-fine steps removed. they only submitted to the gold a phase, where gold mentions and gold anaphors were provided. for discourse deixis, the team employ a multi-task learning approach based on the xu and choi (2020) system, the system first uses heuristics to find the candidate anaphors, then resolve the antecedents and finally uses an anaphora type classifier to filter out the identity, non-referring anaphors. the system also used several linguistic features (e.g. pos, dependency relations) to aid the anaphora type classification. ## conclusion and future work in this paper we presented a general overview of the codi-crac 2022 shared task. like the first shared task in this series, codi-crac 2022 focused on resolving three types of anaphoric relations in dialogues: identity, bridging reference, and discourse deixis. based on the feedback from participants to the first task, in this second event we released the annotation guidelines beforehand so that participants could know exactly how the data had been annotated. in addition, we re-checked the data newly annotated for the first edition (now available for training and development, so that participants could do some in-domain training as well), and using a larger group of annotators, which resulted in an hopefully more objective annotation. new test data in the four new dialogue domains was also annotated. the participant systems outperformed the baselines on virtually all tasks and settings, although a clear difference in performance could be observed for bridging reference between pure resolution and resolution + identification. (interestingly, we didn't observe much difference in performance between the 'gold mention' and 'predicted' settings for either bridging nor discourse deixis.) a clear difference was observed between the results on the ami datasets and on the other datasets for identity anaphora and bridging reference, possibly due to greater length of the documents in ami.