The CODI - CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis in Dialogue (2021)

Abstract: In this paper, we provide an overview of the CODI-CRAC 2021 Shared-Task: Anaphora Resolution in Dialogue. The shared task focuses on detecting anaphoric relations in different genres of conversations. Using five conversational datasets, four of which have been newly annotated with a wide range of anaphoric relations: identity, bridging references and discourse deixis, we defined multiple subtasks focusing individually on these key relations. We discuss the evaluation scripts used to assess the system performance on these subtasks, and provide a brief summary of the participating systems and the results obtained across ?? runs from 5 teams, with most submissions achieving significantly better results than our baseline methods.

## introduction the performance of models for single-antecedent anaphora resolution on the aspects of anaphoric interpretation annotated in the standard ontonotes dataset @xcite has greatly improved in recent years @xcite @xcite @xcite . so the attention of the community has started to turn to more complex cases of anaphora not found or not properly tested in ontonotes. well-known examples of this trend are work on the cases of anaphora whose interpretation requires some form of commonsense knowledge tested by benchmarks for the winograd schema challenge @xcite @xcite , or the pronominal anaphors that cannot be resolved purely using gender, for which benchmarks such as gap have been developed @xcite . gap, however, still focused on identity coreference. in addition, more research has been carried out on aspects of anaphoric interpretation that go beyond identity anaphora but are covered by datasets * work done when the author was a student at cmu such as arrau @xcite . these include, e.g., bridging reference @xcite @xcite @xcite , discourse deixis @xcite @xcite or split-antecedent anaphora @xcite @xcite @xcite . there has been interest in other genres apart from news. this includes substantial research on annotating and resolving coreference in biomedical and other scientific domains @xcite as well as in literary documents @xcite . there are, however, language genres still understudied in the literature on anaphoric reference. arguably the most important among these is conversational language in dialogue. anaphora resolution in dialogue requires systems to handle grammatically incorrect language suffering from disfluencies. dialogue involves much more deictic reference, vaguer anaphoric and discourse deictic reference, speaker grounding of pronouns and long-distance conversation structure. these are complexities that are often missing in news or wikipedia articles, which form a large chunk of current datasets for coreference resolution. there has been some research on coreference in dialogue @xcite @xcite , but very limited in scope (primarily related to pronominal interpretation), due to the lack of suitable corpora. the one language for which substantial corpora of coreference in dialogue exist is french: the ancor corpus @xcite has enabled the development of an end-to-end neural model for coreference interpretation in dialogue by @xcite . for english, the one resource we are aware of fully annotated for anaphoric reference is the trains corpora included in the arrau corpus @xcite . the objective of the codi-crac 2021 shared task in anaphora resolution in dialogue foot_0 was to provide participants with the opportunity to develop automated approaches for coreference resolution that tackle less studied forms of anaphora and generalize on different types of conversational setups. specifically, the shared task is divided in three tasks that individually tackle a particular anaphoric relation: identity, bridging, and discourse deixis. to evaluate the participating systems, we provide development and test sets consisting of four conversational datasets from different domains newly annotated with the above-mentioned relations. this lack of in-domain training data also poses research challenges related to transfer learning and out-of-domain generalization of learned representations. to accommodate for systems that use gold/predicted mentions for bridging and discourse deixis tasks, we set up separate leaderboards for the two settings. our goal in this paper is to present an overview of the codi-crac 2021 shared task. we begin by providing some background in section 2 and introducing the new codi-crac 2021 corpus in section 3. we then provide an extensive overview of the different codi-crac 2021 tasks, markable settings, and evaluation metrics in section 4, and submission details in section 5. this is followed by details of the baselines in section 6 and participating systems in section 7. we present a discussion of the performance of the systems on different tasks and sub-corpora in section 8, and finally conclude this paper in section 9. ## the codi-crac 2021 corpus one of the objectives of the codi-crac 2021 shared task was to annotate new data for studying anaphora in dialogue. the only existing dataset covering the full range of phenomena and with some coverage of dialogue, the arrau data used for the crac 2018 shared task, was used as training material. in addition, new data from dialogue corpora were annotated for development and testing using the same annotation scheme used in ar-rau. ## submission details we used codalab to evaluate submissions and distribute the datasets. in the development phase, the participants only had access to out-of-domain training data (e.g., the arrau corpus) and in-domain validation data. they could submit results to the public leaderboard to evaluate their systems. in addition, we also released the scoring script on github to reduce the dependency on codalab during model development. during the evaluation phase, we released the unseen test set across four sub-corpora. the submissions were evaluated on each sub-corpus individually and the final ranking for each task was performed by taking the mean of the four scores. due to the lack of in-domain training data, the participants were allowed to use additional resources. ## baselines we released one baseline system for each task. we derive the baselines for identity and bridging anaphora from current state-of-the-art methods, and set up a simple yet effective method for discourse deixis. for task 1, we used xu and choi (2020)'s coreference resolution model but without any higherorder inference. 18 we used the ontonotes (english) dataset for training and development. this model is then evaluated on codi-crac 2021 datasets. for task 2, the baseline was derived from @xcite . we used their single-task variant that is only trained on bridging annotations. we evaluated their best-performing model, which was trained on the rst sub-corpus of arrau, on codi-crac 2021 data. 19 the baseline for task 3 leverages a simple heuristic that only considers demonstrative pronouns (this, that) as anaphors and considers the immediately preceding clause/utterance in the conversation to be their antecedent. although simplistic, the algorithm achieves respectable scores on the codi-crac 2021 development corpus. the performance 18 https://github.com/lxucs/coref-hoi/ 19 https://github.com/juntaoy/ dali-bridging of each baseline on different sub-corpora is shown in tables 3 , 4 , and 5. to help participants interested in building upon these baselines, we released the scripts used in the baseline pipeline to convert the codi-crac 2021 data from the conll-ua format to json structure and vice-versa for each task. 20 the scripts contain modules that can also be used independently to transform conversations into a format compatible with different transformer-based encoders, allowing participants to set optional arguments to specify the segment_size and tokenizer for the conversion. ## participating systems a total of 55 individual participants registered for the codi-crac 2021 shared task on codalab. 21 among them, five teams submitted results for task 1, three submitted results for task 2, and two submitted results for task 3. teams utd_nlp, ku_nlp, dfki_talkingrobots, emory_nlp, and inria submitted system description papers. we summarize their approaches below (and in utd_nlp participated in all three tasks. for identity anaphora, they deployed a pipeline architecture consisting of a mention detection component and an entity coreference component. the coreference component extends xu and choi (2020)'s implementation of @xcite by modifying the objective so that it can output singleton clusters, and enforces dialogue-specific constraints. they setup a similar architecture for discourse deixis. however, they slightly modified the objective function in @xcite by classifying each span as a candidate anaphor, a candidate antecedent, or a non-mention in the mention detection stage, and resolving only candidate anaphors to candidate antecedents later. the team used a multi-pass sieve approach for bridging resolution to target same-head bridging links, with @xcite 's model as one of the sieves. in the gold setting, they trained an additional anaphor detection model (adapted from @xcite ) to first identify the bridging anaphors from the gold markables before sending them to the sieves. ku_nlp submitted results for tasks 1 and 2. for identity anaphora, they leveraged @xcite 's model with an electra-large backbone for mention detection. the resulting mention representation, created from the constituent token representations, is then fed to a pointer-network @xcite based coreference resolution model for clustering. they solved the bridging resolution problem using a machine reading comprehension framework, where they constructed a query for each entity of the form -"what is related of en-tity?". the input of their model is the query and the document (i.e., all utterances of dialogue), and the output is the entity span that is the answer for the query. dfki_talkingrobots (dfki) put forward two systems as their final submissions for the identity anaphora task. the workspace coreference system (wcs) attempts to incrementally cluster mentions using semantic similarity based on embeddings combined with lexical feature heuristics, whereas their mention-to-mention (m2m) architecture only focuses on mention pairs to make the decision. wcs and m2m use spacy and bilstm-crf based mention detection components, respectively. for discourse deixis, the team deployed a siamese network to detect discourse anaphorantecedent pairs. they only focused on demonstrative pronouns (as anaphors), which they detected using the spacy nlp pipeline. emory_nlp (emory) only participated in the identity anaphora task. their system was adapted from the end-to-end neural coreference resolution model of @xcite . they recognized singletons, encoded speakers for all turns, and leveraged other out-of-domain datasets during training. inria submitted an end-to-end transformerbased model fine-tuned for the bridging resolution task. they formulated the bridging problem as antecedent selection, and leveraged @xcite ; @xcite 's architecture to find the correct antecedent. ## conclusion and future work we presented a general overview of the codi-crac 2021 shared task. as the first instance in this series, codi-crac 2021 focused on resolving three types of anaphoric relations in dialogues: identity, bridging references, and discourse deixis. in addition, we described the codi-crac 2021 corpus, which contains sub-corpora from different conversation genres newly annotated for the above-mentioned relations. while the teams were encouraged to create systems with generalizable representations using other out-of-domain state-ofthe-art coreference datasets during training, only the best-performing team for the entity coreference track did so. finally, we included a brief summary of the different approaches used by the participants to tackle different tasks within the shared-task. for the shared-task's next installment, we plan to release the full annotation guidelines to reduce the ambiguity about annotation principles. based on suggestions from participating teams (team dfki), we will introduce separate tracks to fairly evaluate the systems trained on the provided data vs. systems (pre-)trained on additional data, and discuss the possibility of using more fine-grained evaluation metrics to differentially evaluate cases of coreference based on their difficulty levels. (e.g., a black cat and the black cat are much easier to resolve compared to a black cat and the dark and furry creature.)