The CODI - CRAC 2021 Shared Task on Anaphora, Bridging, and Discourse Deixis Resolution in Dialogue: A Cross-Team Analysis (2021)

Abstract: The CODI-CRAC 2021 shared task is the first shared task that focuses exclusively on anaphora resolution in dialogue and provides three tracks, namely entity coreference resolution, bridging resolution, and discourse deixis resolution. We perform a cross-task analysis of the systems that participated in the shared task in each of these tracks.

## introduction the codi-crac 2021 shared task @xcite , which focuses on anaphora resolution in dialogue, provides three tracks, namely entity coreference resolution, bridging resolution, and discourse deixis/abstract anaphora resolution. among these three tracks, bridging resolution and discourse deixis resolution are relatively under-studied problems. this is particularly so in the context of dialogue processing. this shared task is therefore of potential interest to researchers in the discourse and dialogue communities, particularly researchers in anaphora resolution who intend to work on problems beyond identity coreference. our goal in this paper is to perform a cross-team analysis of the systems participating in the three tracks of the shared task. our analysis is partly quantitative, where we attempt to draw conclusions based on statistics computed using the outputs of the systems, and partly qualitative, where we discuss the strengths and weaknesses of the systems based on our manual inspection of these outputs. while several attempts have been made to perform an analysis of different coreference systems (e.g., @xcite , @xcite ), we note that conducting an insightful analysis of these systems is inherently challenging for at least two reasons. first, for entity coreference resolution and discourse deixis resolution, the latter of which is treated as a general case of event coreference, * *equal contribution the system outputs on which we perform our analysis is in the form of clusters. hence, we do not have information about which links were posited by a system and used to create a given cluster. this makes it impossible to pinpoint the mistakes (i.e., the erroneous linking decisions) made by a system and fundamentally limits our ability to explain the behavior of a system. second, even if we could pinpoint the mistakes, existing models for anaphora resolution have become so complex that it is virtually impossible to explain why a particular mistake was made. for instance, a mention extraction component is so closely tied to a resolution model that it is not always possible to determine whether a mistake can be attributed to erroneous mention extraction or resolution. worse still, since the participants have the freedom to partition the available training and development datasets in any way they want for model training and parameter tuning and are even allowed to exploit external training corpora, it makes it even harder to determine whether a system performs better because of a particular way of partitioning the data or because external training data are used. the rest of this paper is structured as follows. the next three sections describe our cross-team analysis for the three tracks, namely entity coreference (section 2), bridging (section 3), and discourse deixis (section 4). we present our conclusions and observations in section 5. ## bridging resolution the shared task divides the evaluation of bridging resolution into two phases: (1) the predicted phase, where a system needs to first identify all of the entity mentions that likely correspond to anaphors and antecedents, then perform bridging resolution on the predicted mentions; and (2) the gold phase, which is essentially the same as the predicted phase except that bridging resolution is performed on the given gold mentions. in this section, we analyze the performance of the teams that participated in the bridging resolution track. the utd team @xcite and the ku team @xcite participated in both phases, whereas the inria team @xcite only participated in the gold phase. in other words, two teams participated in the predicted phase, and three teams participated in the gold phase. we will use their team name to refer to the bridging resolution systems they developed. to make it clear which phase a system was developed for, we will augment the team name with a superscript that encodes the phase. for instance, we will use utd p and utd g to refer to the systems the utd team developed for the predicted phase and the gold phase respectively. ## discourse deixis resolution the shared task divides the evaluation of discourse deixis resolution into two phases: (1) the predicted phase, where a system needs to first identify all of the entity mentions that likely correspond to anaphors and antecedents, then perform discourse deixis resolution on the predicted mentions; and in this section, we analyze the performance of the teams that participated in the discourse deixis resolution track. the utd team @xcite participated in both phases, whereas the dfki team @xcite participated in the predicted phase only. in other words, two teams participated in the predicted phase, and one team participated in the gold phase. as in bridging, we will use their team name to refer to the discourse deixis resolution systems they developed, augmenting the team name with a superscript that encodes the phase the system was developed for. ## concluding remarks we presented a cross-team analysis of the systems that participated in each of the three tracks of the codi-crac 2021 shared task. as noted in the introduction, conducting a systematic analysis that can provide insightful observations is by no means a trivial task. we believe that future cross-team analyses can be improved in a number of ways. first, any analysis should be based on the links identified by a system rather than the output clusters they generated. the reason is that a cluster contains both the links identified by a system and the links automatically created via transitive closure. hence, to better understand the mistakes made by a resolver, we should request the teams to provide the links their systems identify in addition to the clusters they produce so that we can conduct cross-team analyses on the links instead. we do note, however, that this may not be easy for entity-based systems, where a link is established between an anaphor and one of its preceding clusters. for these systems, assumptions may need to be made. for example, when merging takes place, we may assume that a link is established between the anaphor and the member of the cluster that is closest to the anaphor. second, to facilitate cross-team comparisons, the teams should be asked to run diagnostic tests provided by the organizers on their systems so that additional insights into their behavior can be gained. for instance, since mention extraction performance has a significant correlation with resolution performance, we will not be able to quantify its impact on resolution performance or directly compare different models in terms of their resolution performance that is not affected by their mention detection performance unless we provide a system with gold mentions. hence, a useful diagnostic test, which was employed in the conll-2012 shared task on unrestricted multilingual coreference @xcite , involves running the systems on the test data when gold mentions are given. another useful diagnostic test, which involves running the bridging resolvers and the discourse deixis resolvers on the test data when gold anaphors are given, would allow us to directly compare the resolvers in terms of their antecedent selection performance. while this shared task has a gold phase for the bridging track and the discourse deixis track in which gold mentions are given, these gold mentions are somewhat different from what one would expect. specifically, while the participants expected to be given task-specific gold mentions, they were instead provided with gold mentions that were created by taking the union of the three sets of gold mentions collected from the three tracks. worse still, the gold mentions provided in the discourse deixis track did not even include the antecedents. unfortunately, this somewhat unconventional definition of gold mentions was not clearly communicated to the participants and has caused some confusion among them. third, our analysis could be improved with an analysis of annotation quality. strictly speaking, an analysis of annotation quality should not be part of a cross-team analysis, but for this shared task annotation quality may have played a role in the performance of the participating systems given that there are linking decisions that we do not agree with based on our casual inspection of the annotated data. having said that, we are not sure whether it is possible for us to assess annotation quality since the annotation guidelines are not made available to the participants. fourth, from an analysis point of view, it may be a good idea to include lea (moosavi and strube, 2016) as one of the evaluation metrics. as moosavi and strube point out, it is not easy to interpret the scores provided by existing scorers such as muc, b 3 , and ceaf e and lea is designed to partially address this problem. fifth, our analysis did not focus on the differences among the four datasets. for instance, in discourse deixis resolution, utd g achieved significantly better results than utd p on persuasion but not the other datasets. a cross-dataset analysis could shed light on why systems exhibit different trends on different datasets. having said that, the organizers should take an active role in explaining to the participants the differences among the different datasets and the unique challenges associated with each of them so that the participants know why these four datasets were chosen, rather than have them figure these differences out on their own. finally, to facilitate cross-team analyses, the organizers should include as much relevant information in the system prediction files that they make available to the participants as possible. currently, these files merely contain the pairwise predictions made by the systems as well as the gold links they missed. some potentially useful information that can also be provided in these files includes the sentence/turn distance between the mention pairs and their surrounding contexts. while this information can be extracted by the participants from the raw system outputs, simply laying the burden on them will likely deter them from conducting an analysis.