The Universal Anaphora Scorer 2.0 (2023)

Abstract: The aim of the Universal Anaphora initiative is to push forward the state of the art both in anaphora (coreference) annotation and in the evaluation of models for anaphora resolution. The first release of the Universal Anaphora Scorer (Yu et al., 2022b) supported the scoring not only of identity anaphora as in the Reference Coreference Scorer (Pradhan et al., 2014) but also of split antecedent anaphoric reference, bridging references, and discourse deixis. That scorer was used in the CODI-CRAC 2021/2022 Shared Tasks on Anaphora Resolution in Dialogues (Khosla et al., 2021; Yu et al., 2022a). A modified version of the scorer supporting discontinuous markables and the COREFUD markup format was also used in the CRAC 2022 Shared Task on Multilingual Coreference Resolution (Zabokrtsky et al., 2022). In this paper, we introduce the second release of the scorer, merging the two previous versions, which can score reference with discontinuous markables and zero anaphora resolution.

## introduction the objective of the universal anaphora initiative, or ua, foot_0 is to coordinate efforts to push forward the state of the art in anaphora and anaphora resolution beyond identity anaphora, foot_1 and also covering genres such as dialogue, exemplified by datasets such as arrau @xcite , the codi-crac 2021/2022 corpora @xcite and gum @xcite for english, the prague dependency treebank (its latest version in @xcite for czech, and ancora for catalan and spanish @xcite . the initiative, modelled on universal dependencies (ud), foot_2 aims to achieve this by expanding the aspects of anaphoric interpretation which are or can be reliably annotated in anaphoric corpora, producing unified standards to annotate and encode these annotations, delivering datasets encoded according to these standards, and developing methods for evaluating this type of interpretation. the universal anaphora effort has proceeded in close collaboration with the core-fud initiative @xcite , whose objective is to facilitate research on coreference and anaphora (possibly along with morphology and dependency syntax) by converting corpora in various languages to a unified markup format, fully compatible with ud standards. an essential prerequisite to make universal anaphora-compatible corpora usable in nlp is the availability of scorers that can evaluate the interpretation produced by a system for, e.g., bridging reference @xcite @xcite @xcite , discourse deixis @xcite @xcite or split-antecedent anaphora @xcite @xcite @xcite . a first step in this direction was the introduction of the universal anaphora scorer for anaphoric interpretation @xcite , the first scorer able to evaluate system performance in all aspects of anaphoric interpretation covered by the current version of the universal anaphora proposal. this scorer was used in the codi-crac 2021/2022 shared tasks in anaphora resolution in dialogue @xcite and a revised version supporting corefud was used in the crac 2022 shared task on multilingual coreference @xcite . in this paper, we introduce the second version of the universal anaphora scorer. this release addresses two key limitations of the first release. the first limitation is the restriction to contiguous mentions, not allowing discontinuous markables such as [a tanker] .. [of orange juice] in (1.1), consisting of two chunks of text separated from s's uttering yeah. discontinuous markables are common in spoken conversations, but are also used in the craft-cr 2019 biomedical corpus @xcite and in corpora such as arrau to encode the conjuncts in noun phrases with coordinated heads such as the students and lecturers from queen mary university, which result in the discontinuous markables [the students] [from queen mary university] and [the][lecturers from queen mary university]. example 1.1 m : ... [a tanker] s : yeah m : [of orange juice] a second limitation of the ua scorer 1.0 is the inability to score the resolution of zero anaphora (unrealized arguments) as in (1.2), except in the 'gold' case in which the zero is explicitly marked in the test set. example 1.2 (it) [giovanni] i è in ritardo, così [∅] i mi ha chiesto se posso incontrar[lo] i al cinema. [en] [john] i is late so [he] i asked me if i can meet [him] i at the movies. zero anaphora is annotated in arabic and chinese ontonotes, and in several of the datasets in the corefud collection @xcite . in arabic and chinese ontonotes, zeros are marked using an asterisk * to indicate the position of the empty category in the training data and in the test data in 'gold' mode, but not in the test data in 'predicted' mode, meaning that to evaluate this second mode the scorer must be able to handle 'insertion' of tokens, resulting in evaluation problems @xcite . the new version of the scorer presented in this paper (i) incorporates the treatment of discontinuous markables developed for the corefud scorer, testing it also on the craft-cr 2019 corpus; (ii) introduces a novel treatment for the basic form of zero anaphora; and (iii) supports both the core-fud and ua markup formats. ## universal anaphora and corefud achievements of the universal anaphora initiative so far include a first proposal concerning the range of phenomena to be covered, as well as a survey of the range of existing anaphoric annotations and two proposals for markup formats extending the conll-u format developed by universal dependencies with mechanisms for marking up the range of anaphoric information covered by ua. ## the universal anaphora scorer 1.0 the universal anaphora (ua) 1.0 scorer @xcite ) is a python scorer for the varieties of anaphoric reference covered by the universal anaphora guidelines: identity anaphora, split antecedent plurals, identification of non-referring expressions, bridging reference, and discourse deixis. for identify reference, the scorer builds on the original reference coreference scorer 10 (pradhan et al., 2014) and its reimplementation in python by moosavi, foot_10 developed for the crac 2018 shared task @xcite . the reference coreference scorer, developed for use in the conll 2011 and 2012 shared tasks on the ontonotes corpus @xcite , implemented the best known metrics for identity anaphora (coreference): muc @xcite , b 3 (bagga and baldwin, 1998), ceaf @xcite , and blanc @xcite . the reference coreference scorer popularized scoring by using the average f1 value of muc, b 3 and ceaf, as originally proposed by @xcite -so much so that this average, originally known as mela, has since become known as the conll metric. moosavi's crac 2018 scorer, apart from being written in python, also implemented the lea metric (moosavi and strube, 2016) and provided a separate score for the interpretation of non-referring expressions. ## the corefud scorer 1.0 corefud scorer 1.0 was used in the crac 2022 shared task on multilingual coreference resolution @xcite . it is based on the universal anaphora scorer 1.0, reusing the implementations of all generally used coreferential measures without any modification. this guarantees that the measures are computed in exactly the same way. nevertheless, corefud scorer is capable of processing the coreference annotation files in the corefud 1.0 format. among other things, it allows evaluation of coreference for zeros. nonetheless, its version 1.0 is not able to handle a response document whose tokens are not completely identical to the tokens in the key document. this holds also for empty tokens, which virtually prevents the scorer to evaluate response documents where the zero expressions are automatically predicted. moreover, the corefud scorer re-defines matching of key and response mentions in the way to be able to process potentially discontinuous mentions, which are present in some corefud datasets. instead of comparing mention boundaries, matching is based on set/subset relations between the tokens of the mentions in question. last but not least, the corefud scorer introduced two new scores. the mor score measures to what extent key and response mentions match, no matter to which coreference entity they belong. the corefud scorer also implements the anaphor-decomposable scoring schema introduced by @xcite and applies it to zeros. this allows for measuring the quality of predicting any of the antecedents of zero anaphors. ## the ua scorer 2.0 the ua scorer 2.0 merges the functions of the ua scorer 1.0 and corefud scorer 1.0 to make them a unified scorer. it also optimises/extends the scorer's ability on handling discontinuous markables and zeros, e.g. the new scorer can handle zeros in the predicted setting and can reproduce the craft-cr 2019 shared task results. we introduce the details of the implementations in the next subsections. ## results in this section we demonstrate the scorer in practice by using it to score the submissions to two shared tasks that involved discontinous markables and zeros, crac 2022 and craft-cr 2019. ## conclusion and future work the new version of the universal anaphora scorer presented in this paper makes further progress towards the goal of providing the community with methods for evaluating systems carrying the full range of anaphoric interpretation. this version builds on the results of three separate shared tasks and additional research that enabled the universal anaphora community to test the scorer not only for a variety of types of anaphoric interpretation, but also for a range of genres covering dialogue @xcite and biomedical text @xcite , and for a variety of languages including arabic @xcite and the 13 languages covered in corefud @xcite . it revealed a number of limitations with the previous version of the scorer that needed addressing. we hope the community will take advantage of the new scorer to broaden the range of research on multilingual, multi-genre anaphoric interpretation.