Anatomy of O nto GUM — A dapting GUM to the O nto N otes Scheme to Evaluate Robustness of SOTA Coreference Algorithms (2021)

Abstract: SOTA coreference resolution produces increasingly impressive scores on the OntoNotes benchmark. However lack of comparable data following the same scheme for more genres makes it difficult to evaluate generalizability to open domain data. Zhu et al. (2021) introduced the creation of the OntoGUM corpus for evaluating geralizability of the latest neural LM-based end-to-end systems. This paper covers details of the mapping process which is a set of deterministic rules applied to the rich syntactic and discourse annotations manually annotated in the GUM corpus. Out-of-domain evaluation across 12 genres shows nearly 15-20% degradation for both deterministic and deep learning systems, indicating a lack of generalizability or covert overfitting in existing coreference resolution models.

## introduction coreference resolution is the task of grouping referring expressions that point to the same entity, such as noun phrases and the pronouns that refer to them. the task entails detecting correct mention or 'markable' boundaries and creating a link with previous mentions, or antecedents. a coreference chain is a series of decisions which groups the markables into clusters. as a key component in natural language understanding (nlu), the task can benefit a series of downstream applications such as entity linking, dialogue systems, machine translation, summarization, and more @xcite . in recent years, deep learning models have achieved high scores in coreference resolution. the end-to-end approach @xcite jointly scoring mention detection and coreference resolution currently not only beats earlier rule-based and statistical methods but also outperforms other deep learning approaches @xcite . additionally, language models trained on billions of words significantly improve performance by providing rich word and context-level information for classifiers @xcite . however, scores on the identity coreference layer of the benchmark ontonotes dataset @xcite do not reflect the generalizability of these systems. @xcite pointed out that lexicalized coreference resolution models, including neural models using word embeddings, face a covert overfitting problem because of a large overlap between the vocabulary of coreferring mentions in the ontonotes training and evaluation sets. this suggests that higher scores on ontonotes-test may not indicate a better solution to the coreference resolution task. to investigate the generalization problem of neural models, several projects have tested other datasets consistent with the ontonotes scheme. @xcite conducted out-ofdomain evaluation on wikicoref @xcite , a small dataset employing the same coreference definitions. results showed that neural models (with fixed embeddings) do not achieve comparable performance (16.8% degradation in score) to scores on ontonotes. more recently, the e2e model using bert @xcite showed gains on the gap corpus @xcite using contextualized embeddings; however gap only contains name-pronoun coreference, a very specific subset of coreference, and is limited in domain to the same single source -wikipedia. though previous work has already identified the overfitting problem, it has three main shortcomings. first, the scale of out-of-domain evaluation has been small and homogeneous: wikicoref only contains 30 documents with ∼60k tokens, much smaller than the ontonotes test set, and the single genre wiki domain in both wikicoref and gap is arguably not very far from some ontonotes materials. second, pretrained lms, e.g. bert (devlin in this paper, we explore the generalizability of existing coreference models on a new benchmark dataset, which we make freely available. compared with work using wikicoref and gap, our contributions can be summarized as follows: ## dataset conversion gum's annotation scheme subsumes all markables and coreference chains annotated in ontonotes, meaning we do not need human annotation to recognize additional mentions in the conversion process, though mention boundaries differ subtly (e.g. for appositions and verbal mentions). since gum has gold syntax trees, we were able to process the entire conversion automatically. additionally, most coreference evaluations use gold speaker information in ontonotes, which is available in gum (for fiction, reddit and spoken data) and could be assembled automatically as well. the conversion is divided into two parts: removing coreference relations not included in the ontonotes scheme, and removing or adjusting markables. for coreference relation deletion, we cut chains by removing expletive cataphora, and identifying the definiteness of nominal markables, since indefinites cannot be anaphors in ontonotes -this was done based on the lemma of determiners attached to the head token of the span (or lack of a determiner) and the pos tag of the head. in addition to modifying existing mention clusters, we order of conversion @xmath0 ## experiments we evaluate two systems on the 12 ontogum genres, using the official conll-2012 scorer @xcite . the primary score is the average f1 of three metrics -muc, b 3 , and ceaf φ4 . ## conclusion this paper presented the mechanics of the conversion of the gum corpus to the ontogum version using the ontonotes coreference scheme, creating the largest open, gold standard coreference dataset following the ontonotes scheme, which adds several new genres (including more spoken data) to the ontonotes family. the corpus is automatically converted from gum by modifying the existing markable spans and coreference relations using multi-layer annotations, such as dependency trees. results showed a lack of generalizability of existing systems, especially in genres low in pronouns and lacking speaker information. we suspect that at least part of the success of sota approaches is due to correct mention detection and high matching scores in genres rich in pronouns, and more so with gold speaker information. success for other types of mentions in ontonotes data appears to be much more sensitive to lexical features, performing well on the benchmark test set with high lexical overlap to the training data, but degrading very substantially outside of it, even on newswire texts from our ontogum data. this supports use of this challenging dataset for future work, which we hope will benefit evaluations of systems targeting the ontonotes standard.