O nto GUM : Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres (2021)

Abstract: SOTA coreference resolution produces increasingly impressive scores on the OntoNotes benchmark. However lack of comparable data following the same scheme for more genres makes it difficult to evaluate generalizability to open domain data. This paper provides a dataset and comprehensive evaluation showing that the latest neural LM based end-to-end systems degrade very substantially out of domain. We make an OntoNotes-like coreference dataset called OntoGUM publicly available, converted from GUM, an English corpus covering 12 genres, using deterministic rules, which we evaluate. Thanks to the rich syntactic and discourse annotations in GUM, we are able to create the largest human-annotated coreference corpus following the OntoNotes guidelines, and the first to be evaluated for consistency with the OntoNotes scheme. Out-of-domain evaluation across 12 genres shows nearly 15-20% degradation for both deterministic and deep learning systems, indicating a lack of generalizability or covert overfitting in existing coreference resolution models.

## introduction coreference resolution is the task of grouping referring expressions that point to the same entity, such as noun phrases and the pronouns that refer to them. the task entails detecting correct mention or 'markable' boundaries and creating a link with previous mentions, or antecedents. a coreference chain is a series of decisions which groups the markables into clusters. as a key component in natural language understanding (nlu), the task can benefit a series of downstream applications such as entity linking, dialogue systems, machine translation, summarization, and more @xcite . in recent years, deep learning models have achieved high scores in coreference resolution. the end-to-end approach @xcite jointly scoring mention detection and resolution currently not only beats earlier rule-based and statistical methods but also outperforms other deep learning approaches @xcite . additionally, language models trained on billions of words significantly improve performance by providing rich word and contextlevel information for classifiers @xcite . however, scores on the identity coreference layer of benchmark ontonotes dataset @xcite do not reflect the generalizability of these systems. @xcite pointed out that lexicalized coreference resolution models, including neural models using word embeddings, face a covert overfitting problem because of a large overlap between the vocabulary of coreferring mentions in the ontonotes training and evaluation sets. this suggests that higher scores on ontonotes-test may not indicate a better solution to the coreference resolution task. to investigate the generalization problem of neural models, several projects have tested other datasets consistent with the ontonotes scheme. @xcite conducted out-ofdomain evaluation on wikicoref @xcite , a small dataset employing the same coreference definitions. results showed that neural models (with fixed embeddings) do not achieve comparable performance (16.8% degradation in score) as on ontonotes. more recently, the e2e model using bert @xcite showed gains on the gap corpus @xcite using contextualized embeddings; however gap only contains name-pronoun coreference, a very specific subset of coreference, and is limited in domain to the same single source -wikipedia. though previous work has already identified the overfitting problem, it also has three main shortcomings. first, the scale of out-of-domain evalua- in this paper, we explore the generalizability of existing coreference models on a new benchmark dataset, which we make freely available. compared with work using wikicoref and gap, our contributions can be summarized as follows: ontonotes and similar corpora ontonotes is a human-annotated corpus with documents annotated with multiple layers of linguistic information including syntax, propositions, named entities, word sense, and within document coreference @xcite . it covers three languages-english, chinese and arabic. the english subcorpus has 3,493 documents and ∼1.6 million words. wikicoref, which is annotated for anaphoric relations, has 30 documents from english wikipedia @xcite , containing 7,955 mentions in 1,785 chains, following ontonotes guidelines. ## dataset conversion gum's annotation scheme subsumes all markables and coreference chains annotated in ontonotes, meaning we do not need human annotation to recognize additional mentions in the conversion process, though mention boundaries differ subtly (e.g. for appositions and verbal mentions). since gum has gold syntax trees, we were able to process the entire conversion automatically. additionally, most coreference evaluations use gold speaker information in ontonotes, which is available in gum (for fiction, reddit and spoken data) and could be assembled automatically as well. the conversion is divided into two parts: removing coreference relations not included in the ontonotes scheme, and removing or adjusting markables. for coreference relation deletion, we cut chains by removing expletive cataphora, and identifying the definiteness of nominal markables, since indefinites cannot be anaphors in ontonotes. in addition to modifying existing mention clusters, we also remove particular coreference relations and mention spans, such as noun-noun compounding (only included in ontonotes for proper-name modi-fiers), bridging anaphora, copula predicates, nested entities ('i-within-i'= single mentions containing coreferring pronouns), and singletons (all not included in ontonotes). we note that singletons are removed as the final step, in order to catch singletons generated during the conversion process. we also contract verbal markable spans to their head verb, and merge appositive constructions into single mentions, following the ontonotes guidelines. to evaluate conversion accuracy, three annotators, including an original ontonotes project member, conducted an agreement study on 3 documents, containing 2,500 tokens and 371 output mentions. re-annotating from scratch based on ontonotes guidelines, the conversion achieves a span detection score of ∼96 and conll coreference score of ∼92, approximately the same as human agreement scores on ontonotes. after adjudication, the conversion was found to make only 8/371 errors, in addition to 2 errors due to mistakes in the original gum data, meaning that degradation due to conversion errors is marginal, and consistency should be close to the variability in ontonotes itself. ## experiments we evaluate two systems on the 12 ontogum genres, using the official conll-2012 scorer @xcite . the primary score is the average f1 of three metrics -muc, b 3 , and ceaf φ4 . ## results ontogum vs. ontonotes the last rows in each half of comparing genres both systems degrade more on specific genres. for example, while vl (with gold speaker information) fares well for both systems, neither does well on tx, and even the sota system falls well below (or around) 60s for the nw, wh and tx genres. this might be surprising for vl, which contains transcripts of spontaneous unedited speech from youtube creative commons vlogs quite unlike ontonotes data; conversely the result is less expected for carefully edited texts which are somewhat similar to data in ontonotes: ontonotes contains roughly 30% newswire text, and it is not immediately clear that gum's nw section, which comes from recent wikinews articles, differs much in genre. examples (1)-(2) illustrate incorrectly predicted coreference chains from both sources and the type of language they contain. tasmania was excluded from the study -where a voluntary iodine fortification program using iodised salt in bread , is ongoing (newswire example, incorrect span and coref: [the studywhere a voluntary...]) these examples show that errors occur readily even in quite characteristic news writing, while genre disparity by itself does not guarantee low performance, as in the case of the vlogs whose lanugage is markedly different. in sum, these observations suggest that accurate coreference for downstream applications cannot be expected in some common well edited genres, despite the prevalence of news data in ontonotes (albeit specifically from the wall street journal, around 1990). this motivates the use of ontogum as a test set for future benchmarking, in order to give the nlp community a realistic idea of the range of performance we may see on contemporary data 'in the wild'. we also suspect that prevalence of pronouns and gold speaker information produce better scores in the results. ## conclusion this paper presented ontogum, the largest open, gold coreference dataset following the ontonotes scheme, adding several new genres (including more spoken data) to the ontonotes family. the corpus is automatically converted from gum by modifying the existing markable spans and coreference relations using multi-layer annotations, such as dependency trees. results showed a lack of generalizability of existing systems, especially in genres low in pronouns and lacking speaker information. we suspect that at least part of the success of sota approaches is due to correct mention detection and high matching scores in genres rich in pronouns, and more so with gold speaker information. success for other types of mentions in ontonotes data appears to be much more sensitive to lexical features, performing well on the benchmark test set with high lexical overlap to the training data, but degrading very substantially outside of it, even on newswire texts from our ontogum data. this supports use of this challenging dataset for future work, which we hope will benefit evaluations of systems targeting the ontonotes standard.