Longtonotes: O nto N otes with Longer Coreference Chains (2023)

Abstract: Ontonotes has served as the most important benchmark for coreference resolution. However, for ease of annotation, several long documents in Ontonotes were split into smaller parts. In this work, we build a corpus of coreference-annotated documents of significantly longer length than what is currently available. We do so by providing an accurate, manually-curated, merging of annotations from documents that were split into multiple parts in the original Ontonotes annotation process. The resulting corpus, which we call LongtoNotes contains documents in multiple genres of the English language with varying lengths, the longest of which are up to 8x the length of documents in Ontonotes, and 2x those in Litbank.We evaluate state-of-the-art neural coreference systems on this new corpus, analyze the relationships between model architectures/hyperparameters and document length on performance and efficiency of the models, and demonstrate areas of improvement in long-document coreference modelling revealed by our new corpus.

## introduction coreference resolution is an important problem in discourse with applications in knowledge-base construction @xcite , question-answering @xcite and reading assistants @xcite . in many such settings, the documents of interest, are significantly longer and/or on wider varieties of domains than the currently available corpora with coreference annotation @xcite @xcite . the ontonotes corpus @xcite is perhaps the most widely used benchmark for coreference (lee et al., 2013a; durrett and klein, 2013; * now at google. long documents in genres such as broadcast conversations (bc) were split into smaller parts in ontonotes. our proposed dataset, longtonotes, restores documents to their original form, revealing dramatic increases in length in certain genres. @xcite @xcite @xcite @xcite . the construction process for ontonotes, however, resulted in documents with an artificially reduced length. for ease of annotation, longer documents were split into smaller parts and each part was annotated separately and treated as an independent document @xcite . the result is a corpus in which certain genres, such as broadcast conversation (bc), have greatly reduced length compared to their original form (figure 1 ). as a result, the long, bursty spread of coreference chains in these documents is missing from the evaluation benchmark. in this work, we present an extension to the ontonotes corpus, called longtonotes. longtonotes combines coreference annotations in various parts of the same document, leading to a full document coreference annotation. a carefully trained annotation team merged coreference annotation following the annotation guidelines laid out in the original ontonotes corpus ( §3). the resulting longtonotes dataset has an average document length that is over 40% longer than the standard ontonotes benchmark. furthermore, longtonotes sees a 25% increase in the average size of coreference chains. while other datasets such as litbank @xcite and @xcite focus on long documents in specialized domains, longtonotes comprises of documents in multiple genres ( to illustrate the usefulness of longtonotes, we evaluate state-of-the-art coreference resolution models @xcite @xcite on the corpus and analyze the performance in terms of document length ( §4.2). we illustrate how model architecture decisions and hyperparameters that support long-range dependencies have the greatest impact on coreference performance and importantly, these differences are only illustrated using longtonotes and are not seen in ontonotes ( §4.3). longtonotes also presents a challenge in scaling coreference models as prediction time and memory requirement increase substantially on the long documents ( §4.4). ## our contribution: longtonotes we present longtonotes, a corpus that extends the english coreference annotation in the ontonotes release 5.0 corpus foot_0 @xcite to provide annotations for longer documents. in the original english ontonotes corpus, the genres such as broadcast conversations (bc) and telephone conversation (tc) contain long documents that were divided into smaller parts to facilitate easier annotation. longtonotes is constructed by collecting annotations to combine within-part coreference chains into coreference chains over the entire long document. the annotation procedure, in which annotators merge coreference chains, is described and analyzed in section 3. the divided parts of a long document in ontonotes are all assigned to the same partition (train/dev/test). this allows longtonotes to maintain the same train/dev/test partition, at the document level, as ontonotes ( ## annotation procedure & quality in this section, we describe and assess the annotation procedure used to build longtonotes. ## empirical analysis with longtonotes we hope to show that longtonotes can facilitate the empirical analysis of coreference models in ways that were not possible with the original ontonotes. we are interested in the following empirical questions using the datasets-ontonotes @xcite , and our proposed longtonotes and longtonotes s : ## conclusion in this paper, we introduced longtonotes, a dataset that merges the coreference annotation of documents that in the original ontonotes dataset were split into multiple independently-annotated parts. longtonotes has longer documents and coreference chains than the original ontonotes dataset. using longtonotes, we demonstrate that scaling current approaches to long documents has significant challenges both in terms of achieving better performance as well as scalability. we demonstrate the merits of using longtonotes as an evaluation benchmark for coreference resolution and encourage future work to do so.