GENTLE : A Genre-Diverse Multilayer Challenge Set for E nglish NLP and Linguistic Evaluation (2023)

Abstract: We present GENTLE, a new mixed-genre English challenge corpus totaling 17K tokens and consisting of 8 unusual text types for out-of-domain evaluation: dictionary entries, esports commentaries, legal documents, medical notes, poetry, mathematical proofs, syllabuses, and threat letters. GENTLE is manually annotated for a variety of popular NLP tasks, including syntactic dependency parsing, entity recognition, coreference resolution, and discourse parsing. We evaluate state-of-the-art NLP systems on GENTLE and find severe degradation for at least some genres in their performance on all tasks, which indicates GENTLE’s utility as an evaluation dataset for NLP systems.

## introduction in the past several years, there have been great advances in nlp system performance on various tasks. however, many of these tasks are still evaluated on in-domain data, i.e. held-out data taken from the same domain as the system's training data. while this methodology is sound, it often overstates systems' ability to perform in real-world settings, where out-of-domain (ood) data can lead to significant degradation @xcite , even when target data comes from a similar domain @xcite . for this reason, it is essential to have evaluation datasets with diverse text types, which can give a more accurate picture of systems' capabilities on ood data, especially for domains that are distant from commonly studied domains or underrepresented in existing training datasets. in this paper, we present gentle (genre tests for linguistic evaluation), a small but "extreme" open-access dataset that can be used for ood evaluation of popular nlp tasks in english, as well as for linguistic analysis of less studied genres. the nlp tasks considered here include morphosyntactic tagging and dependency parsing according to universal dependencies (ud, de @xcite , nested named and non-named entity recognition (nner), coreference resolution, entity linking (wikification), and hierarchical discourse parsing in the framework of rhetorical structure theory (rst, @xcite . 1 our data comes from eight genres explicitly selected to represent unusual and diverse data types not currently included in the english universal dependencies corpora: dictionary entries, transcripts of live esports commentary, legal documents, medical notes, poetry, mathematical proofs, course syllabuses, and threat letters. gentle enables us to answer various questions, including how well state-of-the-art (sota) models can parse ood data and whether or not ood genres are equally difficult for all nlp tasks. apart from nlp performance, we can also see whether the annotation tasks in our challenge genres are difficult for humans and how the difficulties that arise in individual genres in gentle differ from those in existing datasets. the rest of this paper is structured as follows: section 2 presents some related work on ood testing. section 3 presents an overview of the corpus, while section 4 compares the genres in the corpus in detail. section 5 evaluates human agreement and nlp system performance on our data for each task, compared to more standard ud english data. section 6 offers our conclusions. our corpus is available at https://github.com/gucorpling/ gentle . ## related work previous work has focused on the importance of genre diversity and ood evaluation for many of the nlp tasks included in gentle, supporting the general conclusion that nlp system performance tends to degrade on ood data. in coreference resolution, @xcite and @xcite point out that existing models mainly rely on lexical features (e.g. word embeddings) and may face the problem of overfitting because of the large overlap of vocabulary between training and testing data. apart from overfitting, low recall resulting from domain discrepancy is another major problem for named entity recognition (ner, @xcite . despite a recent surge in approaches for discourse-level tasks, there is still room for improvement in this area, especially for ood data @xcite . liu and zeldes (2023) investigate the impact of genre diversity in training data composition for rst discourse parsing, the task of recursively identifying relations between propositions. they show that diverse data is essential for stable and generalizable models for this task. similarly to the present work, kanerva and ginter (2022) conduct an ood evaluation of finnish dependency parsing, including constructing a relatively "extreme" ood treebank, including 5 distinct genres (web documents, clinical, online discussions, tweets, and poetry). 2 their experiments indicate that syntactic parsing performance degrades severely on ood data, particularly on the las (labeled attachment score) metric. data diversity is thus crucial for a range of nlp tasks, but the lack of diverse data available hampers training and evaluation. previous corpus construction efforts cover a wide range of english genres, for example, 5 genres in the english web treebank (ewt, @xcite for syntactic annotations, and 6 in ontonotes @xcite for ner and coreference as well. however, both datasets lack nested, non-named entities, entity linking (wikification), and discourse parsing. more recently, the ud english gum corpus (georgetown university multilayer corpus, zeldes 2017), with data from 12 genres (academic articles, biographies, conversation transcripts, works of fiction, reddit posts, how-to-guides, interviews, news articles, political speeches, textbook excerpts, wikivoyage travel guides, and youtube vlog transcripts), covers all of the annotations examined in this paper, and raises the expectation of being a possibly good training set for ood targets, due to its diverse content. experiments in this paper will therefore use our newly annotated ood gen-tle corpus to evaluate sota models trained on 2 https://github.com/universaldependencies/ud_ finnish-ood/ the already diverse gum corpus and compare their performance on both datasets. ## gentle the gentle corpus is constructed as an ood evaluation dataset, modeled on the test set for the english gum corpus. for nner and coreference resolution, the data includes nested, named and non-named entity annotations. these employ the same scheme used in gum, with 10 entity types, 6-way information status annotations, coreference and bridging links (9 edge types from gum, including split antecedents, discourse deixis, etc., see https:// gucorpling.org/gum/ ). gum-style entity linking (wikification, lin and zeldes 2021) is also provided, with an automatically produced alternate version of the entity/coreference annotations matching the ontonotes scheme @xcite ; see @xcite . the data also includes complete hierarchical discourse trees in rhetorical structure theory (rst, @xcite , following the same scheme as gum. annotation was conducted by the authors of this paper during several hackathon-style annotation sessions. although varying in expertise on each task, every annotator had previous experience annotating every layer of annotation described above. for annotation tools, morphosyntactic layers (xpos tags, lemmas, and basic dependencies), entity layers (entity and coreference), and discourse layers (edu segmentation and discourse relation) were annotated on arborator @xcite and midas loop @xcite , gitdox @xcite , and rstweb @xcite , respectively. we also double annotated a portion of the corpus to measure human agreement, which will be further described in §5. in choosing data, we attempted to select challenging types of spoken and written open-access materials that are maximally different from those already found in gum (cf. §2). texts were selected for each genre from a single source, making sure that (1) the total number of tokens falls between 2k and 2.5k tokens per genre; (2) at least 2 texts are selected to better represent the genre (as mean can only be calculated with 2 or more documents per genre). while the texts were selected randomly for most genres, the texts for some genres were manually selected. for instance, since poetry texts can be extremely short, the documents for this genre were chosen to be varied in length, as to limit the number of documents needed to reach the target token range. the chosen data is broad not only in domain, including medical, legal, and other technical areas, but also in medium (online linked resources such as wiktionary data, spontaneous spoken esports commentary, and threat letters) and communicative intent (e.g. poetry, syllabuses, and mathematical proofs). these genres can also be challenging for both humans and nlp models, as they diverge in various ways from standard training data and ## evaluation to understand how challenging gentle data is for both nlp models and humans, we evaluate representative systems on each task using the entire corpus and conduct an inter-annotator agreement (iaa) experiment by double annotating 10% of the data. however, it is also true that nlp accuracy in document-initial positions diverges from overall accuracy since documents are systematically nonhomogeneous. for example, dictionary entry beginnings are much harder to parse since they contain technical notation, foreign language etymologies, and more, while later sections typically include grammatically simple usage example sentences. therefore, we report nlp accuracy on the double annotated snippets compared to human scores in tokenization, tagging, lemmatization, and dependency parsing we use the widely employed stanza package @xcite to evaluate goldtokenized texts in while these results may be somewhat shocking, human performance is also imperfect, with xpos and lemmatization accuracy in the mid-90s, less than 3 points above stanza for tagging, and neckand-neck for lemmatization, and with human las at 84.66, about 8 points above stanza on average. to illustrate why humans disagree on syntax especially in technical genres, we offer a brief example of parsing a legal case law designation for '410 u.s. 113' in figure 2 . '410 u.s.' is a volume of us supreme court cases, including case '113' (roe v. wade) -one annotator (in black) analyzes '113' (the case) as the head, which is modified by the name of the volume that includes it, while the other treats the volume as the head, with a numerical modifier attached as dep, similar to how gum annotates cases like 'page 5.' without good intuitions about supreme court case nomenclature and very clear guidelines, any chance of perfect agreement is hampered by a myriad of such cases. ## conclusion we have introduced gentle, a new, genrediverse, richly-annotated test corpus for english. while this new resource is relatively small, the challenging genres included in the corpus are diverse not only in topic, but also in terms of medium and communicative intent. the 8 genres have considerably distinct characteristics reflected in metrics and label distributions for individual annotation layers. these genres also differ substantially from the 12 genres in the gum reference corpus. as such, gentle serves as an important complement to gum's test set, and can provide valuable insights into nlp systems' ability to perform on ood data. we found in evaluations that system performance generally degraded on gentle compared to gum, corroborating prior findings that nlp systems degrade on ood data. however, degradation was not uniform, and different genres presented differing degrees of difficulty for different nlp tasks. for dependency parsing, the steepest degradation was in syllabus and proof, while entity recognition saw particularly poor performance in legal and dictionary, and rst parsing performed lowest on esports, dictionary and poetry. it is thus necessary to have a wide variety of genres available for evaluation if one aims for a holistic understanding of the capabilities and limitations of an nlp system. moreover, it is worth noting that the annotation tasks for our challenge genres were not just difficult for the nlp systems, but for our human annotators as well. our iaa experiments showed that human annotation generally outperformed the nlp systems in terms of accuracy. however, some genres stood out as being particularly difficult for humans, such as dictionary, which suggests that it would be beneficial to develop additional annotation guidelines targeting difficult cases that arise from genre-specific phenomena. with the introduction of gentle and the results from the above evaluation experiments, we hope to encourage the use of genre-diverse test corpora for nlp benchmarks. this will allow researchers to obtain realistic measures of how nlp systems will perform on ood data, which is frequently the use case of interest in real-world applications of nlp technologies.