Language Resources From Prominent Born-Digital Humanities Texts are Still Needed in the Age of LLM s (2024)

Abstract: The digital humanities (DH) community fundamentally embraces the use of computerized tools for the study and creation of knowledge related to language, history, culture, and human values, in which natural language plays a prominent role. Many successful DH tools rely heavily on Natural Language Processing methods, and several efforts exist within the DH community to promote the use of newer and better tools. Nevertheless, most NLP research is driven by web corpora that are noticeably different from texts commonly found in DH artifacts, which tend to use richer language and refer to rarer entities. Thus, the near-human performance achieved by state-of-the-art NLP tools on web texts might not be achievable on DH texts. We introduce a dataset carefully created by computer scientists and digital humanists intended to serve as a reference point for the development and evaluation of NLP tools. The dataset is a subset of a born-digital textbase resulting from a prominent and ongoing experiment in digital literary history, containing thousands of multi-sentence excerpts that are suited for information extraction tasks. We fully describe the dataset and show that its language is demonstrably different than the corpora normally used in training language resources in the NLP community.

## introduction the digital humanities (dh) research community makes up a large user base for natural language processing (nlp) tools and algorithms @xcite . digital humanists have long been using cultural heritage data for meaningful nlp work, where nlp in dh includes everything from linguistic analysis of change over time within large linguistic corpora @xcite to narratology @xcite to literary history @xcite and stylometry @xcite . however, there are risks associated with llms that are particularly relevant to dh. unlike the average web document, texts in the humanities tend to use rich and complex writing styles, historical language, and references to under-represented longtail entities @xcite . llms have known problems with bias towards the contemporary and popularity bias @xcite . @xcite warn of a "spiral of silence" where over time, by iteratively training on llm-generated content, llm-based retrieval systems deprioritize accurate human-generated content and lose diversity in the information they return. if future nlp is dominated by llms that ignore the outliers that are so important to humanities scholarship (d @xcite , this will negatively impact humanities research, our sense of history, and the public. as @xcite assert, "marginality and uniqueness are what humanities scholars often seek to discover and analyse". we need curated datasets for evaluating and fine-tuning llms with the priorities and expertise of humanists at their core. for llms to effectively and responsibly leverage this data and become reliable for dh needs, researchers developing these models and the tools that use the models, need to collaborate with data experts. as @xcite point out, there is a need for cross-fertilization of ideas and more communication across the nlp and dh communities. lincs @xcite is an example of computer science (cs) and humanities practitioners working together to extract knowledge from dh texts in the form of linked data connected to web pages to create machine-readable data that could ultimately enhance llms. however, the inability of current systems to handle the ontological nuances of the source data plus the absence of entities from popular knowledge bases (kbs) like wikipedia and wikidata @xcite necessitates manual entity linking, ontology mapping, and data validation. there is an opportunity here for nlp developers to better support such projects with systems optimized for cultural heritage data. to contribute to these efforts, we create an nlp dataset through a collaborative effort between computer scientists and humanists. our dataset, orlando (release) @xcite , consists of 12,627 unique text chunks with over 40,000 entity mentions across four entity types that are manually linked to external entity uris and annotated with 79 unique inter-and cross-sentence relations. the source texts are biographies of historical writers from a large and representative born-digital humanities corpus created by the orlando project @xcite (appendix a). these source biographies are originally expressed as xml documents, written and thoughtfully hand-annotated in english by dh scholars, using language demonstrably richer than that found in typical llm training corpora. we extract our dataset from the source while ensuring a high rate of long-tail entities, and preserving the ontological nuances of the source texts' entity and relationship annotations, which we augment with manually-confirmed entity uris ( §3). this makes the dataset particularly wellsuited for information extraction tasks such as entity linking (el) and relation extraction (re), as elaborated in §5. orlando's text complexity (examples in appendix a.1) makes it an interesting subject of study for what machine-aided tools can process. we conduct a series of linguistic analyses ( §4) to show that, compared to other genres of text such as news, encyclopedia, or web pages, the orlando data is more complex in terms of both lexical and syntactic aspects. in light of that, we test whether the orlando data is out-of-distribution for stateof-the-art large language models such as llama 2 @xcite , using metrics based on mahalanobis distance @xcite and kernel density estimation @xcite . the test highlights that the orlando data is evidently further from the distributions of the training corpora of llms than baseline corpora. this suggests that llms, when used out-of-the-box, may suffer from poorer generalization, lower accuracy and higher aleatoric uncertainty @xcite when processing complex dh text like that from orlando. we hope that this data will encourage nlp tool developers to embrace the challenges posed by dh texts and seek collaboration with the data experts, leading to research, data, tools and systems that would be valuable across disciplines. ## related work our dataset is a unique addition to the important yet disproportionally scarce collection of information extraction datasets derived from humanities texts and created collaboratively by nlp and dh researchers. there are countless datasets created to benchmark information extraction models @xcite and many works that perform such benchmark evaluations @xcite , but they typically lack the humanities perspective. there are exceptions, with examples including but not limited to menini et al. (2022)'s information extraction benchmark relevant to cultural historians interested in textual descriptions of smells in historical documents; delmonte and busetto (2023)'s investigation of bert's limitations when applied to linguistically complex italian poetry; pedinotti et al. (2021)'s diagnostic dataset and evaluation of transformer-based language models on generalized event knowledge; and bamman et al. (2020)'s challenging coreference resolution dataset for literary texts. these works focus on other genres of text than that of orlando, and our domain allows us to provide hand-curated crossdatabase annotations for entity mentions, which is crucial for the evaluation of el. compared to works that evaluate bert and other specialized models, we focus on the suitability of llms for humanities-related information extraction tasks. this is critical as llms are becoming the status quo for many nlp tasks @xcite and llms are often used with a different paradigm: zero-shot prompting rather than fitting to the target domain. there are many other valuable datasets coming out of the humanities 2,3,4 . major differences between these datasets and ours are that most of these projects release their entire research corpora as raw text with humanities research as the target task rather than information extraction or llm benchmarking or fine-tuning. our approach was to look at a prominent humanities dataset with challenging language, consult with dh scholars to understand what level of information was important to keep, and then selectively sample it with the intent of keeping difficult chunks that contain many diverse entities and relations. our text analysis of orlando draws on work in evaluating text readability @xcite , but our work deviates as we apply readability measures to compare corpora used in nlp models. we use the popular flesh-kincaid grade level @xcite which suits our chunklevel data compared to other metrics like @xcite which requires paragraph statistics and discourse coherence. see @xcite work for a corpus linguist's review of computational corpus analysis. our work interrogates whether a corpus is out-of-distribution of an llm's training corpus, picking the best-performing indicators in recent discussions @xcite @xcite . ## creating the orlando dataset derived from the original orlando xml documents, we release a simplified and easily machine processable json dataset, orlando (release). through this collaboration with the data experts, we simplify the complexly nested embedded annotations into an easy-to-use benchmark, without abstracting the nuance of the original entity and relation types. according to the license, we can release 10% of the orlando textbase. instead of uniformly sampling from all sentences or entire documents, we release text chunks of up to 4 sentences each that capture valuable cross-sentence relationships and helpful context for coreference resolution and el tasks. we filter out text chunks containing too few entities or relations to ensure a high density of useful text. when our extractions come from overlapping chunks, we merge smaller ones into larger ones. we select the included text chunks randomly but with constraints to keep the original frequency distribution for relations and to prioritize the inclusion of person mentions with external entity links. our sampling process does not alter the distribution of data as it is uniform sampling in a stratified fashion that preserves the long tail distributions of relation and entity types. orlando (release) has 12,627 unique text chunks with over 40 thousand entity mentions across four entity types, with the majority being person mentions. and presence of external entity links, and breaks down person mentions into the primary subjects of the biographies and other mentioned persons. compared to typical re benchmarks with few broad relations, our dataset contains 79 unique relations, 30 of which are present in at least two contextual categories. the full list of relations and categories with frequency statistics are in the appendix b tables 8 , 9 , and 10. ## corpus comparison we compare orlando with corpora of varying genres to determine its complexity for human readers and automated processing. ## exploring dataset use cases the unique linguistic features of orlando texts have made them subjects of study in applications like text simplification @xcite . our dataset enriches the texts with annotations focused on entity and relation mentions, making it well suited for information extraction tasks. we demonstrate the data's usefulness through off-the-shelf el and re systems simple to use without customization. entity linking we use the zero-shot el system blink (wu et al., 2020, details in appendix d.1), which uses transfer learning and is potentially useful when applied to the humanities because it should not require training data from the target do-bio subjects others by entity 0.89 0.80 by mention 0.92 0.81 ## conclusion we argue that the impressive results reported by fast-paced nlp research might not reach tools in the dh community due to inherent differences in the kinds of texts they use. in particular, we note that llms have been shown to underperform with out-of-distribution inputs compared to experiments where test data comes from the same distribution as the training data (which is the norm in nlp research). while there is currently no machinery to predict the gap in observed performance for a given dataset, we report statistics derived using state-of-the-art methods that indicate noticeable differences between a corpus derived from a prominent borndigital dh textbase and corpora commonly used as training data in nlp research. we contribute this collaboratively developed dataset and argue for its potential to help close the gap between dh scholars and nlp system developers by serving as a benchmark for existing (and future) tools, as well as a resource for tool development.