[Lions: 1] and [Tigers: 2] and [Bears: 3] , Oh My! Literary Coreference Annotation with LLM s (2024)

Abstract: Coreference annotation and resolution is a vital component of computational literary studies. However, it has previously been difficult to build high quality systems for fiction. Coreference requires complicated structured outputs, and literary text involves subtle inferences and highly varied language. New language-model-based seq2seq systems present the opportunity to solve both these problems by learning to directly generate a copy of an input sentence with markdown-like annotations. We create, evaluate, and release several trained models for coreference, as well as a workflow for training new models.

## introduction coreference annotation and entity recognition are key tasks for performing a wide variety of textual analyses. they provide important information about texts as well as serving as the foundation for many more complicated forms of analysis. particularly within the digital humanities (dh), these tasks are often essential for performing large-scale studies of corpora (e.g. @xcite ; @xcite ; @xcite ). however, coreference annotation is considerably more difficult than many binary classification tasks. first, coreference requires nuanced understanding of text, which has been beyond the capabilities of previous nlp. second, coreference requires structured output, such as marking spans for entity mentions and coreferent mentions, which has previously required custom software. generative large language models (llms) have recently demonstrated a capacity to solve both problems @xcite . by leveraging massive pretraining collections and billions of parameters, they can identify the subtle, nuanced patterns of language. in addition, they can generate text that matches specific text markup formats. this capability suggests that non-expert users may be able to use "out of the box" llms to generate complicated marked-up text simply by providing examples of the desired input and output. while we evaluate this process by comparing with existing custom-built coreference systems, we emphasize that the potential impact of this process extends to a much broader class of markup. to explore the promise of fine-tuning generative llms for coreference annotation, we evaluate the capabilities of several models to perform coreference annotation on sentences extracted from literary texts. previous research has shown that literary texts have unique characteristics @xcite ) that make it difficult to adapt generalized nlp models to literary settings. @xcite achieve high performance on the litbank corpus when data from the corpus is included in the fine-tuning dataset; we seek to further explore the capabilities of a model adapted specifically for literary coreference. in this work, we find that a fine-tuned t5-3b model significantly outperforms a state-of-the-art neural model for literary coreference annotation @xcite . in addition, we speculate on the ability of these models to perform more complicated, abstract annotation tasks (e.g. identifying character relationships) given its performance on this task. specifically, in this work we contribute: ## related work many researchers have used neural networks @xcite @xcite or encoder-only transformer models like the bert models as the basis for coreference systems @xcite @xcite . these methods are multistep and perform entity recognition and coreference annotation separately. some studies have explored using generative llms for coreference, but they generally either fine-tune on auxiliary tasks @xcite or use zero-or few-shot prompting @xcite . @xcite fine-tune two sizes of mt5 (xl and xxl) to output coreference annotations for multi-lingual data. they annotate speaker interactions fed to the model one sentence at a time. the model outputs either link or append actions, which are used to annotate the coreference clusters in the next model input. similarly, @xcite find that seq2seq models such as t5 perform well when directly fine-tuned to output sentences annotated for coreference in a format similar to markdown. like @xcite and unlike @xcite , we fine-tune a model to directly produce inline coreference annotations. unlike both papers, we do not attempt to link annotations between sentences. we also focus specifically on literary coreference annotation, which @xcite include but do not foreground, and compare encoder-decoder models to decoder-only models. finally, we perform a qualitative examination of the fine-tuned models' strengths and weaknesses. coreference annotation has been applied to a wide variety of domains, such as movie screenplays @xcite , biomedical journals @xcite , and fiction @xcite . coreference annotation for literary texts in a variety of languages has also received a great deal of attention @xcite @xcite @xcite . however, to our knowledge no work has yet focused on fine-tuning and evaluating generative llms specifically for literary coreference. ## data & methods our training data is drawn from the litbank corpus @xcite , which includes 100 novels written in english before 1923 representing a mix of "high literary style... and popular pulp fiction" ## results one advantage of seq2seq llms is their ability to produce complicated, structured output as text without the need for complex structured prediction model architectures. this means that we can use "off the shelf" transformers and fine-tune them using standard methods to produce coreference annotations. the problem with directly generating marked-up text, however, is that the generated output might not be purely additive: it may change the words in addition to adding annotations. we therefore evaluate each fine-tuned llm using four metrics. we measure the fidelity of the output with average levenshtein distance between the input sentence and the model output stripped of all coreference annotation. we measure annotation accuracy using f1 scores for entity recognition and coreference annotation. finally, we record whether there is an exact string match between the human-annotated output and the model output (not including leading or trailing spaces). this metric is measured as a percentage of sentences instead of a percentage of entities, as the f1 scores are. we expect that the entity and coreference f1 scores will be an underestimate of the true model performance. this is for several reasons. the first is that we only count entities as labeled when the cleaned model output (stripped of entity and coreference annotation) is exactly the same length as the input string. additionally, we only count exact entity or coreference matches. thus, a match is not made when an entity is selected but misspelled (e.g. [helene's: 1] is produced instead of [helen's: 1]) or when a different substring is selected to represent an entity (e.g. [the study behind the dining-room: 1] is selected instead of [the study: 1]). the metric also only counts a coreference annotation as correct if the exact same index is used to identify the cluster. therefore, if an extra entity cluster is labeled or missed (e.g. a sentence is annotated [the lady: 1] in the room picked up [his: 2] hat. instead of [the lady: 1] in [the room: 2] picked up [his: 3] hat.) some annotations may not be counted even though they are technically correctly identified. finally, there are cases where the annotation of an entity is somewhat subjective and human observers may side with the initial annotations or the model output. for example, the litbank annotation for the sentence as it chanced, [dale: 1] lay face down upon the floor of the loft does not mark "the loft" as an entity. however, the model output does, and one could argue that this is a location which should be annotated. for these reasons, we expect that the true performance of these models in the eyes of a human evaluator would be higher than it appears given the strict f1 scores reported. in order to provide more generous accuracy metrics that are comparable across other studies, we also use the corefud-scorer developed by michal novák and martin popel to report the models' performance using seven common coreference evaluation metrics ( ## conclusion fine-tuned generative llms show great promise for coreference annotation. they are simple to apply and can be efficiently trained for specific corpora from open-source base models. the errors made by large models in replicating inputs are minor and they are able to capture great complexity in the entities they annotate. in the future, we hope to extend this method to operate on longer contexts. specifically, we propose to pre-pend all previously identified entities to each successive input. in addition, we believe that the high performance of the large, encoder-decoder models like t5-3b suggests that these models may be capable of performing more complex annotations, such as identifying emotional states or power dynamics between characters.