Improving Automatic Quotation Attribution in Literary Novels (2023)

Abstract: Current models for quotation attribution in literary novels assume varying levels of available information in their training and test data, which poses a challenge for in-the-wild inference. Here, we approach quotation attribution as a set of four interconnected sub-tasks: character identification, coreference resolution, quotation identification, and speaker attribution. We benchmark state-of-the-art models on each of these sub-tasks independently, using a large dataset of annotated coreferences and quotations in literary novels (the Project Dialogism Novel Corpus). We also train and evaluate models for the speaker attribution task in particular, showing that a simple sequential prediction model achieves accuracy scores on par with state-of-the-art models.

## introduction we focus on the task of automatic quotation attribution, or speaker identification, in full-length english-language literary novels. the task involves attributing each quotation (dialogue) in the novel to the character who utters it. the task is complicated by several factors: characters in a novel are referred to by various names and aliases (elizabeth, liz, miss bennet, her sister); these aliases can change and be added over the course of the novel; and authors often employ differing patterns of dialogue in the text, whereby quotations are sometimes attached to the speaker explicitly via a speech verb, and at other times require keeping track of character turns over multiple paragraphs. the development of automated methods has also been hindered by the paucity of annotated datasets on which models can be trained and evaluated. existing methods for quotation attribution fall into one of two groups: those that directly attribute the quotation to a named character entity and those that treat it as a two-step process in which quotations are first attached to the nearest relevant mention of a character and mentions are then resolved to a canonical character name via a coreference resolution model. we contend that most use-cases of a quotation attribution system involve resolving the speaker mention to one among a list of character entities. thus, the usability of these systems is very much dependent on their ability to compile such a list of character entities and to resolve each attributed mention to an entity from this list. here, we use the project dialogism novel corpus @xcite , a large dataset of annotated coreferences and quotations in literary novels, to design and evaluate pipelines of quotation attribution. our analysis shows that state-ofthe-art models are still quite poor at character identification and coreference resolution in this domain, thus hindering functional quotation attribution. in contrast, @xcite link quotations directly to entities, and a list of characters and aliases is required for attribution. this list is generated with a named entity recognition (ner) model to obtain entity terms, which are then grouped together using web resources such as wikipedia. the gutentag package from @xcite contains modules for generating character lists and identifying speakers in literary texts. the former is based on the litner model @xcite , which bootstraps a classifier from a lowdimensional brown clustering of named entities from project gutenberg texts. the speaker attri-bution model is a simple rule-based approach that identifies the nearest named entity. sims and bamman (2020) annotate the first 2000 tokens of 100 novels from the litbank dataset foot_1 . quotations are linked to a unique speaker from a predefined list of entities. litbank also contains annotations for coreference for these tokens @xcite . the booknlp package foot_2 from the same group contains pre-trained models for ner, coreference resolution, and speaker attribution, although the latter is only at the mention-level. cuesta-lazaro et al. ( 2022 ) attempt to reconcile the differences in pre-requisites and methodologies of prior attribution systems by proposing a modularization of the task into three sub-tasks: quotation identification, character identification, and speaker attribution. they evaluate baselines for each component, propose a new state-of-the-art method for speaker attribution, and quantify the relative importance of each module in an end-to-end pipeline. their speaker attribution module, however, considers only named mentions in the text as candidate speakers, leading to a lower performance on implicit and anaphoric quotations. neither their dataset of 15 novels nor their model for speaker attribution have been made public, precluding comparison with our work below. in our work, we follow this modular formulation, with some key differences: (a) we evaluate an additional sub-task of coreference resolution, allowing us to (b) test an attribution model that can work with both named and pronominal candidate mentions surrounding a quotation; and (c) we evaluate our models on a publicly available dataset. ## dataset: pdnc we briefly describe here the project dialogism novel corpus @xcite . pdnc consists of 22 full-length english novels, published in the 19th and 20th centuries, annotated with the following information: characters: a list of characters in the novel. this includes characters who speak, are addressed to, or referred to multiple times in the novel. each character is identified by a main name (e.g., elizabeth bennet), as well as a set of aliases (liz, lizzie, eliza). we do not distinguish between the two, and treat each character entity as identifiable by a set of names (so that elizabeth bennet, liz, lizzie, eliza forms one character entity). quotations: each uttered quotation in the novel is annotated with its speaker and addressee(s); with the referring expression, if any, that indicates who the speaker is; and with internal mentions, i.e., named or pronominal phrases within the quotation that refer to one or more character entities. the annotations in pdnc make it ideal for evaluating several aspects of quotation attribution in novels, including named entity recognition, coreference resolution, and speaker attribution. coreference resolution: the goals here are to identify text spans that refer to a character entity (which we refer to as mentions) and to link each mention to the correct character entity or entities to which it refers. in addition to mentions that are personal pronouns such as he, she, and them, literary texts have an abundance of pronominal phrases that reflect relationships between characters, such as her husband and their father. such phrases can also occur within quotations uttered by a character (e.g., my father), requiring quotation attribution as a prerequisite for complete coreference resolution. ## experimental setup we evaluate the models for character identification, coreference resolution, and quotation identification on the entire set of 22 novels in pdnc, since we are neither training nor fine-tuning these on this dataset. for the speaker attribution models, we define the training setup below. we curate the set of mention candidates for each novel in the following manner: the mention clusters generated by booknlp are used to extract the set of mention spans that could be successfully resolved to a character entity from the annotated pdnc character lists for each novel. we append to this set the annotated mention spans (within quotations) from pdnc, as well as explicit mention spansthat is, text spans that directly match a named alias from the character list. overlaps between the three sets are resolved with a priority ranking, whereby pdnc annotations are considered to be more accurate than explicit name matches, which in turn take precedence over the automated coreference resolution model. we test with 5-fold cross-validation in two ways: splitting the annotated quotations in each novel 80/20 and splitting the set of entire novels 80/20. ## results from the homogeneity; when two named entities are assigned as aliases of each other, it is almost always correct. this shows the advantage of documentlevel named entity clustering as opposed to local span-level mention clustering for character entity recognition. the cluster quality metric, on the other hand, tells us that gutentag still tends to be conservative with its clustering compared to booknlp, which nonetheless is a good strategy for the literary domain, where characters often share surnames. performance of these models on the coreference resolution task is significantly lower ( the importance of the character identification and coreference resolution tasks can be quantified by looking the performance of the speaker attribution models ( ## analysis we briefly go over some qualitative analyses of the errors made by models in the different subtasks, which serves to highlight the challenges presented by literary text and opportunities for future research. ## conclusions and future work in this work, we quantitatively evaluated the key components of a functional quotation attribution system. we showed that the initial task of recognizing characters and their aliases in a novel remains quite a challenge, but doing so greatly improves the performance of speaker attribution by limiting the set of candidate speakers. however, with existing coreference resolution systems, a large portion of mention clusters (around 90%) remain unresolved, so this remains a problem for new research.