What does the sea say to the shore? A BERT based DST style approach for speaker to dialogue attribution in novels (2022)

Abstract: We present a complete pipeline to extract characters in a novel and link them to their direct-speech utterances. Our model is divided into three independent components: extracting direct-speech, compiling a list of characters, and attributing those characters to their utterances. Although we find that existing systems can perform the first two tasks accurately, attributing characters to direct speech is a challenging problem due to the narrator’s lack of explicit character mentions, and the frequent use of nominal and pronominal coreference when such explicit mentions are made. We adapt the progress made on Dialogue State Tracking to tackle a new problem: attributing speakers to dialogues. This is the first application of deep learning to speaker attribution, and it shows that is possible to overcome the need for the hand-crafted features and rules used in the past. Our full pipeline improves the performance of state-of-the-art models by a relative 50% in F1-score.

## introduction natural language processing has enabled a quantitative improvement in the humanities, by allowing for large-scale statistical measurements to be taken over hundreds of thousands of books compared to the order of tenths a human could analyse in a much longer time span. some examples of large-scale literary analyses include studies on characters and their descriptions within the novel, mostly focused on gender differences @xcite , and studies on character's relations by extracting social networks from novels @xcite . most of these studies demand special attention to dialogues, being a major part of character expression and interaction with other characters. dialogues play an instrumental role in plot develop- * work done during internship at amazon ment, frequently encompassing focal plot moments, especially in fiction -which is also the focus for this study. here we aim to identify direct-speech utterances that form part of dialogues and associate them with the speaking characters. such information is not only useful to enable large-scale sociotemporal studies but also crucial to many downstream challenging tasks like narrative understanding @xcite and summarising @xcite . further, the high-quality dialoguecharacter association is pertinent for generating engaging text-to-speech for novels with distinct voice profiles for characters. in the past, models that link direct speech to characters have been dominated by predefined rules @xcite or hand-crafted features @xcite . when evaluating these models the authors also presumed that a character list, together with the character's aliases, has been precompiled and that direct-speech text has been extracted. although extensions to these models that extract speaking characters in a fully automated manner exist, it is unclear what impact does the automation of the aforementioned steps has on the final performance of the model. moreover, the models have only been tested against a small dataset of three books from the same time period. these are the two questions that we aim to answer in this paper: i) how can we build flexible models that can generalise and improve with increasing dataset sizes?, and ii) what is the impact of errors propagating from each component of the pipeline, and thus where should we focus future efforts? to answer these questions, we focus on building deep learning models with the necessary inductive biases and flexibility to learn nuanced features when given a large enough dataset, as opposed to hand-crafted rules that need revisiting to generalise to different time periods, writing styles, genres, or even languages. moreover, we present a separate evaluation of pipeline's each component. ## related work attributing speakers to direct speech is a common problem for two related domains: news and literature. however, previous work (o @xcite has shown that models do not generalise to both domains. their best model obtained an accuracy of 92.4% and 84.1% on their two newswire datasets, whilst they only found a 53.3% accuracy when evaluating the same model on a literature dataset. we therefore focus on summarising progress in quote attribution for literary fiction. the early models targeting literary texts @xcite were based on the identification of speech verbs and their actors. however, the proportion of dialogues accompanied by a speechverb and an explicit mention to a character can be as low as 20% of the total quotes for some books. for this reason, consequent work (o' @xcite shifted their focus to attributing speakers to dialogues where the character is not explicitly mentioned, incorporating rules to exploit the sequential nature of conversations. these models could not improve the results of a simple nearest mention (nm) baseline that obtained a 53.3% accuracy on their test set. finally, current state-of-the-art models @xcite demonstrated how the simple nearest mention baseline could be beaten through a combination of rules and learning. both models present analysis on a limited setup: i) they report performance on a test set comprised of two books -jane austen's emma, and anton chekhov's the steppe, -and have therefore not been bench-marked on a wider range of styles or time periods, and ii) they assume the ideal circumstance of a pre-compiled list of characters, with character aliases and genders provided. we relax the second assumption when we evaluate our model, to estimate the end-to-end performance on a more diverse dataset of fifteen books. the task of speaker attribution is also closely related to other dialogue sequence problems. one such umbrella technique to solve these problems is dialogue state tracking or simply dst, where a system is tasked with estimating some conversation state variables usually the user's goals and intents. we are first to apply dst for the purpose of speaker attribution. our proposed dst-based formulation requires modification to the utterance encoder with focus on non-dialogue context, and state-variable that can generalise to states not seen in the training set. we adapt a bert-based dst model @xcite to track the speaker for every single utterance instead of tracking the user's goals and intents. our work follows a similar line of thought to @xcite ; @xcite , where the model is given a list of candidate intents (speakers in our case) embedded as inputs to the problem so that the model can generalise to unseen goals (speakers) during test time. the task of our model is to generate a score for each utterance against every candidate speaker. to recap, we present an end-to-end pipeline for speaker to dialogue attribution that leverages recent advances in large pretrained language models casting the problem as a dialogue state tracking. we empirically show that our model is capable of generalising to different styles more reliably as compared to prior hand-crafted features-based systems. further, we present this comparative study on literary texts ranging from the 1900s to the 2010s, which are more varied and diverse compared to past studies. note that usually such dataset are effort-intensive to create and not publicly available due to lack of rights to redistribution, which makes the reported result very interesting for the wider community. ## dataset our annotations consist of two independent layers, one focusing on direct speech, and one focusing on clustering mentions that refer to the same character entity. example 1: excerpt from 2001: a space odyssey. annotated direct speech is in bold, and the annotated attributed character entity inside a blue box. poole was asleep, and bowman was reading on the control deck, when hal announced: the second layer of annotations focuses on characters and their mentions. we follow @xcite and distinguish character mentions (e.g. dave, david, bowman, dr. bowman) in the text and character entities (e.g. david bow-man), to which mentions refer to. see the text in italics within example 1 to find some of the annotated character mentions. note that we don't include pronominal mentions. these mentions are then clustered per book into character entities by the annotators. we annotate a collection of 15 books sampled from the most popular titles from time epochs 1881 -2018. the annotation is carried out by 3 expert english native annotators, each reading the book in sequential order, over a brat foot_0 based annotation interface. in case the annotation from any single annotator is different, a master annotator goes through the cases and makes the correction, resulting in a dataset with a very high agreement (cohen's kappa greater than 0.9). across the books, the number of annotated dialogues varies from 200 to 5000 and characters from less than 10 to 200. we would refer these books by ids 1 through 15 and the existing 3 books as e1 (emma), e2 (pride and prejudice) and e3 (the steppe). ## model components we divide the model into three main tasks: identifying quotes, extracting unique characters and their aliases, and attributing dialogues to the extracted characters. our goal is to reduce the amount of hand-crafted rules (usually heavily biased to the small subset of documents of prior studies) where performance can be improved, and allow the model to learn nuanced features that allow it to generalise better when given a large enough dataset. we first introduce our direct speech identification component, which is purely rule-based due to the simplicity of the problem and since improving this aspect is not part of our core contribution. afterward, we focus on identifying characters and compare ner and coreference resolution deep learning models to simple rule-based systems. finally, we discuss the focal aspect of our contribution -a dst architectural adaptation to solve the speaker attribution of quotes. ## evaluation of the model components in this section, we present both the evaluation of each separate component and the final evaluation of the entire pipeline. ## conclusions and future work we have presented a speaker attribution pipeline for novels that does not rely on pre-compiled lists of characters and that performs consistently across different writing styles and time periods. our main contribution has been to develop the first deep learning model for speaker attribution, based on previous dialogue state tracking approaches. our deep learning model has the flexibility to learn nuanced features from data, compared to previous work that relied on rules or hand-crafted features. training our model on a small dataset composed of 15 different novels, we find that it outperforms the model presented in @xcite by an average of 50% f1-score. in the future, we hope to improve our model by: training it on a larger and more varied dataset, and training the model on speaker attribution together with the related task of coreference resolution. we have also presented an error analysis on the different components necessary to perform the endto-end goal of attributing characters to their direct speech utterances in novels: i) direct speech identification, ii) character extraction, and iii) speaker attribution. we have shown the need to produce literature-domain specific models targeting character extraction in order to improve the accuracy of current systems.