Distinguishing Fictional Voices: a Study of Authorship Verification Models for Quotation Attribution (2024)

Abstract: Recent approaches to automatically detect the speaker of an utterance of direct speech often disregard general information about characters in favor of local information found in the context, such as surrounding mentions of entities. In this work, we explore stylistic representations of characters built by encoding their quotes with off-the-shelf pretrained Authorship Verification models in a large corpus of English novels (the Project Dialogism Novel Corpus). Results suggest that the combination of stylistic and topical information captured in some of these models accurately distinguish characters among each other, but does not necessarily improve over semantic-only models when attributing quotes. However, these results vary across novels and more investigation of stylometric models particularly tailored for literary texts and the study of characters should be conducted.

## introduction in prose fiction, entire universes come to life. different techniques are employed by authors to create engaging narratives and use a combination of narrator and character words to build the atmosphere and unveil the story. characters in the fictional world reveal aspects of their personalities through dialogues. in bakhtin's idea of polyphony @xcite , characters participate in dialogues in their own voice, according to their own ideas about themselves and the fictional world. automatically identifying parts of dialogues and attributing them to the character that utters them is central to many studies of large literary corpora @xcite @xcite the detection of direct-speech has been widely performed for english literature, and simple regular expression systems achieve almost perfect performances on well-formatted texts. attributing characters to quotes is more challenging and often requires solving multiple tasks: quotation identification, character identification and speaker attribution @xcite . a speaker is attributed to a quote by training a separate model to find the nearest relevant entity mention, which is then linked to a canonical character with coreference resolution models. figure 1 summarizes this process. although many approaches have been explored in this direction, there is still room for improvement using a recently proposed corpora of english novels annotated with speakers, our current work first investigates to which extent voices of characters in novels are distinguishable using authorship verification approaches applied to character utterances. then, we analyze quotes of characters in this large corpus and evaluate to which extent character-related features encoded by pretrained authorship verification models contain a predictive signal for quotation attribution. our intuition is that character-level information (such as style, preferences in topic, persona) might be used in addition to contextual information to improve quotation at-tribution models. prior stylometric studies have shown that canonical drama authors are able to create memorable characters with distinguishable voices @xcite . nonetheless, the stylometric analysis of characters in novels remains scarce, mainly due to the lack of available corpora annotated with speakers. to the best of our knowledge, exploring this type of character representations for quotation attribution has not been done before. consequently, with this work, we make the following contributions: results suggest that most characters in the pdnc corpus own distinct voices, and that they are best distinguished by models that encode both semantic and stylistic information. semantic-only models, however, seem to be better at attributing quotes than models that encode style. besides, representing characters with the quotes they uttered in a single chapter appear to contain a predictive signal for attributing quotes in other chapters, but this varies per novel. finally, our results suggest that there are semantic variations between explicit quotes (i.e. quotes where the relevant gold mention is a named mention of the speaker) and other type of quotes (anaphoric and implicit quotes, introduced in section 3), and that including stylistic information alleviates the impact of these semantic shifts when distinguishing characters voices based on explicit quotes only. ## experimental setup our goal in this work is to investigate if fictional voices of literary characters in novels are distin-guishable from a stylistic point-of-view. we also want to know if a partial signal of a character's voice derived from its explicit quotes (i.e. the gold mention linked to the quote is any named mention such as "elizabeth") is a good proxy for its overall voice. explicit quotes are straightforward to attribute to characters since they are linked to a named mention, which can then be linked to canonical characters (e.g. with coreference resolution or name clustering) more easily than when dealing with pronominal mentions @xcite . we hypothesize that if we can construct representative character embeddings based on explicit quotes only, then these representations can in turn enhance quotation attribution solutions to detect the speaker of other type of quotes. other types of quotes include anaphoric quotes (i.e the gold mention is a pronoun or noun phrase) and implicit quotes (often happens during a conversation, when no mention is linked to the quote but the speaker can be inferred from the context). finally, using the same set of character representations, we want to evaluate to which extent they contain information to attribute quotes that were not used to build the representations. to evaluate these representations, we formulate the task as the authorship verification task: given a corpus of quotes from character a (the query), a corpus of other quotes from character a and similar corpora for other characters in a given novel (the targets) and a similarity measure, we evaluate the ability of pretrained models to predict if the targets have been written by character a or not. auc is used to assess models' performances, as it accounts for how well models can rank predictions, without concerns of threshold values @xcite . we chose to frame the task as an authorship verification problem rather than closed-set authorship attribution because the number of targets (i.e. number of candidate speakers) vary for each query, which is further described in section 3.4 we first present how character representations are derived from pretrained models, and then describe how we evaluate the capacity of these representations to answer the above questions. we publicly release our code for further research foot_1 . ## discussion we conducted experiments to understand how character representations built from explicit quotes could help to improve quotation attribution. these quotes are particularly easy to attribute to characters and can thus be detected automatically to build informative character representations that can serve as additional inputs to quotation attribution systems. results presented above suggest that explicit quotes might be a good proxy for the voice of fictional characters and that semantic and stylistic information of quotes can help attribute quotes. nonetheless, we think that there might be semantic variations between explicit and other types of quotes and that adding stylistic information in representations of characters alleviates this shift. experiments conducted in this work are focused on intermediate and major characters, i.e. characters that participate more and shape the narrative. although they represent a large number of different characters, minor characters often have less impact on the story and do not contribute significantly to the overall number of quotes that we want to attribute. however, even with characters that are more quoted, we observed discrepancies in authorial patterns of explicit quoting. while some authors quote all their characters explicitly at least 5 times in a chapter, some do not. as a result, we could build queries for only 53% of intermediate and major characters in the pdnc corpus. when looking at whole novels rather than at chapters, only 11% are explicitly quoted less than 5 times, among which 17% are major characters. therefore, we can still build representations for a majority of characters, which motivated our work. we studied stylistic information encoded in two off-the-shelf pretrained authorship verification models, luar and stel. these models have been trained to distinguish thousands of authors of reddit posts, and have been shown to transfer poorly to other domains @xcite . most novels do not contain stylistic traits captured by stel, which probably explains why it is performing badly. we were aware of this limitation at first, but decided to test the model as an off-the-shelf solution to obtain stylometric representations. in the future, we plan to re-train a stel-like model on literary texts such as drama. luar encodes both semantic and stylistic information, it is thus hard to infer the dimensions of content and style it captures as well as their respective contribution to the task. its good performance on the character-character evaluation setup suggests that it gets dimensions of style that make sense in literature. more generally, interpretable authorship verification models @xcite are an interesting direction as they combine the performance of neural approaches with the interpretability of frequency-based methods. the high standard deviations across novels indicate that the task of distinguishing voices of characters is easier in some novels than in others. in the chapterwise experiment (cc evaluation), the auc of luar goes as low as 68% and as high as 91%. ideally, we would like to understand the reasons behind these variations: are some authors better at creating memorable voices? is it easier in a particular genre? interpretability and literary knowledge are key to answer these questions, that we leave for future work. ## conclusion we presented a study of recent neural approaches to authorship verification applied to literary characters. we designed three experiments to assess if such models can be used to create meaningful character representations and to assess if explicit quotes were a good proxy of a character's voice. our first evaluation focuses on the ability of these representations to distinguish characters, while our second quantifies the amount of information they contain to attribute unseen quotes. results at the character level suggest that their voices are better distinguished when using a combination of stylistic and semantic information. using style also helps to reduce the impact of the semantic shift observed between explicit quotes and other types of quotes. when attributing quotes, our results suggest that adding stylistic information does not necessarily improve over semantic-only models. we believe that the main cause is a poor domain transfer from reddit to english novels. in the future, we plan to further analyze representations built from models trained on movie scripts @xcite , which we argue should contain stylistic patterns more similar to the ones found in literary works. we also want to investigate how such representations can be incorporated into quotation attribution systems. finally, we believe our approach could be used at a larger scale to investigate which authors/genre are better at constructing unique voices for their characters. krishnapriya vishnubhotla, adam hammond, and graeme hirst. 2022. the project dialogism novel corpus: a dataset for quotation attribution in literary texts. in proceedings of the thirteenth language resources and evaluation conference, pages 5838-5848, marseille, france. european language resources association. krishnapriya vishnubhotla, frank rudzicz, graeme hirst, and adam hammond. 2023. improving automatic quotation attribution in literary novels. in proceedings of the 61st annual meeting of the association for computational linguistics (volume 2: short papers), pages 737-746, toronto, canada. association for computational linguistics. anna wegmann, marijn schraagen, and dong nguyen. 2022. same author or just same topic? towards content-independent style representations. in proceedings of the 7th workshop on representation learning for nlp, pages 249-268, dublin, ireland. association for computational linguistics. artjoms šel , a, ben nagy, joanna byszuk, laura hernández-lorenzo, botond szemes, and maciej eder. 2023. from stage to page: language independent bootstrap measures of distinctiveness in fictional speech.