Evaluating LLM s for Quotation Attribution in Literary Texts: A Case Study of LL a M a3 (2025)

Abstract: Large Language Models (LLMs) have shown promising results in a variety of literary tasks, often using complex memorized details of narration and fictional characters. In this work, we evaluate the ability of Llama-3 at attributing utterances of direct-speech to their speaker in novels. The LLM shows impressive results on a corpus of 28 novels, surpassing published results with ChatGPT and encoder-based baselines by a large margin. We then validate these results by assessing the impact of book memorization and annotation contamination.We found that these types of memorization do not explain the large performance gain, making Llama-3 the new state-of-the-art for quotation attribution in English literature. We release publicly our code and data.

## introduction quotation attribution, or the automated attribution of utterances to fictional characters, is of crucial importance for character analysis in digital humanities @xcite @xcite . however, quotation attribution remains a challenging task, and recent approaches still struggle to find methods that generalize across writing styles. a few works have explored the use of llms for quotation attribution in novels, by extracting conversations directly with chatgpt @xcite or by asking chatgpt to attribute a single quote given its surrounding context @xcite . yet, these works do not propose a systematic evaluation of llms for quotation attribution in literary works. another significant evaluation drawback in assessing llms is the lack of analysis regarding book memorization and annotation contamination, which can hinder their generalization abilities. book memorization occurs when an llm is able to generate specific passages of texts in a novel, and is correlated with its frequency in pretraining data @xcite . in contrast, data contamination arises when an llm has memorized evaluation data, enabling it to produce labels without reasoning @xcite . to avoid confusion, we refer to data contamination as annotation contamination. addressing both issues is essential when evaluating llms on literary tasks, as they can significantly impact the understanding of its performance on downstream tasks. in this work, we start by evaluating the performance of llama-3 8b on the project dialogism novel corpus (pdnc) @xcite , a corpus of 28 english novels. we selected llama-3 8b due to its popularity, its impressive performance on various tasks @xcite , and because its pretraining corpus only includes data up to march 2023, which makes the second release of pdnc annotations not included in the pretraining data. we carefully designed prompts with chainof-thought reasoning @xcite , and use the larger context size of llms to directly attribute all quotes in a given chapter. our results indicate that this method improves attribution accuracy compared to predicting a single quote in a contextual passage. we next conduct an evaluation of book memorization and annotation contamination to determine whether llama-3's success stems from its reasoning abilities or its capacity to memorize passages and annotations. we found that our llama-3 based approach demonstrates remarkable performance, improving attribution accuracy by 12 points against state-ofthe-art systems on the first 22 novels on pdnc and by 9 points on the remaining novels. besides, we could not find signs of annotation contamination on the first 22 pdnc novels, and we show that although memorization impacts speaker predictions on a subset of quotes, a majority of successful predictions can be attributed to the reasoning ability of llama-3. we validate this finding by evaluating the llm on a recently published novel not included in its pretraining data, where our approach performs on-par with the current state-of-the-art system, booknlp+ @xcite . besides, we found that our approach combined with the larger llama-3 70b reaches an almost perfect accuracy. to sum up, our contributions are: we propose this new measure as other metrics @xcite failed to detect memorization of canonical literature when used with llama-3 8b. we validate our measure following a similar evaluation protocol as @xcite . ## related work llms for literary tasks large language models (llms) have shown promising results in a variety of literary tasks related to narrative understanding @xcite @xcite @xcite or character understanding and profiling @xcite . their capacity of memorizing important details of fictional characters has also been studied for character understanding @xcite @xcite . in this work, we assess llms on the quotation attribution task systematically by accounting for memorization and annotation contamination. for this, we introduce a new measure of book memorization and show that llama-3's state-of-the-art results are not explained by memorization but rather by its reasoning ability. his wife represented to him how absolutely necessary such an attention would be from all the neighbouring gentlemen, on his returning to netherfield. ## data we use the project dialogism novel corpus (pdnc) @xcite , which contains 28 novels published between the 19th and 20th century, resulting in 37,131 quotes annotated manually with quotation attribution. pdnc is currently the largest dataset of quotation attribution. pdnc quotes are categorized into three types: anaphoric quotes, introduced with a speech verb and a pronoun or common noun, implicit quotes, where no narrative details about the speaker are provided and explicit quotes, which occur when the narrator identifies the speaker using a speech verb and a proper named-mention. examples are given in figure 1 . among pdnc novels, 22 novels were released in july 2022 (pdnc 1 ), while 6 novels were added in june 2023 (pdnc 2 ). the latter subset will be crucial to test for annotation contamination since it was released after llama-3 8b's knowledge cutoff (march 2023). additionnaly, we fully annotated a new novel that was published after this cutoff. following pdnc guidelines, one author annotated all quotes and a second author a subset of 5 chapters. the inter-annotator agreement, measured by cohen's κ score, reached 97% indicating almost perfect agreement. a total of 1530 quotes were annotated. we use this recent novel to assess llama-3's generalization ability. ## quotation attribution we divide each novel by chapters, and chunk each chapter using 4096 tokens with a stride of 1024 tokens. we modify the raw text by assigning a unique identifier to each quote starting from 1 to n, where n is the number of quotes in the chunk. we also build a character-to-alias list using the gold character-list from pdnc that we include in the prompt. given the modified text and the list of character aliases, we prompt the model to predict the speaker of quotes 1, . . . , n sequentially. we use llama-3 8b instruct for all experiments, and test the 70b version on the unseen novel as its annotations are not included in the larger model pretraining data. more details are provided in appendix a. baselines we compare to @xcite chat-gpt's (gpt-3.5-turbo-0613) chain-of-thought prompting strategy where the model is prompted with a target quote and its surrounding context. we also compare to the current state-of-the-art on pdnc @xcite . we use the official code to train booknlp+ with the first crossvalidation split of pdnc 1 that we further employ to attribute quotes in pdnc 2 and the unseen novel. evaluation we follow previous works @xcite @xcite , and focus on major and intermediate characters, which are characters that utter at least 10 quotes in a novel. we present attribution accuracy on explicit and other quotes, (including both anaphoric and implicit utterances) @xcite . explicit utterances occur when the narrator indicates the speaker of a quote with a speech verb and a named mention, while anaphoric quotes are introduced with a speech verb and a pronoun or common noun. when no narrative information is given about the speaker of the quote, we refer to those as implicit quotes. results on the unseen novel, booknlp+ performs slightly better than llama-3 8b overall. when increasing the model size to 70b, the performance increases to an almost perfect accuracy, and we identified only 3 wrong predictions out of 1442 quotes (note that we only consider major and interemediate characters). the larger model appears to have improved reasoning abilities, yielding better attribution. while llama-3 shows surprising performance on both subsets of pdnc, we question if those results are due to its reasoning abilities. thus, we analyze the impact of memorization, reasoning and annotation contamination in the next section. ## the impact of memorization the extent to which llms have encountered books and annotations in their training data may influence and bias their assessment on downstream tasks @xcite @xcite . we thus carry out an evaluation of book memorization and annotation contamination. book memorization. we use name-cloze accuracy @xcite to quantify book memorization. this methods prompts an llm to identify a masked character name in a small passage of text. llama-3 8b achieves a 4% average accuracy on pdnc, with 13 novels showing null accuracies. surprisingly, we found null name-cloze accuracies for canonical works such as the picture of dorian gray compared to reported gpt-4 accuracies of 42%. this questions name-cloze's validity for llama-3 8b, leading us to propose a new metric: corrupted-speaker-guessing (csg). we design csg as a speaker-guessing task, providing the model with the book's title, author, a passage, and a target quote. we corrupt the passage by replacing the speaker's name with a different gender-matching name that is not used in the book. this pseudonymization approach has been used for example to build narrative-focused story embeddings @xcite . when making a prediction, the llm must decide whether to use contextual cues (reasoning) or rely on memorized information to identify the correct speaker, de-spite the misleading contextual information. more details and prompt examples are provided in appendix b we validate csg in two ways. first, we follow @xcite and present the spearman ρ correlation between memorization metrics and the average number of search results for 10-grams randomly sampled from a book across google, bing, c4, and the pile. significant correlations were found with all memorization measures (detailed in appendix c). then, we ensured that all memorization metrics returned null accuracies on the unseen novel. ## conclusion we systematically evaluate llama-3's zero-shot performance in quotation attribution, demonstrating that a simple chain-of-thought approach accurately attributes direct-speech utterances from book chapters and significantly surpasses previous state-of-the-art models by a large margin. then, we analyze the reasons behind such performance by evaluating the impact of memorization on the downstream task. our results suggest that neither book memorization nor annotation contamination are key factors contributing to this improvement, suggesting llama-3 as the current best system for quotation attribution in english literature.