B ook W orm: A Dataset for Character Description and Analysis (2024)

Abstract: Characters are at the heart of every story, driving the plot and engaging readers. In this study, we explore the understanding of characters in full-length books, which contain complex narratives and numerous interacting characters. We define two tasks: character description, which generates a brief factual profile, and character analysis, which offers an in-depth interpretation, including character development, personality, and social context. We introduce the BookWorm dataset, pairing books from the Gutenberg Project with human-written descriptions and analyses. Using this dataset, we evaluate state-of-the-art long-context models in zero-shot and fine-tuning settings, utilizing both retrieval-based and hierarchical processing for book-length inputs. Our findings show that retrieval-based approaches outperform hierarchical ones in both tasks. Additionally, fine-tuned models using coreference-based retrieval produce the most factual descriptions, as measured by fact- and entailment-based metrics. We hope our dataset, experiments, and analysis will inspire further research in character-based narrative understanding.

## introduction stories play a key role in shaping our understanding of the world, serving as a medium to share experiences, communicate ideas, teach, and entertain. the two main building blocks of every story are the plot and characters @xcite . characters are particularly important, as they form the primary means through which readers engage and relate to the story. understanding characters is also necessary from a computational perspective, if models are to summarize, analyse, or generate stories effectively. over the past decade, the field of natural language processing has developed computational methods to understand narratives from a character-centric perspective. previous work has focused on detecting characters (chen and choi, 2016), understanding latent personas @xcite , their emotions @xcite , and their relationships @xcite . another line of work has attempted to describe characters with a set of attributes @xcite or personality types @xcite . most prior research studies characters in short stories or adopts relatively simplistic analysis methods (e.g., summaries) when it comes to long narratives. in this paper, we focus on analyzing characters in long-form stories, a relatively understudied area that presents unique challenges not found in short stories. firstly, long stories typically contain a large number of characters with complex relationships and interactions which have a key role in the plot. secondly, characters in long stories are dynamic @xcite : they develop throughout the story and their personalities, motivations, and relationships change as the plot evolves. finally, long narratives exceed the input length that many current transformer-based architectures @xcite can process, making the problem technically challenging. we work towards addressing these challenges and study characters from a text-generation perspective, focusing on two tasks: (1) character description produces a general profile of a character (e.g., their actions, relationships, attributes) and ( 2 ) character analysis produces an in-depth interpretation of a character's personality and behaviour (e.g., how the character's personality develops, their motives, or the social context). the character description task has been introduced with the release of the liscu dataset @xcite , which contains literary book summaries paired with human-written character descriptions. however, using summaries to describe characters significantly simplifies and restricts the task. summaries contain limited information about the overall story, usually book: bleak house by charles dickens, character: esther summerson description: the narrator and protagonist. esther, an orphan, becomes the housekeeper at bleak house when she, ada, and richard are taken in by mr. jarndyce. everyone loves esther, who is selfless and nurturing, and she becomes the confidante of several young women. although she eventually does find her mother, circumstances prevent them from developing a relationship. at first a hesitant, insecure narrator, esther's confidence in her storytelling grows, and she controls the narrative skillfully. analysis: esther summerson, the narrator and protagonist of bleak house, [..] she proves to be a confident narrator who never misses the opportunity to relate others' compliments of her.[..] as her narrative gains breadth and depth, her confidence as a narrator grows. she deliberately withholds information or delays including it to give her story coherence and dramatic effect. and even though she is for the most part a reliable narrator (a narrator we can trust to accurately tell the story), she is less reliable when relaying information about her romantic life. esther nurtures everyone around her, and her first instinct is to be motherly, perhaps because she has never had a caring mother figure of her own. [..] ironically, esther, for all her caring and tenderness, is the unwitting cause of great unhappiness. [..] because of esther's illegitimate birth, lady dedlock was forever estranged from her sister, miss barbary, and was forced to carry a painful secret. because other unhappinesses, [..] we could argue that esther is indirectly responsible for these as well. figure 1 : examples of character description and analysis. both refer to the transformation of esther summerson from a hesitant to a confident narrator. however, the analysis provides more detail focusing on her skill as a narrator (red). the description includes esther's attributes and behaviour, referring to her as a selfless and nurturing figure, while the analysis provides an interpretation of this trait based on her background (green). the character description briefly touches on esther's background, while the analysis demonstrates how, ironically and indirectly, she causes pain to others (grey), adding a moral and psychological dimension. for these reasons, our work focuses exclusively on describing characters attested in full-length books. in addition, we propose character analysis as a new task, which complements and extends character description in that it requires a more indepth understanding of characters. it goes beyond just describing surface-level traits, critically analyzing the character's depth, complexity, and evolution within the narrative context. character analyses also typically explore the social, political, or historical context relevant to understanding a character and their behaviour. we show an example of these two tasks in figure 1 . additional examples can be found in appendix a. while previous work has made a significant effort to understand characters individually, treating them as isolated entities is a simplification. characters have their own arcs in a story, but they are also interconnected -their actions, motivations, and relationships are all intertwined within the narrative @xcite . based on these observations, we introduce joint character description, a variation of the character description task, where the model has to generate a description for every character sequentially. our analysis shows that although current state-of-the-art language models can benefit from knowing all characters in a story, they struggle with joint character understanding. our contributions in this work are as follows: character understanding some prior research has studied narratives from a character-centric perspective, focusing on a variety of tasks: the identification of character personality @xcite , prediction of character emotions (brahman and chaturvedi, 2020) and relationships @xcite , character detection (chen and choi, 2016), grounding @xcite , and the generation of character @xcite . another line of research has used characters as a means to generate new stories @xcite or summaries @xcite of existing narratives. however, most existing research focuses on characters of short stories and studies only specific aspects (e.g., relationships, emotions). in this paper, we focus on characters of full-length books (with an average length of approximately 100k tokens), and study them more holistically, from the perspective of understanding their attributes, actions, and behavior (description task) and how are these interpreted in the context of the narrative (analysis task). character description previous work @xcite has found that character descriptions occur commonly in human-written story summaries, thus advocating the identification of a set of character attributes as a useful intermediate step for automatic summarization. chen and gimpel (2022) introduced tvstorygen, a dataset aiming to generate tv episode recaps based on summaries and character descriptions. carlsson et al. (2021) released gandalf, a dataset containing descriptions paired with multiple character names, with the task of choosing the correct one. brahman et al. (2021) created liscu, a dataset which contains book summaries and human-written character descriptions. a part of this dataset includes full books, however, it has not been publicly released and the work focuses on describing characters from summaries, not books. the current paper contributes to this literature by introducing bookworm, a new dataset for understanding characters based on the full-length books, without assuming that summaries are available. we also introduce the new task of character analysis, which aims to generate a more detailed account of a character's personality, motives, development, and social context. we compare our dataset with liscu in section 3.2. long-context models several papers have focused on alleviating the memory requirements of transformers, which increase quadratically with the input length. sparse attention approaches like big-bird (zaheer et al., 2021), longformer (beltagy et al., 2020), and reformer (kitaev et al., 2020) combine local windowed attention with global attention on a subset of tokens, enabling modelling of much longer sequences. transformer-xl (dai et al., 2019) introduces segment-level recurrence as another technique for capturing longer-range dependencies. another line of research has sought to overcome the limitations of input length by employing retrieval-augmented generation; @xcite show that retrieval can outperform long context transformers even when using shorter input. other work @xcite processes long documents hierarchically by segmenting the input into shorter chunks and generating intermediate responses, which are then aggregated into a final summary. we propose several retrievalaugmented models for our tasks, exploring different content extraction strategies (e.g., based on characters or a retrieval engine like bm25), and show they are superior to hierarchical generation. ## modeling experiments we conduct a series of experiments to benchmark the performance of current models and analyze their abilities across different dimensions. initially, we explore the limits of simple extractive heuristics. next, we evaluate an instruction-tuned model in a zero-shot setting and contrast its performance against fine-tuned models. additionally, across our experiments, we evaluate different retrieval strategies that are generic or rely on domain-specific information and compare them with the hierarchical approach, which uses the full story. ## results there is no lead bias in book-length character understanding. our experimental results are summarized in the extractive oracle heuristic achieves the highest rouge scores across all experiments in both tasks. there is a bigger performance gap when it comes to the analysis task, where oracle experiment is ostensibly better, especially in rouge-1 and rouge-2, compared to zero-shot and fine-tuned models. this result is expected as the oracle model uses gold-standard texts to extract sentences. when considering entity mention recall, we observe that the oracle model is worse at the description task than zero-shot and fine-tuned models. interestingly, in the analysis task, while the oracle model scores lower than zero-shot models in entity mention recall, it surpasses the fine-tuned models in this metric. this result demonstrates that there is still space for improvement in the way that our experiments retrieve context and use salient entities. bertscore results for the oracle model are comparable to the lead and random heuristics, and lower compared to abstractive models. this is an expected outcome, as the oracle fails to capture the semantic information of the reference descriptions, even if it matches the gold-standard tokens. retrieval-augmented models perform best in both character description and analysis. in our zero-shot experiments, we observe that the retrieval-based methods consistently improve performance in both tasks. specifically, the coreference approach outperforms bm25 in the description task while bm25 performs better in the analysis. the hierarchical approach improves results compared to the lead baseline in the description task but does not match the performance of retrieval-based methods. in the analysis task, the performance is slightly worse than in the lead experiment. we hypothesize that this occurs because retrieving relevant information is more crucial than processing the entire story for tasks like character description and analysis, which resemble querybased summarization more than generic summarization. for our fine-tuned models, we observe trends similar to the zero-shot ones. specifically, both the bm25 and coreference-based retrieval lead to better descriptions and analyses, with the exception of longt5 in the analysis task, where the differences are only marginal. llama-3 consistently outperforms longt5 across both tasks. while fine-tuning leads to consistent improvements in the description task, this is not the case for the analysis task where fine-tuning is either comparable or inferior to the zero-shot setting. we hypothesize that there are two reasons for this, the training samples are fewer in the analysis task and the level of data contamination is higher (see appendix c). fine-tuned llama with coreference-based retrieval is the most faithful. we report qa-based and nli-based evaluation results in the nli metric has a clear preference for models fine-tuned on coreference-based input. in particular for llama, we observe a large jump in entailment accuracy over bm25. retrieving relevant context helps achieve higher entailment scores for both tasks. the only exception is the entailment accu facts related to events and relationships are hard to get right. we report the fact-based evaluation in examples of the fact-based evaluation are in appendix d. it is easier to talk about one character than about many. the model generally struggles with the joint task, performing consistently worse across metrics compared to describing each character individually. as we can see in ## discussion our experiments underline the importance of retrieving relevant context; we found that even simple methods such as statistical retrieval with bm25 or coreference-based retrieval lead to consistent improvements in all our experiments. notably, while the hierarchical approach is considered state-ofthe-art for book summarization, our experiments revealed it performs worse than retrieval in both description and the analysis tasks. the difference between retrieval-based and hierarchical approaches is significant even for character analysis. one might conjecture that processing the whole book would be beneficial, however, this is not corroborated by our results. until now, characters have been studied separately in the literature, which is a significant simplification. our experiments with joint understanding of characters show that a separate description model can benefit from knowing all the different characters in a story if we list them in the initial prompt. however, our experiments also demonstrate that models struggle to understand characters jointly, having to "read" a book multiple times to be able to describe each character separately. ## conclusions in this work, we created bookworm, a new dataset which contains books from the gutenberg project and human-written character descriptions and character analyses from literature websites. character descriptions are short and factual, while character analyses are longer; they explore the motives, personality, and development of a character and often also comment on the social, historical, or political context. we established a set of baselines using simple extractive heuristics as well as retrieval-based and hierarchical long-context models, in both zero-shot and fine-tuning settings. our experiments highlight the importance of retrieving relevant context, which leads to consistent improvements and outperforms hierarchical methods. we hope our findings will inspire future research on character analysis, and text generation from long documents more generally. we plan to develop a better suited model for the joint character description task, by keeping track of characters and their relations as the narrative evolves. evaluation is another avenue for future work. entailmentbased metrics are good indicators of model performance for retrieval-augmented approaches, but are computationally challenging for book-length inputs. question-answering evaluation helps assess the factuality of the generated text but is constrained by the availability of references, and can be too punitive (in cases where model predictions have no lexical overlap with the reference). we used a llm as a judge to perform a fact-based evaluation and gain a deeper understanding of the factuality of the different character dimensions. however, these results are again solely based on reference texts. in the future, there is a need to explore evaluation metrics that consider the full input text and are at the same time efficient and scalable.