CHATTER : A character-attribution dataset for narrative understanding (2025)

Abstract: Computational narrative understanding studies the identification, description, and interaction of the elements of a narrative: characters, attributes, events, and relations.Narrative research has given considerable attention to defining and classifying character types.However, these character-type taxonomies do not generalize well because they are small, too simple, or specific to a domain.We require robust and reliable benchmarks to test whether narrative models truly understand the nuances of the character’s development in the story.Our work addresses this by curating the CHATTER dataset that labels whether a character portrays some attribute for 88124 character-attribute pairs, encompassing 2998 characters, 12967 attributes and 660 movies.We validate a subset of CHATTER, called CHATTEREVAL, using human annotations to serve as an evaluation benchmark for the character attribution task in movie scripts.CHATTEREVAL also assesses narrative understanding and the long-context modeling capacity of language models.

## introduction narrativity occurs when characters interact with each other, triggering events that are temporally, spatially, and causally connected. this sequence of events forms the story. @xcite provided a symbolic definition of narrativity in which they asserted that narrativity occurs when the narrator a tells the perceiver b that some agent c performed the action d on another agent e at place f and time g for some reason h. @xcite used this definition to identify four main elements of any narrative: characters, attributes, events, and relations. @xcite explored different types of character interactions and emphasized the central role characters play in narratives. characters drive the plot forward through their actions, develop attributes, arouse tension and emotion in the story by creating conflicts or bonds with other characters, and embody tropes and stereotypes to relate to the audience. the vitality of characters in narratives makes character understanding an essential task in narrative research. narratologists have explored various approaches to operationalize the character-understanding task. for example, @xcite presented character understanding as a suite of document-level tasks that included gender and role identification of the character, cloze tasks, quote attribution, and question answering. @xcite adopted coreference resolution, character linking, and speaker guessing tasks, and @xcite used character relationships and relatedness to evaluate character representations. we organized the characterunderstanding tasks into the following categories. 1) identification tasks find the unique set of characters and their mentions. it includes entity recognition, entity linking, and coreference resolution. the 2) quotation task maps utterances to characters. 3) attribution tasks, such as personality classification, persona modeling, and description generation, describe the character. the 4) cloze task asks the model to fill in the correct character name given an anonymized character description, story summary, or story excerpt. 5) relation tasks, such as relation classification, draw similarities between characters. among these tasks, character attribution is the most challenging because there exists a multitude of ways to qualify a character, such as personality @xcite , adjectives @xcite , persona @xcite , archetypes @xcite , roles @xcite , and descriptions @xcite . each of these approaches has drawbacks. for example, personality scales such as big5 and mbti cannot capture all the variation in characterization and can be difficult to interpret @xcite . adjective descriptors are too general and apply only to a limited context in the story. persona roles are not explicitly defined. archetypes @xcite are abstract concepts specific to the domain of interest. character descriptions are detailed, freeform, and scalable to any number of characters. however, we cannot factor them into simpler components or use them to compare between different characters. we require a robust attribution taxonomy that can scale across different narratives, characters, and domains, is well-defined and discrete for effective character comparison, and necessitates document-level understanding to model them accurately. we developed the chatter dataset to fulfill this need. the chatter dataset uses tropes from the tvtropes website as the attribute type to describe characters. tvtropes editors and moderators comprehensively define every trope with illustrative examples from multiple media sources. tropes cover a wider range of character descriptions than personality types and archetypes, and require longer context-understanding, compared to traits and adjectives, to accurately ascertain if a character portrays some trope. unlike character descriptions, we can efficiently compare characters and their experiences using these well-defined tropes @xcite . the chatter dataset contains labels indicating whether the character portrayed the trope in the movies they appeared in. it contains 88124 character-trope pairs. we drew our characters primarily from hollywood movies across various genres. it also provides the full-length screenplays of the movies, averaging about 25k words per screenplay. we define the character attribution task as a binary classification task where, given a character-trope pair and the screenplays of the movies where the character appears, the model should predict whether the character portrayed the trope. we validate a subset of the chatter data, referred to as chattereval, using human annotations to establish an evaluation benchmark for the character attribution task. we compared the zero-shot and few-shot performance of llms and chatter's labels to assess the suitability of using the chatter dataset as a training set for the attribution task. the dataset is available at https://drive.google.com/drive/folders/ 11egmhs-zkwsase7zjenwhg17-6voexdu?@xmath0. ## evaluation data we can use chatter to train attribution models, but we require a more reliable dataset for evaluation. we annotate a subset of chatter using human raters and create the chattereval dataset. ## data statistics chatter contains 88124 character-trope pairs with an almost 50-50 split between positive and negative pairs. it covers 2998 unique characters from 660 movies and 12967 tropes. chattereval contains 1061 human-annotated character-trope pairs, covering 271 characters, 896 tropes, and 78 movies. it contains 555 positive and 506 negative pairs. tables 2 and 3 show some data statistics of chatter and chattereval. most characters appear in a single movie script in our dataset. chatter contains attribution labels for about 30 tropes for each character. segments indicate the portions of a movie script where a character speaks or is mentioned. as shown in our dataset does not contain scripts for all the movies where the character appears. for example, the character arthur fleck "joker" has appeared in multiple movies, comics, and tv shows, but chatter only contains the movie script of the joker (2019) movie. the tvtropes contributors and our raters draw their knowledge of the character from all sources. however, the attribution model predicts the attribute labels based only on the screenplay and its pretraining data. therefore, intractable character-trope pairs could exist for which the model has insufficient information. in future work, we will investigate ways to find such intractable samples in our dataset. ## related work several past studies have curated character attribution datasets for narrative understanding. finlayson (2015) annotated seven character archetypes in russian folktales to create the propplearner corpus. skowron et al. (2016 ) labeled hero, antagonist, sidekick, spouse and supporting roles in action-genre movies. brahman et al. (2021) collected character descriptions from online study guides such as litchart and sparknotes, and created the liscu dataset for the character identification and description generation task. sang et al. (2022) curated mbti personality types from the personality database for movie characters. yu et al. (2023) annotated reader's notes for character traits in chinese-translated gutenberg books. baruah and narayanan (2024) annotated screenplay excerpts for character attributes and evaluated in-context and chain-of-thought learning methods on the attribution task. ## conclusion we proposed the chatter and chattereval datasets for the narrative character attribution task. we addressed the limitations of previous datasets by curating a resource that is scalable, generalizable, well-defined and discrete. experiments showed that chatter can serve as a reliable training set and chattereval can be used as the evaluation benchmark for character attribution modeling. future work includes developing character attribution models on our datasets to aid creators and writers in analyzing their narratives.