CHIRON : Rich Character Representations in Long-Form Narratives (2024)

Abstract: Characters are integral to long-form narratives, but are poorly understood by existing story analysis and generation systems. While prior work has simplified characters via graph-based methods and brief character descriptions, we aim to better tackle the problem of representing complex characters by taking inspiration from advice given to professional writers. We propose CHIRON, a new ‘character sheet’ based representation that organizes and filters textual information about characters. We construct CHIRON sheets in two steps: a Generation Module that prompts an LLM for character information via question-answering and a Validation Module that uses automated reasoning and a domain-specific entailment model to eliminate false facts about a character. We validate CHIRON via the downstream task of masked-character prediction, where our experiments show CHIRON is better and more flexible than comparable summary-based baselines. We also show that metrics derived from CHIRON can be used to automatically infer character-centricity in stories, and that these metrics align with human judgments.

## introduction computational techniques aiming to analyze and generate literary texts have also emphasized the importance of characters in storytelling by tracking their relationships and emotional trajectories @xcite @xcite , identifying enemies and allies @xcite , villains or heroes @xcite , and representing characters via attributes @xcite , personality types @xcite , or high-level paragraph-style descriptions @xcite @xcite @xcite @xcite . in this work we propose a new 'charactersheet' based representation, chiron, standing for character-informed representations over narratives. chiron consists of structured descriptions of characters along several dimensions, encapsulating character nuance and complexity, while still being interpretable and useful for downstream tasks and analysis. it is composed of two modules: a generation module that generates longform answers to questions based on a snippet of story, and a validation module that verifies the utility and truthfulness of these answers. drawing inspiration from how writers create their own characters @xcite @xcite the editors of writer's digest books, 2014), our character sheets have four broad categories: dialogue (how a character speaks), physical/personality (how a character looks/acts), knowledge (what a character knows), and goals (what a character wants to accomplish/has accomplished thus far). we fill these categories (see story snippet character pre-trained llm generated statements generation prompt story info 1) generation module: generate responses to questions for each chiron category. reponses are statements about the character, based on the snippet. trained entailment llm icl + cot prompts validation prompt question ## storium dataset we base our work on the storium dataset, a collection of 6k stories written collaboratively online @xcite . we filter these for length (avoiding stories with too little content or too long for analysis), english language, and traditional novel writing style (as opposed to game-playing). this leaves us with 529 stories, containing an average of 5.5k words in 42 entries. each entry is from a specific character's perspective (although as in traditional narratives the use of grammatical person and narrator omniscience varies). due to the length of many of these entries, we further split them into snippets, along paragraph lines. we select the top three most frequently occurring characters, giving us 1,403 character-story combinations and 9,021 snippets. further details are presented in appendix a. while our chiron representation can be applied to any story text, we primarily rely on sto-rium for development and validation. as each of these story snippets is centered around the given character, storium is ideal for exploring how characters are represented in narratives. an exam-ple storium snippet and character sheet based on it is shown in ## chiron: character sheets while prior work has used detailed plot outlines and knowledge graphs @xcite , developing rich character representations in a manner similar to writers is an unexplored area of research. such representations should accurately describe all attributes of a given character that may be important for the reader to know, and should be consistent (or change logically) across the story. some attributes (e.g., height, name) are well suited for knowledge graphs and other database-inspired representations, but many of the more subtle facts about characters are too complex to store in this manner (e.g., they are self-conscious about their clothes). inspired by advice for writers @xcite ; the editors of writer's digest books, 2014), we propose instead to represent characters in a long-form character-sheet structure we call chiron. we extrapolated themes and condensed lists of specific questions (e.g. "is this character intelligent, and in what way?") from these resources to create a structure more broadly applicable to any story. as seen in dialogue: how the character speaks; this could include accents, catch-phrases, whether they prefer flowery or direct prose, etc. physical/personality: descriptions of the character's appearance and personality; this could include height, scars, courage, condescension, etc. knowledge: other factual details about this character, as well as knowledge this character has learned about the world; this could include family members, a tragic backstory, where the gold is hidden, etc. goals: goals this character has gained or completed, and changes in their internal motivation; this could include finding the treasure, gaining a quest, wanting revenge, etc. we operationalize chiron's characterrepresentation task as two steps: a generation module and a validation module. (i) the generation step takes in a character and snippet of story, and uses an out-of-the-box llm to answer a series of open-ended questions about the character, and organizes these question-answer pairs into ## validation module given a story snippet, character, and statement (about said character), our validation module predicts whether the statement entails completely from the snippet. as this task is extremely domainspecific, we construct this module from two novel parts: automated reasoning and a trained entail-ment model. in this section, we describe our methodology for collecting data, setting up automated reasoning, and training our domain-specific entailment model. ## masked character prediction in this section we validate chiron on a masked character prediction task @xcite by showing it serves as a better character-representation than a competitive baseline. we define the task as follows: given an unseen storium story, we get the three most common characters by scene appearance and identify story snippets that contain all three characters. we create character sheets for each story up-to this snippet. we mask out character names from the story snippet and, given this masked story snippet and generated character-information, prompt mistral 7b instruct v0.2 @xcite to match each character to its mask-id. filtering for finished, previously unseen stories and scenes with their three most common characters gives us a dataset of 49 entries with 3 characters each, or 147 data points. ## story analysis by character density in addition to being useful for downstream tasks, we wish to show that chiron is useful as a means of automatically analyzing stories from a variety of sources. we create a metric, density, and show that it correlates with human judgements of charactercentricity before applying it to a varied set of hu-man and llm-generated story datasets. we define density as the number of sentences in a finished character sheet divided by the number of sentences in the provided story, averaged across all story-character combinations. let s denote the set of story-character combinations, c is the set of categories (e.g., dialogue, goals), and χ(s, c) is the character sheet category c given a story-character: we interpret this metric as a way of measuring the extent to which characters are the focus and driving force behind a story, or in other words a story's character-centricity. we can also compute density for each category in chiron, quantifying the extent to which a given aspect of characterization is present in a story: correlation with human judgments we evaluate our density metric against human judgments on stories produced by doc, re 3 and their variants @xcite . they are state-of-the art models, exhibiting traditional narrative style. moreover, their stories are relatively short (averaging 2k fewer words than our storium set) and thus amenable to human review. the re 3 framework introduced a promptingbased method for drafting, rewriting and editing generated stories @xcite . doc built on this framework by adding a more detailed plot outline and a controller that attempts to match generated stories to the outline. 'no-control' and 'no-outline' refer to settings without the controller and the detailed outline respectively. both re 3 and doc also have an 'interactive' setting, where humans are involved in the story-planning process @xcite . we collated the output of their systems, a total of 100 stories across 6 settings, which we annotated using a similar prolific setup to our entailment-classifier dataset. annotators were asked to rate (from 1-5) a story across five dimensions meant to encapsulate how well-defined characters are in the story and how central to the plot: roles (does the story have characters with explicitly defined roles?); traits (do the characters have well-defined traits and relationships?); actions (do their actions play a role in the story?); physical/personality (do the characters have defined physical/personality attributes?); and dialogue (do the characters have defined manners of speaking?). we sum up these scores to give us a singular metric of character importance which we correlate with our chiron-based density metric. we find that our automated density metric correlates highly with human judgements of character centricity in stories, achieving a pearson correlation of 0.753 (@xmath0.01). this allows us to use chiron as an analysis tool on datasets where human annotations would be expensive or intractable. we can also determine the correlations between density dialogue and density physical/personality and their respective questions in our annotation task. we find a dialogue correlation of 0.362 (@xmath1.01) and a physical/personality correlation of 0.440 (@xmath2.01), further justifying that the categories within chiron are useful for deeper analysis. comparing stories via character density we briefly show an example of how chiron's density metric can be used to differentiate between different stories and styles. we compute our density metric across the following datasets, for the most common character in each story: storium (akoury et al., 2020): we expect to find a high density from storium stories due to the extreme character focus inherent to the way the stories were created (almost akin to role-playing). we report density based on all-snippets by default as it is the most comparable between datasets, but we also report density on the onlyrole setting. new yorker ttcw (chakrabarty et al., 2024): this dataset is derived from new yorker stories and llm-generated stories from a premise. @xcite showed that the new yorker stories were much more creative (as measured by the ttcw 4 metrics) than the llmgenerated stories, and were in particular more original in their form, thought, theme and content. we expect to find low density from the new yorker stories due to their differences from conventional narratives (a few stories do not even have named 4 ttcw is a short for torrance test of creative writing. ## conclusion in this work we introduce chiron, a novel character-sheet representation for long-form narratives consisting of a zero-shot generation module (in the form of structured question-answering) and a validation module composed of automated reasoning and a trained entailment-model. outof-the-box llms frequently (32.6% of sentences) produce character descriptions inconsistent with the story, but our validation module can differentiate with 0.930 precision between consistent and inconsistent statements. on the downstream task of masked-character prediction, chiron outperforms (11.6% gain) a comparable summary-based baseline, which suggests that the representation would be useful for other character-centric tasks. finally, we show that density, an automated metric derived from chiron, correlates with human judgements of character-centricity and can be used to differentiate story styles. in future work, we plan to leverage chiron to generate more characterdriven stories.