Tracking Evolving Relationship Between Characters in Books in the Era of Large Language Models (2025)

Abstract: This work aims to assess the zero-shot social reasoning capabilities of LLMs by proposing various strategies based on the granularity of information used to track the fine-grained evolution in the relationship between characters in a book. Without gold annotations, we thoroughly analyze the agreements between predictions from multiple LLMs and manually examine their consensus at a local and global level via the task of trope prediction. Our findings reveal low-to-moderate agreement among LLMs and humans, reflecting the complexity of the task. Analysis shows that LLMs are sensitive to subtle contextual changes and often rely on surface-level cues. Humans, too, may interpret relationships differently, leading to disagreements in annotations.

## introduction plots and characters are the two key components of a narrative (among others) that contribute to a good piece of fiction @xcite @xcite . character comprehension is key to understanding narratives in literary, and psychological research @xcite @xcite . particularly, characters and their relationships are one of the basic building blocks of narratives that make them engaging and interesting. such relationships develop chapter-by-chapter in response to various events as the story progresses. for instance, figure 1 depicts how jana and anil's relationship in jana goes wild by farah heron, evolves from intense love to a painful breakup, followed by a separation and re-evaluation of their relationship to fall in love again. humans build mental models for characters and keep updating them as they read a narrative to explain such developing relationships, character's identity, their emotional status @xcite , and future behaviors @xcite he said he has no expectations. he is here only for our daughter and doesn't want to be with me. they met on a date to have a clear communication. jana texted anil daily, but didn't talk about their upcoming date. anil hugged jana and kissed her. they were both desperately trying to fill in the holes left behind. anil looked at jana's eyes, with tears in his eyes. he wrapped her up in a hug and whispered "i love you, too." jana and anil start as romantic partners followed by a tumultuous break-up but years later, co-parenting responsibilities of their daughter force them to confront lingering feelings, reevaluate their past, and rediscover love through shared growth and proximity. ⊕ (⊖) denote a positive (negative) evolution in the relationship. we first characterize evolution in the relationship in terms of predefined relationship types and different ways (such as positive, negative, or stable) in which a relationship can evolve ( §2). then, we formally define the task of tracking evolution in the relationship (see figure 1 ) between two characters ( §3), and propose several strategies based on the granularity of information provided to llms to perform the task ( §4). to address the issue of the unavailability of gold labels, we evaluate the predictions from the proposed strategies by analyzing the agreement between predictions from different families of llms ( §7). low-to-moderate agreement (α = 0.1 -0.6) between predictions from multiple llms suggests that the task is difficult even for llms with increased context window. to provide an upper bound on the performance achievable from the proposed strategies for this task, we manually examine the consensus predictions at both local and global-level ( §8). low-to-moderate agreement between humans reinforces the difficulty of the task. finally, we present a quantitative ( §9.1) and qualitative analysis ( §9.2) of the predictions from llms and disagreement between humans to shed light on the challenging nature of this task. ## characterizing evolution in interpersonal relationships prior works that model the evolution in the relationship between characters use an ontology of relationships that is either coarse-grained (cooperative vs non-cooperative) @xcite or unsupervised @xcite (such as topics from a topic model). however, relationships can be of various types such as familial (e.g., parent and siblings), social (e.g., friends and acquaintance), romantic (e.g., married and engaged), and professional (e.g., boss and colleague) @xcite @xcite . following jurgens et al. category relationship types romantic engaged, married, romantic interest, dating, onesided romantic interest, separated, ex-romantic interest, ex-engaged social stranger, acquaintance, friend, best friend anti-social competitor or enemy ## task of tracking evolution in relationship we consider tracking evolution in the relationship between characters as a classification task formally defined as follows. consider a book @xmath0, p 2 , . . . p n } consisting of n chronologically ordered 2 (in book's passages) nonoverlapping passages of a fixed length, c 1 and p5 p7 pn-1 predictor llm non-overlapping passages from book predicted trajectory p2:k1 p4:k2 p6:kj pn (rk1, ek1) (rk2, ek2) (rkj, ekj) p1 character a character b p2 p3 p4 p6 p8 @xmath1,c2 . . . @xmath2. . . predictor llm predicted trajectory (rk1,ek1) (rk2, ek2) (rkj, ekj) character a character b p1 s1 s1:2 s1:4 s1:k1 . . . s1:k2 . . . . . . . . . . . . . . . s1:n s1:kj . . . . . . c 2 as the two characters, and b c 1 ,c 2 = {p @xmath3. note that b c 1 ,c 2 is a non-contiguous sub-sequence of p 1 , . . . p n . the task is to predict a tuple (r i , e i ) where r @xmath4, respectively, denote the type of relationship and evolution (from a predefined set as described in §2) between the two characters by the end of the passage p @xmath5,c 2 given the passages p 1:i . we define evolution in the relationship between c 1 and c 2 in a book b as a trajectory t c 1 ,c 2 = {(r 1 , e 1 ), (r 2 , e 2 ), . . . , (r j , e j )} of relationship foot_2 and evolution types at each passage p k j where k @xmath6,c 2 }. ## proposed strategies for tracking evolution in relationship automatic tracking of fine-grained evolution in the relationship between characters in a book-length context poses two main challenges: (1) handling long context, and (2) lack of annotated datasets. to address these challenges, we aim to assess the zero-shot social reasoning capabilities of recent large language models @xcite @xcite with increased context window size by proposing various strategies (figure 2 ) based on the granularity of information (i.e., passages p 1:i ) provided to an llm to predict a relationship and evolution type by the end of complete passages. this strategy uses the large context window of llms to provide passages until p @xmath7,c 2 (that can fit in the window) as-is in its highest granularity to predict the status of relationship and evolution type until p i . ## experimental setup we provide details on the source of dataset, preprocessing steps, predictor and summarizer llms. ## evaluation without gold labels one of the major challenges of tracking evolution in the relationship between characters is the unavailability of gold labels and the difficulty in collecting crowd-sourced annotations due to the length of the books; making it extremely expensive, and cognitively challenging. we make a novel contribution by providing insights into the feasibility of this task without gold labels by analyzing the agreement between predictions from multiple llms. additionally, we conduct a quantitative ( §9.1) and qualitative ( §9.2) manual analysis of the predictions. owing to the increasing use of llm-asevaluators @xcite and llmas-annotators @xcite , we hypothesize that if multiple llms agree on a label then it is more likely to be the gold label @xcite . therefore, we thoroughly analyze the quality of the consensus predictions from multiple llms (conditioned on the input) via krippendorff's alpha (α), and compare it against the agreement between human annotators. we also use α scores to compare agreement between different strategies ( ## findings agreement between prediction models. we report the agreement (α scores) between predictions from different llms for summary with passage and complete summary strategies that use summaries generated from various summary models in figure 3 . while this analysis depends on the correctness of the summaries generated from llms, we do not explicitly evaluate their correctness as llm-generated summaries have been shown to be on par with human-written summaries @xcite . low-to-moderate agreement suggests that tracking evolution in the relationship is a challenging task. lower agreement for evolution type than relationship type emphasizes the difficulty of predicting fine-grained evolution. ## manual evaluation how accurate are the consensus predictions from llms? to study the upper bound performance achievable from the proposed strategies, two annotators label the relationship and evolution type conditioned on the input for different strategies. for local-level evaluation, we first select one book per trope (4 books in total) that attains the highest prediction agreement over all the predictors and summarizers across different strategies. then, we sample a maximum of 20 passages (where both the characters are mentioned) per selected book and consensus predictions from the summarizer with the highest agreement 9 for each strategy. passages are selected at random such that the consensus predictions from different strategies are different as it has a two-fold benefit: (1) the accuracy of prediction for the sampled passages acts as a good approximation of overall accuracy as the two strategies will have the same accuracy for the same predictions, and (2) it makes it easier to qualitatively compare the two strategies (as shown in we report the accuracy of prediction averaged over the two annotators and α (in parenthesis) between them in how accurately can trope be identified from the llm predicted trajectory? for global-level evaluation, we present a visualization (similar to figure 1 ) of the trajectory of evolution in the relationship between two characters to annotators 10 and ask them to select the best applicable trope out of enemies-to-lovers, friends-to-lovers, second-9 summaries from llama resulted in higher agreement between predictions as compared to that from gemma or mistral. 10 manual examination is done internally by people who frequently read novels. chance, and forbidden love foot_8 . we keep the book and characters' names anonymous to the annotators to ensure no use of online resources. we provide visuals obtained from the predictor having the highest agreement with the consensus predictions for each strategy and summarizer foot_9 . as mentioned in §5.1, we have gold trope labels for each book to compute the accuracy of trope prediction. ## analysis and discussion we present an ablation study of summary with passage strategy, need for coreference substitutions, and intermediate passages in §9.1 and shed light on the challenges with this task in §9.2. ## related work existing works that examine relationships between characters in narratives either use a fixed set of coarse-grained relations, such as cooperative or non-cooperative @xcite and familial or professional @xcite @xcite or learn a set of relationship descriptors @xcite . others classify emotional relationships between characters in fanfiction stories @xcite and harry potter novel @xcite following @xcite . another line of research analyzes the polarity and intensity of emotions of characters towards each other @xcite in shakespearean plays, or classifies interpersonal relationships from dialogues in tv series @xcite , movie scripts @xcite or detective narratives @xcite . while the above works consider static relationships, @xcite model the evolution of interpersonal relationships in novels in a supervised setting, requiring manual annotations, and model relationships as binary polarities. whereas @xcite introduce an unsupervised method, rmn (relationship modeling network), to model evolving relationships by learning a sequence of discrete states depicting the relationship between the two characters. @xcite employ psychological models to classify movie dialogues into attachment styles and association types to analyze the transformation between relationships. however, we focus on its evolution. ## conclusion this work tracks the evolution in the relationship between characters in books by proposing several strategies that differ in the granularity of information provided to the llms to assess their understanding of social relationships. without gold annotations, our analysis of agreement between predictions from multiple llms shows that providing a running summary of the type and nature of evolution in the relationship between the characters along with a passage is a better approximation of a strategy that uses all the passages until a point than providing a complete summary. overall, lowto-moderate agreement between llms as well as between humans shows the difficulty of the task. while human disagreement can be attributed to their differing interpretations of the context, qualitative analysis reveals that llms adopt surfacelevel cues, and are sensitive to subtle changes in the provided context raising questions on their true understanding of social relationships.