Large Language Models Fall Short: Understanding Complex Relationships in Detective Narratives (2024)

Abstract: Existing datasets for narrative understanding often fail to represent the complexity and uncertainty of relationships in real-life social scenarios. To address this gap, we introduce a new benchmark, Conan, designed for extracting and analysing intricate character relation graphs from detective narratives. Specifically, we designed hierarchical relationship categories and manually extracted and annotated role-oriented relationships from the perspectives of various characters, incorporating both public relationships known to most characters and secret ones known to only a few. Our experiments with advanced Large Language Models (LLMs) like GPT-3.5, GPT-4, and Llama2 reveal their limitations in inferencing complex relationships and handling longer narratives. The combination of the Conan dataset and our pipeline strategy is geared towards understanding the ability of LLMs to comprehend nuanced relational dynamics in narrative contexts.

## introduction tasks like multi-agent interaction @xcite @xcite and charactercentric narrative understanding @xcite have recently gained significant attention. these tasks require a deeper understanding of complex relationships among multiple entities @xcite , thereby serving as a critical benchmark for assessing the reasoning capabilities of llms @xcite . detective stories, where characters often adopt multiple identities or aliases that are revealed at various points, are the most appropriate testbed for assessing llms' capability of deducing complex relationships. however, existing datasets designed for character-centric narrative understanding are either built on well-known stories that llms have been trained on @xcite @xcite , or consist of simpler texts @xcite @xcite ., * equal contribution. i want to kill my husband, so that i can marry his brother! i'm going to pretend to be my twin brother so i can take his money! the unlucky victim of a wrongful killing! figure 1: the example illustrates complex relationships of characters in narratives. gray-colored relationships represent surface-level information, widely known to most characters. orange-colored relationships, on the other hand, are secrets known to only one or very few individuals, often conflicting with the commonly known relationships; these are referred to as secret relationships. red-colored relationships represent inferred information, meaning they are not explicitly stated in any character's story but can be deduced by synthesising information from all characters collectively. llms struggle with such complex relationships in long narratives. however, relationships between characters are often characterised by incomplete and uncertain information in reality. these references may involve descriptions from another character's perspective, as in "a middle-aged man walked out of the main gate, with grey and white hair, and a slender build.". also, in real social scenarios, this is often not the case. for instance, as depicted in figure 1 , husband a may remain unaware that his brother b is the secret lover of his wife c and the biological father of his son d. each individual may have differing interpretations of their relationships, with these perceptions potentially conflicting based on their own perspectives. misidentified relationships can greatly impact the core conflicts and plot of a story, as these complex relationships are often central to the narrative. therefore, we have developed the benchmark, contextual narrative analysis (conan) to understand complex relationships in detective narratives. we have also outlined the desired input-output format, identified three sub-tasks within this framework, and developed hierarchical relationship categories for evaluation, drawing insights from the field of social science. conan is constructed to extract roleoriented relational graphs from detective narratives. it comprises detective narratives from various characters' perspectives, along with annotated relationships that can be deduced from the narrative. as shown in figure 1 , it includes three types of relationships: (1) public relations that are known to most people; (2) secret relations that are known only to a few or even just one person and often conflicting with the widely known relationships; (3) inferred relations, which are not explicitly articulated in any single character's story and must be deduced by combining information from multiple characters. this dataset can act as a benchmark to test the cognitive and inferential abilities of llms. additionally, it holds the potential to improve llm capabilities in areas such as narrative comprehension @xcite , simulation agents @xcite , and game agents @xcite . our quantitative experiments and qualitative analysis show our benchmark is challenging for cutting-edge llms (gpt-3.5, gpt-4, and llama2). our findings reveal that these models struggle primarily due to two reasons: (1) the complexity of information that necessitates inferential reasoning; in summary, we have made the following contributions: as shown in figure 2 , the input narrative n consists of background stories for k characters, represented as c @xmath0, c 2 , ..., c k }. the background story of each character n c i is crafted solely from the perspective of that particular character c i , with all relationships and events in the story framed based on c i 's perception. unlike the conventional narrative structure that revolves around a single protagonist, in our setup, the complete narrative n emerges as a collaborative novel, composed by intertwining the perspectives of these k characters, i.e., @xmath1. as depicted in figure 1 , for the same set of character relationships, there can be instances where some characters remain completely unaware of these connections, or where the perceptions of different characters are in direct contradiction. moreover, the narrative may introduce additional characters beyond these k individuals. some of these extra characters might play peripheral roles, while others could hold pivotal significance, such as the victim. therefore, we define all characters that appeared in the story as @xmath2, c 2 , ..., c k }, where c k ⊆ c. hence, by defining r c as the relationships among this group of characters c, we aim to perform the following three sub-tasks: infer the actual relationships between characters by considering the collection of all character-centric narratives. formally, this corresponds to r c |n . input omniscient perspective directly extract relation graph extract all characters extract relation graph of given characters extract relation of given character pairs merge pairwise relationships relation detection strategy 1 -alltogether strategy 2 -dirrelation strategy 3 -pairrelation output extract all characters congratulations, you are the corpse! don't be surprised, this is easy to explain. you use the tragedy of the twin brother drake li morette... in fact, you did kill your husband hans li morette.in the past three years, you and his brother drake have been deceiving him... you are a very talented surgeon.you have published many articles, researching all over the country... you are the head nurse of the emergency ward. you climbed to this position for your hard work and were proud... you are a priest tom, a well-known television missionaries in the united states, and have a big fan... drake li morette.txt gale li morette.txt andrew paloski.txt sylvia costa.txt father tom.txt you were a doctor at the brighton hospital. after working for more than ten years, you were appointed dean of the hospital... richard berkeley.txt all.json compare with human annotation all.json drake li morette.json drake li morette.json single character's perspective code-driven llm-driven gale li morette.json andrew paloski.json sylvia costa.json father tom.json richard berkeley.json figure 2 : input-output format and benchmark relation detection strategies. the input narrative consists of k background stories n ci that are uniquely created from the perspective of the character c i . for each narrative, we have labelled it k + 1 times, including k relationship graphs from each character's perspective and one relationship graph from omniscient perspective. our objective is to extract all characters from the given story, including those beyond the initial k characters, subsequently detect the relationships among all the extracted characters, even when they involve false or multiple identities, and finally uncover conflicting relationships to deduce the genuine nature of these relationships. to understand the relationships of characters within detective narratives, the challenges mainly manifested in three types, corresponding to the aforementioned three sub-tasks: first, in detective narratives, a character might have different identities or aliases, and these identities may surface at different points in the story. secondly, llms struggle with basic inference based on existing information due to a phenomenon known as reversal curse @xcite . for example, when we know that a is b's father, it does not necessarily imply that b is a's child according to these models. finally, the third challenge involves drawing inferences from the perspectives of multiple characters, which sometimes yield conflicting or inconsistent information. for instance, from a's perspective, b might be regarded as his mother's friend. however, from b's perspective, he has been the lover of a's mother and is actually a's biological father. ## dataset construction to investigate the capabilities of llms in comprehending detective narratives, we developed the first open benchmark dataset for this task. the construction process includes data collection and processing, and annotation of characters and their relationships. ## conclusion and future directions this research introduces a task for llms to comprehend complex character relationships in detective narratives, utilising our newly created conan dataset. this dataset highlights the current challenges for llms and aims to improve their ability in narrative contexts. challenges (1). enhancing inference capabilities of llms. our research reveals that llms face challenges in deciphering complex relationships, particularly when the input narrative contains conflicting information. to address this, recent advancements @xcite @xcite can be leveraged to augment the inference capabilities of llms. (2). optimising key information identification. we observed that llms struggle to pinpoint key information in lengthy inputs. therefore, employing methods like cosine similarity @xcite or retrieval-augmented generation (rag) foot_3 to help llms retrieve and focus on the most relevant or crucial information could be beneficial. applications (1). enhancing narrative understanding. our work can be used to analyse complex narratives in literature, films, and video games. it helps in understanding character dynamics and plot development. (2). interactive agents. ai-driven agents are widely used in many sectors, including chatbots that cater for both professional and emotional needs @xcite , interactive game development @xcite , and digital life simulation @xcite . understanding the relationships between characters in user inputs can enhance conversation quality, making these systems more empathetic and context-aware. (3). theory of mind. our dataset, built on various characters' perspectives, naturally includes insights into how characters perceive relationships and how they think others perceive them @xcite . this makes it well-suitable for theory of mind tasks @xcite .