Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works (2024)

Abstract: Large language models (LLMs) have demonstrated impressive performance and spurred numerous AI applications, in which role-playing agents (RPAs) are particularly popular, especially for fictional characters. The prerequisite for these RPAs lies in the capability of LLMs to understand characters from fictional works. Previous efforts have evaluated this capability via basic classification tasks or characteristic imitation, failing to capture the nuanced character understanding with LLMs. In this paper, we propose evaluating LLMs’ character understanding capability via the character profiling task, i.e., summarizing character profiles from corresponding materials, a widely adopted yet understudied practice for RPA development. Specifically, we construct the CROSS dataset from literature experts and assess the generated profiles by comparing them with ground truth references and evaluating their applicability in downstream tasks. Our experiments, which cover various summarization methods and LLMs, have yielded promising results. These results strongly validate the character understanding capability of LLMs. Resources are available at https://github.com/Joanna0123/character_profiling.

## introduction the recent progress in large language models (llms) @xcite has catalyzed numerous ai applications, among which role-playing agents (rpas) have attracted a wide range of audiences. rpas are interactive ai systems that simulate various personas for applications, including chatbots of fictional characters @xcite , ai none player characters in video games @xcite , and digital replicas of real humans @xcite . in practice, llms are generally prompted with character profiles to role-play fictional characters @xcite , and these profiles are typically generated through the automatic summarization of corresponding literature using advanced llms @xcite . previous efforts have studied llms' capabilities of understanding characters from fictional works. the research on character understanding mainly concentrates on basic classification tasks, such as character prediction @xcite @xcite and personality prediction @xcite , which aims at recognizing characters or predicting their traits from given contexts correspondingly. recently, the research focus has shifted to character role-playing, primarily focusing on the imitation of characteristics such as knowledge @xcite and linguistic style @xcite . hence, these tasks fail to capture the nuanced character understanding of llms. in this paper, we systematically evaluate llms' capability on the character profiling task, i.e., summarizing profiles for characters from fictional works. for research, character profiling is indeed the first task to explore the depth of llms' character understanding via generation. this is more challenging than previous classification tasks, contributing to a more nuanced comprehension of how llms understand the character. in practice, the character profiles generated by llms have been widely adopted for rpa development @xcite @xcite , and have the potential to facilitate human understanding of characters, but their effectiveness remains significantly understudied. our work in this paper aims to evaluate llms' performance on character profiling, of which the challenges mainly include the absence of high-quality datasets and evaluation protocols. to address these challenges, we construct the cross (character profiles from supersummary) dataset for character profiling, and propose two tasks to evaluate the generated profiles. the cross dataset is sourced from supersummary foot_0 , a platform providing summaries for books and characters contributed by literature experts. our evaluation distinguishes four essential dimensions for character profiles: attributes, relationships, events, and personality. we parse the character profiles from supersummary into these dimensions by gpt-4, as the ground truth references. then, the generated profiles are evaluated in either an intrinsic or extrinsic way. the intrinsic evaluation directly employs llama-3-70b (ai@meta, 2024) to compare the generated profiles with the references. for extrinsic evaluation, we propose the motivation recognition task and measure whether the generated profiles can support llms in this task, i.e., identifying the motivations behind characters' decision-making. our experiments cover various summarization methods, including hierarchical merging, incremental updating, and summarizing in one go, implemented on numerous llms. the results reveal that character profiles generated by llms are satisfactory but leave space for further improvement. this suggests the potential information loss in rpas built on these profiles. additionally, the results of motivation recognition demonstrate the importance of each of the four dimensions for character profiles. our contributions are summarized as follows: 1) we present the first work to evaluate llms' capability of character profiling and propose an evaluation framework with detailed dimensions, tasks, and metrics. 2) we introduce cross, a high-quality dataset valuable for character profiling tasks, which is sourced from literature experts. ## related work character role-playing recent advancements in llms have significantly enhanced the capabilities of role-playing agents (rpas) across various aspects. currently, many role-playing tasks require interactive ai systems to act as assigned personas, including celebrities and fictional characters. in these studies, researchers have utilized various methods to develop rpas, which can be divided into three categories: 1) manual construction @xcite , which employs book fans or professional annotators to label information related to characters; 2) online resource collection @xcite , which collects character profiles from online resources, e.g., wikipedia foot_1 , and baidu baike foot_2 ; 3) automatic extraction @xcite , which utilizes llms to extract character dialogues from origin books or scripts. in this paper, we explore the capabilities of llms in generating character profiles for rpas construction. ## conclusion we introduce the first task for assessing the character profiling ability of large language models (llms), using a dataset of 126 character profiles from novels. our evaluation, which includes the factual consistency examination and motivation recognition, reveals that llms generally perform well. however, even the most advanced models occasionally generate hallucinations and errors, particularly with complex narratives, highlighting the need for further improvement.