C haracter C raft: Bridging the Literature-Reality Dialogue Gap for Practical Role-Playing Agents (2025)

Abstract: Recent advancements in large language models (LLMs) have given rise to the emergence of role-playing agents (RPAs). The development of high-quality dialogue datasets is critical for advancing RPAs. However, existing datasets have two main issues: (1) the bias between query distributions and real-world user language usage, and (2) the challenge of ensuring responses accurately reflect character traits.To address these issues, we propose CharacterCraft, a novel framework designed for practical RPAs, comprising a tailored Chinese role-playing dataset and a robust evaluation method. First, we develop a specialized model for Chinese dialogue extraction, achieving state-of-the-art performance. Using this model, we then extract a large amount of character dialogue from novels, ensuring high data quality (issue 2).To mitigate the literature-reality dialogue bias in extracted dialogue (issue 1), we introduce an iterative augmentation-reconstruction method, which revises queries to better align with common language usage. Additionally, we propose a context-aware memory retrieval module for fine-grained alignment with the character and introduce a reference-guided LLM-as-a-judge evaluation method for more reliable assessments by comparing their responses to source material dialogues.Our automated pipeline produces a large-scale high-quality Chinese role-playing dataset with 21,392 samples and 121,418 utterances. The experimental results demonstrate the effectiveness of our framework and reveal the limitations of existing RPAs when faced with diverse scenes.Our repository is at https://github.com/yin214/CharacterCraft.

## introduction the rapid advancement of large language models (llms) has spurred significant innovations across various real-world applications @xcite (a) synthetic dialogue i'm not coming back hermione. i've got to finish whatever dumbledore started, and i don't know where that'll lead me, but i'll let you and ron know where i am when i can. i've always admired your courage harry, but sometimes you can be really thick. you don't really think you're going to be able to find all those horcruxes by yourself do you? you need us harry. ## problem definition the role-playing agent (rpa) is designed to allow users to interact with immersive and highly personalized characters. in practical applications, a human user interacts with an rpa designed to imitate a specific character c. the character c is defined by a profile p, which includes essential attributes such as identity, linguistic style, personality, etc. the interaction takes place in a scene s and a dialogue context @xmath0, r 1 , q 2 , r 2 , . . . , q n ] where q i and r i represent the i-th utterances from the human user and the rpa, respectively. the goal of the rpa is to generate a response r @xmath1, s, d) which is consistent with the character's profile p. notably, in real-world applications, human users may also obtain a specific character identity to enhance interactions. however, users typically do not consciously align their language style with the role they are playing. thus, we assume that users' linguistic patterns exhibit alignment with daily communication or common language usage. ## charactercraft corpora as presented in figure 2a , we construct character-craft dataset, a large-scale, high-quality chinese role-playing collection that aligns with real-world user interactions. all prompts used in this section are detailed in §d. ## charactercraft-eval in this section, we begin by outlining a series of evaluation dimensions from three aspects, as shown in figure 2b . we then introduce our referenceguided llm-as-a-judge method, which provides stable and comprehensive assessments of rpas. ## conclusion in this work, we highlight the bias between current role-playing datasets and real-world applications. we introduced charactercraft, a novel framework designed to better align with real-world application scenarios. by integrating high-quality literary dialogue extraction with an iterative augmentationreconstruction method, we construct a large-scale, high-quality chinese role-playing dataset. to enhance rpas' contextual understanding, we proposed a context-aware memory retrieval module. additionally, our reference-guided llm-asa-judge evaluation improves role-playing assessments by incorporating source material references. experiments underscore rpas' challenges in capturing dynamic character traits and show our framework's effectiveness.