RMTB ench: Benchmarking LLM s Through Multi-Turn User-Centric Role-Playing (2025)

Abstract: Recent advancements in Large Language Models (LLMs) have shown outstanding potential for role-playing applications. Evaluating these capabilities is becoming crucial yet remains challenging. Existing benchmarks mostly adopt a character-centric approach, simplify user-character interactions to isolated Q&A tasks, and fail to reflect real-world applications. To address this limitation, we introduce RMTBench, a comprehensive user-centric bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds. RMTBench includes custom characters with detailed backgrounds and abstract characters defined by simple traits, enabling evaluation across various user scenarios. Our benchmark constructs dialogues based on explicit user motivations rather than character descriptions, ensuring alignment with practical user applications. Furthermore, we construct an authentic multi-turn dialogue simulation mechanism. With carefully selected evaluation dimensions and LLM-based scoring, this mechanism captures the complex intention of conversations between the user and the character. By shifting focus from character background to user intention fulfillment, RMTBench bridges the gap between academic evaluation and practical deployment requirements, offering a more effective framework for assessing role-playing capabilities in LLMs. All code and datasets will be released soon.

## introduction recent breakthroughs in large language models (llms) have demonstrated significant potential for role-playing conversational agents. these models effectively simulate diverse character identities, making them valuable in entertainment, education, and emotional support applications. platforms like character.ai, which attract millions of active * corresponding authors. users, underscore the growing importance of roleplaying llms in interactive ai systems. despite this widespread adoption, a systematic evaluation of llms' role-playing capabilities remains essential to guide future technological advancements and enhance conversational applications. researchers have conducted assessments on dimensions such as self-awareness and conversational ability @xcite @xcite . however, these approaches overlook the critical role of user intentions and focus excessively on character information. users engage with role-playing systems for diverse purposes-entertainment, emotional support, or knowledge acquisition, but existing evaluations simplify these complex intentions into simple knowledge question answering. as shown on the left side of figure 1 , these character-centric approaches create dialogues from isolated q&a pairs, disconnected from the user intentions that indeed drive meaningful interactions. furthermore, most benchmarks exacerbate this issue by employing static single-turn evaluations with optional preset histories, which poorly align with real-world user interaction scenarios and fail to capture the dynamic context of conversations. these limitations create a substantial gap between academic evaluation and practical applications. to address these limitations, we propose rmt-bench, a user-centric role-playing benchmark comprising 80 diverse characters and 8,000+ dialogue rounds in both english and chinese. as shown on the right side of figure 1 , our user-centric dialogues are built around virtual users with clear intentions, enhancing continuity across multi-turn interactions and better reflecting real-world applications. our contributions include: through this user-centric design paradigm, rmtbench shifts evaluation focus from characters to users, providing a more effective reference point for research and practical applications. our benchmark bridges the gap between academic evaluation and real-world deployment requirements by assessing how well llms fulfill diverse user intentions within role-playing contexts, rather than merely how accurately they maintain character identities. role-playing benchmark. current role-playing evaluation approaches include generative evaluation using llms as judges @xcite and mcq-based formats @xcite . more recent work uses dialogues to examine character consistency @xcite ; @xcite ; @xcite . however, these benchmarks use a character-centric approach in data construction and simplify complex interactions to isolated q&a pairs without coherent contexts and user intentions, struggling to capture llms' performance under authentic interaction. our rmtbench shifts focus from characters to users, generating dialogue data based on user intentions rather than character information, better aligning with real-world applications. ## rmtbench framework we introduce rmtbench, a comprehensive bilingual benchmark for role-playing large language models. this benchmark focuses on user-centric scenarios, a dimension often underexplored in prior research, and consists of five distinct role-playing scenarios. based on these scenarios, we automatically constructed an evaluation dataset that contains 80 characters and more than 8,000 utterances in english and chinese. through strict quality control mechanisms and multi-dimensional evaluation, rmtbench effectively complements performance assessment for role-playing llms. ## rmtbench construction this section details the construction methodology of the rmtbench. ## discussion in this section, we present additional experimental results on various variants of our benchmark evaluation to validate the effectiveness of the proposed multi-turn benchmark in real-world applications. ## conclusion this paper presents rmtbench, a bilingual multiturn benchmark designed to address critical gaps in evaluating the role-playing capabilities. unlike traditional character-centric approaches, rmtbench adopts a user-centric evaluation paradigm that better aligns with real-world applications. by constructing scenarios around diverse user intentions, the benchmark evaluates models across five key user intents, 80 distinct characters, and over 8,000 dialogue rounds. experimental results highlight the importance of assessing multi-turn, authentic interactions. rmtbench contributes to the system-atic evaluation of role-playing capabilities and provides a scalable framework adaptable to the rapidly evolving domain of conversational ai.