Reasoning Does Not Necessarily Improve Role-Playing Ability (2025)

Abstract: The application of role-playing large language models (LLMs) is rapidly expanding in both academic and commercial domains, driving an increasing demand for high-precision role-playing models. Simultaneously, the rapid advancement of reasoning techniques has continuously pushed the performance boundaries of LLMs. This intersection of practical role-playing demands and evolving reasoning capabilities raises an important research question: Can reasoning techniques enhance the role-playing capabilities of LLMs?” To address this, we conduct a comprehensive study using 6 role-playing benchmarks, 24 LLMs, and 3 distinct role-playing strategies, comparing the effectiveness of direct zero-shot role-playing, role-playing with Chain-of-Thought (CoT), and role-playing using reasoning-optimized LLMs. Our findings reveal that CoT may reduce role-playing performance, reasoning-optimized LLMs are unsuitable for role-playing, reasoning ability disrupts the role-playing scaling law, and large models still lack proficiency in advanced role-playing. Furthermore, based on extensive experimental results, we propose two promising future research directions: Role-aware Chain-of-Thought for improving role-playing LLMs and Reinforcement Learning for role-playing LLMs, aiming to enhance the adaptability, consistency, and effectiveness of role-playing LLMs for both research and real-world applications.

## introduction large language models (llms), leveraging their strong foundational capabilities and advanced instruction-following abilities, have become the cornerstone of role-playing ai models @xcite . these role-playing llms present new opportunities in both research context tom: captain, your boat has been stolen. zero-shot role-playing with gpt-4-turbo not stolen, mate-temporarily misplaced. and i always get my ship back, savvy? role-playing with chain-of-thought using gpt-4-turbo [reasoning process omitted] so the pearl's decided to go on a little adventure without me-no matter, i'll just steal her back. role-playing with openai o1-mini [reasoning process omitted] stolen, you say, mate-why, i'd be far more worried for the soul foolish enough to filch the pearl, savvy? this situation prompts us to explore a previously unanswered research question: "can reasoning techniques enhance the role-playing capabilities of large language models?" the answer to this question could provide valuable insights for the future application of reasoning techniques in role-playing llms, potentially paving a new path for their development and advancement. to address this question, we carefully select six role-playing datasets and conduct experiments us-ing 24 widely used open-source and proprietary models. these models include api-based closedsource models such as gpt-4-turbo @xcite , popular open-source models like qwen2.5 @xcite , and reinforcement learning-optimized reasoning models such as @xcite . all experiments are conducted using opencompass @xcite to ensure evaluation consistency, stability, and reproducibility. specifically, we utilize six role-playing benchmarks: rolebench @xcite , incharacter @xcite , so-cialbench @xcite , charactereval @xcite , hpd @xcite , and cross-mr @xcite . these benchmarks cover both multiple-choice and text generation tasks and include english and chinese, making the evaluation linguistically diverse. for evaluation, we employ both traditional metrics (e.g., accuracy, rouge, and exact match) and llm-as-a-judge methods (e.g., prompt-based evaluation and reward model scoring), ensuring a broad assessment scope to enhance the robustness of our conclusions. furthermore, we define three role-playing approaches: direct zero-shot role-playing, role-playing with chain-of-thought (cot), and role-playing using reasoning-optimized llms. this comprehensive setup allows us to systematically investigate the impact of reasoning techniques on role-playing performance. through extensive experiments, our findings reveal several key insights that can inform the future design of role-playing llms: (1) cot may reduce role-playing performance; (2) reasoningoptimized llms are unsuitable for role-playing; our study makes three key contributions. first, through extensive experiments on six role-playing datasets and 24 models, we reveal that reasoning techniques do not necessarily enhance the roleplaying capabilities of llms. second, our results provide a comprehensive analysis of the current state and limitations of role-playing llms, leading to two promising research directions: role-aware cot for improving role-playing llms and rein-forcement learning for role-playing llms. third, to address the fragmented nature of prior work with inconsistent evaluation standards, we integrate all datasets and experimental methods into opencompass, enabling one-click execution and providing a standardized experimental framework and codebase for future research foot_0 . ## evaluation framework the evaluation framework for llm-based roleplaying agents primarily consists of the following components: the llm itself (m), character profile (p), evaluation task (t ), evaluation metric (e), and, when applicable, a reference standard answer (a). in this study, we focus on whether reasoning techniques (r) can enhance the role-playing capabilities of llms. therefore, the overall evaluation process can be formalized as follows: given an llm m, a predefined character profile p, an evaluation task t , and an evaluation metric e, the role-playing performance of m is assessed by generating responses conditioned on p and t . the generated outputs are then evaluated using e, which quantifies alignment with the intended role. additionally, reasoning techniques r are incorporated into m to examine their impact on enhancing role-playing abilities. formally, the evaluation process can be expressed as: @xmath0, t , r), a) where s represents the final performance score, capturing the effectiveness of m in role-playing under the given conditions. a higher s indicates stronger role-playing capabilities. if a is unavailable or unnecessary, the evaluation metric e can be adapted to rely on alternative assessment criteria, such as llm-as-a-judge. ## experimental setting in this section, we introduce the benchmarks, models, metrics, and reasoning methods used in our experiments. all relevant code has been integrated into the opencompass @xcite library to facilitate efficient replication by researchers. ## results and findings in this section, we present our experimental analysis and key findings. ## related works in recent years, large language models (llms) have revolutionized the paradigm of artificial intelligence, achieving human-like performance in mathematics @xcite , coding @xcite @xcite , embodied intelligence @xcite , game intelligence @xcite , and advanced reasoning @xcite . the comprehensive enhancement of foundational capabilities in these models has opened new opportunities for the development of role-playing agents @xcite , resulting in a proliferation of such applications based on llms. on one hand, this has catalyzed innovations in fields such as psychological counseling, anthropomorphic companionship, and game npcs @xcite . on the other hand, it has provided fresh avenues for advancing social simulations @xcite . concurrently, reasoning techniques in llms have flourished, with methods like chainof-thought reasoning @xcite , tree-ofthought reasoning @xcite , and o1-style reasoning @xcite progressively maximizing the potential of these models, particularly achieving remarkable outcomes in tasks involving mathematics and coding @xcite . while prior research has predominantly focused on role-playing applications and their evaluation, this study is positioned at the intersection of llm-based role-playing agents and reasoning techniques. it seeks to address the scientific question of whether reasoning techniques enhance the role-playing capabilities of llms. by doing so, this work aims to provide guidance for developing more realistic and reliable role-playing agents powered by llms. ## future directions based on our experimental findings, we propose two potential research directions that integrate roleplaying and reasoning techniques for future exploration. role-aware chain-of-thought (cot) for improving role-playing llms. one promising direction for enhancing the role-playing ability of llms is role-aware cot reasoning. while standard cot enables step-by-step logical inference, it often disregards the persona-specific con-straints that are essential for consistent role-playing. a role-aware cot approach would integrate persona attributes, narrative constraints, and characterspecific perspectives into the reasoning process, ensuring that logical deductions align with the character's predefined traits. for instance, a historical figure simulated in an llm should reason within the knowledge and biases of their era rather than applying contemporary logic. this requires incorporating dynamic memory structures that retain persona information throughout multi-turn interactions, mitigating the risk of breaking character or adopting inconsistent reasoning patterns. ## conclusion this study aims to address the research question: "can reasoning techniques enhance the role-playing capabilities of large language models (llms)?" to this end, we conduct extensive experiments using 6 role-playing benchmarks, 24 llms, and 3 distinct role-playing methods. our experimental results lead to the following key findings: cot may reduce role-playing performance, reasoning-optimized llms are unsuitable for roleplaying, reasoning ability disrupts the role-playing scaling law, the qwen series is well-suited for roleplaying tasks, chinese role-playing performance surpasses english role-playing performance, and large models still lack proficiency in advanced roleplaying. all code is integrated into opencompass, ensuring reproducibility and facilitating further research. we hope that our findings provide new perspectives for future studies on role-playing llms.