E mo C haracter: Evaluating the Emotional Fidelity of Role-Playing Agents in Dialogues (2025)

Abstract: Role-playing agents (RPAs) powered by large language models (LLMs) have been widely utilized in dialogue systems for their capability to deliver personalized interactions. Current evaluations of RPAs mainly focus on personality fidelity, tone imitation, and knowledge consistency, while overlooking emotional fidelity, a key factor that affects user experience. To this end, we propose a benchmark called EmoCharacter to assess emotional fidelity of RPAs in dialogues. EmoCharacter includes two benchmark datasets (single-turn and multi-turn dialogues), three evaluation settings, and six metrics to measure the emotional fidelity between RPAs and the characters they portray. Based on EmoCharacter, we conduct extensive evaluations on RPAs powered by seven widely used LLMs with representative role-playing methods. Our empirical findings reveal that: (1) Contrary to intuition, current role-playing methods often reduce the emotional fidelity of LLMs in dialogues; (2) Enhancing the general capabilities of LLMs does not necessarily improve the emotional fidelity of RPAs; (3) Fine-tuning or In-Context Learning based on real dialogue data can enhance emotional fidelity.

## introduction the rise of large language models (llms) has revolutionized dialogue systems, enabling more dynamic and personalized interactions @xcite . among these advancements, role-playing agents (rpas) have emerged, with agents assigned specific characters or personas. rpas have been successfully deployed in various applications, ranging from entertainment to mental health support @xcite @xcite . current evaluations of rpas primarily focus on personality fidelity, tone imitation, and knowledge consistency @xcite @xcite ), * corresponding authors. emotional fidelity refers to the degree to which an rpa can simulate the emotional dynamics of the character it portrays. human beings naturally display emotional shifts based on the context, tone, and flow of conversation @xcite . psychological research indicates that personality is closely linked to emotions, with different characters exhibiting varying emotional dynamics in dialogues @xcite . the ability to exhibit emotional variability and coherence throughout a dialogue is essential for achieving realistic and engaging role-playing experiences. however, existing rpas often fail to capture these nuances, offering static emotional responses or inaccurately portraying emotions, thereby breaking the illusion of authenticity. to address this gap, we propose emocharacter, a novel benchmark designed to assess the emotional fidelity of rpas. emocharacter includes two benchmark datasets derived from classic tv dialogues and introduces six metrics across three evaluation settings to quantitatively measure how well rpas maintain emotional fidelity with the characters they portray. through extensive experiments ( §4) and analysis ( §5) of rpas powered by seven widely used llms with three different role-playing methods (none, rolellm and chat-haruhi), we pinpoint three key insights: (1) contrary to intuition, roleplaying methods often reduce the emotional fidelity of llms in dialogues; (2) enhancing the general capabilities of llms does not necessarily improve the emotional fidelity in rpas; (3) supervised finetuning (sft) or in-context learning (icl) using real dialogue data annotated with emotional states can enhance emotional fidelity. our contributions are as follows: ## emocharacter to evaluate the emotional fidelity between rpas and their assigned characters, emocharacter incorporates three components, as shown in figure 1 : evaluation task and settings, single-turn and multiturn dialogue datasets, and evaluation metrics. ## conclusion we proposed emocharacter to evaluate the emotional fidelity of rpas from both macro and micro perspectives. with extensive evaluations of multiple llms and role-playing methods, we pinpointed that even state-of-the-art llms like gpt-4o experience a decline in emotional fidelity during role-playing dialogues. our case studies revealed that llms often use overly distinct emotions to express character traits, leading to low emotional fidelity. additionally, we found that fine-tuning with real dialogue data and using in-context learning both benefit the rpas' emotional fidelity. we hope emocharacter will facilitate research on building rpas with greater emotional intelligence.