R ole MRC : A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following (2025)

Abstract: Role-playing is important for Large Language Models (LLMs) to follow diverse instructions while maintaining role identity and the role’s pre-defined ability limits. Existing role-playing datasets mostly contribute to controlling role style and knowledge boundaries, but overlook role-playing in instruction-following scenarios. We introduce a fine-grained role-playing and instruction-following composite benchmark, named RoleMRC, including: (1) Multi-turn dialogues between ideal roles and humans, including free chats or discussions upon given passages; (2) Role-playing machine reading comprehension, involving response, refusal, and attempts according to passage answerability and role ability; (3) More complex scenarios with nested, multi-turn and prioritized instructions. The final RoleMRC features a 10.2k role profile meta-pool, 37.9k well-synthesized role-playing instructions, and 1.4k testing samples. We develop a pipeline to quantitatively evaluate the fine-grained role-playing and instruction-following capabilities of several mainstream LLMs, as well as models that are fine-tuned on our data. Moreover, cross-evaluation on external role-playing datasets confirms that models fine-tuned on RoleMRC enhances instruction-following without compromising general role-playing and reasoning capabilities. We also probe the neural-level activation maps of different capabilities over post-tuned LLMs. Access to our RoleMRC, RoleMRC-mix and Codes: https://github.com/LuJunru/RoleMRC.

## introduction role-playing is one of the key capabilities of llms. modern llms are designed to interact with human users under certain role-playing settings @xcite . in this context, llms respond to various instructions, serving as ai assistants @xcite , personalized agents @xcite , 2023; @xcite , leisure partners @xcite , content creators @xcite @xcite , social experimental simulator @xcite among other roles @xcite . figure 1 demonstrates an example of llm roleplaying. in the first turn of dialogue, when asked to give advice on paper writing, the llm should respond based on the pre-defined role profile (shown at the top of figure 1 ). among the responses, the reply "[a]" completely ignored the role setting, "[b]" misinterpreted the role and thus did not respond well, only "[c]" correctly gave suggestions in a cat girl style. in the second turn of dialogue (continuing with "[c]"), the user not only asked a new question, but also modified the role setting (adding a heart emoji at the beginning of the answer). while both replies "[d]" and "[e]" maintained the initial cat girl style, only "[e]" correctly incorporated the additional role-playing instruction. scenarios dataset data scale role num. #turns #words per reply free chat on scene ruled chat characterllm (shao et al., 2023) 14.2k 9 13.2 36 ✔ ✘ ✘ chatharuhi (li et al., 2023)* 11.6k 35 5.5 7 ✔ ✘ ✘ rolellm (wang et al., 2023) 168.1k 100 1 30.5 ✔ ✘ ✘ characterglm (zhou et al., 2023b) 1k 250 15.8 24.3 ✔ ✘ ✘ charactereval (tu et al., 2024) 1.8k 77 9.3 27.4 ✘ ✔ ✘ ditto (lu et al., 2024b) 7.2k 4k 5.1 -* ✔ ✘ ✘ character100 (wang et al., 2024a) 10.6k 106 1 74.1 ✘ ✔ ✘ mmrole (dai et al., 2024) 14.3k 85 4.15 66.8 ✘ ✔ ✘ rolemrc (ours) 37.9k 10.2k 3.5 (9.5) 40.6 ✔ ✔ ✔ rolemrc-mix (ours) 107.7k 10.2k 2 (9.5) 67.1 ✔ ✔ ✔ ## related work role-playing datasets are the basis for relevant research. existing role-playing datasets can be categorized into two types: character-centric and individual-centric @xcite . by mining public information from social experience @xcite @xcite , literary works @xcite , or books @xcite @xcite , the character-centric branch extracts roles with distinctive characteristics to form open characters (e.g., eskimos, harry potter, or beethoven). on the contrary, the individual-centric datasets are derived from personalized user data @xcite @xcite aiming to create digital clones or personal assistants. rolemrc is a character-centric dataset. llm's role-playing capabilities have made great strides in the past years. characterllm @xcite collected nine portraits from wikipedia and fine-tuned llms to be a simulation of the roles, then assessed their character consistency through interviews. rolellm @xcite employed gpt-4 @xcite for extracting role profiles from scripts and synthesizing rolespecific dialogues, then evaluated the accuracy, style, and understanding of role knowledge of the role-playing llms. charactereval @xcite evaluated the llm's role-playing capability via four aspects: conversation, consistency, attractiveness, and personality. specifically, our rolemrc is the first large-scale, fine-grained role-playing instruction-following dataset, equipped with an evaluation pipeline consisting of seven heuristic metrics, a five-dimension llm-as-a-judge @xcite ) framework, and neural probes. ## evaluation on inner rolemrc test set by leveraging the above reference-based metrics and reference-free llm-as-a-judge approaches, we report evaluation on rolemrc in what follows. performance of proprietary llms. as shown in ## evaluation on external benchmarks we present cross-evaluation on external datasets. ## analysis on alignment tax despite all the other role-playing and instructionfollowing abilities of the llms are enhanced during the dpo alignment, we observe a slight yet common deterioration in multi-turn instructionfollowing performance (appendix f). we refer to this phenomenon as an "alignment tax", which is characterized by a gradual forgetting of knowledge acquired during pre-training (ouyang et al., 2022). neuron-level localization. to identify the underlying cause of this alignment tax, we examine the neuron activation patterns of our rolemrc models (llama3.1-8b sft vs. dpo). following tang et al. (2024) , we probe and collect activations from each attention layer, focusing on highly activated neurons by selecting the top 20% of activations. specifically, for each input instruction, we measure activations when first forwarding the instruction. we then group the activation maps by the evaluation dimension of the test instruction, generating layer-specific differences in neuron usage. next, we count the activation frequency of each neuron and normalize it by the total number of test cases. figure 5 visualizes the resulting discrepancy between the sft and dpo models. layers 3-11 exhibit minimal changes, whereas layers beyond the 13th show substantial activation differences, with layers 12-31 (highlighted in red) differing the most. notably, layer 19 is significantly more active in multi-turn instruction. this observation aligns with @xcite , who found that only the top and bottom layers of a language model are primarily used for language processing. these shifts in neuron activations suggest that certain neurons are activated very differently between the sft and dpo models. further details and results are provided in appendix h. neuron-level restraint. after identifying these critical neuron subsets, we apply a minor scaling restraint (multiplicative factor 1 -10 -6 ) to modulate their impact. as shown in ## conclusion we introduce rolemrc, a large-scale fine-grained benchmark designed to improve and evaluate the role-playing and instruction-following abilities of llms. rolemrc uniquely integrates rolespecific multi-turn dialogues, mrc, and complex instruction-following scenarios. experiments show that rolemrc-aligned models outperform existing baselines in both reference-based and referencefree evaluations, and also perform well on both ood role-playing and general-purpose benchmarks. we further conduct a neuron-level analysis to identify specific neurons with significant activation changes and apply targeted constraints to alleviate the alignment tax, thereby improving evaluation metrics without additional retraining.