C haracter E val: A C hinese Benchmark for Role-Playing Conversational Agent Evaluation (2024)

Abstract: Recently, the advent of large language models (LLMs) has revolutionized generative agents. Among them, Role-Playing Conversational Agents (RPCAs) attract considerable attention due to their ability to emotionally engage users. However, the absence of a comprehensive benchmark impedes progress in this field. To bridge this gap, we introduce CharacterEval , a Chinese benchmark for comprehensive RPCA assessment, complemented by a tailored high-quality dataset. The dataset comprises 1,785 multi-turn role-playing dialogues, encompassing 11,376 examples and featuring 77 characters derived from Chinese novels and scripts. It was carefully constructed, beginning with initial dialogue extraction via GPT-4, followed by rigorous human-led quality control, and enhanced with in-depth character profiles sourced from Baidu Baike. CharacterEval employs a multifaceted evaluation approach, encompassing thirteen targeted metrics on four dimensions. To facilitate the convenient evaluation for these subjective metrics in CharacterEval , we further developed CharacterRM, a role-playing reward model based on human annotations, which has a higher correlation with human judgment compared to GPT-4. Comprehensive experiments on CharacterEval demonstrate that Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation.

## introduction the development of large language models (llms) has marked the beginning of a new era in conversational ai @xcite , and opened up a wide range of application possibilities, particularly in agent-based interactions (park * *corresponding authors:xin gao and rui yan (xin.gao@kaust.edu.sa, ruiyan @ruc.edu.cn) @xcite @xcite . the automated agents, equipped with the emerging capabilities of llms such as planning @xcite @xcite , reasoning @xcite , and in-context learning @xcite , can perform complex tasks for humans without any supervision. among the diverse agents, the role-playing conversational agent (rpca), designed to offer emotional value instead of productivity, attracts an amount of interest. rpca represents a unique category within the realm of conversational agents, distinguished by their capability for immersive interaction @xcite . different from traditional dialogue systems, which typically focus on chit-chat @xcite , knowledge-based @xcite , personalized @xcite and empathetic dialogue @xcite , rpcas engage users in dynamic scenarios, where llm agents are assumed as specific characters or roles, often derived from existing composition such as novels, films, cartoons, and games. the development of connections between fictional characters and humans has the potential to not only deepen the impact of cultural works but also improve human engagement. furthermore, rpcas hold significant application value in their ability to offer emotional value to users, positioning fictional characters as virtual friends. the multifaceted nature of rpcas has sparked considerable attention, leading to a surge in both research @xcite @xcite and application development (e.g., character ai foot_0 , tongyi xingchen foot_1 and glow foot_2 ). however, these implementations of rpcas vary significantly in both approach and objectives, presenting a challenge in systematically assessing and comparing their capabilities. there- to develop a benchmark, the primary problem is the construction of a dataset. while there are existing datasets @xcite @xcite @xcite , their quality is concerning, which are either generated by llms or suffering from significant noise due to the extractive methods. these limitations render the evaluation results unreliable for the rpca's actual capabilities. to address it, we constructed a chinese role-playing conversation dataset comprising 1,785 multi-turn roleplaying dialogues, encompassing 11,376 examples and 77 leading characters, drawn from diverse chinese novels and scripts. our process began with the collection of well-known sources across various genres. after that, gpt-4 was employed to extract dialogue scenes, utterances, and behaviors of the leading roles of these sources. following basic preprocessing and the removal of dialogues with fewer turns, we invited annotators to assess the quality of the dialogues. their task was to identify and retain high-quality dialogues while discarding those of lower quality. additionally, we crawled detailed character profiles from baidu baike foot_3 , composing a comprehensive dataset for rpca evaluation. the example from the dataset is as figure 1 shows. otherwise, role-playing conversation is a compli-cated task that requires not only mimicking a character's behavior and utterance but also maintaining the character's knowledge, as well as excellent multi-turn ability. considering this, we proposed a multifaceted evaluation approach including thirteen specific metrics on four dimensions for a fair and thorough assessment of rpcas, our evaluation approach considered conversational ability, character consistency, and role-playing attractiveness, and utilized a personality back-testing method to evaluate the personality accuracy of an rpca. to assess conversational ability, we measured conversational fluency, coherence, and consistency at both the sentence and conversation levels @xcite . character consistency is the most crucial in role-playing conversation. hence, we evaluated knowledge and persona consistency to measure how vividly an rpca can simulate a character. this involves assessing knowledge exposure, accuracy, and hallucination for knowledge consistency, and evaluating behavior and utterance consistency for persona consistency. considering that rpcas are entertainment-oriented, role-playing attractiveness is also an important element. we assessed this through human-likeness, communication skills, expression diversity, and empathy. finally, we introduced personality back-testing. with the collected myers-briggs type indicator(mbti) (myers, 1962) personality types as a reference, we let rpcas do the mbti assessment and calculate the mbti accuracy (personality back-test) as implemented in @xcite . in summary, our contributions are as follows: ## 11838 the role-playing conversational agent (rpca) is designed to engage in conversations with users by emulating specific characters. these characters are defined by their knowledge, behavior, and style of response. to achieve this, the rpca utilizes a character profile, denoted as p , and the current dialogue context represented as here, q i and r i correspond to the i-th question and response in the dialogue, respectively. the goal for the rpca is to generate a response r n that is consistent with the character's profile, which can be represented as: ## data collection in this section, we detail the methodology for constructing the character-based, multi-turn dialogue dataset with high quality. prior to initiating data collection, adherence to the following four principles is important: the pipeline of data collection includes four steps: plot division, dialogue extraction, quality filtering, and human annotation. ## evaluation metric different from traditional chatbots, we contend that rpcas require a more comprehensive evaluation framework to assess their role-playing capabilities. as shown in figure 2 , we have devised a four-dimensional evaluation system, which includes conversational ability, character consistency, role-playing attractiveness, and personality backtesting, including thirteen metrics. ## conclusion in this work, we aim to build a comprehensive benchmark to evaluate recent role-playing conversational agents (rpcas). we introduce gpt-4 to extract the dialogues from the existing novels and scripts, proceeding with strict human filtering. after a series of processing, we release a high-quality multi-turn role-playing dataset. besides, we construct a comprehensive evaluation system to assess the multi-dimensional ability of rpcas. we also collect human annotation to train a character-based