S ocial B ench: Sociality Evaluation of Role-Playing Conversational Agents (2024)

Abstract: Large language models (LLMs) have advanced the development of various AI conversational agents, including role-playing agents that mimic diverse characters and human behaviors. While prior research has predominantly focused on enhancing the conversational capability, role-specific knowledge and style of these agents, there has been a noticeable gap in assessing their social intelligence. In this paper, we introduce SocialBench, the first benchmark designed to systematically evaluate the sociality of role-playing agents at both individual and group levels of social interactions. SocialBench is constructed from various sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. We conduct comprehensive evaluations on this benchmark using mainstream LLMs. We find that agents excelling in individual level does not imply their proficiency in group level. Experimental results on SocialBench confirm its significance as a testbed for assessing the social interaction of role-playing agents. The benchmark is publicly accessible at https://github.com/X-PLUG/RoleInteract.

## introduction recently, role-playing applications powered by llms, such as character.ai foot_0 , have gained significant attention. a growing number of research efforts have been dedicated to developing llmbased role-playing agents, aiming to mimic diverse characters and human behavior @xcite @xcite . as an rapidly developing area, the evaluation of role-playing agents is becoming increasingly important. @xcite collected a rolespecific dataset and utilized llms to assess the model's role-specific knowledge and speaking style. @xcite proposed a chinese benchmark and trained a reward model to measure the model's conversational ability and attractiveness. while these works mainly focus on evaluating the agent's individual abilities, this study aims to explore and measure the sociality of role-playing agents, another pivotal dimension for assessing how role-playing agents behave in a social environment. therefore, we introduce socialbench, the first evaluation benchmark designed to systematically assess the social interaction of role-playing agents. as introduced in @xcite , the agent society represents a complex system comprising individual and group social activities. following this definition, socialbench assesses the sociality at both the individual and group levels, as illustrated in figure 1 . at the individual level, the agent should possess the basic social intelligence as individuals, such as self-awareness on role description @xcite , emotional perception on environment @xcite , and longterm conversation memory @xcite . each of these aspects contributes to the nuanced understanding of how the agents manifest their individual social behaviors. moreover, we further examine the social intelligence of the role-playing agents within group social interactions, which require the agents to possess certain social preferences towards group dynamics @xcite . socialbench is carefully constructed from diverse english and chinese books, movies, and novels, covering a wide range of 500 characters and 6,000 questions, and 30,800 multi-turn role-playing utterances. we conducted extensive evaluations on socialbench using mainstream open-source and closed-source llms. we find that agents excelling at the individual level do not necessarily demonstrate proficiency at the group level. moreover, the behavior of individuals may drift as a result of the influence exerted by other agents within the group. we hope our findings will inspire future research. not match match figure 1 : an example about socialbench, which is partially constructed from the film "the great gatsby". ## sociality of role-playing agent given a character profile and context, the sociality of role-playing agents focuses on imitating typical social interactions from individual to group level. ## socialbench in this section, we introduce the construction of so-cialbench. refer to appendix a for more details. ## conclusion in this paper, we introduced socialbench, the first evaluation benchmark designed to systematically assess the social intelligence of role-playing agents at both individual and group levels. we constructed diverse question prompts on a wide range of characters covering comprehensive dimensions. moreover, rigorous human verification ensures the questions' difficulty and validity. we conducted extensive experiments and analysis on socialbench. while role-playing agents perform well at the individual level, their social interaction capabilities at the group level are lacking. this highlights the need for further exploration in future research.