I n C haracter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews (2024)

Abstract: Role-playing agents (RPAs), powered by large language models, have emerged as a flourishing field of applications. However, a key challenge lies in assessing whether RPAs accurately reproduce the personas of target characters, namely their character fidelity. Existing methods mainly focus on the knowledge and linguistic patterns of characters. This paper, instead, introduces a novel perspective to evaluate the personality fidelity of RPAs with psychological scales. Overcoming drawbacks of previous self-report assessments on RPAs, we propose InCharacter, namely **In**terviewing **Character** agents for personality tests. Experiments include various types of RPAs and LLMs, covering 32 distinct characters on 14 widely used psychological scales. The results validate the effectiveness of InCharacter in measuring RPA personalities. Then, with InCharacter, we show that state-of-the-art RPAs exhibit personalities highly aligned with the human-perceived personalities of the characters, achieving an accuracy up to 80.7%.

## introduction recent advancements in large language models (llms) have catalyzed the emergence of roleplaying agents (rpas). rpas are interactive ai systems simulating diverse roles or characters. rpa applications have been extended to diverse contexts, such as ai agents of fictional characters @xcite , digital clones for humans @xcite , and ai non-player characters in video games @xcite . recent research trends have increasingly focused on the development of to evaluate the personality fidelity of rpas, we apply various scales to measure their personalities and compare the results with the personality labels of the characters. rpas, including building rpas for specific characters @xcite and improving the role-playing abilities of foundation models @xcite . however, evaluating character fidelity in rpas remains a relatively underexplored area. prior research mainly concentrates on the replication of knowledge, experience, and linguistic patterns of characters @xcite , which manifests in two primary issues: (1) they necessitate character-specific datasets, thereby complicating the evaluation of new characters. (2) they overlook evaluating rpas' thoughts and underlying mindsets. towards these issues, we propose to evaluate if rpas faithfully reproduce the personalities of target characters, i.e., personality fidelity, as depicted in figure 1 . personality tests, administered by psychological scales, measure an individual's interrelated behavioral, cognitive, and emotional patterns @xcite . by measuring the personalities of rpas and com-paring them with the personalities of the characters, we can attain a more nuanced understanding of rpas' character fidelity. prior studies on llm personalities are mainly based on self-report scales, which prompt llms to select options or assign ratings to specific items @xcite . however, this method suffers from several limitations for rpas. therefore, we propose incharacter, a novel approach to interviews character agents for personality tests. while self-report scales are popular in humans for their cost-effectiveness, interviewbased scales evaluated by experts offer a more comprehensive analysis @xcite . self-reports are sometimes influenced by an individual's lack of insight, denial, or bias. in contrast, an interviewer can be a guide to elicit thoughts of individuals, effectively identifying and addressing the nuances via conversations to overcome the previously mentioned limitations. in-character employs this interview-based procedure @xcite on rpas, which includes two stages: (1) interview: rpas are engaged with open-ended questions derived from psychological scales to elicit rpas' mindsets and behaviors. (2) assessment: we utilize llms to interpret the responses collected from the first stage. this can involve converting the responses to likert levels or using llms to simulate a psychiatrist's role in judging rpa personalities. we apply incharacter to various rpas on 14 personality tests, including the big five inventory (bfi), 16personalities foot_0 (16p), and dark triad dirty dozen (dtdd). the personality labels for the bfi and 16p are accessible through the personality database (pdb) 3 . additionally, we engage human annotators familiar with the characters to label them on other scales, thereby creating a comprehensive benchmark for evaluating rpa personalities. our experiments include various types of existing rpas. the results demonstrate that the in-character effectively simulates interview-based tests conducted by human experts and yields rpas personality measurement better aligned with the characters than self-report methods. additionally, we find that the conversation data generated in our interview phase are of high quality and ideal for fine-tuning foundation models for rpas. hence, we release a dataset comprising 18,304 interview dialogues to facilitate future research. the contributions of this paper are mainly threefold: 1) we introduce a novel aspect for rpa evaluation, i.e., personality fidelity, based on psychological scales. 2) we propose incharacter, an interview-based framework for personality tests on rpas and collect the first benchmark for rpa personality evaluation, facilitating future research on developing better rpas. 3) our experiments on various rpas and psychological scales demonstrate the efficacy of incharacter. ## incharacter this section introduces incharacter, a novel personality assessment methodology designed explicitly for rpas, utilizing an interview-based procedure. figure 2 illustrates our two-stage framework. the interview stage is detailed in §3.1, followed by an elaboration of the assessment stage in §3.2. ## conclusion in this study, we investigate the personality fidelity in rpas, i.e., whether rpas reproduce personalities of their intended characters. addressing the shortcomings of previous methods on rpas, we propose incharacter, an interview-based approach that accurately measures rpa personalities based on their elicited mindsets and behaviors. our experiments span various types of rpas, covering 32 characters on 14 psychological scales. the results validate the effectiveness of incharacter in measuring rpa personalities. afterward, with incharacter, we comprehensively evaluate personality fidelity in existing rpas, discovering that state-of-the-art rpas successfully portray many personality traits of the characters.