Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment (2024)

Abstract: Considerable efforts have been invested in augmenting the role-playing proficiency of open-source large language models (LLMs) by emulating proprietary counterparts. Nevertheless, we posit that LLMs inherently harbor role-play capabilities, owing to the extensive knowledge of characters and potential dialogues ingrained in their vast training corpora. Thus, we introduce Ditto, the first self-alignment method for role-play, which encourages an instruction-following LLM to simulate role-play dialogues as a variant of reading comprehension, and creates a role-play training set comprising 4000 characters, surpassing the scale of currently available datasets by tenfold regarding the number of roles. Subsequently, we fine-tune the LLM using this self-generated dataset to augment its role-playing capabilities. Upon evaluating our meticulously constructed role-play benchmark and the roleplay subset of MT-Bench, Ditto, in various parameter scales, consistently maintains a consistent role identity and provides accurate role-specific knowledge in multi-turn role-play conversations, outperforming all open-source role-play baselines. Furthermore, we present the first cross-supervision role-play experiment, revealing that the role-play styles can be easily acquired, while the intrinsic capabilities of LLMs confine the knowledge within role-play.

## introduction large language models (llms) have showcased unparalleled proficiency in understanding intent @xcite , following instructions @xcite , and solving tasks across a diverse range of applications @xcite . however, designed as universal task assistants, llms typically differ from humanlike interlocutors, lacking experiential events and emotions @xcite . consequently, they face limitations in facilitating engaging and extensive conversations with users @xcite . to infuse emotional value into user interactions, role-play llms empower users to define and create profiles for their preferred characters @xcite . nonetheless, existing works cheaply imitate the proprietary model (gpt-4)'s role-play capabilities using a weaker opensource model @xcite @xcite @xcite , as gpt-4 has already demonstrated outstanding role-playing abilities @xcite . this approach presents challenges, assuming the existence of a more proficient role-play model, and we currently lack a clear understanding of how to build such a model from scratch, apart from manually annotating extensive datasets. furthermore, imitation models excel at mimicking gpt-4's style but fall short in replicating its factuality, introducing increased hallucina-tion as a trade-off @xcite , and are subject to openai's terms of use foot_0 @xcite . in this work, for the first time, we enable llm role-play through self-alignment and named this method ditto, eliminating the need for distilling outputs from more potent role-play models. llms, extensively trained on a vast corpus of humangenerated text @xcite , encapsulate a rich array of character experiences, events, personalities, and dialogues, as illustrated in fig. 1 . taking a nuanced perspective, we perceive an llm as a superposition of characters @xcite . this implies that llms are essentially equipped with the conversational styles necessary for role-playing and possess knowledge about numerous famous characters, albeit exhibiting an average of these roles. to elicit such role-play capabilities in a general llm, only two steps are required: meanwhile, the efficient and reproducible evaluation of role-play remains elusive. recent efforts heavily rely on manual annotations @xcite @xcite . however, the costly manual labeling prevents previous works from comprehensively comparing the performance of all relevant models. moreover, the high variance in manual annotations hinders subsequent work from consistently replicating previous evaluation results. as to the observation in the previous paragraph that role-play can be decomposed into conventional style and character knowledge, we aim to simplify role-play evaluation so that llms can automatically score. specifically, we assess: (1) whether the model can maintain consistent role identity. we provide a role-play dialogue to an llm judger and four character options, requiring the judger to determine which character is being portrayed. if the conversation successfully mimics the role, it should be straightforward for the judger to select the correct role. (2) whether the model can provide accurate role-related knowledge. we present a role-play dialogue to the judger and the underlying golden knowledge supporting the dialogue. the judger is tasked with determining whether the knowledge implied in the dialogue is consistent with the provided golden knowledge. we apply ditto on qwen-chat models @xcite in four different parameter scales to examine it empirically. extensive experiments show ditto effectively empowers llms with roleplay capabilities without distilling from advanced chatbots. ditto based on qwen-72b-chat even achieves 90% on role identity consistency, showing robust self-awareness in role-play. the general performance of qwen-72b-chat on our evaluation can be on par advanced chatbots, such as gpt-3.5-turbo, but slightly falls short on accurate rolerelated knowledge than gpt-4 and qwen-max, as qwen-max achieves the highest scores on the current role-play benchmark and the role-play subset of mt-bench. furthermore, we comprehensively analyze the dissection of role-play by extending our self-alignment setting to cross-supervision. experiments show consistent role identity can benefit from imitation learning even with worse supervision, while knowledge-related metrics do not. at the same time, we observe knowledge in role-play is bounded by the inherent capabilities of llms in strong-to-weak settings, and we notice consistent weak-to-strong generalizations on knowledgerelated metrics. such observations provide a deep and solid understanding of llm role-play and alignment, suggesting the knowledge of seed llms and proper demonstration, such as simulation data from ditto, are the key to impressive role-play capabilities. our contributions are mainly three-fold: chen et al. (2023) focused on evaluating how well a llm can align with a specific character, using harry potter as a case study. wang et al. (2023c) introduced the first fine-grained role-playing dataset containing 100 roles via prompting to gpt-3.5. li et al. (2023a) incorporated substantial prompts about the character's background, personality, and prior conversations, leveraging chatgpt to generate dialogues of 32 characters. zhou et al. (2023) prompted gpt-4 to expand the scale and diversity of human-annotated role-playing data, resulting in 1,034 dialogues of 250 characters. shao et al. (2023) also prompted gpt-3.5 to become the roleplay data generator. zhou et al. (2023) proposed drawing role-playing dialogues from diverse chinese novels and scripts with the help of gpt-4. however, limited by data sources, they could only construct a chinese dataset containing 77 roles. in this work, different from previous works, we completely abandon imitating proprietary llms and build role-playing training data entirely through self-alignment. our method separates character knowledge and conversation style, allowing it to be used with any llm capable of following instructions. it is highly scalable, creating the first multilingual dataset with 4,000 roles, 16 times the number in previous works. we demonstrate that our model achieves the best role-playing ability to date through self-alignment, surpassing even proprietary llms like gpt-3.5-turbo. ## evaluation efficient evaluation for open-ended problems, such as role-play, is significantly understudied. recent work depends on heavy manual annotations for conducting multifaceted role-play evaluations @xcite @xcite . however, though human evaluation is promising, it is label-intensive and cannot be exactly reproduced, impairing the further development of this field. this work proposes an objective assessment instead of previous preference annotations to evaluate basic role-play capabilities. we first design three core metrics for role-play and implement a trustworthy evaluation recipe for each based on "llms as judges" @xcite . during the evaluation, we only provide a brief introduction of the character profile, as shown in fig. 3 , such as "you are edward iii of england, king of england.". such a recipe evaluates whether llms can excavate inherent knowledge for roleplay. ## experiments in this section, we present experimental setup ( §5.1), main results of ditto ( §5.2), and further analyses ( §6). ## dissecting role-play by cross supervision we have observed in ## conclusion in this paper, we present for the first time a llm endowed with instruction-following capabilities, can achieve role-play proficiency through selfalignment without the need to distill proprietary counterparts like gpt-4. experimental results demonstrate the effectiveness of our proposed selfalignment strategy ditto, across four llm sizes ranging from 1.8b to 72b. it consistently outperforms all existing open-source role-play models, even without relying on distillation data. notably, it showcases performance levels comparable to proprietary llms such as gpt-4-turbo. furthermore, we delve into the decomposition of role-play into two distinct sub-abilities: role-specific knowledge and conversational style. the former is inherently constrained by the lm's knowledge, while the latter displays a spectrum of weak-to-strong generalization, facilitating easy acquisition from a smallersized model. our intention with this paper is to stimulate researchers to reconsider the foundational roots of role-play alignment capabilities.