LLM Roleplay: Simulating Human-Chatbot Interaction (2025)

Abstract: The development of chatbots requires collecting a large number of human-chatbot dialogues to reflect the breadth of users’ sociodemographic backgrounds and conversational goals. However, the resource requirements to conduct the respective user studies can be prohibitively high and often only allow for a narrow analysis of specific dialogue goals and participant demographics. In this paper, we propose LLM Roleplay, the first comprehensive method integrating multi-turn human-chatbot interaction simulation, explicit persona construction from sociodemographic traits, goal-driven dialogue planning, and robust handling of conversational failures, enabling broad utility and reliable dialogue generation. To validate our method, we collect natural human-chatbot dialogues from different sociodemographic groups and conduct a user study to compare these with our generated dialogues. We evaluate the capabilities of state-of-the-art LLMs in maintaining a conversation during their embodiment of a specific persona and find that our method can simulate human-chatbot dialogues with a high indistinguishability rate.

## introduction collecting human-chatbot dialogues requires recruiting and managing a large number of human annotators, which can pose prohibitive obstacles to researchers who aim to develop conversational ai agents (i.e., chatbots). to circumvent the latter and the limitations of publicly available data, numerous methods employing chatbots to generate dialogues have been introduced lately @xcite @xcite @xcite . these methods can generate dialogues much faster and more cost-effectively while still approximating the quality and variety of human annotators @xcite . master's degree and english is not their native language" finish. figure 1: schematic illustration of our method: a textual description of a persona and a goal (top) is used to instruct the inquirer (s i ) model to embody the given persona (left) and engage in a dialogue with the responder (s r ) chatbot (right). we show that dialogues simulated by the inquirer llm and the responder chatbot can effectively simulate human-chatbot interaction. recently, several studies have leveraged such synthetic data generated through distillation and self-improvement techniques to enhance the capabilities of large language models (llms) @xcite . by employing instruction tuning and fine-tuning techniques, these efforts aim to ensure that llms generate useful and safe responses that adhere to provided instructions @xcite @xcite @xcite . however, current dialogue generation methods have two critical limitations. first, the set of possible user responses for a given chatbot utterance 1 is large while the existing datasets typically only cover a single or few annotated user responses. previously generated datasets are constrained by the number of dialogues and turns, making it difficult to extend them to novel domains or conversational goals. second, existing methods rarely account for subjective response behavior and implicitly assume an average user. this can create a gap in representation @xcite and pose the risk of evaluating and improving chatbots for a specific sociodemographic group while neglecting others. the quality of existing datasets is primarily assessed through model performance in a supervised fine-tuning (sft) setup, which is influenced by various properties of the generated dataset. for example, @xcite argue that the initial prompt plays a crucial role in the quality of the generated dialogue. on the other hand, @xcite demonstrate that the length of the response is a key factor, and with significantly fewer samples, it outperforms methods that focus solely on quality. meanwhile, @xcite emphasizes the importance of mimicking human-style interactions in the dialogues. to mitigate the shortcomings of existing dialogue generation methods and respond to the needs identified in prior work for sft, we introduce llm roleplay, the first comprehensive method integrating multi-turn human-chatbot interaction simulation, explicit persona construction from sociodemographic traits, goal-driven dialogue planning, and robust handling of conversational failures, enabling broad utility and reliable dialogue generation. our method is the first to instruct an llm (inquirer) to adopt a specific persona and prompt a chatbot (responder) to achieve a given conversational goal, thereby eliciting realistic human-ai interactions with a particular llm. figure 1 illustrates an exemplary application of our method. in this work, we address the following two research questions: (i) to what extent can we simulate real human-chatbot dialogues using llmchatbot dialogues? (ii) how do various llms perform as inquirers within this setup? to answer those research questions, we conduct two user studies. first, we collect real human-chatbot dialogues by asking participants to reach various conversational goals and link the collected dialogues with participants' sociodemographic backgrounds. next, we use llm roleplay to simulate dialogues with the same set of personas and goals. second, we conduct a human evaluation with another pool of participants to assess how well the generated dialogues mimic the collected ones. specifically, we present participants with two dialogues: one collected from the first study and one simulated using llm roleplay, both involving the same persona and the conversation goal. we then ask the participants to identify which of the dialogues was simulated. we find that llm roleplay approximates natural human-chatbot dialogues with a high level of indistinguishability. overall, this paper contributes: (i) a novel method to simulate human-chatbot dialogues with an arbitrary choice of personas and conversational goals; (ii) a human evaluation confirming our method's potential to closely resemble real dialogues; (iii) a dataset of goal-oriented humanchatbot and model-chatbot dialogues using our hand-crafted multi-hop goals involving four stateof-the-art llms, (iv) an in-depth comparison of open-source and proprietary llms in maintaining conversations while embodying specific personas, (v) an open-source implementation of our plug-andplay method, readily applicable for simulating dialogues with any combination of model and chatbot across various conversational goals and personas. ## large language model roleplay in the following, we introduce the notation that we are following throughout this paper. refer to figure 1 for an annotated example. persona (p) is the composition of sociodemographic features. we conceptualize persona as a written description of an individual from a specific sociodemographic group. we follow @xcite to define the sociodemographic features and the options of the latter. ## discussion and future work in this work, we propose the novel llm roleplay method: an automatic, model-agnostic approach for eliciting multi-turn, goal-oriented, personabased simulated human-chatbot dialogues. we develop and validate our method through two user studies involving 40 participants. our findings show that up to 44% of these dialogues, where 50% represents perfect indistinguishability, are indistinguishable from real human-chatbot interactions and feature more turns than previous datasets. building on this work's findings, several future research avenues warrant exploration. first, our method can generate user-specific conversational datasets for targeted alignment and domainadaptation (e.g., using rlhf @xcite ). second, it can improve dialogue evaluation by providing ample realistic conversational data. while this paper presents important findings on llm roleplay and provides the first evidence that our method can approximate human-chatbot dialogues, the sociodemographic representativeness of our method still needs assessment and advancement. we release our dialogue dataset, method code, and synthesized dialogues to support future research in this promising field. ## conclusion we present our novel comprehensive method llm roleplay integrating multi-turn human-chatbot interaction simulation, explicit persona construction from sociodemographic traits, goal-driven dialogue planning, and robust handling of conversational failures, enabling broad utility and reliable dialogue generation. in a series of two user studies, we collect real human-chatbot dialogues and demonstrate that llm roleplay can generate diverse multi-turn conversations that approximate natural human-chatbot dialogues with a high level of indistinguishability. our findings highlight the potential of llms in simulating human-chatbot interactions to synthesize realistic dialogues that create new opportunities for real-time model evaluation and training data generation for model finetuning.