Generating Diverse Personas for User Simulators to Test Interview Dialogue Systems (2025)

Abstract: This paper addresses the issue of the significant labor required to test interview dialogue systems. While interview dialogue systems are expected to be useful in various scenarios, like other dialogue systems, testing them with human users requires significant effort and cost. Therefore, testing with user simulators can be beneficial. Since most conventional user simulators have been primarily designed for training task-oriented dialogue systems, little attention has been paid to the personas of the simulated users. During development, testing interview dialogue systems requires simulating a wide range of user behaviors, but manually creating a large number of personas is labor-intensive. We propose a method that automatically generates personas for user simulators using a large language model. Furthermore, by assigning personality traits related to communication styles when generating personas, we aim to increase the diversity of communication styles in the user simulator. Experimental results show that the proposed method enables the user simulator to generate utterances with greater variation.

## introduction interview dialogue systems, which can efficiently gather information from many users, are a promising application of dialogue system technology and have attracted increasing research interest in recent years @xcite @xcite . however, building dialogue systems, including interview dialogue systems, incurs significant costs. one major cost is the effort required for testing. the goal of our study is to alleviate the manual effort for testing interview dialogue systems. to reduce testing costs, user simulation can be used instead of manual testing. however, existing user simulators have primarily been developed for the training and evaluation of task-oriented dialogue systems. thus, we aim to develop a simu-lator that is useful for testing interview dialogue systems. when testing interview dialogue systems, simulating a wide variety of users is crucial. to increase variation, it is beneficial to have a large number of user personas @xcite @xcite . however, manually creating numerous personas is labor-intensive. while using a large set of pre-constructed personas has been considered @xcite , this approach is unsuitable for interview dialogues, where personas need to be aligned with the specific interview topics. therefore, this paper proposes a method to generate diverse personas using large language models (llms). in this method, a few manually created personas are provided as examples, and the llm is tasked with generating a large number of additional personas. to ensure diversity in speaking styles and attitudes toward the system, personality traits are specified during generation. evaluation experiments demonstrated that the proposed method enables the creation of diverse dialogues through the generation of many personas. this increases the likelihood of discovering system issues. note that detecting system issues from simulated dialogues is beyond the scope of our study. manual inspection is one possible approach, but automated methods using llms are also conceivable @xcite . the contributions of our study are as follows: ## proposed method in the proposed method, we generate simulated user personas using an llm. it is necessary to seed personas personality traits generated personas llm-based user simulator (simulated interviewees) interview dialogue system to test interact for testing prompt for simulation prompt for persona generation llm-based persona generation embedded in the prompt ensure that the generated personas include information relevant to the domain of the target interview dialogue system. to achieve this, we perform incontext learning using a small number of manually written personas (called seed personas hereafter) as few-shot examples. furthermore, to facilitate the discovery of potential issues by introducing greater variation in user utterances during simulation, we provide personality traits that define communication styles when generating personas with the llm. by embedding the generated personas and personality traits into the prompt of an llm-based user simulator, we can generate diverse dialogues. figure 1 illustrates the proposed method. we use the following two axes as personality traits. degree of anthropomorphism one axis concerns whether the user treats the system as an object or as a human. users who treat the system as an object aim to use the dialogue system efficiently and tend to use expressions that are easy for the system to understand. in contrast, users who treat the system as a human speak to it as if they were speaking to a person, without considering whether the system can understand them. this behavior can be interpreted as users anthropomorphizing the system @xcite . we refer to this axis as the degree of anthropomorphism. degree of elaborateness the other axis concerns whether the user engages in redundant communication or direct communication. @xcite and @xcite classify users' communication styles into elaborateness and directness. figure 2 : topics the travel interview dialogue system asks the user about. we adopt the same classification in our study. although the big five personality traits are wellknown, we did not use them because it is considered more effective to directly specify users' speaking styles is considered more effective for our purpose. ## evaluation we conducted an evaluation experiment to investigate whether the proposed method is effective for testing interview dialogue systems. note that this experiment deals only with japanese systems, and that all subsequent figures are translations from japanese. ## discussion as discussed above, the experimental results suggest that generating personas with an llm enables the generation of dialogues with greater variation. although the quantitative metrics do not indicate this very clearly, you can see the variation by looking at the generated dialogues. this could lead to the discovery of more problems in dialogue systems. furthermore, specifying personality traits related to communication style can introduce greater variation in utterance lengths, which may help uncover additional problems. in the current user simulator prompt we used, relatively similar utterances tend to be generated consecutively. this might be different from the behaviors of human users. it is possible that generating various types of utterances within a single dialogue could help reveal different types of issues. in future work, we will explore methods for increas-el high: ## concluding remarks this paper proposed a persona generation method for user simulation using large language models (llms). we also introduced a method to incorporate personality traits related to communication style during persona generation. this enables diverse testing of interview dialogue systems without human involvement, significantly reducing development costs. although the evaluation experiments were limited and some issues remain, the results suggest the effectiveness of the proposed method. therefore, we believe it is worthwhile to share our proposed method and experimental findings. in the future, we plan to develop user simulators for interview dialogue systems with speech input/output and multimodal input/output. we also aim to extend our approach to build user simulators for various types of dialogue systems beyond interview dialogues.