P lato LM : Teaching LLM s in Multi-Round Dialogue via a User Simulator (2024)

Abstract: The unparalleled performance of closed-sourced ChatGPT has sparked efforts towards its democratization, with notable strides made by leveraging real user and ChatGPT dialogues, as evidenced by Vicuna. However, due to challenges in gathering dialogues involving human participation, current endeavors like Baize and UltraChat rely on ChatGPT conducting roleplay to simulate humans based on instructions, resulting in overdependence on seeds, diminished human-likeness, limited topic diversity, and an absence of genuine multi-round conversational dynamics. To address the above issues, we propose a paradigm to simulate human behavior better and explore the benefits of incorporating more human-like questions in multi-turn conversations. Specifically, we directly target human questions extracted from genuine human-machine conversations as a learning goal and provide a novel user simulator called ‘Socratic‘. The experimental results show our response model, ‘PlatoLM‘, achieves SoTA performance among LLaMA-based 7B models in MT-Bench. Our findings further demonstrate that our method introduces highly human-like questioning patterns and rich topic structures, which can teach the response model better than previous works in multi-round conversations.

## introduction large language models (llms) such as chat-gpt (openai, 2023) have made great strides in the dialogue domain. although chatgpt and its successor gpt-4 @xcite are successful, they remain proprietary and non-replicable. recent democratization efforts @xcite @xcite @xcite , in addition to focusing on distilling the responses of chatgpt through various means such as self-instruction @xcite ## background previous works typically focus on leveraging user simulators to generate large amounts of data with limited samples @xcite @xcite or enhancing the performance of the assistant's response through feedback from the user simulator via reinforcement learning @xcite @xcite in closed-domain conversations. shifting to open-domain conversations, without a specific task goal among multi-round conversations, it becomes challenging to ascertain user feedback. consequently, researchers have shown great interest in distilling data with seeds on selected domains via chatgpt's static simulation. baize @xcite ) distilled 100k samples called 'self-chat', and ultralm @xcite distilled 1,468k samples named 'ultrachat' based on seeds from humans or even chatgpt. we argue that while the large quantity of dialogue datasets signifies a substantial contribution, the key challenge lies in aligning the dialogues closer to real human-machine interaction scenarios, while reducing over-reliance on chatgpt. ## methodology as shown in figure 2 , the pipeline of our methodology consists of three steps: (1) fine-tune the user simulator socratic to raise questions; (2) generate the synthetic multi-round dataset called socratic-chat via iteratively calling socratic and chatgpt; ## analysis to further explore why questions from real human can teach the response model better, we conducted an in-depth analysis of the above 10k datasets. ## conclusion in this paper, we propose a straightforward yet effective paradigm for simulating users better than the traditional static simulation relying on chat-gpt. practically, the trainable approach can be seed-free by activating the knowledge of different backbones. theoretically, it captures the thinking patterns of genuine users questioning and leading the richer topic structures, which has been quantitatively proven to teach the response model better than the static simulation based on chatgpt in dynamic multi-round conversations. further experiments demonstrate the transferability, scalability, and versatility of this paradigm across various scenarios, as well as its ethical friendliness. in the future, we intend to research user simulators for some specific domains.