KAUCUS - Knowledgeable User Simulators for Training Large Language Models (2024)

Abstract: An effective multi-turn instruction-following assistant can be developed by creating a simulator that can generate useful interaction data. Apart from relying on its intrinsic weights, an ideal user simulator should also be able to bootstrap external knowledge rapidly in its raw form to simulate the multifarious diversity of text available over the internet. Previous user simulators generally lacked diversity, were mostly closed domain, and necessitated rigid schema making them inefficient to rapidly scale to incorporate external knowledge. In this regard, we introduce Kaucus, a Knowledge-Augmented User Simulator framework, to outline a process of creating diverse user simulators, that can seamlessly exploit external knowledge as well as benefit downstream assistant model training. Through two GPT-J based simulators viz., a Retrieval Augmented Simulator and a Summary Controlled Simulator we generate diverse simulator-assistant interactions. Through reward and preference model-based evaluations, we find that these interactions serve as useful training data and create more helpful downstream assistants. We also find that incorporating knowledge through retrieval augmentation or summary control helps create better assistants.

## introduction significant advancements in large language models (llms) have made them exceptionally adept in conversational applications like virtual assistants @xcite @xcite . this proficiency is largely attributed to the notably parallelizable transformer architecture @xcite enabling these models to utilize extensive pre-training datasets effectively @xcite . to create effective assistants, llms are then further enhanced by learning from human interactions including popular paradigms such as rlhf @xcite @xcite . such conversational human alignment of assistants requires large amounts of interactive dialog data, both for training as well as testing. however, interactive data collection is a manual and slow process, particularly (a) for covering a wide range of user behaviors as well as (b) for diverse adversarial and behavior testing. these challenges can be mitigated by simulating user behaviors by automating the generation of interactive data, reducing both time and cost, while maintaining control over the interactions. simulated interactions can be executed at a much faster pace than manual collection efforts, limited only by the speed of inference. yet, current user simulators lack diversity, are mostly closed domain, and require rigid schema for control or conversation grounding. the necessity of intermediate schema in the form of a knowledge base @xcite or handcrafted rules (like user persona or specific behaviors) while being excellent drivers to ground conversations, make it hard to develop scalable simulators -that can utilize natural text freely available on the internet and rapidly create corresponding assistant models. a simulator should be able to exploit external knowledge rapidly and also be controllable without a rigid schema. we argue that such a knowledge simulator can be helpful in two ways -it can seamlessly convert free-form text to useful training data without user intervention as well as provide a natural control to direct simulators for specific behaviors @xcite . hence, in this work, we propose kaucus, a knowledge augmented simulator framework foot_0 . through this framework, we demonstrate the usage of external sources of knowledge -viz. retrieval augmentation and summary control -for creating collect and augment demonstration data collect human-machine demonstration data use external knowledge to augment the data generate demonstration summaries from external summariser create a simulator and generate interaction data use the collected demonstration data to train a simulator use free flowing text to interact with a partner assistant model and create simulated interactions train an assistant and evaluate using a reward model use the interactions to train a downstream assistant generate interactions by talking to any simulator use a reward model trained on human preferences to evaluate user simulators that can incorporate free-flowing text and result in better assistant training. the paper is organized as follows: in section 2, we first discuss existing work related to user simulators. in section 3, we define simulators and introduce kaucus, through two knowledge simulators. we further describe the efficacy of each through training and evaluating downstream assistant models. our retrieval augmented simulator, srag shows how retrieving relevant passages with a simple bm25 retriever can be used to improve intrinsic metrics as well as provide useful training data to train helpful assistants. we also introduce the summary-controlled setting, sctrl to build scalable simulators to exploit freely available text and further measure their performance with and without retrieval. ## related work user simulators have been studied in various settings. aher et al. (2023) create four simulators that elicit behavior to judge an assistant's fairness, rationality, grammaticality, and general knowledge, and then measure them qualitatively. their simulators are models with different prompt templates. training multi-agent interactions has been a popular choice in reinforcement learning. horton (2023); argyle et al. (2023) create simulations for economic purposes by endowing gpt3 @xcite with demographic characteristics and then get responses in various scenarios that match what is seen empirically. @xcite in ai safety has proposed using self-play and self-debate to train ai agents to pursue human goals and preferences. two tasks in the collaborative benchmark, big-bench @xcite evaluate the model's ability for self-evaluation by simulating specific human professions. they make the models to act as lawyers, tutors foot_1 , judges foot_2 , students, etc. and then have separate model instances to evaluate the conversation. each of the roles is invoked by user-specific prompts like "you are a lawyer" and a subsequent model-based evaluation is performed by prompting to seek numerical ratings. other studies augment simulators with emotions @xcite and trusting behaviours @xcite . for instance, @xcite simulate user emotions alongside user behavior based on the user goal, the dialogue history, and persona. @xcite utilize gpt-based models for scientific simulations while @xcite explore llms to simulate biological systems. with the popularity of large language models deployed in closed-source settings, bootstrapping training data from them has become useful. taori et al. (2023) create downstream assistant models by training llama-7b and 13b models @xcite on 52k single-turn instruction following demonstrations generated through self-instruct @xcite ) from text-davinci-003 @xcite . @xcite create a dialog corpus by extending the same to the multi-turn setting. @xcite show improved conversation retrieval by proposing a mechanism to convert wikipedia passages to dialog. on the other hand, retrieval augmentation has been the focus of many recent efforts @xcite @xcite as it offers advantages such as up-todate information access beyond an llm's training dataset, incorporation of proprietary or domainspecific data at runtime, and enhanced factuality in outputs compared to standard llms. studies have been performed by training rag systems end-toend @xcite as well as using retrieval in context for various tasks @xcite @xcite . ## the kaucus framework in this section, we introduce kaucus, a 3-stage framework, and outline the process of creating knowledge-augmented simulators as shown in figure 1. our approach involves the following steps: ## methods and experiments we now specifically describe the two types of knowledge-augmented simulators, viz. utterance human:```a monopoly is distinguished from a monopsony, in which there is only one buyer of a product or service; a monopoly may also have monopsony control of a sector of a market.```can you write a short introduction about the relevance of the term "monopsony" in economics? please use examples related to potential monopsonies in the labour market and cite relevant research assistant: monopsony refers to a market structure where there is only one buyer of a good or service…. human:```the term may also be used to describe the restriction of certain access rights when an employee has changed roles within the organization. onboarding and offboarding procedures must be documented to ensure compliance with regulatory requirements.```what can be done at a regulatory level to ensure, in a monopsony, that the power….", ## conclusion simulators provide a way to generate data to create downstream assistant models saving human time and effort. through our framework kaucus, we further showed that augmenting simulators by exploiting external knowledge helps generate diverse interactions and as well as creates more helpful assistants than vanilla simulators. we describe two types of knowledge-augmented simulators, a retrieval augmented simulator, srag, and a summary-controlled simulator, sctrl both of which consume external knowledge in unique ways. raw text is more prevalent than the conversational counterparts. controlling simulators through conversational summaries or external documents can be a quick and powerful tool to convert public text to trainable interaction data and create more helpful assistants. it provisions the simulator to generate interactions for novel information outside the scope of an llm's intrinsic parameters. we hope kaucus will help encourage the development of automated techniques to be able to incorporate the vast amount of text produced rapidly over the internet and align assistant models better with newer data as well as be able to control the distribution of training data without the need for a rigid schema.