Bootstrapping LLM -based Task-Oriented Dialogue Agents via Self-Talk (2024)

Abstract: Large language models (LLMs) are powerful dialogue agents, but specializing them towards fulfilling a specific function can be challenging. Instructing tuning, i.e. tuning models on instruction and sample responses generated by humans (Ouyang et al., 2022), has proven as an effective method to do so, yet requires a number of data samples that a) might not be available or b) costly to generate. Furthermore, this cost increases when the goal is to make the LLM follow a specific workflow within a dialogue instead of single instructions. Inspired by the self-play technique in reinforcement learning and the use of LLMs to simulate human agents, we propose a more effective method for data collection through LLMs engaging in a conversation in various roles. This approach generates a training data via “self-talk” of LLMs that can be refined and utilized for supervised fine-tuning. We introduce an automated way to measure the (partial) success of a dialogue. This metric is used to filter the generated conversational data that is fed back in LLM for training. Based on our automated and human evaluations of conversation quality, we demonstrate that such self-talk data improves results. In addition, we examine the various characteristics that showcase the quality of generated dialogues and how they can be connected to their potential utility as training data.

## introduction llms have recently proven their viability to be deployed as conversational agents in a variety of scenarios, promising novel possibilities in user-facing applications, such as working as a personal assistant or customer support. in these scenarios, dialogue agents have to be able to quickly adapt to new tasks, which in the current paradigm implies either the use of already-existing or the creation of new datasets. naturally, this two llms, called a client and an agent, are prompted to converse with each other in different roles, with the agent asked to follow a specific narrative structure. generated conversations will then be filtered by quality and used for supervised finetuning on the agent model until it adapts to the intended dialogue structure. in reinforcement learning (rl) for games, selfplay has proven to be an effective mean to bootstrap powerful systems that can beat human experts @xcite , even when no initial humangenerated data @xcite or rules @xcite are supplied. at the same time, increasingly sized language models trained with growing datasets have shown to develop limited but notable extents of language abilities @xcite and some degree of theory-of-mind, i.e. reasoning over other agents, their behavior and the world state @xcite . this has prompted studies in which they simulate single users @xcite or even interacting groups of agents @xcite . recent work by @xcite also demonstrated how llms can be iteratively improved on their own, generating data for language modeling. we take these developments as encouragement to explore llm self-talk as a way to bootstrap taskspecific finetuning data from llms themselves. specifically, we focus on task-oriented dialogue, a very relevant problem for applications such as customer service, pair programming and digital personal assistants. we propose the following procedure: two versions of an llms take part in a dialogue, impersonating a client and an agent. both are given a description of how to act within the dialogue: the client is given a persona and motivation, while the agent is equipped with a fixed set of behaviors, depending on the request of the agent. they then engage in a conversation, which is recorded as a potential training sample. some of these conversations will be successful, however, many of them will not be useful, since they will be off-topic or of low-quality. for this reason, we employ an additional filtering step to avoid training the model on counterproductive samples. a schematic depiction of our method is shown in figure 1. our contributions are as follows: 1 we propose a new method to bootstrap training data for task-oriented dialogue agents. 2 we explore different quality assurance options. 3 we also introduce automated evaluation metrics to evaluate dialogue success and conversational consistency and 4 assess our trained agents in a human evaluation study. llms as agents. the quality of dialogue and reasoning in current llms also has enabled research on simulating human agents: for instance, @xcite demonstrate that suitable prompts can make llms mimic the answers of different test subject demographics, and @xcite create a dialogue user-simulator. @xcite put llm agents equipped with memory into a grid world and stimulate discussions between them about sensitive topics to aid the emergence of social norms. by further having llms act with different personas and enabling them to retrieving relevant memories, @xcite are able to simulate the social dynamics in a simulated village. llm self-improvement. while naïvely retraining a generative model on its own outputs might produce degenerate solutions @xcite , llms have been successfully shown to produce data augmentations for other models @xcite . furthermore, a recent line of work is exploring the limits of using llms to provide signal for their own training: @xcite partially automate the process of ranking model responses through human feedback by having models critique their generations according to different harmfulness criteria, while @xcite have the llm generate multiple refinements of the output based on the human feedback. a further improvement in responses has been observed through the use of an external critique model @xcite , or through self-critiquing for a diverse set of tasks such as information extraction @xcite , code generation @xcite and summarization @xcite . we see the recent work of @xcite as closest to ours; the authors show how after generating data during a grow step, different tranches can be utilized to incrementally train the model during an improve step. in contrast, our approach focuses on a single improve step and employs a conceptually simpler supervised finetuning strategy over a rl objective. furthermore, gulcehre et al. focus their work on conditional language modeling, while we experi-ment within the domain of task-oriented dialogue. this also motivates our framework of using two separate models for the dialogue participants, albeit that it also admits the use of two copies of the same model ( see next section we take these research lines as encouraging pieces evidence for the following points: a) neural networks, and particular conversational agents, can be used to bootstrap their own training data under certain conditions and b) llms have the ability to provide learning signals for themselves. these insights are synthesized into our proposed method. ## method we first introduce some notation for our method. we start with two parameter vectors θ client and θ agent of pretrained llms for the client and agent model, respectively. since we might update the parameters multiple times, we denote θ n } from this distribution at every time step by having the models converse in roles specified by their prompts. while we can expect some of the generated dialogues to be successful given a sufficient model and high-quality prompts, there is no guarantee for them to achieve novel dialogue goals. for this reason, we employ a filter function to create a subset t) of the generated dialogues. an example for a filter would be to only include successful conversations, identified using a method we introduce later in section 4.1. the filtered examples are used to finetune the models by minimizing the negative log-likelihood ## experiments dataset. for the characters that the llms are supposed to embody in our conversations, we source descriptions from the light dataset by @xcite . originally used for agents that solve tasks in a fantasy-world environment, it contains different 7487 quest descriptions, including characters and their personas. for instance, the role of genie from the lamp has the persona of "i am a genie and i enjoy playing tricks on those who release me from my lamp. i was imprisoned in this lamp by an evil witch. i am very bitter that i have to spend my days as a prisoner". we choose a set of five characters for potential roles of the agent (shop keeper, doctor, witch, tavern owner, genie from lamp) and select another set of the 16 most frequently occurring characters as potential clients (see appendix a.1). to create dialogue workflows, we then then prompt a 13 billion parameter openllama variant @xcite @xcite foot_0 to generate three distinct work-flows per agent character respectively including four, six and seven questions, which are then manually edited for quality. an overview and sample of these workflows is presented in appendix a.1. self-talk loop. for the self-talk loop in figure 1 , we choose the 30 billion parameter mosaicai chat model (mosaicml nlp team, 2023) for the client and the 7 billion parameter model from the same family for the agent. we found the client model to already work quite consistently using only prompting, so we choose to only finetune the agent model. for every dialogue generation round, we first sample at random a client character, and agent character and a corresponding agent workflow. then, we generate the conversation utterance by utterance for up to eight turns (one turn including one utterance per character). we also check whether a conversation has ended by using a combination of string matching and prompting a llm along with some postprocessing to ensure the quality of generation, all of which are described in appendix a.2. filters. generated dialogues are then filtered and used to finetune the agent model for a single epoch. we compare the impact of multiple different filtering procedures, with baselines using the entire generated set of dialogues for finetuning (none) and sampling a random p % subset (p-random). we also explore whether at least k subgoals or p % of subgoals in the workflow have been achieved (k-subgoals and p-subgoals, see next section). ## discussion our experimental results have demonstrated the challenges that arise when implementing self-talk in a task-oriented dialogue setup. these reach from the ability of the agent to follow the workflow carefully and selecting the next steps reliablya problem that is at least partially rooted in the open-ended and paraphrastic nature of conversations. nevertheless, we have provided insights that a proper filtering technique isolating high-quality data constitutes a positive training signal for the agent llm, that can be used to improve its performance. this conclusion is in line with the emerging literature on llm self-critique and the concurrent work of @xcite on using self-generated data for finetuning. an open question in this vein is whether this process can reliably repeated multiple times, and determining the conditions under which it leads to an improvement or model degradation. in some tentative experiments, we found multi-loops of self-talk to lead to minor improvements at first, but also to be unstable if the number of filtered finetuning samples was too low. we leave further explorations of this question to future work and collect other, smaller insights from our experiments in appendix b.3. ## conclusion despite previous works articulating worries about the distributional collapse of model trained on their own generated data @xcite , we demonstrated thatgiven the right setup-language models can selfimprove by being trained on their own generation and become better task-oriented dialogue agents. as we showed in section 4.2, this depends on the setup; using different models and varying prompts helps to create sufficiently diverse data, with additional filters ensuring sufficient quality. we especially found the filters that strike a balance between quality and number of total data points to work best for selecting finetuning data and validated these insights with the human evaluation in section 4.3. in addition, we established new methods for automatic evaluation of agents in a task-oriented dialogue setting and showed that they can be used to quantify the progress of our finetuned model. future work. this work focused on self-talk for task-oriented dialogue agents. since we are finetuning on bootstrapped conversations, we expect the model to loose some general conversational abilities that are not relevant for the given task. future work could explore how we can nevertheless maintain these abilities. furthermore, this version of self-talk could still be improved by generating higher-quality conversations and identifying the conversations are most informative to the model for finetuning, reminiscent of work in active learning (see e.g. zhang et al., 2022). lastly, our setup intuitively lends itself well to also using negative signals, since low-quality conversation could be used for negative rewards. the agent could then be optimized with an objective similar to direct preference optimization @xcite or using natural language policy optimization @xcite by embedding it in a rl environment.