Enhancing Dialogue State Tracking Models through LLM -backed User-Agents Simulation (2024)

Abstract: Dialogue State Tracking (DST) is designed to monitor the evolving dialogue state in the conversations and plays a pivotal role in developing task-oriented dialogue systems. However, obtaining the annotated data for the DST task is usually a costly endeavor. In this paper, we focus on employing LLMs to generate dialogue data to reduce dialogue collection and annotation costs. Specifically, GPT-4 is used to simulate the user and agent interaction, generating thousands of dialogues annotated with DST labels. Then a two-stage fine-tuning on LLaMA 2 is performed on the generated data and the real data for the DST prediction. Experimental results on two public DST benchmarks show that with the generated dialogue data, our model performs better than the baseline trained solely on real data. In addition, our approach is also capable of adapting to the dynamic demands in real-world scenarios, generating dialogues in new domains swiftly. After replacing dialogue segments in any domain with the corresponding generated ones, the model achieves comparable performance to the model trained on real data. The source code and generated dialogue data are available at https://github.com/ParticleMedia/LUAS.

## introduction dialogue state tracking (dst) is a critical component of task-oriented dialogue systems, serving to track users' goals and system actions in the conversation and facilitate precise information handling for communicating with external apis @xcite @xcite . dst task usually applies the form of key-value pairs, where the keys are denoted as slots which are defined in the system schema, outlining the specific information that the system aims to track or extract information during the whole conversation @xcite . the design of dst approaches could be broadly categorized into two main types, the classification-based dst models and generation-based dst models. classification-based models choose slot values from candidates @xcite , assuming that the dialogue ontology is pre-defined and hence lacking generalization capability @xcite . generation-based models directly generate the slot values to handle unseen domains and values @xcite @xcite . @xcite proposes a novel framework, ldst, based on llama @xcite . by using an instruction tuning method, ldst achieves performance on par with @xcite . despite dst showing promising results, a significant challenge is that the annotation of dialogues entails significant costs. furthermore, the dynamic nature of real-world demands highlights the urgent need to quickly generate more utterances for new domains. compared to the other types of nlp data, collecting the authentic dialogue data is particularly challenging. this difficulty is partly due to the dialogues frequently containing personal or sensitive information, which complicates data collection and sharing efforts. in response to these two challenges, and inspired by the recent advancements of large language models (llms) @xcite @xcite , we try the application of these llms for generating annotated dst data for data augmentation. by leveraging llm's cross-domain generation capability, we aim to create synthetic dialogues that could serve as replacements for manually annotated data, significantly alleviating both financial cost and time constraints. in this paper, we propose a llm-backed user-agents simulation (luas) algorithm to enhance dst. the process begins with the llm generating the user profile that details the individual's preferences for different tasks. following this initial step, the llm is prompted to simulate a conversation between the user and the agent. in these simulations, the user simulator makes requests and seeks recommendations or assistance, while the agent responds by understanding the user's needs, providing suggestions, and taking appropriate actions. through iterative conversations between the user and agent, complemented by a slot extractor also prompted by the llm, we could generate a substantial corpus of labeled, multi-turn dialogue data. in particular, it is essential to perform consistency checks between the extracted slots and the true values of the user's requests or entity attributes to ensure the quality of the generated dialogue data. to verify the effectiveness of our approach and the quality of the generated data, experiments are conducted on two public dst datasets, multiwoz 2.2 @xcite and multiwoz 2.4 @xcite datasets. following @xcite , llama 2 is finetuned with real data as the strong baseline. by using both the generated data and real data, finetuning llama 2 can further improve the performance. besides, by replacing dialogue segments of any domain with the generated data, the newly trained model achieves comparable performance to the model trained on the real data, which shows the capability of our approach to meet the dynamic requirements of real-world scenarios, generating dialogues in new domains and preserving the promising high performance. in summary, the contributions of our work can be categorized into the following aspects: ## method in this section, we will begin with the basic problem definition ( §3.1). then, we introduce our pro- ## conclusion in this paper, we propose a novel approach utilizing gpt-4 to simulate conversations between users and agents and generate dialogues with dialogue state labels. we then conduct a two-stage fine-tuning of llama 2 on both the generated and real data for dst prediction. experimental results on two public dst benchmarks demonstrate that our method, augmented with the generated data, surpasses the baseline trained solely on real data. furthermore, detailed analysis confirms the adaptability of our approach, effectively meeting the dynamic requirements of transitioning to new domains in real-world scenarios. we also believe that this approach can be extended into a generalizable framework, offering benefits to a wide range of dialogue-related tasks.