Reliable LLM -based User Simulator for Task-Oriented Dialogue Systems (2024)

Abstract: In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator’s coherence with user goals, effectively mitigating hallucinations—a major source of inconsistencies in simulator responses.

## introduction the field of dialogue systems has seen a notable surge in the utilization of user simulation approaches, primarily for the evaluation and enhancement of conversational search systems @xcite and task-oriented dialogue (tod) systems @xcite . user simulation plays a pivotal role in replicating the nuanced interactions of real users with these systems, enabling a wide range of applications such as synthetic data augmentation, error detection, and evaluation @xcite @xcite @xcite . the significance of user simulation in the development and evaluation of dialogue systems is undeniable. however, the prevailing methodologies often rely on rudimentary rule-and template-based approaches, which can limit their adaptability and effectiveness @xcite . furthermore, certain user simulation methods require a substantial amount of annotated data @xcite @xcite , or a deep understanding of the internal workings of the dialogue system they interact with @xcite . the rise of generative capabilities of large language models (llms) enabled user simulators to generate contextually appropriate responses in natural language, without the need for predefined rules @xcite . this shift offers distinct advantages over traditional approaches: i) no human effort is needed to construct the rules; ii) it introduces lexical diversity into utterance generation to assess the robustness of downstream natural language understanding and enables testing of system's robustness to different dialogue paths. however, llms are susceptible to hallucinations @xcite , resulting in inconsistency across dialogue turns or the generation of irrelevant information to the user's goal. in this paper, we introduce daus, a generative user simulator for tod systems. as depicted in figure 1 , once initialized with the user goal description, daus engages with the system across multiple turns, providing information to fulfill the user's objectives. our aim is to minimize the commonly observed user simulator hallucinations and incorrect responses (right-hand side of figure 1 ), with an ultimate objective of enabling detection of common errors in tod systems (left-hand side of figure 1 ). our approach is straightforward yet effective: we build upon the foundation of llmbased simulators @xcite and extend such approach by finetuning the llm on in-domain dialogues, annotated with their user goals. notably, daus does not require insights into the inner-workings of the tod system, its policy, nor system-specific functionalities, as it interacts with the tod system strictly through natural language. figure 1 : example conversation between user simulator and tod system. we aim to minimize common simulator's hallucinations (right) and thus ease the detection of tod system failures (left). we summarize our contributions and findings as follows: ## generative user simulator in this section, we define the task of generative user simulation for tod systems. moreover, we describe our approach, based on fine-tuned llms. ## experimental setting in this section, we describe datasets, implementation details, and experimental setting for simulatorsystem interactions. we consider two data sources to evaluate our approach. first, we experiment on internal dialogue data of user-tod system phone call interactions within the automotive industry, dubbed automo-tivedata. foot_0 second, we use the well-established dataset of multi-domain tod systems -multi-woz 2.1 @xcite . both data sources contain user goal g in natural language and multiturn dialogues (compliant with section 3.1). for each dataset, we randomly sample 2,500 dialogues for training, 300 for testing and 300 for validation. the statistics of the resulting datasets are reported in ## evaluation we comprehensively evaluate our method, aiming to assess its ability to achieve designated user goals in dialogues and its impact on lexical diversity when aligning with real user language patterns. moreover, we perform qualitative analysis of simulated dialogues via human evaluation. in this section, we detail these evaluation procedures. additionally, we examine utterance-level metrics, comparing generated utterances with those in the target dataset, using both general natural language generation and domain-specific entity-based metrics. however, we found that these metrics poorly correlate with the simulator's task completion. detailed information about these metrics and their results can be found in appendix c. ## results in this section, we examine our study's findings across three main threads. first, we investigate the impact of fine-tuning llms with domain-specific data on goal fulfillment in dialog interactions (section 6.1). next, we explore the link between finetuning and the lexical diversity of generated utterances (section 6.2). finally, we assess whether the adaptability of llm-based user simulators to unseen user tasks is influenced by the diversity of subtask types in their training data (section 6.3). ## conclusions the use of a domain-aware llm-based user simulator, such as daus, shows promising results in multi-turn interactions with tod systems. daus can fulfill user goals by generating consistent and faithful utterances. compared to previous llm-based approaches @xcite , our method has demonstrated superior performance, as measured by multiple metrics designed to capture the fulfillment of the given goal, as well as faithfulness across the dialogue. this indicates that daus is capable of effectively simulating user behavior and can serve as a valuable tool for testing and evaluating tod systems. moreover, our approach requires relatively small training dataset and imposes modest computational demands, thanks to parameter-efficient fine-tuning. this discovery aligns with findings in related research that contrasts in-context learning with parameter-efficient fine-tuning @xcite . consequently, our approach emerges as a pragmatic choice for broader adoption within the nlp and conversational ai community. the potential applications of llm-based user simulators are synthetic data augmentation @xcite , supporting reinforcement learning approaches @xcite , and tod system evaluation @xcite ). daus's reliability and consistency to the user goal make it particularly suitable for tod system evaluation. as we have seen previously, an incomplete user goal can mainly imply two scenarios: a user simulator who hallucinates or a tod system that is not able to understand the user's requirements. therefore, the presence of a reliable user simulator is crucial: it allows us to identify the tod system's errors with high accuracy. moreover, we stress that at the center of our approach is an llm, leading to potentially different generations given the same input, depending on the sampling method. this means that daus is more flexible than certain agenda-based simulators, which usually rely on template-based responses. as such, we are able to simulate a dialogue with the same user goal multiple times, which results in multiple different attempts of the simulator to fulfill its goal, going through potentially different conversational paths. therefore, we are able to test the robustness of the tod system to different expressions of the same user goal.