One Cannot Stand for Everyone! Leveraging Multiple User Simulators to train Task-oriented Dialogue Systems (2023)

Abstract: User simulators are agents designed to imitate human users; recent advances have found that Task-oriented Dialogue (ToD) systems optimized toward a user simulator could better satisfy the need of human users. However, this might result in a sub-optimal ToD system if it is tailored to only one ad hoc user simulator, since human users can behave differently. In this paper, we propose a framework called MUST to optimize ToD systems via leveraging Multiple User SimulaTors. The main challenges of implementing MUST fall in 1) how to adaptively determine which user simulator to interact with the ToD system at each optimization step, since the ToD system might be over-fitted to some specific user simulators, and simultaneously under-fitted to some others; 2) how to avoid catastrophic forgetting of the adaption for a simulator that is not selected for several consecutive optimization steps.To tackle these challenges, we formulate MUST as a Multi-armed bandits (MAB) problem and provide a method called MUST _{\mathrm{adaptive}} that balances i ) the boosting adaption for adaptive interactions between different user simulators and the ToD system and ii ) the uniform adaption to avoid the catastrophic forgetting issue.With both automatic evaluations and human evaluations, our experimental results on MultiWOZ show that the dialogue system trained by MUST achieves a better performance than those trained by a single user simulator. It also has a better generalization ability when testing with unseen user simulators.

## introduction task-oriented dialogue systems aim to help users accomplish their various tasks (e.g., restaurant reservations) through natural language conversations. training task-oriented dialogue systems in supervised learning approaches often requires a large amount of expert-labeled dialogues, however collecting these dialogues is usually expensive and time-consuming. moreover, even with a large amount of dialogue data, some dialogue states may not be explored sufficiently for dialogue systems foot_1 @xcite . to this end, many researchers try to build user simulators to mimic human users for generating reasonable and natural conversations. by using a user simulator and sampling user goals, we can train the dialogue system from scratch with reinforcement learning (rl) algorithms. previous works tend to design better user simulator models @xcite @xcite @xcite . especially, @xcite builds various user simulators and analyzes the behavior of each user simulator in the popular restaurant search task from multiwoz @xcite . in real scenarios, dialogue systems need to face various types of users. a single ad hoc user simulator can only represent one or a group of users, while other users might be under-represented. instead of choosing the best-performing one from many dialogue systems trained by different single user simulators, we believe that it is worth trying to train a dialogue system by leveraging all user simulators simultaneously. in this paper, we propose a framework called must to utilize multiple user simulators simultaneously to obtain a better system agent. there exist several simple ways to implement the must framework, including a merging strategy, a continual reinforcement learning (crl) strategy, and a uniform adaption strategy, namely must merging , must crl , and must uniform respectively (see §3.2). however, none of them could effectively tackle the challenges: 1) how to efficiently leverage multiple user simulators to train the dialogue system since the system might be easily over-fitted to some specific user simulators and simultaneously under-fitted to some others, and 2) it should avoid a catastrophic forgetting issue. to tackle them effectively, we first formulate the problem as a multi-armed bandits (mab) problem @xcite ; similar to the exploitation vs exploration trade-off, specifying multiple user simulators should trade off a boosting adaption (tackling challenge 1) and a uniform adaption (tackling challenge 2), see §4.1 for more details. then we implement a new method called must adaptive to utilize an adaptively-updated distribution among all user simulators to sample them when training the dialogue system in the rl training. our contributions are three-fold: (1) to the best of our knowledge, our proposed must is the first developed work to improve the dialogue system by using multiple user simulators simultaneously; (2) we design several ways to implement the must. especially, we formulate must as a multi-armed bandits (mab) problem, based on which we provide a novel method must adaptive ; and (3) the results show that dialogue systems trained with must consistently outperform those trained with a single user simulator through automatic and human evaluations, showing its potential for robustness to the diversity of user simulators. importantly, it significantly improves the performance of the dialogue system tested on out-of-domain evaluation. moreover, our results show that our method must adaptive can efficiently leverage multiple user simulators to train the dialogue system in terms of convergence speed. user simulator. the user simulator is also an agent but plays a user role. different from dialogue systems, the user agent has a goal describing a target entity (e.g., a restaurant at a specific location) and should express its goal completely in an organized way by interacting with the system agent @xcite . therefore, besides the modules of nlu, dm, and nlg like dialogue systems, the user agent should have another module called goal generator @xcite , which is responsible for generating the user's goal. building a user simulator could usually use an agenda-based approach @xcite designing handcrafted rules to mimic user behaviors or a model-based approach such as neural networks @xcite @xcite learned on a corpus of dialogues. training dialogue systems with a user simulator. to start a dialogue, a user agent will have an initial goal from its goal generator and then expresses its goal in natural languages. however, users' goals are invisible to the system agent. then the system agent tends to gradually understand the users' utterances, query the database to find entities, and provide useful information to accomplish users' task. when the database result returned by the system agent is empty, the user agent should learn to compromise and change its goal with the help of goal generator. when the dialogue ends, the user simulator will reward the system agent according to if it accomplishes the task. then we could use the reward to update the system agent with rl algorithms @xcite . ## must: a framework to leverage one cannot stand for everyone. users might behave differently, one could design different user simulators with specific user dialog act distributions, see @xcite . a single user simulator learned on a task-oriented dialogue corpus can just represent one or a group of users, while the dialogue system needs to accomplish tasks from various human users in real scenarios. we argue that it is beneficial to utilize all different user simulators to train the dialogue system. by leveraging multiple user simulators that have different user dialog act distributions, the dialogue systems can explore a larger dialogue state space, which might 4 the dialogue policy learning module is essential in both dialogue systems and user simulators. a policy module corresponds to a dialog act distribution since it decides to take which dialog act to respond to the current dialogue state. the user dialog act distribution behind a user simulator determines the diversity of the dialogue state space explored by dialogue systems; therefore it might affect the system performances. ## must as a mab problem to tackle the challenges in must, we first formulate must as a multi-armed bandit (mab) problem, see §4.1. in §4.2, we propose a method called must adaptive to use an adaptively-updated distribution to replace the uniform distribution under the must uniform for accelerating the must training. we briefly compare these different implementations of must in tab. 1. ## experiments to verify the effectiveness of must, we benchmark the system agents trained either with a single user simulator or multiple user simulators (including must merging , must uniform , and must adaptive ). see must crl in the app. c. ## conclusion in this paper, we propose a framework named must to improve dialogue systems by using multiple user simulators simultaneously. we discuss several simple methods to implement must, which is either inflexible or inefficient. therefore, we formulate must as a multi-armed bandits (mab) problem, based on which we propose a novel implementation called must adaptive . the experimental results on the restaurant search task from multiwoz demonstrate that must can largely improve the system agent upon baselines, especially when tested with unseen user simulators. moreover, must adaptive is more efficient than other implementations.