Transferable Dialogue Systems and User Simulators (2021)

Abstract: One of the difficulties in training dialogue systems is the lack of training data. We explore the possibility of creating dialogue data through the interaction between a dialogue system and a user simulator. Our goal is to develop a modelling framework that can incorporate new dialogue scenarios through self-play between the two agents. In this framework, we first pre-train the two agents on a collection of source domain dialogues, which equips the agents to converse with each other via natural language. With further fine-tuning on a small amount of target domain data, the agents continue to interact with the aim of improving their behaviors using reinforcement learning with structured reward functions. In experiments on the MultiWOZ dataset, two practical transfer learning problems are investigated: 1) domain adaptation and 2) single-to-multiple domain transfer. We demonstrate that the proposed framework is highly effective in bootstrapping the performance of the two agents in transfer learning. We also show that our method leads to improvements in dialogue system performance on complete datasets.

## introduction this work aims to develop a modelling framework in which dialogue systems (dss) converse with user simulators (uss) about complex topics using natural language. although the idea of joint learning of two such agents has been proposed before, this paper is the first to successfully train both agents on complex multi-domain human-human dialogues and to demonstrate a capacity for transfer learning to low-resource scenarios without requiring re-redesign or re-training of the models. one of the challenges in task-oriented dialogue modelling is to obtain adequate and relevant training data. a practical approach in moving to a new domain is via transfer learning, where pre-training on a general domain with rich data is first performed and then fine-tuning the model on the target domain. end-to-end ds @xcite @xcite are particularly suitable for transfer learning, in that such models are optimised as a single system. by comparison, pipe-lined based dss with multiple individual components @xcite require fine-tuning of each component system. these separate steps can be done independently, but it becomes difficult to ensure optimality of the overall system. a similar problem arises in the data-driven us as commonly used in interaction with the ds. though many uss have been proposed and been widely studied, they usually operate at the level of semantic representation @xcite . these models can capture user intent, but are otherwise somewhat artificial as user simulators in that they do not consume and produce natural language. as discussed above for dss, the end-to-end architecture for the us also offers simplicity in transfer learning across domains. there are also potential advantages to continued joint training of the ds and the us. if a user model is less than perfectly optimised after supervised learning over a fixed training corpus, further learning through interaction between the two agents offers the us the opportunity to refine its behavior. prior work has shown benefits from this approach to dialogue policy learning, with a higher success rate at dialogue level @xcite @xcite , but there has not been previous work that addresses multi-domain end-to-end dialogue modelling for both agents. @xcite address refinement of the dialogue policy alone at the semantic level, but do not address end-to-end system architectures. @xcite ; @xcite address single-domain dialogues @xcite , but not the more realistic and complex multi-domain dialogues. this paper proposes a novel learning framework for developing dialogue systems that performs joint optimisation with a user simulator (joust). 1 through the pre-training on complex multi-domain datasets, two agents are able to interact using natural language, and further create more diverse and rich dialogues. using reinforcement learning (rl) to optimise both agents enables them to depart from known strategies learned from a fixed limited corpus, to explore new, potentially better policies. importantly, the end-to-end designs in the framework makes it easier for transfer learning of two agents from one domain to another. we also investigate and compare two reward designs within this framework: 1) the common choice of task success at dialogue level; 2) a fine-grained reward that operates at turn level. results on mul-tiwoz dataset @xcite show that our method is effective in boosting the performance of the ds in complicated multi-domain conversation. to further test our method in more realistic scenarios, we design specific experiments on two low-resource setups that address different aspects of data sparsity. our contributions can be summarised as follows: ## pre-training the dialogue system and user simulator in our joint learning framework, we first pre-train the ds and us using supervised learning so that two models are able to interact via natural language. this section presents the architectures of two agents, illustrated in fig. 1 , and the objectives used for supervised learning. ## rl optimisation of the dialogue system and user simulator after the ds and us models are pre-trained from the corpus using supervised learning, they are finetuned using reinforcement learning (rl) based on the dialogues generated during their interactions. two reward designs are presented after which the optimisation strategy is given. ## related work in the emerging field of end-to-end dss, in which all components of a system are trained jointly @xcite @xcite . rl methods have been used effectively to optimize end-to-end dss in @xcite @xcite , although using rule-based uss or a fixed corpus for interaction. recent works utilise powerful transformers such as gpt-2 @xcite or t5 @xcite for dialogue modeling and reach stateof-the-art performance; however, the area of having a user simulator involved during training is unexplored. by comparison, this work uses a learned us as the environment for rl. the two agents we propose are able to generate abundant high-quality dialog examples and they can be extended easily to unseen domains. by utilizing an interactive envi-ronment instead of a fixed corpus, more dialogue strategies are explored and more dialogue states are visited. there have been various approaches to building uss. in the research literature of uss, one line of research is rule-based simulation such as the agenda-based user simulator (abus) @xcite . the abus's structure is such that it has to be re-designed for different tasks, which presents challenges in shifting to new scenarios. another line of work is datadriven modelling. el @xcite modelled user simulation as a seq2seq task, where the output is a sequence of user dialogue acts the level of semantics. @xcite proposed a variational hierarchical seq2seq framework to introduce more diversity in generating the user dialogue act. @xcite introduced the neural user simulator (nus), a seq2seq model that learns the user behaviour entirely from a corpus, generates natural language instead of dialogue acts and possesses an explicit goal representation. the nus outperformed the abus on several metrics. @xcite also compared the nus and abus to a combination of the abus with an nlg component. however, none of these prior works are suitable for modelling complex, multi-domain dialogues in an end-to-end fashion. by contrast, the user model proposed here consumes and generates text and so can be directly employed to interact with the ds, communicating via natural language. the literature on joint optimization of the ds and the us is line of research most relevant to our work. @xcite proposed a hybrid value network using marl @xcite with roleaware reward decomposition used in optimising the dialogue manager. however, their model requires separate nlu/nlg models to interact via natural language, which hinders its application in the transfer learning to new domains. @xcite lane (2017b); papangelis et al. (2019 ) learn both the ds and the us in a (partially) end-to-end manner. however, their systems are designed for the single-domain dataset (dstc2) and cannot handle the complexity of multi-domain dialogues: 1) their models can only predict one dialogue act per turn, which is not sophisticated enough for modelling multiple concurrent dialogue acts; 2) the simple dst components cannot achieve satisfactory performance in the multi-domain setup; 3) the user goal change is not modelled along the dialogue proceeds, which we found in our experiments very important for learning complex behaviors of user simulators. relative to these three publications, this paper focuses on joint training of two fully end-to-end agents that are able to participate in complex multi-domain dialogues. more importantly, it is shown that the proposed framework is highly effective for transfer learning, which is a novel contribution relative to previous work. ## conclusion and future work we propose a novel joint learning framework of training both the ds and the us for complex multidomain dialogues. under the low-resource scenarios, the two agents can generate more dialogue data through interacting with each other and their behaviors can be significantly improved using rl through this self-play strategy. two types of reward are investigated and the turn-level reward benefits more due to its fine-grained structure. experiments shows that our framework outperforms previously published results on the multiwoz dataset. in two transfer learning setups, our method can further improves the well-performed ewc models and bootstraps the final performance largely. future work will focus on improving the two agents' underlying capability with the powerful transformer-based models.