Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems (2021)

Abstract: Dialogue policy optimisation via reinforcement learning requires a large number of training interactions, which makes learning with real users time consuming and expensive. Many set-ups therefore rely on a user simulator instead of humans. These user simulators have their own problems. While hand-coded, rule-based user simulators have been shown to be sufficient in small, simple domains, for complex domains the number of rules quickly becomes intractable. State-of-the-art data-driven user simulators, on the other hand, are still domain-dependent. This means that adaptation to each new domain requires redesigning and retraining. In this work, we propose a domain-independent transformer-based user simulator (TUS). The structure of TUS is not tied to a specific domain, enabling domain generalization and the learning of cross-domain user behaviour from data. We compare TUS with the state-of-the-art using automatic as well as human evaluations. TUS can compete with rule-based user simulators on pre-defined domains and is able to generalize to unseen domains in a zero-shot fashion.

## introduction task-oriented dialogue systems are designed to help users accomplish specific goals within a particular task such as hotel booking or finding a flight. solving this problem typically requires tracking and planning @xcite . in tracking, the system keeps track of information about the user goal from the beginning of the dialogue until the current dialogue turn. in planning, the dialogue policy makes decisions at each turn to maximise future rewards at the end of the dialogue @xcite . the system typically needs thousands of interactions to train a usable policy @xcite @xcite . the amount of interactions required makes learning from real users time-consuming and costly. it is therefore appealing to automatically generate a large number of dialogues with a user simulator (us) 1 @xcite . rule-based uss are interpretable and have shown success when applied in small, simple domains. however, expert knowledge is required to design their rules and the number of rules needed for complex domains quickly becomes intractable @xcite . in addition, handcrafted rules are unable to capture human behaviour to its fullest extent, leading to suboptimal performance when interacting with real users @xcite . data-driven uss on the other hand can learn user behaviour directly from a corpus. however, they are still domain-dependent. this means that in order to accommodate an unseen domain one needs to collect and annotate a new dataset, and retrain or even re-engineer the simulator. we propose a transformer-based domainindependent user simulator @xcite . unlike existing data-driven simulators, we design the feature representation to be domain-independent, allowing the simulator to easily generalise to new domains without modifying or retraining the model. we utilise a transformer architecture @xcite so that the input sequence can have a variable length and dynamic order. the dynamic order takes into account the user's priorities and the varying input length enables the us to incorporate system actions in a seamless manner. tus predicts the value of each slot and the domains of the current turn, allowing the model to optimise its performance in multiple granularities. by disentangling the user behaviour from the domains, tus can learn a more general user policy to train the dialogue policy. we compare policies trained with our tus to policies trained with other uss through indirect and direct evaluation as well as human evaluation. the results show that policies trained with tus outperform those that are trained with another data-driven us and are on par with policies trained with the agenda-based us (abus). moreover, the policy generalises better when evaluated with a different us. automatic and human evaluations on our zeroshot study show that leave-one-domain-out tus is able to generalise to unseen domains while maintaining a comparable performance to abus and tus trained on the full training data. ## related work the quality of a us has a significant impact on the performance of a reinforcement-learning based task-oriented dialogue system @xcite . one of the early models include an n-gram user simulator proposed by @xcite . it uses a 2-gram model p (a u |a m ) to predict the user action a u according to the system action a m . since it only has access to the latest system action, its behaviour can be illogical if the goal changes. therefore, models which can take into account a given user goal were introduced @xcite . the bayesian model of @xcite predicts the user action based on the user goal, and hidden markov models are used to model the user and the system behaviour @xcite . the graph-based us of @xcite combines all possible dialogue paths in a graph. it can generate reasonable and consistent behaviour, but is impractical to implement, since extensive domain knowledge is required. the agenda-based user simulator (abus) @xcite ) models the user state as a stack-like agenda, ordered according to the priority of the user actions. the probabilities of updating the agenda and choosing user actions are set manually or learned from data @xcite . still, the stacking and popping rules are domain-dependent and need to be designed carefully. to build a data-driven model, the sequence-tosequence (seq2seq) model structure is widely used. el @xcite propose a seq2seq semantic level us with an encoder-decoder structure. each turn is fed into the encoder recurrent neural network (rnn) and embedded as a context vector. then this context vector is passed to the decoder rnn to generate user actions. to add new domains, it is necessary to modify the domain-dependent feature representation and retrain the model. instead of generating semantic level output, the neural user simulator (nus) by @xcite generates responses in natural language, thus requiring less labeling, at the expense of interpretability. however, its feature representation is still domain-dependent. a variational hierarchical seq2seq user simulator (vhus) is proposed by @xcite . instead of designing dialogue history features, the model encodes the user goal and system actions with a vector using an rnn, which alleviates the need of heavy feature engineering. however, the inputs are represented as one-hot encodings, which are also dependent on the ontology. in addition, the output generator is not constrained by the ontology in any way, so it can generate impossible actions. as shown in fig. 1 , abus and graph-based models are domain-dependent and require significant design efforts. data-driven models such as seq2seq, nus, and vhus can learn from data, but are constrained by the underlying domain. nus generates natural language responses, which requires less labeling, but comes with reduced interpretability. @xcite compared different ways to build a us and indicated that the data-driven models suffer from bias in the corpus. if some actions are rare in the corpus, the model cannot capture them. thus, the dialogue policy cannot explore all possible paths during training with the data-driven uss. it is important to learn more general human behaviour to reduce the impact of the corpus bias. ## problem description task-oriented dialogue systems are defined by a given ontology, which specifies the concepts that the system can handle. the ontology can include multiple domains. in each domain, there are informable slots, which are the attributes that users can assign values to, and requestable slots, which are the attributes that users can query. for example, in fig. 2 the user goal has two domains, "hotel" and "restaurant". the slot area is an informable slot with the value north in domain "hotel" and addr is a requestable slot in domain "restaurant". the system state records the slots and values mentioned in the dialogue history. a us for task-oriented dialogue systems needs to provide coherent responses according to a given user goal @xmath0: [(slot 1 , value 1 ), (slot 2 , value 2 ), . . . ], . . . }. the domains, slots and values are selected from the ontology. the user action is composed of user intents, domains, slots, and values. we consider user intents that appear in the multiwoz dataset @xcite . it is of course possible to consider arbitrary intents within the same model architecture, as long as they are defined a priori foot_2 . the two possible user intents we consider are inform and request. with inform, the user can provide information, correct the system or confirm the system's recommendations. when a user goal cannot be fulfilled, the user can also randomly select a value from the ontology and change the goal. with request, the user can request information about certain slots. the system action is similar to the user action, but there exist more (system) intents. for example, the system can provide suggestions to users with the intent recommendation and make reservations for users with the intent book. more system intents can be found in appendix a. we view user simulation in a task-oriented dialogue as a sequence-to-sequence problem. for each turn t, we extract the input feature vectors v t of the input list of slots s @xmath1, s 2 , . . . ], which is composed of the slots from the user goal and the system action. the output sequence o @xmath2, o t 2 , . . . ] is then generated by the model, where o t i shows how the value for slot s i is obtained. the input feature representation and the output target should be user goal info: hotel-@xmath3, rest-@xmath4: hotel-name, rest-addr conversation turn 0 usr: i want to find a hotel in the north and a nearby restaurant. inform(hotel-@xmath5, rest-@xmath6: there are some good hotels in the south. which price range do you prefer? would you mind providing more information? recom(hotel-@xmath7, request(hotel-price), general-reqmore() turn 1 usr: no, i want one in the north and i don't care about the price range. inform(hotel-@xmath8, hotel-@xmath9: an example dialogue with a multi-domain goal. ## transformer-based domain-independent user simulator the tus model structure is shown in fig. 3 . for each turn t, the list of input feature vectors the user policy network is a transformer @xcite . we choose this structure because transformers are able to handle input sequences of arbitrary lengths and to capture the relationship between slots thanks ## system action features the system action feature vector v system action encodes system actions in each turn. there are two kinds of system actions, general actions and domainspecific actions. the general actions are composed only with general intents, such as "reqmore" and "bye". for example, general-reqmore(). the feature vector of general actions v gen is a multihot encoding of whether or not a general intent appears in the dialogue. with a total number of n gen general intents, for each @xmath10, . . . , n gen }, the k-th entry of v gen is set to 1 if the k-th general intent is part of the system act. on the other hand, domain-specific actions are composed with domains, slots, values, and domainspecific intents such as "recommend" and "select". for example, recom(hotel-@xmath11. each domain-specific action vector v spec j with the domain-specific j-th intent, @xmath12, . . . , n spec }, where n spec is the total number of domain-specific intents, is represented by a 3-dimensional onehot encoding that describes whether the value is "none", "?" or "other values". the final action representation v system action is formed by concatenating n spec domain-specific action representations together with the general action representation, i.e., ## conclusion we propose a domain-independent user simulator with transformers, tus. we design ontologyindependent input and output feature representations. tus outperforms the data-driven vhus and it has a comparable performance to the rule-based abus in cross-model evaluation. human evaluation confirms that tus can compete with abus even though abus is based on carefully designed domain-dependent rules. our ablation study shows that the proposed features and loss functions are essential to model natural user behavior from data. lastly, our zero-shot study shows that tus can handle new domains without feature modification or model retraining, even with substantially fewer training samples. in future work, we would like to learn the order of slots and add output language generation to make the behaviour of tus more human-like. applying reinforcement learning to this model would also be of interest. proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 100-110, hong kong, china. association for computational linguistics. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, ł ukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems, volume 30. curran associates, inc. steve young. 2002. talking to machines (statistically speaking). in seventh international conference on spoken language processing. qi zhu, zheng zhang, yan fang, xiang li, ryuichi takanobu, jinchao li, baolin peng, jianfeng gao, xiaoyan zhu, and minlie huang. 2020. convlab-2: an open-source toolkit for building, evaluating, and diagnosing dialogue systems. in proceedings of the 58th annual meeting of the association for computational linguistics.