G en TUS : Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers (2022)

Abstract: User simulators (USs) are commonly used to train task-oriented dialogue systems via reinforcement learning. The interactions often take place on semantic level for efficiency, but there is still a gap from semantic actions to natural language, which causes a mismatch between training and deployment environment. Incorporating a natural language generation (NLG) module with USs during training can partly deal with this problem. However, since the policy and NLG of USs are optimised separately, these simulated user utterances may not be natural enough in a given context. In this work, we propose a generative transformer-based user simulator (GenTUS). GenTUS consists of an encoder-decoder structure, which means it can optimise both the user policy and natural language generation jointly. GenTUS generates both semantic actions and natural language utterances, preserving interpretability and enhancing language variation. In addition, by representing the inputs and outputs as word sequences and by using a large pre-trained language model we can achieve generalisability in feature representation. We evaluate GenTUS with automatic metrics and human evaluation. Our results show that GenTUS generates more natural language and is able to transfer to an unseen ontology in a zero-shot fashion. In addition, its behaviour can be further shaped with reinforcement learning opening the door to training specialised user simulators.

## introduction task-oriented dialogue systems (dss) assist their users in accomplishing a goal, such as booking a flight ticket or making a payment. this should be done through natural language interactions between the system and the user, whilst the system interacts with various external databases and api calls in the background. the core component of such a ds is the dialogue policy module, which decides what should be said to the user next. this module can be trained via interaction with users, through reinforcement learning (rl). however, this creates a conflict between the high cost of interacting with real users and the large amount of interactions required for rl. as a result, user simulators (uss) are often utilised instead to train dialogue policies, as they make it possible for the system to learn from a large number of interactions in a controlled environment at a fraction of the cost. rule-based uss are widely used both in research and industry because they are interpretable and can be built without a labelled dataset. however, designing the rules demands expert knowledge and creating these rules becomes intractable on complex domains, making them only suitable for small and simple domains. in addition, human behaviour is too complex and diverse to be manually described by rules, leading to sub-optimal performance of dss in deployment scenarios @xcite . on the other hand, data-driven uss can be built with less expert involvement. however, these models are either ontology-dependent @xcite @xcite , which means adapting to a new domain requires reengineering the feature representation or re-training the model, or they do not model the language of the user @xcite . both shortcomings are serious. the user simulator needs to support zeroshot transfer across ontologies, as it is difficult to collect enough labelled data for each new domain. the ability to produce natural language output is also critical as it makes the training and testing environment more challenging and similar to the real user scenario. therefore, models that can attain both properties are much needed. in this work, we propose a model that has both desired properties. more specifically, our contributions are as follows: the rest of the paper is organised as follows. in section 2, we review the related work. section 3 describes in detail the proposed simulation framework. in section 4, we present the experimental set-up, followed by the experimental results in section 5. we conclude with section 6. ## related work the performance of a task-oriented dialogue policy trained by rl is significantly affected by the quality of the us used to generate the interactions @xcite . an n-gram user simulator proposed by @xcite is one of the earliest data-driven models. this model predicts the user action a u according to the system action a m based on a bi-gram model p (a u |a m ). its behaviour is often unreasonable since it only takes the latest system action as input without any information about the user goal. therefore, models which can act on a given user goal were introduced @xcite . a bayesian user simulation model which predicts the user action based on the user goal is proposed by @xcite . in @xcite , the user and the system behaviour are modelled by hidden markov models. a graph-based us, which constructs a graph from all possible dialogue paths, is proposed by @xcite . this simulator can act reasonably and consistently, but it is not practical to implement in a complex scenario, as it requires extensive domain knowledge. the agenda-based user simulator (abus) @xcite is widely used to train tourist-information dss. its behaviour is based on hand-crafted stacking and popping of rules with a stack-like agenda user goal, ordered by the priority of the user actions. it is difficult to transfer this model to a new ontology because the rules need to be redesigned. moreover, it only provides semantic-level dialogue acts. to reduce the involvement of experts, further data-driven user simulator approaches have been proposed. the sequence-to-sequence (seq2seq) model structure is the most common framework. a semantic level seq2seq user simulator with an encoder-decoder structure is proposed by @xcite . this model embeds the dialogue history into a context vector via a recurrent neural network (rnn) encoder. its decoder then generates user actions based on the context embedding vector. instead of generating dialogue acts, the neural user simulator (nus) of @xcite can generate responses in natural language. however, this model has limited interpretability because it does not provide semantic-level outputs and its input representation is domain-dependent. the variational hierarchical seq2seq user simulator (vhus) proposed by @xcite encodes the system actions and the user goal by rnns instead of complex dialogue history features and generates semantic user actions. its features are still domain-dependent as system actions and user goals are represented by domain-dependent onehot encodings. as vhus has no constraints in the decoding process, it often generates impossible actions under the given ontology. a domain-independent transformer-based user simulator (tus) is proposed by @xcite . with domain-independent input and output feature representations, tus can adapt to an unseen domain in a zero-shot fashion. however, it does not model natural language output. moreover, all intents are part of the model, which makes transfer to an unrelated ontology, i.e. the one with a different sets of intents, difficult. to convert the dialogue acts from the semantic level to natural language, a user simulator commonly includes an nlg module connected to the semantic level user policy. although templatebased nlgs are widely used in research, creating templates for every dialogue act is labour-intensive and lacks language variation. data-driven nlg models, such as sc-lstm @xcite and sc-gpt @xcite can generate natural language utterances conditioned on given semantic actions. however, taking only semantic actions as input, their results may not be sufficiently natural in a given context. in addition, the user policy and nlg model cannot be optimised jointly within the modular architecture. an end-to-end us which generates both dialogue acts and utterances is proposed by @xcite , although in their evaluation they train a ds using only the semantic actions from the us. the nlg of this us is based on a simple delexicalised lstm model. the user goal is represented as a binary vector, with each dimension representing a domain-slot pair in the ontology. this creates several obstacles for transfer to an unseen ontology: such a transfer would require further hand-coded lexicalisation rules for the nlg component, modifications of the feature representations and further fine-tuning of the us policy. ## generative transformer-based user simulation task-oriented dss are expected to handle the requests of real users in natural language. therefore, when designing uss, it is important to endow them with the ability to converse with the system via natural language as well. in this way, we can study, for example, the robustness of the systems towards misunderstandings that may occur when conversing with real users. on the other hand, users rarely misunderstand the ds response. it is hence reasonable to assume that the input to the us may be on the semantic level. this is also practical in such cases as when dss need to execute api calls, such as playing a song or turning off the light. task-oriented dss are built upon an ontology which includes all possible intents that the user or the system can exhibit in their actions and domains, which describes the entities the user or the system can talk about. domains are further characterised by a number of slots and each slot can take a number of values. in task-oriented ds we assume that the user has a particular goal they want to achieve. we define goal as the following set @xmath0: [(slot 1 , value 1 ), (slot 2 , value 2 ), . . . ], domain 2 : [(slot 3 , value 3 ), . . . ], . . . }, where domains, slots and values are selected from the ontology. the semantic user action and system action are composed of several tuples of the following structure: (intent, domain, slot, value). users and systems may have different intents, e.g., systems can recommend an option and users can negate the recommended offer. a semantic action can be converted into a natural language utterance, which we denote with text usr in the case of a user action. user simulation in a task-oriented dialogue can be modelled as a sequence-to-sequence problem. for each turn, gentus takes the context information as an input sequence, including the system action, the user history, the user goal, and turn information, and generates the semantic action and the natural language response as the output sequence. in following sections, we provide more details. ## experimental setup the objective of our experiments is four-fold. first, we want to show that when trained and tested on the same ontology, the user simulator can adequately capture the semantics represented in the real user data. at the same time, we also want to examine its zero-shot capability by conducting the evaluation on another unseen ontology. second, as natural language output is an important component of the proposed model, we evaluate it separately using both automatic measures and a human preference test. third, we jointly evaluate the gentus dia3. ## experimental results our experimental results can be divided into five parts. in section 5.1, we analysis the impact of different features with an ablation study. in section 5.2, we conduct the direct evaluation by measuring automatic metrics (ser, sacre-bleu, and self-bleu) and human ratings (informativeness and naturalness) from the preference test. in section 5.3, we focus on the generalisability of gen-tus by a zero-shot ontology transfer experiment, measured by semantic level and language level metrics on two different corpora. the indirect evaluation is in section 5.4. we compare dss trained by different uss with cross-model evaluation. the result from the interactive human trial is discussed in section 5.5. in section 5.6, we show that it is possible to further configure the behaviour of gentus via rl. ## conclusion we propose a generative transformer-based user simulator (gentus), which achieves high interpretability and linguistic variation by generating both semantic actions and natural language utterances. moreover, it produces generalisable feature representation by treating the inputs and outputs as word sequences and leveraging a large pre-trained language model. our results show that gentus generates more natural language than sc-gpt in a given context and it can transfer to an unseen ontology in a zero-shot fashion. we consolidate our findings by a number of automatic as well as human evaluations. in addition, the gentus behaviour can be further configured by rl with different reward functions, providing an opportunity to build specialised uss. in future work, we hope to modify also the nlg of gentus via rl in order to model user sentiment or personality. a intents and domains in multiwoz and sgd type system user general welcome, reqmore, bye, thank, greet bye, thank, greet domainspecific recommend, inform, request, select, book, nobook, offerbook, offerbooked, nooffer inform, request all intents in the multiwoz dataset are listed in