Meta-Reinforced Multi-Domain State Generator for Dialogue Systems (2020)

Abstract: A Dialogue State Tracker (DST) is a core component of a modular task-oriented dialogue system. Tremendous progress has been made in recent years. However, the major challenges remain. The state-of-the-art accuracy for DST is below 50% for a multi-domain dialogue task. A learnable DST for any new domain requires a large amount of labeled in-domain data and training from scratch. In this paper, we propose a Meta-Reinforced Multi-Domain State Generator (MERET). Our first contribution is to improve the DST accuracy. We enhance a neural model based DST generator with a reward manager, which is built on policy gradient reinforcement learning (RL) to fine-tune the generator. With this change, we are able to improve the joint accuracy of DST from 48.79% to 50.91% on the MultiWOZ corpus. Second, we explore to train a DST meta-learning model with a few domains as source domains and a new domain as target domain. We apply the model-agnostic meta-learning algorithm (MAML) to DST and the obtained meta-learning model is used for new domain adaptation. Our experimental results show this solution is able to outperform the traditional training approach with extremely less training data in target domain.

## introduction a dialogue state tracker (dst) is a core component of a modular task-oriented dialogue system @xcite . for each dialogue turn, a dst module takes the user utterance and the dialogue history as input, and outputs a belief estimate of the dialogue state. the dialogue state as of today is simplified as a set of requests and goals, both of which are represented as (slot, value) pairs such as (area, centre), (food, chinese) for a user request i'm looking for a chinese restaurant in the centre of the city. a highly accurate dst is crucial to ensure moderate price south. the hotel figure 1 : an example of dialogue state tracking process for booking a hotel, looking for an attraction and booking a taxi between them. each turn contains a user utterance (grey) and a system utterance (blue). the dialogue state tracker (yellow) tracks all the (domain, slot, value) until the current turn. blue color texts indicate mentions of slot values appeared at that turn. best viewed in color. the quality and smoothness of a human-machine dialogue. @xcite recently introduced a multi-domain dialogue dataset multi-domain wizard-of-oz (multiwoz), which is more than one order of magnitude larger than all previous annotated task-oriented corpora with around 10k dialogues and involves more than 7 domains. a domain of a task-oriented system is often defined by an ontology, which defines all entity attributes called slots and all possible values for each slot. multiwoz presents conversation scenarios much similar to those in real industrial applications. figure 1 shows an example of a multi-domain dialogue, where a user starts a conversation about hotel reservation and moves on to look for attractions nearby of his interest. it adds a layer of complexity to the dst and brings new challenges. the first new challenge is how to appropriately model dst for a multi-domain dialogue task. multi-domain dst is in its infancy before multi-woz @xcite . most previous work on dst focus on one given domain @xcite @xcite @xcite . as @xcite pointed out, to process the multiwoz data, the dst model has to determine a triplet (domain, slot, value) instead of a pair (slot, value) at each turn of dialogue. multiwoz contains 30 (domain, slot) pairs over 4,500 possible slot values in total. the prediction space is significantly larger. this change seems quantitative. however, it challenges the foundation of most successful dst models, where dst is casted as a neural model based classification problem, each (slot, value) pair is an independent class and the number of classes is relatively limited. when the number of classes is large enough as the case in multiwoz, classification-based approaches are not applicable. in real industry scenarios, the prediction space is even larger and it is often not possible to have full ontology available in advance @xcite ). it's hard to enumerate all possible values for each slot. the second challenge is how to model the commonality and differences among domains. the number of domains is unlimited in real-life. it won't be able to scale up if each new domain requires a large amount of annotated data. to overcome these challenges, @xcite proposed a transferable dialogue state generator (trade) that generates dialogue states from utterances using a copy mechanism, facilitating knowledge transfer between domains. the prominent difference from previous one-domain dst models is that trade is based on a generation approach instead of a close-set classification approach. the generation model parameters are shared among various domains and slots. trade is able to help boost the dst accuracy up to 48.62% with the multiwoz corpus. it is obvious this accuracy is far from being acceptable. in this paper, we are motivated to enhance this generation-based approach for two objectives, higher accuracy and better domain adaptability. to improve dst accuracy, we propose a new framework which contains the state generator and reward manager. the state generator follows the same setup of trade. the reward manager calculates the reward to fine-tune the generator through policy gradient reinforcement learning (pgrl). we use the reward manager to help the generator alleviate the objective mismatch challenge. objective mismatch is a limitation of encoder-decoder generation approaches, where the training process is set to maximize the log likelihood, but it doesn't assure producing the best results on discrete evaluation metrics such as the dst accuracy. since multiwoz provides data for multiple domains, it enables us to study the long-standing domain adaptability problem. it is a hope we can train a general dst model from multi-domain data and this model can be adapted to a new domain with minimal examples from a new domain. we apply the metalearning algorithm, maml, for this study. our key contributions in this paper are as follows: ## model meret the overview of our model is illustrated in figure 2 . it consists of a generator model and a reward manager. ## maml-adaptive dst the traditional paradigm of supervised learning is to train a model for a specific task with plenty of annotated data. meta-learning aims at learning new tasks with few steps and little data based on existing tasks. maml @xcite is the most popular meta-learning algorithm. it has been successfully employed in various tasks. we propose to apply maml to perform dialogue state tracking for new domains. the maml algorithm tries to build an internal representation of multiple tasks and maximize the sensitivity of the loss function when applied to new tasks, so that small update of parameters could lead to large improvement of new task loss value. in this paper, we explore how it works with dst, a key component in task-oriented dialogue systems. algorithm 1 reinforce algorithm input: dialogue history sequence x, ground-truth output slot value sequences y , a pre-trained model π θ . output: trained model π θ with reinforce algorithm. 1: training steps: 2: initialize π θ with random weights θ; 3: pre-train π θ using cross-entropy loss of generator and classifier on dataset (x, y ); 4: initialize π θ = π θ . 5: while not done do 6: select a batch of size n from x and y ; 7: for each slot do 8: sample {y @xmath0, • • • , y s |y j | )} n 1 from the final probability distribution of vocabulary; 9: compute reward {r(y s 1 ), • • • , r(y s |y j | )} n 1 defined in the reward manager; 10: end for 11: compute l rl and l using eq (6) and eq (7); 12: update the parameters of network with learning rate ρ, θ ← θ + ρ∇ θ l θ ; 13: end while 14: testing steps: 15: for batch of x and y do 16: generate the output y ; 17: end for 18: return the evaluated model π θ ; maml is compatible for any model training based on gradient descent. we can denote the baseline model as m . training a typical gradient descent model m involves (1) providing training data and initializing parameters of m ; (2) computing a given objective loss; (3) applying gradient descent to the loss to update m parameters. with maml, the training steps becomes: (1) initialize m and making nd copies of m to be m d ; (2) select training data from each domain and updating m d parameters based on gradient descent and a loss function; (3) calculate a loss for each domain with their updated temporary model m d ; (4) sum up the new loss from each training domain to be a total loss; (5) update parameters of the original m based on the total loss; (6) repeat above steps until m converges. algorithm 2 shows step-by-step how maml combines with our model meret. suppose we consider nd dialogue domains, we take ntr do-mains as source domains for meta-training and nts domains as target domains for meta-testing. for each source domain, we divide the source domain data into d train d as the support dataset and d valid d as the query dataset, d is the domain index. α, β are two hyper-parameters for maml, α as the learning rate for each domain and β as the learning rate for meta-learning update. there are two cycles. the outer cycle is for metalearning, updating model parameters of m . the inner cycle is for task learning, updating the temporary model m d of each domain d. for task learning, we select k examples from d train d for each domain d, evaluate the gradient of the loss function as eq ( 7) , update the parameters θ d with respect the k examples (step 4). after each domain model is updated once, the m model parameters are updated using the sum of the loss with respect to k examples sampled from each d valid d . specifically, we sum the loss of m d in each domain to obtain the meta loss l m , finally, we minimize the meta loss for updating the current model m until an ideal meta-learned model m is achieved, to adapt to a new domain, we start with the metalearned model m instead of initializing randomly, new-domain training data is used to update model parameters as multiple batches and the learnt task model is fit for the new domain. ## conclusion we introduce an end-to-end generative framework with pre-trained language model and copymechanism, using rl-based generator to encourage higher semantic relevance in greater exploration space for dst. experiments on multidomain dataset show that our proposed model achieves state-of-the-art performance on the dst task, exceeding current best result by over 2%. in addition, we train the dialogue state tracker using multiple single-domain dialogue data with richresource by using the maml. the model is capable of learning a competitive and scalable dst on a new domain with only a few training examples in an efficient manner. empirical results on mul-tiwoz datasets indicate that our solution outperforms non-meta-learning baselines training from scratch, adapting to new few-shot domains with less data and faster convergence rate. in future work, we intend to explore more with the combination of rl and dst on the basis of reward designing, trying to explore more in the internal mechanism. in the long run, we are interested in combing many tasks into one learning process with meta-learning.