Modeling Long Context for Task-Oriented Dialogue State Generation (2020)

Abstract: Based on the recently proposed transferable dialogue state generator (TRADE) that predicts dialogue states from utterance-concatenated dialogue context, we propose a multi-task learning model with a simple yet effective utterance tagging technique and a bidirectional language model as an auxiliary task for task-oriented dialogue state generation. By enabling the model to learn a better representation of the long dialogue context, our approaches attempt to solve the problem that the performance of the baseline significantly drops when the input dialogue context sequence is long. In our experiments, our proposed model achieves a 7.03% relative improvement over the baseline, establishing a new state-of-the-art joint goal accuracy of 52.04% on the MultiWOZ 2.0 dataset.

## introduction dialogue state tracking (dst, also known as belief tracking) predicts user's goals in task-oriented dialogue system, where dialogue states are normally represented in the form of a set of slot-value pairs. a variety of approaches to dialogue state tracking are devoted to dealing with two different settings: dst over a predefined domain ontology and dst with slot-value candidates from an open vocabulary. most of the previous work is based on the first setting, assuming that all possible slot-value candidates are provided in a domain ontology in advance. the task of the dialogue state tracking with this setting is therefore largely simplified to score all predefined slot-value pairs and select the value with the highest score for each slot as the final prediction. although predefined ontology-based approaches are successfully used on datasets with small ontologies, such as dstc2 @xcite and woz2.0 @xcite , they are * corresponding author quite limited in both scalability to scenarios with infinite slot values and prediction of unseen slot values. in order to address these issues of dst over predefined ontologies, recent efforts have been made to predict slot-value pairs in open vocabularies. among them, trade @xcite proposes to encode the entire dialogue context and to predict the value for each slot using a copy-augmented decoder, achieving state-of-the-art results on the multiwoz 2.0 dataset @xcite . as trade simply concatenates all the system and user utterances in previous turns into a single sequence as the dialogue context for slot-value prediction, it is difficult for the model to identify whether an utterance in the dialogue context is from system or user when the concatenated sequence becomes long. we observe that the longest dialogue context after concatenation on the multiwoz 2.0 dataset contains 880 tokens. our experiments also demonstrate that the longer the dialogue context sequence is, the worse trade performs. to deal with this problem, we propose two approaches to modeling long context for better dialogue state tracking. the first method is tagging. while constructing the dialogue context sequence, we insert a tag of [sys] symbol in front of each system utterance, and a tag of [usr] symbol in front of each user utterance. the purpose of adding such symbolic tags in the concatenated dialogue context sequence is to explicitly enhance the capability of the model in distinguishing system and user utterances. in the second method, we propose to integrate a bi-directional language modeling module into the upstream of the model as an auxiliary task to gain better understanding and representation of the dialogue context. the bi-directional language modeling task is to predict the next word by using forward hidden states and the previous word by using backward hidden states based on the dialogue context sequence without any annotation. with these two approaches, we perform dialogue state tracking in a multi-task learning architecture. in summary, the contributions of our work are as follows: ## related work predefined ontology-based dst assumes that all slot-value pairs are provided in an ontology. @xcite propose a neural belief tracker (nbt) to leverage semantic information from word embeddings by using distributional representation learning for dst. an extension to the nbt is then proposed by @xcite , which learns to update belief states automatically. zhong et al. (2018) use slot-specific local modules to learn slot features and propose a global-locally self-attentive dialogue state tracker (glad). nouri and hosseini-asl (2018) propose gce model based on glad by using only one recurrent networks with global conditioning. ramadan et al. ( 2018 ) introduce an approach that fully utilizes semantic similarity between dialogue utterances and the ontology terms. @xcite propose statenet which generates a fixed-length representation of the dialogue context and compares the distances between this representation and the value vectors in the candidate set for making prediction. these predefined ontology-based dst approaches suffer from their weak scalability to large ontologies and cannot deal with previously unseen slot values. in open vocabulary-based dst, @xcite propose a model that learns to predict unknown values by using the index-based pointer network for different slots. @xcite apply an encoder-decoder architecture to generate dialogue states with the copy mechanism. however, their method simply concatenates the whole dialogue context as input and does not perform well when the dialogue context is long. we study this problem and propose methods to help the dst model better model long context. inspired by @xcite who use an additional language model in question generation, we attempt to incorporate language modeling into dialogue state tracking as an auxiliary task. ## our methods in this section, we describe our proposed methods. first, section 3.1 briefly introduces the recent trade model @xcite as background knowledge, followed by our methods: utterance tagging in section 3.2 and multi-task learning with language modeling in section 3.3. ## experiments in this section, we evaluated our proposed methods on the public dataset. ## conclusion in this paper, we have presented the utterance tagging and auxiliary bi-directional language modeling in a multi-task learning framework to model long dialogue context for open vocabulary-based dst. experiments on the multiwoz 2.0 dataset show that our model significantly outperforms the baselines and achieves new state-of-the-art results.