Slot Attention with Value Normalization for Multi-Domain Dialogue State Tracking (2020)

Abstract: Incompleteness of domain ontology and unavailability of some values are two inevitable problems of dialogue state tracking (DST). Existing approaches generally fall into two extremes: choosing models without ontology or embedding ontology in models leading to over-dependence. In this paper, we propose a new architecture to cleverly exploit ontology, which consists of Slot Attention (SA) and Value Normalization (VN), referred to as SAVN. Moreover, we supplement the annotation of supporting span for MultiWOZ 2.1, which is the shortest span in utterances to support the labeled value. SA shares knowledge between slots and utterances and only needs a simple structure to predict the supporting span. VN is designed specifically for the use of ontology, which can convert supporting spans to the values. Empirical results demonstrate that SAVN achieves the state-of-the-art joint accuracy of 54.52% on MultiWOZ 2.0 and 54.86% on MultiWOZ 2.1. Besides, we evaluate VN with incomplete ontology. The results show that even if only 30% ontology is used, VN can also contribute to our model.

## introduction dialogue state tracking (dst) is a core component in the pipeline-based task-oriented dialog systems. the goal of dst is to extract the dialogue states which are indicated by a set of (domain, slot, value) triples during conversation. the (domain, slot, value) triple represents that previous conversation involves the slot of the domain and the specific content is the value. for example, as shown in figure 1 , (restaurant, price, expensive) triple means that user wants to reserve an expensive restaurant. a highquality dst model plays a significant role in the * corresponding author. traditional dst approaches generally rely on ontology already defined, where all slots and their possible values are given. with a predefined ontology, dst is simplified to a classification problem. the goal is to choose the most appropriate value from ontology for the slot @xcite . however, in practical applications, a complete ontology is almost impossible to be defined in advance. to overcome the drawback, span-based @xcite and generation @xcite approaches spring up. the second problem is that some values required by dst cannot be found in utterances due to the diverse descriptions during a conversation. as shown in figure 1 , value expensive was expressed as high end in the first turn. the problem gives rise to the powerlessness of span-based approaches. @xcite show a dual strategy that combines the advantages of both the picklist-based and span-based methods. they use ontology in spanbased approaches to deal with the problem and achieve the sota performance, which also shows that the ontology is powerful. @xcite introduced a largescale multi-turn dialogue dataset (multiwoz) spanning over several domains and topics. as shown in figure 1 , the user initially wants to make a restaurant reservation, then requests information about attractions close to the restaurant, and finally books a taxi. during the conversation, the models for dst should determine whether each (domain, slot) pair has a value in each turn to obtain the most relevant (domain, slot, value) triples. however, unlike single domain dst problems, in which only a few slots need to be tracked, such as four slots in woz @xcite , there are a total of 30 (domain, slot) pairs of five domains in multiwoz, which user system dialogue state i would like to find a high end restaurant in the center of the city. (restaurant, price, expensive) (restaurant, area, centre) there is an expensive restaurant called the ugly duckling. can i get a reservation for 7 at 14:00 this coming friday? (restaurant, price, expensive), … (restaurant, name, the ugly ducking), (restaurant, people, 7), (restaurant, time, 14:00), (restaurant, day, friday) i have successfully booked your reservation. your reference number is ynceb914. will this be all? i am also looking for some entertainment close to the restaurant. to tackle these challenges, we emphasize that dst models should optimize the structure of slots determination and utilize ontology more flexibly rather than abandon it. in this paper, we propose to divide the model of dst into slot attention (sa) and value normalization (vn). simple and efficient processing of slots and flexible use of ontology are the main advantages of savn. contributions in this work are summarized as † : ## savn model the overview of the model is shown in figure 2 , which consists of slot attention and value normalization. sa outputs the supporting span (sp) from utterances for each (domain, slot) pair. and vn chooses to output supporting span directly or convert supporting span to the value in ontology according to the gate. ## annotation for supporting span our annotation work is based on the multiwoz 2.1 dataset @xcite , which is a fixed version of the multiwoz 2.0 dataset @xcite . multiwoz 2.1 dataset is a large-scale collection of human-human written conversations over multiple domains and topics, which has labeled 63,662 (conversation, domain, slot, value) quadruples (except "none" value) in the training set. annotation for supporting span is mainly to address the problem that some labeled value can not be found in the conversations. the causes of this problem can be divided into three categories: varied expressions, spelling mistakes, and annotation errors. the criterion of annotations is to find the shortest span in the conversations, which can help us get the labeled value. based on the criterion, we annotate 936 (supporting span, value) pairs on multiwoz 2.1 training set, in which varied expressions account for 637 (68%), spelling mistakes account for 123 (13%), and annotation errors account for 176 @xcite . after annotation, we can change (domain, slot, value) triples in training set to (domain, slot, supporting span, value) quadruples, where the supporting span will be equal to the value if the value can be found in the conversations. then we employ (domain, slot, supporting span) triples to train sa and (supporting span, value) pairs to train vn. specifically, we do not use the annotation of annotation errors to train vn, for it should not convert saturday to sunday. ## experiments we evaluate our model on two publicly available datasets: multiwoz 2.0 and multiwoz 2.1, both of which are fully-labeled task-oriented corpora comprised of human-human written conversations and contain 8,438 multi-turn dialogues with each dialogue having 13.68 turns on average in training set @xcite . the difference between multiwoz 2.0 and multiwoz 2.1 is that multiwoz 2.1 has changed more than 32% of state annotations across 40% of the dialogue turns to fix the noisy state annotations in multiwoz 2.0 @xcite . following previous work @xcite , only five domains (i.e., restaurant, hotel, attraction, taxi, and train) are employed in our experiments because the dialogues that belong to the other two domains (i.e., hospital and police) are rare in the training set and do not appear in the test set. as introduced in section 3, we get a new training set by using the supporting span annotations, which can be called the sp training set. additionally, there are no changes to the test set and the dev set. ## error analysis an error analysis of sa sp with vn 1 on multiwoz 2.1 is shown in figure 5 . the three slots with the highest error rates are restaurant-name with 6.37%, hotel-type with 6.08% and attraction-name with 5.93%. through the detailed analysis of error samples, we observe many labeled states do not include the name that only appears in system response. these states are similar to the example in figure 1 with the restaurant-name and attractionname removed. once the difference occurs, it will lead to errors in the subsequent dialogue states, resulting in high error rates. and the labels of hotel-type are found to be confusing. for instance, for the sentence "i am looking for a hotel with ...", sometimes the label of hotel-type is hotel and sometimes it is none. compared with sa, savn has significantly lower error rates on attraction-type and attractionname. the improvement of attraction-name is mainly due to the repair of spelling mistakes, and the improvement of attraction-type mainly benefits from the normalization of varied expressions. it is worth mentioning that vn can not improve the accuracy of some slots, which only need to be filled with yes or no, such as hotel-internet and hotel-parking. ## related work traditional dialogue state tracking models extract utterance semantics by hand-crafted features and complex domain-specific lexicons @xcite @xcite to predict the dialogue states, which is hard to adapt to new domains. then, to overcome this drawback, @xcite propose a novel neural belief tracking (nbt) framework with learning n-gram representation of utterance by using a convolutional neural network, and achieve better performance. at the same time, models for multi-domain dst have then been proposed. rastogi et al. (2017) after the pre-trained bert model showed superior performance, encoding by bert has become the mainstream. @xcite encode the slots and utterances with bert, and then compute the similarity between possible values and utterances after a multi-head attention layer. and zhang et al. (2019) also employ bert to encode the utterances. the difference is that they combine the picklistbased and span-based methods and get higher performance. in order to eliminate the dependence on ontology, @xcite propose an encoderdecoder architecture with a pointer network to generate the value for each slot. and zhou and small (2019) formulate multi-domain dst as a question answering problem and learn relationships between slots by a dynamically-evolving knowledge graph. most recently, @xcite propose to use copy mechanisms to fill slots with values, which combine span-based methods with memory methods to avoid the use of value picklists. ## conclusion we introduce a new architecture that divides the prediction of slots and the use of ontology. sa shares parameters not only among all slots but also between slots and utterances. and vn can handle ontology flexibly with a simple and effective structure, which is able to work with incomplete ontology. combining sa with vn, savn has shown excellent performance on both multiwoz 2.0 and multiwoz 2.1. and we also introduce the annotation of supporting span. in future work, the supporting span annotation can be added to the datasets of a task-oriented dialog system, for the reason that supporting span serves as a bridge between diverse descriptions of users and the normative values in the system. furthermore, dst models with supporting span allow for a fairer comparison regardless of whether the ontology is used. for computational linguistics, pages 808-819, florence, italy. association for computational linguistics. puyang xu and qi hu. 2018. an end-to-end approach for handling unknown slot values in dialogue state tracking. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1448-1457, melbourne, australia. association for computational linguistics. jian-guo zhang, kazuma hashimoto, chien-sheng wu, yao wan, philip s. yu, richard socher, and caiming xiong. 2019. find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. victor zhong, caiming xiong, and richard socher. 2018. global-locally self-attentive encoder for dialogue state tracking. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1458-1467, melbourne, australia. association for computational linguistics. li zhou and kevin small. 2019. multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering.