Multi-Domain Dialogue State Tracking with Disentangled Domain-Slot Attention (2023)

Abstract: As the core of task-oriented dialogue systems, dialogue state tracking (DST) is designed to track the dialogue state through the conversation between users and systems. Multi-domain DST has been an important challenge in which the dialogue states across multiple domains need to consider. In recent mainstream approaches, each domain and slot are aggregated and regarded as a single query feeding into attention with the dialogue history to obtain domain-slot specific representations. In this work, we propose disentangled domain-slot attention for multi-domain dialogue state tracking. The proposed approach disentangles the domain-slot specific information extraction in a flexible and context-dependent manner by separating the query about domains and slots in the attention component. Through a series of experiments on MultiWOZ 2.0 and MultiWOZ 2.4 datasets, we demonstrate that our proposed approach outperforms the standard multi-head attention with aggregated domain-slot query.

## introduction task-oriented dialogue system is designed to assist users to accomplish sorts of certain tasks. for example, by using dialogue-based automated customer service, users can online query information and make reservations. multi-domain dialogue state tracking has been an important challenge introduced by @xcite , in which numerous mixed-domain conversations are involved. in this case, dst has to track the dialogue states at each turn through the conversation, which contains a huge space involving the combinations of the ontology of different domains, slots, and values. it is a challenging task since spoken language is not formal, in which ellipsis and cross-reference are barrier to handling the correlations among different domains and slots. several studies have explored sorts of approaches to handle the correlations among domains and slots. in recent mainstream approaches, each domain and slot are aggregated into a single vector regarded as a query. the query and the dialogue history are fed into attention to generate domain-slot specific representations @xcite . then the information interchange across different domains and slots are performed with them to model the correlation among different domain and slots @xcite @xcite . however, these approaches introduce too much human prior knowledge and they only consider the correlations among domains and slots names or overestimate these correlations @xcite . to tackle this problem, we propose a disentangled domain-slot attention (ddsa), which disentangles information extraction about domains and slots in a flexible and context-dependent manner. in detail, we disentangle the query about domains and slots in the domain-slot attention component. firstly, domain specific representations are obtained using the domain query and the dialogue history. then the model utilizes these representations and slot query to retrieve slot specific information (in this context, slot means the slot only) and generate domain-slot specific representations. finally, state prediction is performed with these domain-slot specific representations. we conduct experiments to verify our approach on multiwoz 2.0 and multiwoz 2.4 datasets. the experimental results show that the proposed approach can effectively improve the performance of multi-domain dialogue state tracking. the contributions of this work can be addressed as follows. ## related works dialogue state tracking (dst) is the core of taskoriented dialogue systems. in the early years, dst highly relies on hand-crafted semantic features to predict the dialogue states @xcite @xcite , which is hard to handle lexical and morphological variations in spoken language @xcite . benefiting from the rapid development of deep learning methods, neural networkbased dst models have been explored. @xcite proposes a novel neural belief tracking (nbt) framework with learning n-gram representations of utterances. inspired by it, a lot of neural network models are investigated @xcite @xcite @xcite and achieve further improvement. pre-trained models have brought natural language processing to a new era in recent years. many substantial works have shown that the pretrained models can learn universal language representations, which are beneficial for downstream tasks @xcite @xcite @xcite . more recently, the very deep pre-trained language models, such as bidirectional encoder representation from transformer (bert) @xcite and generative pre-training (gpt) @xcite , trained with an increasing number of selfsupervised tasks have been proposed to make the models capturing more knowledge from a large scale of corpora, which have shown their abilities to produce promising results. in view of it, many pieces of studies about dst have explored to establish the models on the basis of these pre-trained language models @xcite @xcite @xcite @xcite @xcite . related to handling the correlations among domains and slots in multi-domain dst, several approaches have been investigated. in recent mainstream approaches, domain-slot specific representations are first achieved using attention mechanism with aggregated domain-slot query, and then the correlations are modeled with them. @xcite utilizes domain and slot information to extract both domain and slot specific representations and then combines such representations to predict the values. @xcite manually constructs a schema graph modeling the dependencies of different slots and introduces a graph attention matching network to mix the information from utterances and graphs to control the state updating. @xcite introduces a matrix representing the similarity among different slots and then perform slot information sharing among similar slots. the above two approaches are name-based since they only consider the semantics dependencies of slot names to measure the correlation among different slots, which may result in overlooking the dependencies of some slots. more recently, @xcite proposes a data-driven approach to handle these correlations, in which slot self-attention is introduced. however, this approach may inevitably result in overestimating some correlations @xcite . ## dialogue state tracking with disentangled domain-slot attention figure 1(a) presents the overview of the proposed model. it consists of a dialogue encoder, a domain, slot and value encoder, disentangled domain-slot attention (ddsa), and slot value matching. the context representations of dialogue history, domains, slots and values are firstly obtained by feeding dialogue history, domains, slots and values into encoders respectively. and then these representations are passed to our proposed disentangled domainslot attention, as shown detailedly in figure 1b , to achieve domain-slot specific representations. finally, the corresponding values are chosen to predict the state values with these representations and slot value matching. ## experimental settings we conduct the experiments using multiwoz 2.0 and multiwoz 2.4 datasets in this work. multi-woz 2.0 @xcite is one of the largest open-source human-human conversational datasets of multiple domains. it contains over 10,000 dialogues in which each dialogue averages 13.68 turns. multiwoz 2.4 is the latest refined version @xcite . it mainly fixes the annotation errors in the validation and test set. to make a fair comparison with the models evaluated on these two datasets, we follow the pre-processing and evaluation procedure in several previous works @xcite @xcite to keep consistent. we present the settings of the model in appendix a. ## conclusion in this work, we propose a model based on disentangled domain-slot attention for multi-domain dialogue state tracking to handle the correlation among different domains and slots. unlike the conventional approach in recent mainstream models, we disentangle the query about domains and slots in a flexible and context-dependent manner. the experimental results on multiwoz 2.0 and mul-tiwoz 2.4 datasets show that, comparing to the models based on conventional approaches of slot attention using the aggregated domain-slot pairs, our approach effectively improves the performance of multi-domain dialogue state tracking. in future works, we will investigate to utilize the proposed approach to generative models and generalize them to more complicated scenarios.