Efficient Context and Schema Fusion Networks for Multi-Domain Dialogue State Tracking (2020)

Abstract: Dialogue state tracking (DST) aims at estimating the current dialogue state given all the preceding conversation. For multi-domain DST, the data sparsity problem is a major obstacle due to increased numbers of state candidates and dialogue lengths. To encode the dialogue context efficiently, we utilize the previous dialogue state (predicted) and the current dialogue utterance as the input for DST. To consider relations among different domain-slots, the schema graph involving prior knowledge is exploited. In this paper, a novel context and schema fusion network is proposed to encode the dialogue context and schema graph by using internal and external attention mechanisms. Experiment results show that our approach can outperform strong baselines, and the previous state-of-the-art method (SOM-DST) can also be improved by our proposed schema graph.

## introduction dialogue state tracking (dst) is a key component in task-oriented dialogue systems which cover certain narrow domains (e.g., booking hotel and travel planning). as a kind of context-aware language understanding task, dst aims to extract user goals or intents hidden in human-machine conversation and represent them as a compact dialogue state, i.e., a set of slots and their corresponding values. for example, as illustrated in fig. 1 , (slot, value) pairs like (name, huntingdon marriott hotel) are extracted from the dialogue. it is essential to build an accurate dst for dialogue management @xcite , where dialogue state determines the next machine action and response. recently, motivated by the tremendous growth of commercial dialogue systems like apple siri, microsoft cortana, amazon alexa, or google assistant, multi-domain dst becomes crucial to help users across different domains @xcite . as shown in fig. 1 , the dialogue covers three domains (i.e., hotel, attraction and taxi). the goal of multidomain dst is to predict the value (including none) for each domain-slot pair based on all the preceding dialogue utterances. however, due to increasing numbers of dialogue turns and domainslot pairs, the data sparsity problem becomes the main issue in this field. to tackle the above problem, we emphasize that dst models should support open-vocabulary based value decoding, encode context efficiently and incorporate domain-slot relations: to this end, we propose a multi-domain dialogue state tracker with context and schema fusion networks (csfn-dst). the fusion network is exploited to jointly encode the previous dialogue state, the current turn dialogue and the schema graph by internal and external attention mechanisms. after multiple layers of attention networks, the final representation of each domain-slot node is utilized to predict the corresponding value, involving context and schema information. for the value prediction, a slot gate classifier is applied to decide whether a domain-slot is mentioned in the conversation, and then an rnn-based value decoder is exploited to generate the corresponding value. our proposed csfn-dst is evaluated on mul-tiwoz 2.0 and multiwoz 2.1 benchmarks. ablation study on each component further reveals that both context and schema are essential. contributions in this work are summarized as: ## related work traditional dst models rely on semantics extracted by natural language understanding to predict the current dialogue states @xcite @xcite @xcite , or jointly learn language understanding in an end-to-end way @xcite . these methods heavily rely on hand-crafted features and complex domain-specific lexicons for delexicalization, which are difficult to extend to new domains. recently, most works about dst focus on encoding dialogue context with deep neural networks (such as cnn, rnn, lstm-rnn, etc.) and predicting a value for each possible slot @xcite @xcite . multi-domain dst most traditional state tracking approaches focus on a single domain, which extract value for each slot in the domain @xcite . they can be directly adapted to multi/mixed-domain conversations by replacing slots in a single domain with domain-slot pairs (i.e. domain-specific slots) @xcite @xcite @xcite . despite its simplicity, this approach for multi-domain dst extracts value for each domain-slot independently, which may fail to capture features from slot co-occurrences. for example, hotels with higher stars are usually more expensive (price range). predefined ontology-based dst most of the previous works assume that a predefined ontology is provided in advance, i.e., all slots and their values of each domain are known and fixed @xcite . predefined ontology-based dst can be simplified into a value classification task for each slot @xcite @xcite @xcite . it has the advantage of access to the known candidate set of each slot, but these approaches may not be applicable in the real scenario. since a full ontology is hard to obtain in advance @xcite , and the number of possible slot values could be substantial and variable (e.g., song names), even if a full ontology exists @xcite . open-vocabulary dst without a predefined ontology, some works choose to directly generate or extract values for each slot from the dialogue context, by using the encoder-decoder architecture @xcite or the pointer network @xcite @xcite . they can improve the scalability and robustness to unseen slot values, while most of them are not efficient in context encoding since they encode all the previous utterances at each dialogue turn. notably, a multi-domain dialogue could involve quite a long history, e.g., multiwoz dataset @xcite ) contains about 13 turns per dialogue on average. ## problem formulation in a multi-domain dialogue state tracking problem, we assume that there are m domains (e.g. taxi, hotel) involved, since different domains may contain a same slot, we denote all distinct n slots as @xmath0, s 2 , • • • , s n }, where @xmath1. a dialogue can be formally represented as {(a 1 , u 1 , b 1 ), (a 2 , u 2 , b 2 ), • • • , (a t , u t , b t )}, where a t is what the agent says at the t-th turn, u t is the user utterance at t turn, and b t denotes the corresponding dialogue state. a t and u t are word sequences, while b t is a set of domain-slot-value triplets, e.g., (hotel, price range, expensive). value v tj is a word sequence for j-th domain-slot pair at the t-th turn. the goal of dst is to correctly predict the value for each domain-slot pair, given the dialogue history. as the input. however, this may lead to increased computation time. in this work, we propose to utilize only the current dialogue turn a t , u t and the previous dialogue state b t-1 to predict the new state b t . during the training, we use the ground truth of b t-1 , while the previous predicted dialogue state would be used in the inference stage. schema graph to consider relations between 1 for open-vocabulary dst, possible values for each slot @xmath2. different domain-slot pairs and exploit them as an additional input to guide the context encoding, we formulate them as a schema graph @xmath3, e) with node set v and edge set e. fig. 2 shows an example of schema graph. in the graph, there are three kinds of nodes to denote all domains d, slots s, and domain-slot pairs o, i.e., @xmath4. four types of undirected edges between different nodes are exploited to encode prior knowledge: ## context and schema fusion networks for multi-domain dst in this section, we will introduce our approach for multi-domain dst, which jointly encodes the current dialogue turn (a t and u t ), the previous dialogue state b t-1 and the schema graph g by fusion networks. after that, we can obtain contextaware and schema-aware node embeddings for all j domain-slot pairs. finally, a slot-gate classifier and rnn-based value decoder are exploited to extract the value for each domain-slot pair. the architecture of csfn-dst is illustrated in fig. 3 , which consists of input embeddings, context schema fusion network and state prediction modules. ## conclusion and future work we introduce a multi-domain dialogue state tracker with context and schema fusion networks, which involves slot relations and learns deep representations for each domain-slot pair dependently. slots from different domains and their relations are organized as a schema graph. our approach outperforms strong baselines on both multiwoz 2.0 and 2.1 benchmarks. ablation studies also show that the effectiveness of the schema graph. it will be a future work to incorporate relations among dialogue states, utterances and domain schemata. to further mitigate the data sparsity problem of multi-domain dst, it would be also interesting to incorporate data augmentations @xcite and semi-supervised learnings @xcite .