Dynamic Fusion Network for Multi-Domain End-to-end Task-Oriented Dialog (2020)

Abstract: Recent studies have shown remarkable success in end-to-end task-oriented dialog system. However, most neural models rely on large training data, which are only available for a certain number of task domains, such as navigation and scheduling. This makes it difficult to scalable for a new domain with limited labeled data. However, there has been relatively little research on how to effectively use data from all domains to improve the performance of each domain and also unseen domains. To this end, we investigate methods that can make explicit use of domain knowledge and introduce a shared-private network to learn shared and specific knowledge. In addition, we propose a novel Dynamic Fusion Network (DF-Net) which automatically exploit the relevance between the target domain and each domain. Results show that our models outperforms existing methods on multi-domain dialogue, giving the state-of-the-art in the literature. Besides, with little training data, we show its transferability by outperforming prior best model by 13.9% on average.

## introduction task-oriented dialogue systems @xcite help users to achieve specific goals such as restaurant reservation or navigation inquiry. in recent years, end-to-end methods in the literature usually take the sequence-to-sequence (seq2seq) model to generate a response from a dialogue history @xcite @xcite @xcite @xcite . taking the dialogue in figure 1 as an example, to answer the driver's query about the "gas station", the end-to-end dialogue system directly generates system response given the query and a corresponding knowledge base (kb). figure 1 : example of a task-oriented dialogue that incorporates a knowledge base (kb) from the smd dataset @xcite . words with the same color refers queried entity from the kb. better viewed in color. though achieving promising performance, endto-end models rely on a considerable amount of labeled data, which limits their usefulness for new and extended domains. in practice, we cannot collect rich datasets for each new domain. hence, it is important to consider methods that can effectively transfer knowledge from a source domain with sufficient labeled data to a target domain with limited or little labeled data. existing work can be classified into two main categories. as shown in figure 2 (a), the first strand of work @xcite @xcite simply combines multi-domain datasets for training. such methods can implicitly extract the shared features but fail to effectively capture domain-specific knowledge. as shown in figure 2 (b), the second strand of work @xcite trains model separately for each domain, which can better capture domain-specific features. however, those methods ignore shared knowledge between different domains (e.g. the location word exists in both schedule domain and navigation domain). we consider addressing the limitation of existing work by modeling knowledge connections between domains explicitly. in particular, a simple baseline to incorporate domain-shared and domain-private features is shared-private framework @xcite @xcite . shown in figure 2 (c), it includes a shared module to capture domain-shared feature and a private module for each domain. the method explicitly differentiates shared and private knowledge. however, this framework still has two issues: (1) given a new domain with extremely little data, the private module can fail to effectively extract the corresponding domain knowledge. (2) the framework neglects the fine-grained relevance across certain subsets of domains. (e.g. schedule domain is more relevant to the navigation than to the weather domain.) to address the above issues, we further propose a novel dynamic fusion network (df-net), which is shown in figure 2 (d ). in contrast to the sharedprivate model, a dynamic fusion module (see §2.3) is further introduced to explicitly capture the correlation between domains. in particular, a gate is leveraged to automatically find the correlation between a current input and all domain-specific models, so that a weight can be assigned to each domain for extracting knowledge. such a mechanism is adopted for both the encoder and the decoder, and also a memory module to query knowledge base features. given a new domain with little or no training data, our model can still make the best use of existing domains, which cannot be achieved by the baseline model. we conduct experiments on two public benchmarks, namely smd @xcite and multi-woz 2.1 @xcite . results show that our framework consistently and significantly outperforms the current state-of-the-art methods. with limited training data, our framework outperforms the prior best methods by 13.9% on average. to our best of knowledge, this is the first work to effectively explore shared-private framework in multi-domain end-to-end task-oriented dialog. in addition, when given a new domain which with few or zero shot data, our extended dynamic fusion framework can utilize fine-grained knowledge to obtain desirable accuracies, which makes it more adaptable to new domains. all datasets and code are publicly available at: ## model architecture we build our model based on a seq2seq dialogue generation model ( §2.1), as shown in figure 3(a) . to explicitly integrate domain awareness, as shown in figure 3(b) we first propose to use a sharedprivate framework ( §2.2) to learn shared and the corresponding domain-specific features. next, we further use a dynamic fusion network ( §2.3) to dynamically exploit the correlation between all domains for fine-grained knowledge transfer, which is shown in figure 3(c ). in addition, adversarial training is applied to encourage shared module generate domain-shared feature. ## related work existing end-to-end task-oriented systems can be classified into two main classes. a series of work trains a single model on the mixed multi-domain dataset. @xcite augments the vocabulary distribution by concatenating kb attention to generatge entities. @xcite first integrates track dialogue believes in end-to-end task-oriented dialog. @xcite combines end-toend memory network @xcite into sequence generation. gangi @xcite proposes a multi-level memory architecture which first addresses queries, followed by results and finally each key-value pair within a result. @xcite proposes a global-to-locally pointer mechanism to query the knowledge base. compared with their models, our framework can not only explicitly utilize domain-specific knowledge but also consider different relevance between each domain. another series of work trains a model on each domain separately. @xcite leverages dialogue state representation to retrieve the kb implicitly. @xcite first adopts the kb-retriever to explicitly query the knowledge base. their works consider only domain-specific features. in contrast, our framework explicitly leverages domain-shared features across domains. the shared-private framework has been explored in many other task-oriented dialog components. @xcite applies a shared-private lstm to generate shared and domain-specific features. @xcite proposes a global-local architecture to learn shared feature across all slots and specific feature for each slot. more recently, @xcite utilizes the shared-private model for text style adaption. in our work, we explore shared-private framework in end-to-end taskoriented dialog to better transfer domain knowledge for querying knowledge base. in addition, we take inspiration from @xcite , who successfully apply the mix-of-the-experts (moe) mechanism in multi-sources domain and cross-lingual adaption tasks. our model not only combines the strengths of moe to incorporate domain-specific feature, but also applies adversarial training to encourage generating shared feature. to our best of knowledge, we are the first to effectively explore shared-private framework in multi-domain end-toend task oriented dialog. ## conclusion in this paper, we propose to use a shared-private model to investigate explicit modeling domain knowledge for multi-domain dialog. in addition, a dynamic fusion layer is proposed to dynamically capture the correlation between a target domain and all source domains. experiments on two datasets show the effectiveness of the proposed models. besides, our model can quickly adapt to a new domain with little annotated data.