A Template-guided Hybrid Pointer Network for Knowledge-based Task-oriented Dialogue Systems (2021)

Abstract: Most existing neural network based task-oriented dialog systems follow encoder-decoder paradigm, where the decoder purely depends on the source texts to generate a sequence of words, usually suffering from instability and poor readability. Inspired by the traditional template-based generation approaches, we propose a template-guided hybrid pointer network for knowledge-based task-oriented dialog systems, which retrieves several potentially relevant answers from a pre-constructed domain-specific conversational repository as guidance answers, and incorporates the guidance answers into both the encoding and decoding processes. Specifically, we design a memory pointer network model with a gating mechanism to fully exploit the semantic correlation between the retrieved answers and the ground-truth response. We evaluate our model on four widely used task-oriented datasets, including one simulated and three manually created datasets. The experimental results demonstrate that the proposed model achieves significantly better performance than the state-of-the-art methods over different automatic evaluation metrics.

## introduction task oriented dialogue systems have attracted increasing attention recently due to broad applications such as reserving restaurants and booking flights. conventional task-oriented dialogue systems are mainly implemented by rule-based methods @xcite , which rely heavily on the hand-crafted features, establishing significant barriers for adapting the dialogue systems to new domains. motivated by the great success of deep learning in various nlp tasks, the neural network based methods @xcite @xcite have dominated the study since these methods can be trained in an end-to-end manner and scaled to different domains. despite the remarkable progress of previous studies, the performance of task-oriented dialogue systems is still far from satisfactory. on one hand, due to the exposure bias problem @xcite , the neural network based models, e.g., the sequence to sequence models (seq2seq), tend to accumulate errors with increasing length of the generation. concretely, the first several generated words can be reasonable, while the quality of the generated sequence deteriorates quickly once the decoder produces a "bad" word. on the other hand, as shown in previous works @xcite , the seq2seq models are likely to generate non-committal or similar responses that often involve high-frequency words or phrases. these responses are usually of low informativeness or readability. this may be because that arbitrarylength sequences can be generated, and it is not enough for the decoder to be purely based on the source input sentence to generate informative and fluent responses. we demonstrate empirically that in task-oriented dialogue systems, the responses for the requests with similar types often follow the same sentence structure except that different named entities are used according to the specific dialogue context. in this paper, we propose a template-guided hybrid pointer network (thpn to generate the response given a user-issued query, in which the domain specific knowledge base (kb) and potentially relevant answers are leveraged as extra input to enrich the input representations of the decoder. here, knowledge base refers to the database to store the relevant and necessary information for supporting the model in accomplishing the given tasks. we follow previous works and use a triple (subject, relation, object) representation. for example, the triple (starbucks, address, 792 bedoin st) is an example in kb representing the information related to the starbucks. specifically, given a query, we first retrieve top-n answer candidates from a pre-constructed conversational repository with question-answer pairs using bert @xcite . then, we extend memory networks @xcite to incorporate the commonsense knowledge from kb to learn the knowledge-enhanced representations of the dialogue history. finally, we introduce a gating mechanism to effectively utilize candidate answers and improve the decoding process. the main contributions of this paper can be summarized as follows: ## related work task-oriented dialogue systems are mainly studied via two different approaches: pipeline based and end-to-end. pipeline based models @xcite achieve good stability but need domain-specific knowledge and handcrafted labels. end-to-end methods have shown promising results recently and attracted more attention since they are easily adapted to a new domain. neural network based dialogue systems can avoid the laborious feature engineering since the neural networks have great ability to learn the latent representations of the input text. however, as revealed by previous studies @xcite @xcite , the performance of the sequence to sequence model deteriorates quickly with the increase of the length of generation. therefore, how to improve the stability and readability of the neural network models has attracted increasing attention. @xcite proposed a copy augmented seq2seq model by copying relevant information directly from the kb information. @xcite proposed a generative model by employing the multi-hop attention over memories with the idea of pointer network. @xcite proposes a global-tolocally pointer mechanism to effectively utilize the knowledge base information, which improves the quality of the generated response. previous proposed neural approaches have shown the importance of external knowledge in the sequence generation @xcite @xcite @xcite , especially in the task-oriented dialogue systems where an appropriate response usually requires correctly extracting knowledge from the domain-specific or commonsense knowledge base @xcite @xcite . however, it is still under great exploration with regard with the inclusion of external knowledge into the model. yan et al. (2016); song et al. (2018) argue that retrieval and generative methods have their own demerits and merits, and they have achieved good performance in the chit-chat response generation by incorporating the retrieved results in the seq2seq based models. @xcite proposed an adversarial training approach, which is enhanced by retrieving some related candidate answers in the neural response generation, and @xcite also applies a similar method in the neural conversation model. in addition, in task-oriented dialogue tasks, the copy mechanism @xcite has also been widely utilized @xcite , which shows the superiority of generation based methods with copy strategy. ## methodology we build our model based on a seq2seq dialogue generation mode, and the overall architecture is exhibited in figure 1 . each module will be elaborated in the following subsections. ## conclusion in task-oriented dialog systems, the words and sentence structures are relatively limited and fixed, thus it is intuitive that the retrieved results can provide valuable information in guiding the response generation. in this paper, we retrieve several potentially relevant answers from a pre-constructed domain-specific conversation repository as guidance answers, and incorporate the guidance answers into both the encoding and decoding processes. we copy the words from the previous context and the retrieved answers directly, and generate words from the vocabulary. experimental results over four datasets have demonstrated the effectiveness of our model in generating informative responses. in the future, we plan to leverage the dialogue context information to retrieve candidate answers turn by turn in multi-turn scenarios.