Contextualize Knowledge Bases with Transformer for End-to-end Task-Oriented Dialogue Systems (2021)

Abstract: Incorporating knowledge bases (KB) into end-to-end task-oriented dialogue systems is challenging, since it requires to properly represent the entity of KB, which is associated with its KB context and dialogue context. The existing works represent the entity with only perceiving a part of its KB context, which can lead to the less effective representation due to the information loss, and adversely favor KB reasoning and response generation. To tackle this issue, we explore to fully contextualize the entity representation by dynamically perceiving all the relevant entities and dialogue history. To achieve this, we propose a COntext-aware Memory Enhanced Transformer framework (COMET), which treats the KB as a sequence and leverages a novel Memory Mask to enforce the entity to only focus on its relevant entities and dialogue history, while avoiding the distraction from the irrelevant entities. Through extensive experiments, we show that our COMET framework can achieve superior performance over the state of the arts.

## introduction task-oriented dialogue systems aim to achieve specific goals such as hotel booking and restaurant reservation. the traditional pipelines @xcite consist of natural language understanding, dialogue management, and natural language generation modules. however, designing these modules often requires additional annotations such as dialogue states. to simplify this procedure, the end-to-end dialogue systems @xcite are proposed to incorporate the kb (normally relational databases) into the learning framework, where the kb and dialogue history can be directly modeled for response generation, without the explicit dialogue state or dialogue action. ## methodology in this section, we first introduce the general workflow for this task. then, we elaborate on each part of comet, i.e., the dialogue history encoder, context-aware memory generation, and response generation decoder (as depicted in fig. 2 ). finally, the objective function will be introduced. ## related work task-oriented dialogue systems can be mainly categorized into two parts: modularized @xcite and end-toend @xcite . for the end-to-end task-oriented dialogue systems, @xcite first explores the end-to-end method for the task-oriented dialogue systems. however, it can only link to the entities in the dialogue context and no kb is incorporated. to effectively incorporate the external kb, @xcite proposes a keyvalue retrieval mechanism to sustain the grounded multi-domain discourse. @xcite augments the dialogue systems with end-to-end memory networks @xcite . @xcite ) models a dialogue state as a fixed-size distributed representation and uses this representation to query kb. @xcite designs belief spans to track dialogue believes, allowing task-oriented dialogue systems to be modeled in a sequence-tosequence way. @xcite proposes a multi-level memory to better leverage the external kb. @xcite proposes a global-to-local memory pointer network to reduce the noise caused by kb. @xcite proposes heterogeneous memory networks to handle the heterogeneous information from different sources. @xcite proposes a dynamic fusion mechanism to transfer the knowledge among different domains. @xcite exploits the graph structural informa-tion in kb and the dialogue. other works also explore how to combine the pre-trained model @xcite with the endto-end task-oriented dialogue systems. @xcite directly embeds the kb into the parameters of gpt-2 @xcite via finetuning. @xcite proposes a dialogue model that is built with a fixed pre-trained conversational model and multiple trainable light-weight adapters. we also notice that some existing works also combine transformer with the memory component, e.g., @xcite . however, our method is distinguishable from them, since the existing works like @xcite simply inject the memory component into transformer. in contrast, inspired by the dynamic generation mechanism @xcite , the memory in comet (i.e., the entity representation) is dynamically generated by fully contextualizing the kb and dialogue context via the memory-masked transformer. ## conclusion in this work, we propose a novel context-aware memory enhanced transformer (comet) for the end-to-end task-oriented dialogue systems. by the designed memory mask scheme, comet can fully contextualize the entity with all its kb and dialogue contexts, and generate the (n + 1)-tuple representations of the entities. the generated entity representations can further augment the framework and lead to better capabilities of response generation and entity linking. the extensive experiments demonstrate the effectiveness of our method.