Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented System (2023)

Abstract: End-to-end task-oriented dialogue (TOD) systems have achieved promising performance by leveraging sophisticated natural language understanding and natural language generation capabilities of pre-trained models. This work enables the TOD systems with more flexibility through a simple cache. The cache provides the flexibility to dynamically update the TOD systems and handle both existing and unseen dialogue scenarios. Towards this end, we first fine-tune a retrieval module to effectively retrieve the most relevant information entries from the cache. We then train end-to-end TOD models that can refer to and ground on both dialogue history and retrieved information during TOD generation. The introduced cache is straightforward to construct, and the backbone models of TOD systems are compatible with existing pre-trained generative models. Extensive experiments demonstrate the superior performance of our framework, with a notable improvement in non-empty joint goal accuracy by 6.7% compared to strong baselines.

## introduction task-oriented dialogue (tod) systems play an important role in various applications, such as restaurants booking, alarm setting, and recommendations @xcite . these systems can be broadly categorized into two groups: pipeline-based dialogue systems and end-to-end dialogue systems. pipeline-based dialogue systems consist of four separate modules, namely a natural language understanding (nlu) module for detecting user intents, a dialogue state tracking (dst) module to track user belief states across dialogue turns, a dialogue management (dm) module for system actions based on dialogue states, and a natural language generation (nlg) module for generating natural-language responses. however, the pipeline-based approach is label-intensive, prone to error propagation, and challenging to scale @xcite @xcite . recently, various approaches have been proposed to utilize sequence-to-sequence models for generating dialogue states and responses in an endto-end manner @xcite @xcite @xcite @xcite @xcite . compared with the pipeline-based systems, these approaches demonstrate effectiveness on public datasets with fewer direct annotations required, such as user intents and dialogue acts. additionally, they leverage the capabilities of large-scale pre-trained language models, such as gpt-2 @xcite , t5 @xcite , and bart @xcite , for improved performance in nlu and nlg tasks. however, these approaches are limited in their ability to dynamically handle existing, unseen, or emerging intents and slots, particularly in the context of unseen dialogue scenarios such as new domains and services @xcite @xcite . in parallel, research on open-domain question answering and dialogue systems has explored the use of retrieval-augmented models. these models retrieve relevant information from a passage, database, apis, etc., and incorporate it into the generation process, improving answer quality or dialogue responses @xcite @xcite @xcite . inspired by these ideas, we combine both worlds and propose an end-to-end tod framework with a retrieval system that addresses the challenge of handling both existing and zero-shot unseen dialogue scenarios. our approach involves training the end-to-end tod models with a cache that contains accessible domains, intents, slots and apis. the cache can be constructed based on the schema or database, or by extracting information from accessible dialogues when the schema or database is not fully accessible. the cache serves as a reference point, allowing the models to ground their responses in the retrieved information. by incorporating a retrieval module and leveraging this cache of knowledge, our system enhances the flexibility and adaptability to handle both existing and unseen intents and slots, and enables robust performance even in novel dialogue domains and services where the model has not been explicitly trained. figure 1 shows an illustrative example of our approach, demonstrating how the retrieval module retrieves relevant information, such as slots in this case, from the cache to enrich the system's understanding and generate more accurate responses. the apicall represents the dialogue states from the system side, and apiresp returns information from external api interactions between the system and system databases. to build an accurate end-to-end tod system with the benefits of a simple cache, we fine-tune a retrieval module to effectively retrieve the most relevant and informative information from the cache, using a top-n retrieval strategy. then we integrate the retrieval module into the generative model to facilitate end-to-end tod generation. we evaluate our approach on the publicly available google schema-guided dialogue dataset (sgd) @xcite , which includes a significant number of unseen dialogue domains and services in the development and test sets. the contributions of this paper are as follows: (1) we design a simple yet effective end-to-end tod framework with a cache that enables dynamic handling of intents and slots. the framework is compatible with existing pre-trained generative models, and enhances the system's robustness. (2) experimental results demonstrate the superior performance of our approach compared to strong baselines. it achieves 6.7% improvement in non-empty joint goal accuracy, demonstrating the effectiveness in handling various dialogue scenarios, including the challenging zero-shot unseen dialogues. (3) to advance future research in accurate end-to-end tod systems, we conduct comprehensive ablation studies and analyses to provide insights into the impact of different components and design choices within our framework. retrieval-augmented models retrieval augmented approaches have been widely used in open-domain question answering. for instance, @xcite propose a bert-based @xcite dual-encoder framework to retrieve passages from wikipedia, which is further incorporated into open-domain conversations to reduce hallucination and enrich engagement with users @xcite . these models retrieve information related to the query from a knowledge base of sentences and ground the generation response on this information @xcite . inspired by these works, we explore the integration of retrieval modules into end-to-end tod systems, leveraging the retrieval-augmented approach to enhance the system's performance in handling both existing and novel dialogue scenarios. ## tod systems with a simple cache we present an end-to-end transformer-based framework with a simple cache that is compatible with multiple generative models, including bart, t5, gpt2, etc. our framework enables dynamic handling of intents, slots, and apis while maintaining flexibility in choosing the backbone model. generally, our framework consists of two parts: a retrieval model for retrieving the most relevant and informative information from the cache, and an end-to-end tod model that generates apicalls and system responses based on the dialogue history and the retrieved information. the retrieval model functions by retrieving intents, slots, apis, and other relevant information from the cache. figure 2 illustrates one simple variant of our framework, which is an encoder-decoder architecture. in this variant, the retrieved information such as slots are stacked together. we also introduce another variant in sec. 4.2, where each retrieved information is concatenated with the dialogue history and then all the information are concatenated together before being sent to the decoder. ## conclusion this paper aims to improve performance of end-toend tod systems with a simple cache. we first construct a simple cache with intents and slots and finetune a retrieval module to retrieve most relevant information entries. we then train the end-to-end tod model to reference and ground the dialogue history and the retrieved information while performing tod generation. experimental results on a large-scale sgd dataset show that our approach has superior performance over strong baselines.