RAC : Retrieval-augmented Conversation Dataset for Open-domain Question Answering in Conversational Settings (2024)

Abstract: In recent years, significant advancements in conversational question and answering (CQA) have been driven by the exponential growth of large language models and the integration of retrieval mechanisms that leverage external knowledge to generate accurate and contextually relevant responses. Consequently, the fields of conversational search and retrieval-augmented generation (RAG) have obtained substantial attention for their capacity to address two key challenges: query rewriting within conversational histories for better retrieval performance and generating responses by employing retrieved knowledge. However, both fields are often independently studied, and comprehensive study on entire systems remains underexplored. In this work, we present a novel retrieval-augmented conversation (RAC) dataset and develop a baseline system comprising query rewriting, retrieval, reranking, and response generation stages. Experimental results demonstrate the competitiveness of the system and extensive analyses are conducted to apprehend the impact of retrieval results to response generation.

## introduction conversational question answering (cqa), also known as interactive or sequential qa, focuses on answering questions within a conversational context @xcite @xcite . however, existing studies often constrain questions and answers within predefined contexts, excluding the retrieval process @xcite . this limitation creates a gap between the ideal and actual cqa environment. a more realistic scenario is to retrieve relevant passages related to a question each turn of the conversation and use these passages to provide answers. we refer this new task as retrieval-augmented conversation (rac). ## data construction to comprehensively address the requirement of rac, a dataset must comprise conversations with referenced passages for response generation, as well as passage collections for retrieval purposes. however, existing cqa datasets are insufficient for the entire rac because they provide questions and answers constrained on given contexts or do not cover an answering stage. neither conversational search nor rag datasets are also inadequate, as they primarily focus on query rewriting to improve retrieval performance and response generation using retrieved knowledge, respectively. to bridge the gap, we utilize the knowledge-retrieval conversation dataset and address its limitations. the dataset contains conversations between a user and an expert on several topics, including supporting factors configuring the responses by the expert and documents referenced for the supporting factors. ## retrieval-augmented conversation although some studies about rag construct endto-end training systems that backpropagates loss of response generation to the retrieval model, we break the entire process into pipelines to alleviate the difficulties of systems as a beginning of rac. consequently, the overall system is divided up to query rewriting, retrieval, reranking, and response generation stages. the encoder-decoder model, such as t5 @xcite or bart @xcite , was adopted as a backbone for both query rewriting and response generation because the architecture is particularly beneficial for the subtasks by crossattending to given inputs after the self-attention layer and ensuring that input contexts well affect to the generated tokens. the details of the baseline system is specifically explained in the following subsections. ## conclusion in this work, we introduced rac and presented the new dataset that satisfies its requirements. with the comprehensive dataset, a strong baseline system comprising query rewriting, retrieval, reranking, and response generation was constructed. specifically, the query rewriting model was trained following the generative retrieval approach and also used for reranking stage by leveraging the ability of query generation from passages, resulting in significant improvement of the retrieval performance. our empirical experiments and analyses discover the challenges of rac and enlighten the future direction of the entire system.