CONVERSER : Few-shot Conversational Dense Retrieval with Synthetic Data Generation (2023)

Abstract: Conversational search provides a natural interface for information retrieval (IR). Recent approaches have demonstrated promising results in applying dense retrieval to conversational IR. However, training dense retrievers requires large amounts of in-domain paired data. This hinders the development of conversational dense retrievers, as abundant in-domain conversations are expensive to collect. In this paper, we propose Converser, a framework for training conversational dense retrievers with at most 6 examples of in-domain dialogues. Specifically, we utilize the in-context learning capability of large language models to generate conversational queries given a passage in the retrieval corpus. Experimental results on conversational retrieval benchmarks OR-QuAC and TREC CAsT 19 show that the proposed Converser achieves comparable performance to fully-supervised models, demonstrating the effectiveness of our proposed framework in few-shot conversational dense retrieval. All source code and generated datasets are available: https://github.com/MiuLab/CONVERSER

## introduction conversational information retrieval (cir) has been an important area of research in recent years, aiming to retrieve relevant information from a large corpus of text in a conversational format. it has gained considerable interest due to its potential to deliver information in a natural format in response to a user's queries. unlike traditional ir, cir poses distinctive challenges, including its multiturn and context-dependent nature, which require more nuanced approaches @xcite . dense retrieval methods have demonstrated their ability to understand the semantics of complex user queries and shown promising performance on opendomain retrieval @xcite . one of the major obstacles to conversational dense retrieval is the scarcity of training data, given the high cost and extensive time to collect high-quality information-seeking conversations @xcite . previous work has explored various approaches to address this issue @xcite . however, most methods still rely on the assumption that a large amount of in-domain data is present and build data augmentation models upon it. in this paper, we aim to develop a few-shot conversational dense retrieval model that can effectively retrieve relevant passages based on a small number of in-domain dialogues. to achieve this, we leverage the in-context learning capability of large language models (llms) to generate synthetic passage-dialogue pairs with few-shot demonstrations. specifically, in-domain passages are sampled from the retrieval corpus, and dialogues are synthesized by asking llms to generate a series of queries based on a few examples. we also employ a self-consistency filtering mechanism to automatically discard inconsistent generated queries, ensuring the accuracy and reliability of the generations. we conduct experiments on two benchmark datasets, including or-quac @xcite and trec cast 19 @xcite . the experimental results demonstrate that our proposed framework, converser, performs comparably to fully-supervised models that are trained on thousands of annotated dialogues while using only 6 examples at most. furthermore, analyses show that converser rivals other data augmentation methods that utilize full in-domain datasets, demonstrating its effectiveness. more recently, qu et al. (2020) proposed or-convqa, which extends cqa to the opendomain setting where a retrieval module is required. @xcite utilizes an adhoc dense retriever and manually rewritten contextindependent queries for training few-shot retrievers and rerankers, while our method does not require an ad-hoc model and additional annotation. others have explored various methods for encoding conversational queries @xcite @xcite , which are orthogonal to our work. ## proposed method: converser we propose few-shot conversational dense retrieval with synthetic data generation, converser, which aims to generate synthetic conversational queries given few examples. more formally, given a conversational retrieval task t , its retrieval corpus p t , and k examples, we aim to generate synthetic conversational query-passage pairs { ĉ1 , • • • , ĉn } for training dense retrievers. ## experiments to evaluate if our generated conversational questions can help train a conversational retriever, we conduct experiments on a conversational question answering dataset, or-quac (qu et al., 2020), and a conversational search benchmark, trec cast-19 @xcite . ## qualitative study we present a generated example in ## conclusion this paper introduces converser, a synthetic data generation method for training few-shot conversational dense retrievers. we leverage the incontext learning capability of llms and propose techniques that are designed for generating conversational queries. experimental results demonstrate that our proposed converser achieves comparable performance to fully-supervised models while only requiring 6 annotated examples. further analyses demonstrate that our method outperforms a fully-supervised data augmentation method. future work could explore instruction-following llms, better filtering mechanisms, and synthesizing specialized data for conversational dense retrieval, such as query rewrites.