I nstructo R : Instructing Unsupervised Conversational Dense Retrieval with Large Language Models (2023)

Abstract: Compared to traditional single-turn ad-hoc retrieval, conversational retrieval needs to handle the multi-turn conversation and understand the user’s real query intent. However, most existing methods simply fine-tune the pre-trained ad-hoc retriever on limited supervised data, making it challenging for the retriever to fully grasp the entirety of the conversation. In this paper, we find that large language models (LLMs) can accurately discover the user’s query intent from the complex conversation context and provide the supervised signal to instruct the retriever in an unsupervised manner. Therefore, we propose a novel method termed InstructoR to Instruct unsupervised conversational dense Retrieval with LLMs. We design an unsupervised training framework that employs LLMs to estimate the session-passage relevance score as the soft label to guide the retriever’s training. Specially, we devise three instructing strategies from context, query and response perspectives to calculate the relevance score more precisely, including conversational retrieval as conversation generation, question rewrite as latent variable and question response as posterior guide. Experimental results show InstructoR can bring significant improvements across various ad-hoc retrievers, even surpassing the current supervised state-of-the-art method. We also demonstrate the effectiveness of our method under low-resource and zero-shot settings. Our code is publicly available at https://github.com/jinzhuoran/InstructoR/.

## introduction the development of generative language models @xcite has sparked a paradigm shift in information-seeking, transitioning from using search engines to interacting with conversational assistants (e.g., chatgpt). conversational assistants offer a more flexible and userfriendly experience by addressing users' queries through multi-turn conversations. however, they still struggle to provide highly accurate and upto-date responses. to mitigate the problems, one promising solution @xcite is to employ a retrieve-then-generate pipeline. as illustrated in figure 1(a ), to answer the user's conversational query "did he like collecting things?" in the conversation, the pipeline first retrieves a small set of reference passages about "nicolas-claude fabri de peiresc" via conversational retrieval and then generates the response conditioned on these passages along with the ongoing conversation. conversational retrieval, which focuses on retrieving relevant passages from a large passage collection based on the given conversation session, has gained considerable attention from researchers. unlike traditional single-turn ad-hoc retrieval, conversational retrieval needs to deal with the complex multi-turn conversation session which consists of both the historical conversation context and the user's current query. in contrast to standalone queries, conversational queries may contain more complicated linguistic phenomena, such as omissions, coreferences, and ambiguities shown in @xcite . besides, topic-switching behaviour is natural in information-seeking conversations @xcite , introducing a significant amount of noise in the conversation context that is unrelated to the user's query. therefore, conversational retrieval is much more challenging than ad-hoc retrieval. existing methods can be mainly categorized into conversational query rewriting and conversational dense retrieval. conversational query rewriting methods @xcite utilize the rewriting models to explicitly reformulate the conversational queries into de-contextualized rewrites and then perform ad-hoc retrieval. while such a two-stage approach offers strong interpretability, its rewriting stage poses difficulties in direct optimization for retrieval performance and leads to increased retrieval latency @xcite . recently, conversational dense retrieval methods @xcite ) have shown better retrieval effectiveness. most existing supervised methods leverage the pre-trained ad-hoc retriever to encode the entire conversation session into a dense embedding and then fine-tune the retriever on conversational retrieval data. however, they still face the following two challenges: (1) fine-tuning the retriever often requires a large number of labeled session-passage pairs. in practical applications, annotating session-passage pairs is much more difficult than collecting conversation data. hence, there is a need to train a conversational dense retriever in an unsupervised manner, i.e., without using session-passage pairs; (2) simply compressing the session into one single vector may mix irrelevant context and neglect crucial information, making it hard for the retriever to comprehend the user's query in the complex conversation. in this paper, we propose a novel method termed instructor to instruct unsupervised conversational dense retrieval with large language models (llms). the main insight of our instruc-tor is that the powerful llms can help us train adhoc retrievers for conversational retrieval without supervision. on the one hand, some research @xcite @xcite has demonstrated that llms can generate high-quality annotated data with only a few or even without demonstration examples. on the other hand, we find that llms have powerful linguistic capabilities @xcite and can accurately grasp the user's intent from the complex and noisy conversation context. based on these findings, we unleash the power of llms to judge the relevance between sessions and passages without requiring any training. specifically, we design a three-stage unsupervised training framework for instructor shown in figure 1 to capture relevance precisely, we find that question response can provide the relevant signal when the passages related to the session are unknown. therefore, we treat question responses as the posterior guide to further enhance the retriever. we conduct extensive experiments on four datasets and prove all three strategies can bring significant improvements across various ad-hoc retrievers. moreover, our method surpasses the current supervised state-ofthe-art model without using labeled training data. our contributions are summarized as follows: ## methodology: instructor in this paper, we consider a more realistic setting which is to train a conversational dense retriever in an unsupervised manner, i.e., without using sessionpassage pairs. we only have a certain number of conversation sessions and a large passage collection during the training. our instructor uses the frozen llms' conditional generation log likelihood as the supervised signal to guide the training of the retriever without direct supervision. according to sachan et al. (2022b), we adopt an unsupervised training framework to distill knowledge from llm to retriever. moreover, we devise three instructing strategies from diverse perspectives in the conversation to calculate the relevance score of session-passage pairs more accurately. ## conclusion in this paper, we propose a novel method termed instructor to instruct unsupervised conversational dense retrieval with llms. instructor calculates the session-passage relevance score with frozen llms as the supervised signal to guide the ad-hoc retriever's training. to estimate the relevance score more precisely, we devise three strategies from diverse perspectives: conversational retrieval as conversation generation, question rewrite as latent variable and question response as posterior guide. experimental results show that instruc-tor can bring significant improvements across various ad-hoc retrievers, even surpassing the current supervised state-of-the-art method.