C hat R etriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval (2024)

Abstract: Conversational search requires accurate interpretation of user intent from complex multi-turn contexts. This paper presents ChatRetriever, which inherits the strong generalization capability of large language models to robustly represent complex conversational sessions for dense retrieval. To achieve this, we propose a simple and effective dual-learning approach that adapts LLM for retrieval via contrastive learning while enhancing the complex session understanding through masked instruction tuning on high-quality conversational instruction tuning data. Extensive experiments on five conversational search benchmarks demonstrate that ChatRetriever significantly outperforms existing conversational dense retrievers, achieving state-of-the-art performance on par with LLM-based rewriting approaches. Furthermore, ChatRetriever exhibits superior robustness in handling diverse conversational contexts. Our work highlights the potential of adapting LLMs for retrieval with complex inputs like conversational search sessions and proposes an effective approach to advance this research direction.

## introduction conversational search is rapidly gaining prominence and reshaping how users interact with search engines to foster a more natural informationseeking experience. at the heart of a conversational search system lie two key components: retrieval and generation @xcite . the retrieval process is tasked with sourcing relevant passages, which the generation component then uses to craft the final response. conversational retrieval plays a crucial role in ensuring the accuracy and reliability of the system responses by providing relevant passages @xcite . compared to traditional ad-hoc web search, conversational retrieval requires an accurate under-how does the bottom of ocean water freeze? standing of the user's real search intent within longer, noisier, and more complex conversational contexts. a "shortcut" approach is to transform the conversational session into a standalone query rewrite, enabling the usage of ad-hoc retrievers for conversational retrieval. however, the additionally introduced rewriting process is hard to directly optimize towards better retrieval, and it also introduces extra search latency from the rewriting step @xcite . in contrast, the end-to-end conversational dense retrieval appears to be more promising, as it directly encodes the original conversational search session and passages into dense representations without additional input processing and can enjoy the efficiency benefit from advanced approximate nearest neighbor search algorithms (e.g. faiss @xcite ). nonetheless, the effectiveness of existing conversational dense retrievers largely trails behind state-of-the-art conversational query rewriting approaches, which leverage large language models (llms). owing to their strong text understanding and generation capabilities, llm-based rewriters @xcite have demonstrated exceptional effectiveness, even outperforming human rewrites. given that llms are inherently generative models, they can naturally serve as a high-quality conversational rewriter just through prompting (figure 1 ). the question that remains is: whether the potent capabilities of llms can be harnessed to substantially enhance the performance of conversational dense retrievers. several studies have explored tuning llms for dense retrieval but with a primary focus on ad-hoc search @xcite @xcite @xcite . while in conversational search, the multi-turn sessions exhibit greater diversity, complex expressions, and longer-tail intents compared to singleturn ad-hoc queries, posing severe challenges to the session representation learning. additionally, these approaches often rely on manually designed and fixed instruction templates, which can considerably limit their ability to generalize and handle intricate conversational scenarios. in this work, we propose adapting llm itself to serve as a powerful conversational dense retriever. to achieve this, we select high-quality conversational instruction tuning data @xcite as our training data and propose a simple dual-learning approach called contrastive session-masked instruction tuning (csit) for the model training. specifically, we adopt the classical contrastive ranking loss function @xcite to fine-tune llm from a generative model to a retrieval (or representational) model on the multiturn instruction (i.e., session)-response pairs, using the special tokens at the end of the input text to represent the entire text. meanwhile, we mix the basic contrastive learning with a session-masked instruction tuning objective, where we mask all tokens except the special tokens of the session when computing the language modeling loss of the response tokens. the incorporation of this generative instruction tuning loss forces a strong enhancement in the learning of the complex session representation since the response tokens have to be generated solely based on the special tokens representing the session. furthermore, it also helps retain the strong generalization capability of llm for retrieval. our resulting model, which we call chatretriever, can inherit the strong generalization capability of llm to robustly represent complex conversational sessions for dense retrieval. we conducted extensive experiments across five conversational search benchmarks, where chatretriever substantially outperforms existing conversational dense retrievers. notably, it achieves absolute ndcg@3 improvements of 6.8% and 12.2% on cast-20 and cast-21, respectively, matching the performance of the leading llm-based conversational query rewriting methods. beyond standard evaluations using fixed conversational trajectories, we also developed two robustness evaluation methods to assess the resilience of conversational retrieval approaches by altering the historical context. cha-tretriever demonstrates markedly more stable performance in our robustness test, showcasing its superior robustness in comparison to baselines when faced with varied contexts. our contributions can be summarized as: ## related work conversational search has seen the development of two primary approaches: conversational query rewriting (cqr) and conversational dense retrieval (cdr). the former approach transforms the conversational search problem into a traditional ad-hoc search problem by reformulating the conversational context into a standalone query. techniques in this area range from selecting useful tokens from the context @xcite to training generative rewriters based on session-rewrite pairs @xcite @xcite . inspired by the strong language generation capability of llms, some studies @xcite @xcite propose to leverage llms as query rewriters and achieve amazing performance. conversational dense retrieval (cdr), on the other hand, directly encodes the entire conversational session for end-to-end dense retrieval @xcite . efforts in this direction have focused on improving session representation through various perspectives such as context denoising @xcite @xcite , data augmentation using other corpus and llms @xcite @xcite @xcite , and hard negative mining @xcite . llm-based and instruction-aware retrieval. existing research has demonstrated that similar to the scaling laws @xcite observed in llms, increasing the scale of models, data, and computing resources can also enhance the performance of retrieval models @xcite . to incorporate the ability to follow instructions into retrievers, some studies @xcite propose the creation of fixed instruction templates for various retrieval tasks, and use these instruction-enhanced datasets to train the retrievers. moreover, there have been efforts to adapt llms for retrieval purposes by training on improved search data @xcite or developing new search-oriented training objectives @xcite . however, these approaches often rely on manually designed and fixed instruction templates, which can limit the generalization capabilities of the retrievers across diverse instructions. additionally, they are typically designed for single-turn ad-hoc search, lacking the capability to comprehend long and complex search sessions. in contrast to llms, which can smoothly understand a wide range of complex user inputs, existing llm-based retrievers still exhibit a large gap in their generalization capabilities, particularly in the context of conversational search. ## methodology we describe our simple and effective dual-learning approach, contrastive session-masked instruction tuning (csit), which is designed to adapt llm to a generalized and robust conversational dense retriever. an overview is shown in figure 2 . ## influence of number of special tokens in figure 4 , we present the performance of chatretriever when varying the number of special tokens used for text representation. our findings suggest that the inclusion of additional special tokens generally enhances retrieval performance. this improvement may be attributed to the fact that a sequence of consecutive special tokens can serve as a form of representational-level cot, effectively expanding the learning space. however, we observe that performance plateaus when the number of special tokens exceeds three. consequently, we finally append three special tokens in our implementation. ## conclusion in this paper, we introduce chatretriever, a large conversational retrieval model adapted from llm. we propose a novel contrastive session-masked instruction tuning approach for this adaptation and fine-tune llm on high-quality conversational instruction tuning data. experimental results on five conversational retrieval datasets demonstrate the superior performance and robustness of chatretriever. looking ahead, we aim to further explore and expand the generalization capabilities of cha-tretriever in a broader range of complex ir scenarios beyond conversational search, such as legal case retrieval, product search, and other instructionfollowed search tasks. we envision chatretriever to be as versatile as llms, capable of accepting and understanding any conversational inputs and retrieving useful information for those inputs.