Bridging the Preference Gap between Retrievers and LLM s (2024)

Abstract: Large Language Models (LLMs) have demonstrated superior results across a wide range of tasks, and Retrieval-augmented Generation (RAG) is an effective way to enhance the performance by locating relevant information and placing it into the context window of the LLM. However, the relationship between retrievers and LLMs in a RAG is still under-investigated. Most existing work treats the retriever and the LLM as independent components and leaves a gap between retrieving human-”friendly” information and assembling a LLM-”friendly” context. In this work, we examine a novel bridge mechanism. We validate the ranking and selection assumptions of retrievers in the context of RAG and propose a framework that chains together supervised and reinforcement learning to train a bridge model that optimizes the connection between the retriever and the LLM. Empirical results demonstrate the effectiveness of our method in both question-answering and personalized generation tasks.

## introduction large language models (llms) such as @xcite and palm 2 @xcite , have demonstrated impressive performance on a wide variety of tasks. retrieval-augmented generation (rag), which retrieves knowledge items from an external data source and puts it into the context window of llms, has produced significantly enhanced results in many nlp tasks @xcite @xcite . however, most works on rag study retrievers and llms separately. on one hand, most retrievers are designed to be human-friendly, usually based on the general belief in classic information * the work was done during internship at google research. we empirically investigate this preference gap, focusing specifically on ranking and selection. as shown in fig. 1 , when we randomize the ordering of top-5 retrieved items (in our case, passages), the performance of rag only varies by around 1% foot_0 . however, the variation exceeds 5% when the llm is only presented with the top-1 passage under each order (therefore it encounters different selections of information). this indicates the general belief in ranking does not always apply to llms, and that the selection of information could be more crucial. this finding confirms the existence of the preference gap between retrievers and llms, and it is critical to bridge this preference gap to enhance the performance of rag. to the best of our knowledge, this is a novel insight that may guide future designs of rag systems. existing work has tried to finetune the llms to align with the retriever or adjust the retriever to align with the llm. however, finetuning llms, especially at the scale of gpt-4 or palm 2, is often expensive. similarly, it is difficult to adjust production-level retrievers such as google or bing. even when the retriever is adjustable, existing efforts often focus on re-ranking the retrieved results and fail to address other aspects of preference such as selection or repetition. instead, we propose a novel and practical framework called bgm (bridging the gap between retrievers and llms), which keeps the retriever and llm fixed and trains a bridge model in between. the bridge model aims to transform the retrieved information into a format that llms prefers and can effectively work with. without loss of generality, we structure the bridge model as a sequence-to-sequence (seq2seq) model, which allows dynamically selecting items, re-ranking them, and potentially broader operations like repeating some of them in the retrieval-augmented prompt. training such a bridge model is challenging as there are usually no ground truth labels on ideal item sequences for retrieval augmentation. existing work has tried to derive supervisory signals for ranking from rag's downstream task performance, such as perplexity distillation @xcite . nevertheless, these methods only provide pointwise supervisory signals for each item independently. directly applying the same idea to obtain sequential supervision is infeasible, since it would require feeding all possible item sequences into the llm to obtain perplexity. we developed a greedy search approach to solve this problem (sec. 4.1). moreover, we find sequential supervision can be too sparse to effectively train such a seq2seq model ( it is important to note that bgm transcends the limitations of traditional re-rankers. typically, rerankers most often overlook the selection process (i.e., they simply return the top-k results) and score passages independently, without considering collective properties such as result diversity. in contrast, bgm addresses both ranking and selection jointly, and enhances collective aspects like diversification. furthermore, bgm often reduces the api cost of the llm, and decreases its overall runtime. this is because bgm significantly reduces the number of tokens fed into the final llm by selecting relevant passages only. this approach is similar in spirit to a multi-stage retriever (i.e., a retriever plus re-ranker), which is more cost-effective. our experiments reveal that bgm can enhance the performance of various downstream tasks, such as question answering (qa) and personalized generation, across a spectrum of datasets, from public qa and amazon reviews to private email conversations. notably, the modified passages retrieved by bgm surpass the performance of strong retrievers and baseline reranking models. this underscores the significance and promise of the "bridge" approach in the realm of rag. in summary, our contributions can be summarized as follows: ## related work retrieval-augmented generation (rag). augmenting llms with relevant information retrieved from various knowledge sources is proven effective across numerous nlp tasks, including language modeling @xcite @xcite , question answering @xcite @xcite @xcite @xcite , fact versification @xcite ) and text generation @xcite . specifically, rag utilizes input as a query and comprises two main components: (1) a retriever retrieves a set of items from a side corpus. particular items may vary across different tasks, including documents, passages, or even tokens. in this study, we focus on retrieving passages; and (2) a llm incorporates the retrieved items, as additional information in the input context, and makes final predictions. a fundamental question in this process arises regarding the disparate preferences between llms and retrievers, as llms performing optimally only when their preferences are satisfied. bridging the preference gap is crucial. depending on which components are subject to updates, this challenge can be categorized into three families. finetuning retrievers and llms jointly. this is the most widely used setting of rag @xcite @xcite @xcite . however, most prior work is based on relative small lms (< 1b). for example, altas @xcite finetunes llm (t5 @xcite ) and retriever (contriever @xcite ) jointly by leveraging the llm to provide supervisory signal to train the retriever. rag @xcite uses a tunnable query encoder and dpr @xcite as retriever, bart as llm, and design an end-to-end schema to train the query encoder and the llm. finetuning llms only. updating retrievers is not always desirable as it is costly and requires the document index to be periodically updated. to bridge the preference gap, it is also possible to only update the llms. fid @xcite takes the retrieved documents and query as input, finetunes the llm to adapt to the external information. similarly, lummen @xcite and @xcite improve fid via adding reranker and pre-encoding memeory. ## problem formulation retriever. given an input x, the retriever aims to retrieve a ranked list of passages from a corpus @xmath0. in this work, we assume a typical scenario of employing a frozen dense retriever. typically, a dual encoder architecture is applied, where an encoder is used to encode both the input context x and the passage d. specifically, the encoder maps each passage to an embedding e(d). the similarity between input and passage embedding is computed by their cosine similarity, the top-k passages that have the highest similarity scores when compared with the input x are retrieved in this step, bridge model for rag. the retrieved top-k passages provide richer information about the original input/query x to help the llm to make a better prediction on downstream tasks. a bridge model b adapts the retrieved passages to a sequence of passages that is llm-friendly. as mentioned in the sec. 1, the bridge model is a seq2seq model. it takes all the retrieved passages (d retr. this formulation is general enough as the seq2seq model automatically considers ranking by generating the next token based on the preceding one, selection by placing the end-of-sentence token in the appropriate position, and repetion by generating the same passage id (as explained in the following paragraph). note that n may be smaller or larger than k due to selection and repetition. before concatenating the query and passages as bridge model's input, we prepend each passage with a unique sentinel token as its passage id, e.g., [query][id 1 ]d retr. retrieval-augmented generation with bridge. we concatenate adapted passages from the bridge model, (d bdr. j ) n @xmath1, with the input x, and fed the resulting long sequence into the llm as context to obtain the output for downstream tasks. ## conclusion this paper demonstrates the need to bridge the preference gap between retrievers and llms, which has various levels of impact on ranking and selection in rag systems. we propose a bridge model, bgm, to adapt the output of a frozen retriever for frozen llms, formatting the task as a seq2seq problem. bgm chains supervised and reinforcement learning, for dense supervisions and end-to-end training. our extensive experiments validate bgm's effectiveness.