R a F e: Ranking Feedback Improves Query Rewriting for RAG (2024)

Abstract: As Large Language Models (LLMs) and Retrieval Augmentation Generation (RAG) techniques have evolved, query rewriting has been widely incorporated into the RAG system for downstream tasks like open-domain QA to enhance document retrieval by reformulating queries. Many works have attempted to improve query rewriting in smaller models to avoid rewriting with costly LLMs, and the most common method is to employ reinforcement learning for feedback training. However, current methods require annotations (labeled relevant documents or downstream answers) or predesigned rewards for feedback, lack generalization, and fail to utilize signals tailored for query rewriting. In this paper, we propose RaFe, a framework for training query rewriting models. By leveraging reranker, RaFe provides ranking feedback aligned well with the rewriting objectives without needing signals from annotations and supports both online and offline training models. Experimental results demonstrate that with a general and publicly available reranker, RaFe can effectively steer the training for rewrite models.

## introduction large language models (llms) have demonstrated strong capacities to solve a variety of tasks @xcite . however, they still encounter the challenges of hallucinations @xcite @xcite or outdated knowledge @xcite . recently, retrieval augmentation generation (rag) @xcite has become an important technology to enhance llms' abilities, by incorporating external knowledge. for instance, in open-domain qa, llms can first retrieve related documents and then generate answers. nonetheless, directly retrieving by original query does not always achieve correct and relevant documents. therefore, query rewriting @xcite has been widely employed to reformulate the query to expand the retrieved documents for a better response as illustrated in figure 1 . many efforts have been proposed to leverage the powerful llms to generate rewrites @xcite directly. while in practical applications, it is more prevalent to implement specific small query rewriting models to avoid the costly use of llms @xcite . to improve the performance of query rewriting, reinforcement learning (rl) with feedback @xcite can be utilized as a typical solution. for instance, nogueira and cho ( 2017 note that these feedback-driven query rewriting methods rely on either annotated labels such as relevant documents or answers, or pre-designed rewards tailored to specific domains. however, they often lack the utilization of effective and general signals for query rewriting. meanwhile, consider-able efforts have been made to harness diverse feedback mechanisms across various domains @xcite . notably, @xcite effectively integrates unit testing feedback into code generation, yielding significant efficacy. drawing from these, in this paper we attempt to (i) reduce the cost of annotations for feedback; and (ii) identify a signal that better aligns with the objectives of the query rewriting task. to address these issues, we introduce rafe (ranking feedback improves query rewriting), a novel framework that leverages feedback from the reranker to train query rewriting models. this approach is inspired by the reranker module in traditional information retrieval (ir) systems, which scores and sorts retrieved documents based on the query. intuitively, query rewriting aims to retrieve documents relevant to the original query, which aligns perfectly with the goal of the reranker. specifically, the reranker is capable of scoring documents without requiring additional labels. thus, we incorporate a reranker to provide feedback for the query rewriting model. rafe comprises a two-stage process. we first train an initial query rewriting model by standard supervised fine-tuning. subsequently, we utilize the ranking scores from the reranker to conduct feedback training on the query rewriting model. rafe supports both offline and online rl feedback training. empirically, we demonstrate that utilizing a general, publicly available reranker, rafe can drive the training of the query rewriting model, indicating the effectiveness and potential generalizability of the proposed approach. the main contributions of our paper can be summarized as follows: ## experimental setup as we attempt to improve query rewriting for better rag, we conduct our experiments on the typical rag scenarios, open-domain question answering (odqa). the process of rag for odqa can be formulated as f([d : q]), where f denotes the llms, q is the original query from datasets and d is the documents concatenated for augmentation. ## conclusion and future work this paper proposes a novel feedback training framework named rafe for query rewriting, based on the effectiveness of the reranker in enhancing document ranking during the information retrieval process. by leveraging the feedback signals from reranker, rafe is capable of effectively and generally conducting feedback training for rewrite models, yielding great improvements. experimental results indicate that our method achieves exemplary performance across cross-linguistic datasets. in the future, we plan to conduct joint training of reranker and rewrite models, which may yield substantial benefits for rag.