M r R ank: Improving Question Answering Retrieval System through Multi-Result Ranking Model (2024)

Abstract: Large Language Models (LLMs) often struggle with hallucinations and outdated information. To address this, Information Retrieval (IR) systems can be employed to augment LLMs with up-to-date knowledge. However, existing IR techniques contain deficiencies, posing a performance bottleneck. Given the extensive array of IR systems, combining diverse approaches presents a viable strategy. Nevertheless, prior attempts have yielded restricted efficacy. In this work, we propose an approach that leverages learning-to-rank techniques to combine heterogeneous IR systems. We demonstrate the method on two Retrieval Question Answering (ReQA) tasks. Our empirical findings exhibit a significant performance enhancement, outperforming previous approaches and achieving state-of-the-art results on ReQA SQuAD.

## introduction large language models (llms) have demonstrated proficiency in comprehending human language across diverse domains. among these advancements, question answering (qa) systems integrated with llms have gained significant attention. nonetheless, concerns persist regarding the susceptibility of llms to hallucinations, potentially leading to inaccuracies in their responses. to address the aforementioned challenge, the retrieval augmented generation (rag) @xcite is proposed. the rag system generally consists of two parts: an information retrieval (ir) engine which proposes the top k candidate pieces from a document corpus for the query, and the qa engine which is responsible for deriving the final answer from retrieved information. although llms demonstrate exceptional proficiency in qa tasks, contemporary text embedding techniques employed in ir remain underdeveloped. this deficiency poses a performance bottleneck for rag, as shown by the results in the massive text embedding benchmark @xcite . to improve retrieval performance, advanced finetuning techniques applied to individual retrieval systems @xcite or the enhancement of model components @xcite emerge as a prominent area of investigation. while finetuning approaches have demonstrated efficacy in this domain, they commonly require substantial resources for training, posing a challenge to realworld adoption. combining results from multiple ir systems can be quite effective, as demonstrated in the work of @xcite , reciprocal rank fusion (rrf) is a technique that utilizes ranking information to rerank the overall ranking, which has become the industry standard for combining multiple ranking systems. although the implementation of rrf is straightforward, a notable limitation arises due to its failure to leverage output scores from individual retrievers. modern techniques have been developed to overcome this limitation; flexneuart @xcite employs the learning to rank (ltr) framework to amalgamate scores from diverse retrievers into a unified score. however, the application of ltr in the qa retrieval setting requires a modification of the training process to avoid irrelevant samples. moreover, the exploration and optimization of these techniques within the domain of question answering retrieval settings remain largely unaddressed. in the work of @xcite , a routing method for combining multiple retrievers in the qa setting is introduced. this model employs a routing classifier to select between a neural retriever and a traditional bm25 retriever. however, this method heavily relies on the score generated by bm25 and disregards ranking information from the neural model. consequently, its performance is limited in modern question answering systems where the neural model typically outperforms bm25. we propose re-ranking techniques specifically trained to improve question answering retrieval performance. our investigations involve combining two different types of model architecture (term weighting and neural networks). additionally, we conducted experiments on two distinct styles of reqa datasets to demonstrate that combining multiple models using this setup can leverage the advantages of both models through ranking loss objective training. our findings show that the proposed methodology yields enhanced retrieval performance and ultimately improves the downstream qa task. our approach outperforms the current state-of-the-art @xcite on reqa squad, surpassing all individual retrieval models, rrf, and the statistical routing strategy, yielding an average enhancement of 13.6% in the mean reciprocal rank (mrr) across datasets. ## proposed method to construct a cohesive framework for merging multiple sets of ranking information, we adopt a re-ranking approach. we have designated a neural-based model as the primary retriever, complemented by bm25 as a supporting model. this choice is based on the observed consistency of the neural-based model, which demonstrates superior performance on r@10. this metric is particularly crucial for assessing the upper limits of performance gain in re-ranking performance. the rankings obtained from the main retriever serve as the initial rankings, subsequently augmented by incorporating scores derived from the support retriever. these concatenated scores are then fed into the re-ranker model to predict the final rankings. the overall process is illustrated in figure 1 . the frameworks consist of two stages: 1) retrieval stage: off-the-shelf retrievers are used in order to generate a candidate pool. we select top candidate retrievers from mteb benchmarks within the retrieval and re-ranking category. the performance is shown in ## results we assess different retrieval models, including bm25, use-qa, sgpt-5.8b-msmarco and mpnet-qa @xcite . our baselines include average scoring, thresholding, and rank reciprocal fusion (rrf), a technique commonly employed within modern multi-retriever systems. we compare our proposed approach mainly on the reqa squad and reqa nq, aligning with the original reqa paper. additionally, we conduct experiments on seven question answering datasets from mteb to assess its effectiveness in contemporary retrieval tasks. the stanford question answering dataset (squad) @xcite consists of questions on a set of wikipedia articles, whereas natural questions (nq) @xcite involves user questions issued to google search, with corresponding answers sourced from wikipedia by annotators. these two datasets represent distinct characteristics related to lexical overlap between questions and contexts, with squad representing a low-overlap scenario while nq exhibits the opposite. the publicly available version of both datasets consists of two partitions: a training set and a development set. the number of questions and documents is shown in the development set was used as the evaluation set for both datasets. we focused on the sentencelevel retrieval task, following the method outlined in the original reqa paper. however, our implementation of bm25 included the complete vocabulary of the training documents, deviating slightly from the original reqa paper which only used the first 10,000 questions. this difference resulted in better performance for our bm25 implementation. see "bm25 retriever" in section 3.1 for additional details. ## conclusion in this paper, we propose an effective method to combine term weighting and neural-based reqa systems. our approach involves training re-ranker neural networks using a modified pairwise reranking objective. through empirical evaluations, the combined system exhibits a significant performance improvement over other combining strategies, particularly when confronted with significant performance disparities among the individual models. it is noteworthy that this technique is not restricted to a specific type or quantity of retrieval models. our method focuses on improving the qa system by enhancing the retrieval which we identified as a bottleneck. further investigation into improving the overall performance by directly optimizing the reader along with the retrieval model might improve the overall pipeline.