DSLR : Document Refinement with Sentence-Level Re-ranking and Reconstruction to Enhance Retrieval-Augmented Generation (2024)

Abstract: Recent advancements in Large Language Models (LLMs) have significantly improved their performance across various Natural Language Processing (NLP) tasks.However, LLMs still struggle with generating non-factual responses due to limitations in their parametric memory.Retrieval-Augmented Generation (RAG) systems address this issue by incorporating external knowledge with a retrieval module.Despite their successes, however, current RAG systems face challenges with retrieval failures and the limited ability of LLMs to filter out irrelevant information.Therefore, in this work, we propose ( D ocument Refinement with S entence- L evel R e-ranking and Reconstruction), an unsupervised framework that decomposes retrieved documents into sentences, filters out irrelevant sentences, and reconstructs them again into coherent passages.We experimentally validate DSLR on multiple open-domain QA datasets and the results demonstrate that DSLR significantly enhances the RAG performance over conventional fixed-size passage.Furthermore, our DSLR enhances performance in specific, yet realistic scenarios without the need for additional training, providing an effective and efficient solution for refining retrieved documents in RAG systems.

## introduction recent advancements in large language models (llms) @xcite @xcite have significantly expanded their capabilities across diverse knowledge-intensive tasks in natural language processing (nlp), such as question answering (qa) @xcite @xcite . however, despite these capabilities, llms still face challenges such as generating plausible yet non-factual responses, known as hallucination, due to their reliance on limited parametric memory * corresponding author @xcite . also, it is noted that this parametric memory is static, as llms can learn knowledge only up to the specific date on which the training was completed. therefore, these limitations restrict their adaptability to long-tailed or ever-evolving domains @xcite and to unseen knowledge outside their training data @xcite . retrieval-augmented generation (rag) @xcite @xcite has been introduced as an effective solution to address such problems. specifically, rag enhances llms by integrating non-parametric memories fetched from external knowledge bases using a retrieval module, which helps llms' responses grounded on factual evidence and makes them more up-to-date. while the efficacy of rag depends on the performance of the retrieval module, the instability of llms in incorporating the retrieved knowledge is also a critical challenge to rag. to be specific, retrieved documents sometimes contain irrelevant information @xcite , and llms often struggle to effectively filter out such redundant details and focus on the most query-relevant knowledge @xcite @xcite , which leads to the failure of the overall rag systems. therefore, it is crucial to investigate how to effectively refine retrieved documents before augmenting them with llms, ensuring that the llms are not distracted by irrelevant information within retrieved documents. re-ranking the order of the retrieved document set @xcite or refining them into new documents @xcite can be considered as solutions. however, they generally require high computational costs for training additional re-ranking or refining models. another proposed solution is to reduce the retrieval granularity from passage-level to sentence-level which can help eliminate redundant information within passages @xcite . however, this might also inadvertently remove important contextual information, which is crucial for accurately answering the given queries @xcite . therefore, we should explore a novel method that can effectively and efficiently filter out irrelevant information while maintaining the necessary contextual details. in this work, we introduce an unsupervised dslr (document refinement with sentence-level re-ranking and reconstruction) framework that consists of three steps: 1) decomposition, 2) reranking, and 3) reconstruction. specifically, after retrieving the passage-level document, the dslr framework operates by first decomposing the retrieved document into sentences for finer granularity and then filtering out the irrelevant sentences based on their re-ranking scores from the ranking models, including off-the-shelf retrievers and re-rankers. finally, the remaining sentences are reconstructed into a single document to preserve the original contextual information. note that dslr is an unsupervised refinement framework, which does not require any additional training for re-ranking or reconstruction steps. the overall dslr framework is illustrated in figure 1 . we validate our framework across a diverse range of open-domain qa benchmarks, which include three general qa datasets and three specific qa datasets that require domain-specific or ever-evolving knowledge. our experimental results show that dslr significantly enhances the overall rag performance and is comparable to, or even outperforms, the supervised baseline approaches. specifically, when evaluated with specific qa datasets, dslr shows high robustness in realistic settings. furthermore, a detailed analysis demonstrates the effectiveness of each proposed step and how it contributes to the overall performance. our contributions in this work are threefold: in order to further enhance retrieval performance, additional strategies have been proposed. specifically, the re-ranking strategy improves retrieval per-formance by recalculating relevance scores using an additional re-ranking model (nogueira and @xcite @xcite , and then reordering the documents based on these scores. recently, llms have shown remarkable re-ranking performance by generating relevance labels without requiring further fine-tuning @xcite . while the aforementioned work on ir @xcite generally assumes fixed-size, 100-word passages as the document length, some work has explored an optimal level of retrieval granularity @xcite @xcite . these approaches validate that a finegrained level of granularity, containing only the knowledge needed to answer the query, can enhance the overall performance by excluding redundant details in the lengthy retrieved documents. however, reducing retrieval granularity to the sentence level can disrupt the original context and result in a loss of the document's coherence @xcite . in addition, sentence-level retrieval generally requires a much larger index size compared to passage-level retrieval @xcite . by contrast, we investigate a novel framework for effectively re-ranking sentences within retrieved passage-level documents and then reconstructing the re-ranked sentences to preserve contextual integrity. retrieval-augmented generation. rag has emerged as a promising solution for addressing llms' hallucination issues by leveraging external knowledge fetched by the retrieval module. specifically, rag incorporates retrieval modules that reduce the need to update the parameters of llms and help them generate accurate and reliable responses @xcite @xcite . additionally, various real-world applications integrate rag as a core component when deploying llmbased services @xcite @xcite . however, they still have limitations due to the imperfections of the retrieval module within rag, where the retrieved documents containing query-irrelevant information can negatively lead the llms to generate inaccurate answers. to address them, several studies have attempted to leverage the capabilities of llms to enhance their resilience against irrelevant knowledge. these approaches include crafting specialized query: how many episodes in "grace and frankie" season 1? ## method in this section, we describe a novel framework dslr for enhancing the precision of retrieval results through sentence-level ranking and reconstruction, integrated into the rag system. note that dslr does not require additional training. ## experiment setups in this section, we describe the experimental setup for evaluating dslr across various scenarios. we provide additional details in appendix a. ## experimental results and analyses in this section, we show the overall experimental results with in-depth analyses of our framework. main results. first of all, dslr in multiple passages. to assess the effectiveness and efficiency of dslr in multiple passages, we gradually increased the number of documents n and compared the performance, token count, and end-to-end (e2e) latency foot_1 of the original top-n documents with those refined by dslr. as shown in the left panel of figure 3 , both sets of documents show consistent performance improvements as n increases. however, dslr consistently outperforms the original documents across all n levels, with more notable differences 10 20 30 40 50 60 70 80 at lower n values. this suggests that dslr can significantly enhance performance in rag, even as the number of documents increases. due to the quadratic increase in memory and time requirements with the number of tokens in transformer-based llms, reducing the token count is crucial for improving efficiency @xcite . as depicted in the center and right panels of figure 3 , dslr substantially reduces the token count compared to the original documents, with the difference becoming more significant as n increases. this reduction in tokens also decreases e2e latency in all scenarios except top-1. notably, at top-10, while the performance difference is minimal (39.6 vs. 39.7), the token count reduction from 1,713 to 577 (nearly 2.97 times) and the corresponding e2e latency reduction from 7.382 seconds to 5.422 seconds (nearly 2 seconds) demonstrate that dslr can enhance both performance and efficiency in rag. detailed results are available in impact of threshold adjustment. to examine the impact of varying t , we adjusted the threshold in increments of 10, starting from the 10th percentile, and measured the resulting performance. additionally, to explore the theoretical maximum performance of our method, we configured an oracle setting where any correct response, regardless of the threshold setting, was counted as correct. as shown in figure 4 , increasing the threshold t generally improves performance by removing irrelevant content, thus reducing the number of tokens. however, our experimental results revealed that the performance at the 90th percentile threshold was 29.4, while a lower 80th percentile threshold yielded better performance at 29.9. this indicates that an overly stringent threshold can also remove essential information, suggesting that task-specific threshold fine-tuning could improve results. bm25 contriever dpr monot5 rankt5 rg 40 45 50 acc passage sentence figure 6: comparative average performance of sentence-level and passage-level re-ranking across the dataset with a context length of 100 words. furthermore, in the oracle setting, accuracy significantly improved to 34.1, and the token count was reduced to 77. this shows a marked performance improvement over the best performing threshold (80th percentile), with a similar reduction in tokens. this result implies that dynamically adjusting the threshold based on the query could achieve substantial performance improvements with a comparable number of tokens, suggesting an area for future work. detailed results are available in ## conclusion in this work, we present dslr, a novel unsupervised document refinement framework that enhances the performance of rag systems. the dslr framework aids rag systems to generate more accurate answers by decomposing passages into sentences, re-ranking them based on each relevance score, and then reconstructing them to preserve the continuity and coherence of the context. our comprehensive experiments on multiple qa datasets show that dslr consistently outperforms the conventional approaches of using fixedsize passage in rag, especially in ever-evolving and domain-specific contexts. our ablation studies highlight the importance of sentence-level reranking and contextual reconstruction for improvement on rag. we believe that dslr suggests a promising research direction for refining document retrieval without additional training, together with potential applications across a wide range of knowledge-intensive nlp tasks by integrating more diverse retrieval or ranking models.