Towards Reliable Retrieval in RAG Systems for Large Legal Datasets (2025)

Abstract: Retrieval-Augmented Generation (RAG) is a promising approach to mitigate hallucinations in Large Language Models (LLMs) for legal applications, but its reliability is critically dependent on the accuracy of the retrieval step. This is particularly challenging in the legal domain, where large databases of structurally similar documents often cause retrieval systems to fail. In this paper, we address this challenge by first identifying and quantifying a critical failure mode we term Document-Level Retrieval Mismatch (DRM), where the retriever selects information from entirely incorrect source documents. To mitigate DRM, we investigate a simple and computationally efficient technique which we refer to as Summary-Augmented Chunking (SAC). This method enhances each text chunk with a document-level synthetic summary, thereby injecting crucial global context that would otherwise be lost during a standard chunking process. Our experiments on a diverse set of legal information retrieval tasks show that SAC greatly reduces DRM and, consequently, also improves text-level retrieval precision and recall. Interestingly, we find that a generic summarization strategy outperforms an approach that incorporates legal expert domain knowledge to target specific legal elements. Our work provides evidence that this practical, scalable, and easily integrable technique enhances the reliability of RAG systems when applied to large-scale legal document datasets.

## introduction large language models (llms) are increasingly adopted in high-stakes domains such as law. yet, they remain critically limited by the phenomenon of hallucination: incorrect outputs that are fabricated or deviate from the provided source material, posing severe risks in legal applications @xcite @xcite . recent studies report hallucination rates between 58-80% for general-purpose llms on legal tasks @xcite , highlighting how factual reliability is not just desirable but essential for deploying llms in the legal domain. this challenge is amplified by forward-looking proposals for how legal documentation itself may evolve. for instance, @xcite suggest that privacy policies may intentionally become longer and more comprehensive to ensure they are legally complete. in such a future, llms are expected to serve as the designated "readers" of these texts, extracting and summarizing information for human users. this vision, however, can only be realized if the systems are highly reliable. retrieval-augmented generation (rag) @xcite ) has emerged as one of the leading approaches to improving reliability. using a trusted text corpus to provide factual evidence, rag guides the llm's output, reducing hallucinations and ensuring closer alignment with the source material @xcite . in the context of long, structurally similar legal documents, identifying the relevant text passage as "needle in the haystack" becomes a top priority that we aim to address. on a technical level, we quantify the retrieval quality with our document-level retrieval mismatch (drm) metric and the character-level precision and recall. then, we investigate a simple yet effective technique to improve retrieval quality, summary-augmented chunking (sac). we enrich text chunks in the trusted text corpus with document-level summaries. this preserves global context, lost in standard chunking, guiding the retriever toward the correct document without altering the underlying retrieval pipeline. this method is applied to question-answering tasks across a diverse set of legal documents, including privacy policies, non-disclosure agreements, and merger-andacquisition contracts. key contributions: (i) first, we define and quantify document-level retrieval mismatch (drm), a key failure mode we observe in standard rag pipelines where the retrieved information originates from the entirely wrong source document. (ii) we propose summary-augmented chunking (sac) as a lightweight and modular solution that strongly reduces drm by injecting global context directly into each chunk. we experimentally validate sac on legalbench-rag @xcite , showing substantial improvements over standard chunking. (iii) additionally, we explore how to utilize legal domain knowledge by evaluating both a generic and an expert-guided summarization strategy. interestingly, we find that simple, generalpurpose summarization yield the best retrieval performance. rag has been adapted for a wide array of legal tasks @xcite , including case reasoning @xcite , legal judgment prediction @xcite , and legal question-answering @xcite . recent approaches have focused on improving reliability by imposing more structure on the knowledge source, for instance, through knowledge graphs @xcite or structured case databases @xcite . despite its demonstrated ability to improve factual accuracy in context-sensitive tasks @xcite , the effectiveness of rag is critically dependent on the quality of the initial retrieval step @xcite . if the retriever fails to select information that is relevant or complete, the generator may produce factually unsupported responses. accordingly, prior studies demonstrate that rag is not a guaranteed solution, as legal rag systems continue to generate a considerable amount of hallucinated content, particularly when the retrieval mechanism is ineffective @xcite @xcite . our work addresses this challenge by focusing on the pre-retrieval stage, the engineering of the knowledge base, which forms the foundation of any reliable rag system. ## experimental setup we evaluate the performance of our methods covering a broad picture of retrieval quality: as our baseline, we implemented a standard rag pipeline using a recursive character splitting strategy with a chunk size of 500 characters and without overlap. the document summaries were generated using gpt-4o-mini @xcite and we instructed and processed the summaries to be about 150 characters long (details in the appendix a). the concatenated texts were embedded with thenlper/gte-large foot_4 @xcite (see appendix c) and indexed in a faiss @xcite vector database with cosine similarity as retrieval metric. dense semantic search excels at capturing conceptual similarity but may overlook exact lexical matches. in contrast, bm25 @xcite is well established for keywordbased retrieval and can be effective for queries with unique identifiers or technical terminology. we therefore experimented with a hybrid dense+sparse retrieval. however, results showed that bm25 (sparse) improved drm but decreased precision/recall while introducing additional computational overhead (see the appendix b), so we decided to only use dense retrieval in the main experiments. for all systems, we report performance across a range of top-k retrieved chunks. this reflects realworld deployment scenarios, where practitioners must balance precision and recall depending on application needs. reporting the full curve enables a more informative assessment of trade-offs across retrieval strategies. ## conclusion we addressed the critical challenge of retrieval reliability in rag systems operating on large, structurally similar legal document databases. we identified and quantified document-level retrieval mismatch (drm) as a dominant failure mode, where retrievers are often easily confused by legal boilerplate language and select text from entirely incorrect documents. targeting this issue, we investigate summary-augmented chunking (sac), a simple and computationally efficient technique that prepends document-level summaries to each text chunk. by injecting global context, sac drastically reduces drm and consequently improves text-level retrieval precision and recall. a key finding is that generic summaries outperform expert-guided ones focusing on key legal variables. for the purpose of guiding retrievers, broad semantic cues appear more robust and generalizable than dense, structured, legally precise summaries. this demonstrates that meaningful retrieval performance gains are achievable without heavy domain-specific engineering. while sac is not a full solution on its own, it offers a practical, scalable intervention for more reliable legal rag systems. by improving the crucial retrieval step for legal information, our work brings us closer to a future where ai can truly serve as a trusted partner in the legal profession. bm25 @xcite is a well-established keyword-based (sparse) retrieval method. a common assumption in rag research is that hybrid approaches, combining dense vector similarity with keyword matching, often yield the best results @xcite . following this intuition, we augmented our dense retriever with a bm25 component, allowing the system to explicitly match salient terms (e.g., party names) from the prepended summaries with the same terms in the user query. results are reported in as expected, adding sparse retrieval improved document selection, reducing our document-level mismatch (drm). however, it lowered text-level precision and recall. a closer inspection suggests a likely explanation: the summaries are highly structured and rich in identifiers, where sparse keyword matching excels, whereas the chunked document bodies contain more natural language and nearly no direct keywords to match. for pinpointing the relevant passage, semantic similarity between the query and text is more informative than keyword overlap, explaining the higher text-level precision and recall without the bm25 component. in conclusion, adding bm25 search only contributed slightly to finding the correct document (improved drm), but tended to result in poorer search results within a document (reduced precision and recall). also considering the computational overhead of the bm25 algorithm, we decided to rely exclusively on semantic search.