Document Segmentation Matters for Retrieval-Augmented Generation (2025)

Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) by integrating external knowledge. A critical yet underexplored challenge in RAG is document segmentation, also known as document chunking. Existing widely-used rule-based chunking methods usually lead to suboptimal splits, where overly large chunks introduce irrelevant information and small chunks lack semantic coherence. Existing semantic-based approaches either require costly LLM calls or fail to adaptively group contextually related sentences. To address these limitations, we propose PIC, Pseudo-Instruction for document Chunking), a simple yet effective method that leverages document summaries as pseudo-instructions to guide chunking. By computing semantic similarity between sentences and the summary, PIC dynamically groups sentences into chunks that align with the document’s key themes, ensuring semantic completeness and relevance to potential user instructions. Experiments on multiple open-domain question-answering benchmarks demonstrate that PIC can significantly improve retrieval accuracy (Hits@k) and end-to-end QA performance (Exact Match) without any additional training.

## introduction retrieval-augmented generation (rag) integrates large language models (llms) with large-scale textual knowledge to enhance response quality and mitigate hallucinations in llms @xcite @xcite @xcite @xcite . by dynamically retrieving real-time information from databases such as wikipedia, rag enables more relevant and accurate responses. rag has been widely applied across domains such as law @xcite , medicine @xcite , and finance @xcite , making it an essential component in real-world applications. a standard rag pipeline comprises three key stages: (1) document segmentation into manageable units, named chunks; (2) knowledge retrieval for input instructions; (3) response generation based on retrieved chunks. while existing research mainly focuses on retriever improvements @xcite and system architecture design @xcite , the foundational challenge of document segmentation remains critically underexplored. effective document segmentation is crucial, as lengthy source documents in knowledge bases like wikipedia exceed the processing capacities of both retrievers and llms. the majority of chunking methods employed in rag systems are rule-based, typically relying on heuristics such as fixed-length segmentation @xcite @xcite . however, these methods usually suffer from inherent limitations in selecting appropriate chunk sizes. larger chunks may include irrelevant information, which can both interfere with retrieval and generation, leading to hallucinations. conversely, smaller chunks may lack sufficient semantic information, making it challenging for the llm to generate accurate and coherent responses. to address the above issues, certain studies have explored semantic-based chunking methods. some research attempts to rewrite documents. for example, @xcite tries to transform the original text into independent, contextualized, and self-contained segments. this approach can integrate contextual knowledge, but it still relies on rule-based context window segmentation and fails to consider the adaptive combination of sentences. other studies attempt to identify semantic breakpoints. greg (2024) seperates adjacent sentences with the least semantic similarity, yet it lacks a global consideration of the entire document's con-text, leading to suboptimal chunking. @xcite and @xcite leverage llms' understanding of semantics for segmentation, but these models require extensive calls to large models, making them costly, time-consuming, and difficult to apply to large-scale text knowledge bases. we aim to explore a new perspective on what constitutes good chunking. when a user inputs an instruction, our goal is to ensure that sentences relevant to the instruction are grouped within the same chunk, while unrelated sentences are placed in other chunks. this approach improves retrieval accuracy and prevents unrelated information from affecting llm generation. however, in practical rag applications, predefining chunks based on specific instructions is infeasible, as the instruction distribution is unknown when constructing chunks. moreover, dynamically resegmenting a document for each instruction would incur unacceptable computational overhead. to address the challenge of document segmentation without prior knowledge of real user instruction distributions, we propose pic (pseudo-instruction for document chunking), which utilizes document summaries as pseudo-instructions to guide segmentation. by generating a summary that preserves key document information (often central to real-world user queries), we compute semantic similarity scores between each sentence and the summary, then group sentences with scores above/below the average similarity threshold into coherent chunks. this approach ensures semantic completeness within chunks while isolating irrelevant content, thereby improving retrieval relevance and reducing hallucinations in llms. the method aligns chunk distributions with potential user instructions by prioritizing summary-aligned content aggregation and minimizing unrelated information within individual chunks. to evaluate the effectiveness of different chunking methods, we process the latest english wikidump, segmenting each document into chunks to serve as the knowledge base for rag. we conduct experiments on multiple open-domain question-answering benchmarks, where our proposed method outperforms existing chunking approaches in terms of retrieval performance (hits@k) and end-to-end qa performance (exact match). the results demonstrate that pic achieves promising performance improvements across various experimental settings, proving to be an effective and generalizable chunking method. our main contributions are summarized as follows: ## methodology in the context of rag, an effective chunking strategy is critical for both retrieval and generation performance. given a document @xmath0, s 2 , ..., s n } containing n sentences, we aim to segment d into a set of semantically coherent chunks @xmath1, c 2 , .., c m }, where each chunk c i consists of one or more combined sentences, and m represents the total number of chunks. our method consists of two key steps: 1) generating pseudo-instructions. 2) performing semantic chunking based on pseudo-instructions, as shown in figure 1 . we will provide a detailed explanation of both steps below. ## experimental settings to evaluate the performance of different document chunking methods in rag scenarios, we adopt wikipedia as the knowledge corpus and conduct experiments on several knowledge-intensive tasks using two different llms. in this section, we present the details and results of our extensive experiments. ## conclusions in this paper, we identify document segmentation as a critical yet underexplored challenge in retrieval-augmented generation (rag) systems. we propose pic, a novel document chunking method that leverages document summaries as pseudo-instructions to guide semantic chunking. by dynamically grouping sentences based on their relevance to document topic, pic creates chunks that better align with potential user instructions while maintaining semantic coherence. extensive experiments across six qa benchmarks demonstrate that our method achieves significant improvements in both retrieval accuracy and end-to-end qa performance compared to existing approaches, without requiring additional model retraining.