L umber C hunker: Long-Form Narrative Document Segmentation (2024)

Abstract: Modern NLP tasks increasingly rely on dense retrieval methods to access up-to-date and relevant contextual information. We are motivated by the premise that retrieval benefits from segments that can vary in size such that a content’s semantic independence is better captured. We propose LumberChunker, a method leveraging an LLM to dynamically segment documents, which iteratively prompts the LLM to identify the point within a group of sequential passages where the content begins to shift. To evaluate our method, we introduce GutenQA, a benchmark with 3000 “needle in a haystack” type of question-answer pairs derived from 100 public domain narrative books available on Project Gutenberg. Our experiments show that LumberChunker not only outperforms the most competitive baseline by 7.37% in retrieval performance (DCG@20) but also that, when integrated into a RAG pipeline, LumberChunker proves to be more effective than other chunking methods and competitive baselines, such as the Gemini 1.5M Pro.

## introduction the rapid expansion of large language models (llms) has paved the way for tackling a wide range of tasks, including translation @xcite , summarization @xcite , and question answering @xcite , among others @xcite . however, a significant issue arises when these models are tasked with generating responses based on information they lack, often resulting in "hallucinations" -responses that, while seemingly plausible, are factually incorrect @xcite . given the broad accessibility of these models, less informed users may accept all generated content as accurate, potentially causing severe misinterpretations and adverse conse-quences, like the recent incident where a lawyer cited fictitious cases produced by @xcite in court, resulting in sanctions for the lawyer and highlighting the severe risks of unverified ai-generated information @xcite . in the field of question answering, where precision and accuracy of information are paramount, retrieval augmented generation (rag) systems present a viable solution to hallucinations by grounding the model's generation on contextually relevant documents @xcite . one often overlooked part of the rag pipeline is how textual content is segmented into 'chunks', which can significantly impact the dense retrieval quality @xcite . real-world applications tend to simplify this step and consider sentences, paragraphs, or propositions as the usual granularity level @xcite . in this paper, we propose lumberchunker, a novel text segmentation method based on the principle that retrieval efficiency improves when content chunks are as independent as possible from one another. this independence is best achieved by allowing chunks to be of dynamic sizes. given that language models excel at analyzing text, we leverage their capabilities to identify optimal segmentation points. specifically, we repeatedly instruct a language model to receive a series of continuous paragraphs and determine the precise paragraph within the sequence where the content starts diverging. this approach ensures that each segment is contextually coherent yet distinct from adjacent segments, thereby enhancing the effectiveness of information retrieval. we introduce a new benchmark: gutenqa, which consists of 100 public domain narrative books manually extracted from project gutenberg foot_1 . we create 3000 high-quality question-answer pairs from these books to evaluate the impact of lumber-chunker on retrieval. finally, we integrate lum-berchunker into a rag pipeline for a downstream qa task to assess its effect on the accuracy of the generated outputs. our main contributions are as follows: ## background the retrieval granularity at which a document is segmented plays an essential role as ineffective chunking strategies can lead to chunks with incomplete context or excessive irrelevant information, which damage the performance of retriever models @xcite . beyond typical granularity levels like sentences or paragraphs @xcite , other advanced methods can be employed. recursive character splitting @xcite ) segments text based on a hierarchy of separators such as paragraph breaks, new lines, spaces, and individual characters. while this method better respects the document's structure, it may lack contextual understanding. to address this, semantic-based splitting @xcite utilizes embeddings to cluster semantically similar text segments. this approach ensures that chunks maintain meaningful context and coherence by identifying breakpoints based on significant changes in embedding distances. the recent work by @xcite introduces a novel retrieval granularity termed propositionsminimal textual units, each conveying an individual fact in a clear, self-sufficient natural language format. while this concept is valid for contexts with fact-based information, like wikipedia, it may be less effective for narrative texts where the flow and contextual continuity play a critical role (as illustrated in appendix a). furthermore, the shift in the search space from the original text to these new representations also places considerable reliance on the generating llm to ensure their quality. retrieval granularity is often viewed at the document level, but it can also involve adjusting the query itself. @xcite suggests the hypothetical document embeddings (hyde) method, where an llm transforms the query into a potential answer document. ## experiments we evaluate lumberchunker using a series of diverse experiments. the key questions that guide our experimental evaluation are as follows: the number of tokens in the prompt provided to the llm directly correlates with the token count in the resulting chunk. intuitively, one might hypothesize that increasing the prompt length reduces retrieval difficulty, as larger chunks reduce the overall search space, thereby increasing the likelihood of retrieving the correct chunk. to test this hypothesis, we analyze how lumberchunker's dcg@k scores, where k is the number of chunks being retrieved, are influenced by different prompt lengths θ ∈ [450, 1000] tokens. we evaluate lumberchunker's ability to locate highly specific information within the documents, as represented by our gutenqa questions. we compare its dcg@k and recall@k scores against other baseline methods, such as semantic or proposition-level chunking. we compare our approach with other rag pipeline variants using different chunking techniques, including manually created chunks, which we consider the gold standard for optimal retrieval. we also employ non-rag baselines like gemini 1.5 pro @xcite , capable of processing up to 1.5 million input tokens, and a closed-book setup where gemini pro relies solely on internal knowledge. ## conclusions in this study, we introduce lumberchunker, a novel text segmentation method leveraging llm textual capabilities to dynamically segment documents into semantically coherent chunks. we also present gutenqa, a collection of 100 carefully parsed public domain books, segmented with lumberchunker, for which we create 3000 curated question-answer pairs. our experiments demonstrate that lumberchunker significantly improves retrieval performance over competitive baselines, and when integrated into a rag pipeline, it also enhances the accuracy of generated answers.