MC -indexing: Effective Long Document Retrieval via Multi-view Content-aware Indexing (2024)

Abstract: Long document question answering (DocQA) aims to answer questions from long documents over 10k words. They usually contain content structures such as sections, sub-sections, and paragraph demarcations. However, the indexing methods of long documents remain under-explored, while existing systems generally employ fixed-length chunking. As they do not consider content structures, the resultant chunks can exclude vital information or include irrelevant content. Motivated by this, we propose the **M**ulti-view **C**ontent-aware indexing (**MC-indexing**) for more effective long DocQA via (i) segment structured document into content chunks, and (ii) represent each content chunk in raw-text, keywords, and summary views. We highlight that MC-indexing requires neither training nor fine-tuning. Having plug-and-play capability, it can be seamlessly integrated with any retrievers to boost their performance. Besides, we propose a long DocQA dataset that includes not only question-answer pair, but also document structure and answer scope. When compared to state-of-art chunking schemes, MC-indexing has significantly increased the recall by **42.8%**, **30.0%**, **23.9%**, and **16.3%** via top k = 1.5, 3, 5, and 10 respectively. These improved scores are the average of 8 widely used retrievers (2 sparse and 6 dense) via extensive experiments.

## introduction document question answering (docqa) is a pivotal task in natural language processing (nlp) that involves responding to questions using textual documents as the reference answer scope. conventional docqa systems (as depicted in figure 2a ) comprise three key components: (i) an indexer that segments the document into manageable text chunks indexed with embeddings, (ii) a retriever that identifies and fetches the most relevant chunks desired reference text: you can bake a chocolate cake by ... : 1. preparation: ... 2. gather ingredients: ... 3. dry ingredients mixture: ... 4. wet ingredients mixture: ... 5. combine mixtures: ... 6. bake the cake: ... (500 words) actual chunks retrieved: ... you can bake a chocolate cake by: 1. preparation: ... ( 100 words to the corresponding question, and (iii) a reader that digests the retrieved answer scope and generates an accurate answer. unlike the retriever @xcite @xcite and reader @xcite @xcite that are vastly studied, the indexer received relatively less attention. existing indexing schemes overlook the importance of content structures when dealing with long documents, as they are usually organized into chapters, sections, subsections, and paragraphs @xcite , i.e., structured. the widely used fixed-length chunking strategy can easily break the contextual relevance between text chunks for long documents. such chunking errors can be further aggravated by the retriever and the reader. moreover, determining the boundary between chunks can be tricky, requiring delicate design to prevent contextual coherence disruption. ideally, each chunk should represent a coherent and content-relevant textual span. otherwise, it can lead to the exclusion of relevant information or the inclusion of irrelevant text, as exemplified in figure 1 . our empirical study on fixed-length chunking reveals that setting the chunk length to 100 results in over 70% of long answers/supporting evidence being truncated, i.e., incomplete. such incompleteness still exists at 45%, despite an increase of chunk length to 200. 1 meanwhile, most existing retrieval systems rely solely on the raw text of chunks to determine relevance to a query. while raw-text-based semantic embeddings effectively address queries seeking specific short-form details, they often fail to capture complete semantic essence of the text. when inquiring high-level information, such as event summaries or comparisons, raw-text embeddings may fall short. additionally, reliance on raw text poses practical constraints, as models e.g., dpr @xcite , e5 @xcite , bge @xcite based on bert @xcite typically have a token limit of 512. this leads to potential truncation and loss of information during the indexing process. @xcite attempt to embed the entire document with multiple representations, however, these embeddings are not applicable to individual chunks. to mitigate aforementioned gaps, we present multi-view content-aware indexing, termed mcindexing, for more effective retrieval over long documents. our method involves content-aware chunking of structured long documents, whereby, instead of employing naïve fixed-length chunking, the document is segmented into section chunks. the content-aware chunking can effective eliminate chunking errors. each of these section chunks is then indexed in three different views, representing each chunk with raw-text, a list of keywords, and a summary. the keyword and summary view can provide richer but more concise representation of section chunks, thereby significantly enhancing the semantic richness of each chunk. for retrieval, we aggregate the top relevant chunks from each view. note that the entire process of mc-indexing 1 more statistics of chunking errors are in appendix a. to our best knowledge, existing docqa datasets do not provide content structure. hence, we transform an existing long documents dataset, namely wikiweb2m @xcite , into a qa dataset, by adding annotations to the documents. in addition, we complement natural questions dataset @xcite with content structure, and filter only long documents for our experiment. distinct from other qa datasets, our documents are longer (averaging at 15k tokens) and contain detailed content structure. our contributions are in fourfold: • we propose a long document qa dataset annotated with question-answer pair, document content structure, and scope of answer. • we propose multi-view content-aware indexing (mc-indexing), that can (i) segment the long documents according to their content structures, and (ii) represent each chunk in three views, i.e., raw-text, keywords, and summary. • mc-indexing requires neither training nor finetuning, and can seamlessly act as a plug-and-play indexer to enhance any existing retrievers. • through extensive experiments and analysis, we demonstrate that mc-indexing can significantly improve retrieval performance of eight commonly-used retrievers (2 sparse and 6 dense) on two long docqa datasets. 2 related work chunking methods. chunking is a crucial step in either qa or retrieval-augmented generation (rag). when dealing with ultra-long text documents, chunk optimization involves breaking the document into smaller chunks. existing systems focus on how to retrieve relevant chunks, but neglecting how text content is chunked. in practice, fixed-length chunking is a commonly used method that is easy to be implemented. it chunks text at a fixed length, e.g., 200 words. sentence chunking involves dividing textual content based on sentences. recursive chunking employs various delimiters, such as paragraph separators, newline characters, or spaces, to recursively segment the text. chen et al. (2023b) use propositions for dense retrieval, where each proposition is a concise and self-contained factoid. raina and gales (2024) propose to represent each chunk as a set of atomic pieces of informa-fixed length chunking documents ...... fixed length snippets indexing ... retrieval snippet indexings retrieved snippet(s) answer generation question ...... document indexing retrieval & question answering question question answering (a) conventional docqa system: @xmath0... ... ... keywords view summary view raw-text view ...... section content retrieval retrieved content(s) answer generation question ...... question multi-view content indexing document indexing content-aware chunking question answering retrieval & question answering tion. however, these methods often fail to preserve semantic integrity of critical content. in contrast, advanced chunking methods provided by layoutpdfreader 2 , docugami 3 , and mineru (contributors, 2024) are able to detect the layout of input document and then chunk the text by the smallest subdivision according to the document's content structure. we refer these techniques as content-aware chunking (see section 3.2), which ensures each chunk to be semantically coherent, thus reducing chunking error. ## dataset construction in our work, we focus on long and structured document, thus we collect dataset corpus based on the following two factors. (1) presence of structured information: the content of long documents is usually divided into multiple sections. for example, a research paper is organized into various sections such as abstract, introduction, methodology and conclusion. structured documents have explicitly labelled sections along their corresponding text. most of the existing qa datasets (e.g., squad @xcite , triviaqa @xcite , ms macro @xcite ) do not include the content structure of source documents. due to the absence of structure information, they are not considered in our work. according to these criteria, we select wikipedia webpage 2m (wikiweb2m) @xcite and natural questions (nq) @xcite datasets. we discuss dataset processing and annotations on these datasets in finer detail. ## conclusion in this paper, we propose a new approach: multiview content-aware indexing (mc-indexing) for more effective long document question answering. specially, we propose a long document qa dataset which annotates not only the questionanswer pair, but also the document structure and the document scope to answer this question. we propose a content-aware chunking method to segment the document into content chunks according to its organizational content structure. we design a multi-view indexing method to represent each content chunk in raw-text, keywords, and summary views. through extensive experiments, we demonstrate that content-aware chunking can eliminate chunking errors, and multi-view indexing can significantly benefit long docqa. for future work, we would like to explore how to use the hierarchical document structure for more effective retrieval. moreover, we would like to train or finetune a retriever that can generate more fine-grained or nuanced embeddings across multiple views.