Equipping Retrieval-Augmented Large Language Models with Document Structure Awareness (2025)

Abstract: While large language models (LLMs) demonstrate impressive capabilities, their reliance on parametric knowledge often leads to factual inaccuracies. Retrieval-Augmented Generation (RAG) mitigates this by leveraging external documents, yet existing approaches treat retrieved passages as isolated chunks, ignoring valuable structure that is crucial for document organization. Motivated by this gap, we propose ( RDR2 ), a novel framework that explicitly incorporates structural information throughout the RAG process. RDR2 employs an LLM-based router to dynamically navigate document structure trees, jointly evaluating content relevance and hierarchical relationships to assemble optimal evidence. Our key innovation lies in formulating document routing as a trainable task, with automatic action curation and structure-aware passage selection inspired by human reading strategies. Through comprehensive evaluation on five challenging datasets, RDR2 achieves state-of-the-art performance, demonstrating that explicit structural awareness significantly enhances RAG systems’ ability to acquire and utilize knowledge, particularly in complex scenarios requiring multi-document synthesis.

## introduction large language models (llms) @xcite have demonstrated remarkable capabilities across a wide range of natural language processing (nlp) tasks, yet even state-of-the-art models continue to generate factually incorrect responses @xcite @xcite despite their growing scale and capability @xcite . retrieval-augmented generation (rag) @xcite @xcite addresses these limitations through a retrieve-and-read paradigm, which first retrieves relevant passages then uses them as context for generation @xcite @xcite . this approach combines the strengths of information retrieval and generative models, proving particularly effective for atomicfact question answering (qa) @xcite @xcite where a single precise retrieval suffices to answer clear information needs. recent advances in rag have extended its capabilities to complex knowledge-intensive scenarios requiring multi-perspective responses, particularly for factual-inductive queries that demand coherent synthesis of multiple knowledge fragments @xcite @xcite . however, current rag frameworks process retrieved passages as isolated chunks, discarding their inherent document structure -a limitation stemming from both structure-agnostic pipeline design and the flat-context paradigm of standard retrieval methods. while fixed chunking ensures retrieval efficiency, it restricts query-adaptive content selection, discarding the document's native organization which humans naturally exploit for information navigation and relational reasoning. at the reading phase, retrieved passages are simply ordered by relevance scores, potentially disrupting their original sequence in the source document. even with useful information, this loss of structural priors forces the model to implicitly reconstruct relationships that were explicitly encoded in the source hierarchy. this structural blindness constrains rag's knowledge acquisition and synthesis capabilities. in this paper we ask: can llms leverage document structural information, and can rag systems benefit from such structural awareness? we propose retrieve-documentroute-read (rdr 2 ), where a structure-aware llm performs document routing through three actions inspired by how humans selectively read sections, expand promising headings, and skip irrelevant parts when browsing articles. through this process, rdr 2 dynamically assembles query-oriented passages for better knowledge acquisition and utilization. we evaluate rdr 2 on five qa datasets representing diverse formats: triviaqa @xcite (single-answer), hotpotqa @xcite (multi-hop), qampari @xcite (list-style), asqa @xcite (ambiguous), and eli5 @xcite (indepth). as shown in figure 1 , rdr 2 achieves new state-of-the-art results with only the router trained on questions from the asqa training set (without answer supervision), while keeping the retriever and reader off-the-shelf. additionally, rdr 2 enables test-time scaling without weight updates and demonstrates generalization across different rag components (i.e., retrievers and readers). our main contributions are: ## related work retrieval-augmented generation @xcite @xcite (rag) augments language models with nonparametric knowledge through retrieved passages, demonstrating significant improvements in knowledge-intensive tasks @xcite . the standard retrieve-and-read framework operates in two stages: (1) a dense retriever, typically employing a bi-encoder architecture @xcite @xcite , retrieves passages relevant to the input question, and (2) an lm reader processes these passages, either as an off-the-shelf model @xcite or through taskspecific fine-tuning @xcite @xcite @xcite , to generate grounded responses. while effective for simple tasks with clear information needs, rag systems show limitations in complex scenarios, necessitating more advanced methods. ## methodology in this section, we present rdr foot_0 (retrieve-documentroute-read), a novel framework that endows the retrieval-augmented systems with explicit awareness of document structure. we first introduce the overview of our framework. then we define tree structures to represent the document hierarchy, ensuring stable scope and adaptive contextual focus. finally, we introduce the document routing task and the scheme for training a structure-aware llm router. ## results and analysis we first present overall results, then perform ablation studies to assess the contribution of each key component. finally, we examine the framework's scalability under different ## conclusion this work introduces rdr 2 , a novel framework that explicitly incorporates document structure throughout the rag process. our approach dynamically navigates document structure trees using an llm-based router, which jointly considers content relevance and hierarchical relationships to assemble optimal evidence. comprehensive evaluations across five datasets demonstrate that document structure awareness brings significant and consistent gains to rag systems, especially in scenarios requiring multi-document synthesis.