S uper RAG : Beyond RAG with Layout-Aware Graph Modeling (2025)

Abstract: This paper introduces layout-aware graph modeling for multimodal RAG. Different from traditional RAG methods that only deal with flat text chunks, the proposed method takes into account the relationship of multimodalities by using a graph structure. To do that, a graph modeling structure is defined based on document layout parsing. The structure of an input document is retained with the connection of text chunks, tables, and figures. This representation allows the method to handle complex questions that require information from multimodalities. To confirm the efficiency of the graph modeling, a flexible RAG pipeline is developed using robust components. Experimental results on four benchmark test sets confirm the contribution of the layout-aware modeling for performance improvement of the RAG pipeline.

## introduction retrieval augmented generation (rag) @xcite @xcite is a new paradigm that helps to reduce the hallucination of large language models (llms) @xcite @xcite by providing additional contexts for prompting llms @xcite . recently, the approach has gained considerable attention due to its effectiveness in enhancing the capabilities of llms @xcite @xcite @xcite . within this domain, graph-based rag has emerged, introducing a novel perspective that leverages structured knowledge to improve further performance and interpretability @xcite @xcite @xcite . unlike non-graph-based rag methods that directly use raw data as individual chunks of text for downstream reasoning or question-answering tasks, the graph-based rag approach can represent input data as a graph that considers the relationship among text chunks @xcite @xcite . we argue that while most rag-based pipelines perform effectively within the text modality, handling multimodal inputs-common in real-world business applications-poses substantial challenges to these systems, potentially limiting their broader applicability and impact. the challenge comes from two main reasons. first, input documents contain diverse layouts, structures, and multimodalities that need to be captured in a rag pipeline. the information on the layout plays an important role, helping llms understand the document. also, the document contains text, tables, and figures which should be encoded into prompts for llms' reasoning @xcite . second, an input question may require information in different modalities. let's consider the question: "please list the standard steps for creating internet navigware teaching materials". it requires information in the flow chart on page 27, and text on pages 28, and 29. 1 this paper introduces a novel graph-based rag scheme that addresses the two challenges above for actual multimodal qa cases. the pipeline includes four steps: document parsing, data modeling, advanced information retrieval, and reasoning. the document parsing can handle multiple input types using in-house and third-party readers. for data modeling, we introduce a new knowledge graph (kg) that retains the layout and structure of input documents. this is because the layout and structure are important to comprehend the meaning of input documents which enhances the performance of the information retrieval (ir) step. data modeling in the form of a kg is combined with full-text and vector search to create an advanced ir module that uses re-ranking to retrieve the most relevant contexts. the combination of multiple retrievers allows the proposed pipeline to retrieve more relevant information from the contexts. the reasoning step combines an input query and the relevant contexts to form a prompt feed to an llm for achieving the final answer. in summary, this paper makes three main contributions as follows. graph-based rag the graph structure has been adapted to capture relationships among concepts such as connected papers tool, foot_1 a tree of summary nodes for long context @xcite , or multimodal kgs for storing text, diagrams, and source code @xcite . the graph has also been used to improve the quality of rag in different ways such as hyper-relational kg @xcite , graph-based agents for long contexts @xcite , kg for summarization @xcite , or graph neural networks @xcite . however, we observed that most of these efforts have focused on the text modelity. we follow the direction of building multimodal kgs for rag @xcite . while prior works have explored hierarchical document parsing for rag, superrag differentiates by emphasizing structured granularity and document layout analysis. we introduce a modern, generalized data model, incorporating table of contents (toc) and master sections to improve retrieval for large documents. these enhancements preserve document structure, enhancing retrieval accuracy and effectiveness. we also share the idea of using the structure of documents for rag @xcite ; however, our method empowered by an in-house reader that can handle diverse document types with table and chart understanding rather than only processing the text structure of pdf files as @xcite . ## layout-aware graph modeling layout-aware graph modeling (lagm) is designed to effectively represent input documents while preserving their original layout and structure. this approach is motivated by the need to enhance the comprehensibility and manageability of property graphs, particularly for applications involving multimodal and complex data. for example, if the query asks for information in a table or chart, the rag pipeline needs to know which section or subsection it belongs to. ## applications figure 3 shows the pipeline of lagm that integrates multiple retrievers and re-rankers, combining heuristic graph traversal, similarity search, and language model-based techniques for efficient retrieval and ranking. the pipeline is flexible in several ways. first, it merges cross-page context using the graph representation. second, a toc retriever is included for documents with structured information, improving context quality for specific queries. additionally, the pipeline uses diagram/table expansion for queries needing information from tables and diagrams, with a self-reflection layer to ## results and discussion this section first reports the performance comparison of superrag with other strong rag-based methods, and then shows the ablation study, output observation. it finally describes the demo system. ## conclusion the paper introduces layout-aware graph modeling for multimodal data construction used by rag. the modeling takes into account the structure of input documents for building a graph that contains the relationship among text chunks, tables, and figures. a rag pipeline has also been developed to confirm the efficiency of the modeling. experimental results on four public test sets show two important points. first, layout-aware modeling is beneficial for improving the performance of rag compared to non-layout-aware and strong other rag pipelines. second, the designed rag pipeline is flexible, and adding more sophisticated ragrelated components improves the performance of the system. the modeling and rag pipeline are practical for business scenarios.