V is D o M : Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation (2025)

Abstract: Understanding information from a collection of multiple documents, particularly those with visually rich elements, is important for document-grounded question answering. This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings with rich multimodal content, including tables, charts, and presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG, combining robust visual retrieval capabilities with sophisticated linguistic reasoning. VisDoMRAG employs a multi-step reasoning process encompassing evidence curation and chain-of-thought reasoning for concurrent textual and visual RAG pipelines. A key novelty of VisDoMRAG is its consistency-constrained modality fusion mechanism, which aligns the reasoning processes across modalities at inference time to produce a coherent final answer. This leads to enhanced accuracy in scenarios where critical information is distributed across modalities and improved answer verifiability through implicit context attribution. Through extensive experiments involving open-source and proprietary large language models, we benchmark state-of-the-art document QA methods on VisDoMBench. Extensive results show that VisDoMRAG outperforms unimodal and long-context LLM baselines for end-to-end multimodal document QA by 12-20%.

## introduction in today's information-rich landscape, pdf documents play a crucial role in storing and disseminating information across various domains, including finance, legal, scientific research, and more. these documents often contain a rich blend of textual, visual, and tabular data, making them a unique * primary research mentor challenge for information retrieval systems. unlike structured formats like databases, pdfs are inherently unstructured, with diverse layouts combining paragraphs, images, charts, and tables. this complexity demands sophisticated multimodal processing techniques capable of interpreting both the textual and visual content. effective handling of multimodal content from pdfs is essential for downstream tasks such as question-answering @xcite , summarization @xcite , and knowledge extraction @xcite , where accurate and context-aware data extraction can significantly enhance decisionmaking processes. as a result, developing advanced methods that can fully leverage the multimodal nature of pdf documents has become a critical research challenge. in real-world document qa systems, queries are often directed over a collection of source documents rather than a single source, requiring the system to identify the document that contains the relevant answer. this reflects common scenarios in domains such as finance, science, and policy analysis, where users interact with large, varied document sets to find specific information. in these cases, the challenge lies in effectively localizing context relevant to the query, from a large volume of information distributed across multiple documents (akin to finding a "needle in a haystack" @xcite ). multi-document qa datasets are scarce, with existing multi-document benchmarks @xcite , predominantly focused on textual information, often overlooking the diverse content forms found in real-world documents, such as tables, charts, and visual elements. visually rich elements, such as tables, charts, and slides, provide structured data and visual summaries that are critical for answering certain types of questions. tables often present dense, organized information that cannot be captured through plain text. at the same time, charts and slides can visually depict trends, relationships, or distributions that require interpretation beyond textual descriptions. the absence of datasets that include these modalities limits the ability of current qa models to address complex, multimodal questions. for instance, answering a financial or scientific question may require interpreting both numerical data in tables and trends in graphs alongside the surrounding text. in the context of visually rich content-based documents, existing rag systems face a critical limitation due to their reliance on a singular modality (either text or vision) for retrieval. text-based systems are proficient in linguistic reasoning but often overlook vital visual elements, such as tables and figures, that may contain key information. conversely, multimodal rag @xcite systems that leverage vision-based retrieval can effectively extract visual data but are often constrained in endto-end performance by the llm's visual reasoning abilities, as text often performs better than visual input when given the same context @xcite , which can be attributed to language bias in visual llms @xcite , and visual hallucination @xcite . ## problem formulation given a query q, we have a collection of m documents @xmath0, d 2 , . . . , d m }, wherein each document d i may consist of a set of n i pages represented by p @xmath1, p i 2 , . . . , p i n i }. we aim to generate text â for each query q that accurately answers the user query. the answer generation relies on retrieving relevant evidence context from one or more documents. each query q may require information spread across different pages from one or more of the associated documents in d. we aim to propose a framework that can accurately answer questions over a collection of multi-page documents where the system first retrieves relevant evidence at the level of individual pages, paragraphs or text chunks, followed by using the retrieved context to generate answer text. ## visdombench every data point in visdombench can be expressed as triple (q, d, â), where a question q is posed to a set of documents d, with groundtruth answer â. we re-purpose five existing document-qa datasets to form our benchmark. ## experiments in our experiments, we first evaluate different retrieval and indexing models on our benchmark, followed by end-to-end qa evaluation using the identified optimal retrieval models with different llms. the experiments, baselines and evaluation are discussed below: 6.1 retrieval baselines: we use popular text based retrieval models: bm25 @xcite a statistical baseline, and , mpnet @xcite , minilm @xcite , and bge-1.5 @xcite , which represent sota dense retrieval baselines. text extraction from pdf documents is performed using pytesseract. the extracted text is then segmented into 3000-character chunks using the recursive-split method @xcite , with a 10% overlap to mitigate information loss. for visual retrieval, we utilize recent advances late interaction based multi-vector retrieval models built on top of llms @xcite , namely colpali and colqwen2, which have paligemma @xcite and @xcite as their base llms. readers are encouraged to refer to the appendix for further details of these models. evaluation: evidence extraction is assessed using anlcs between ground truth evidence and retrieved chunks/pages. document identification evaluates the retrievers' ability to select the correct source document in a multi-document setup. we report the rate of instances where the ground truth document is the source of the majority of the retrieved context. ## conclusion and future work in this work, we introduced visdombench, the first qa dataset designed to evaluate multi-document systems incorporating visually rich elements such as tables, charts, and slides. by targeting docu-ments that require both textual and visual comprehension, visdombench offers a novel benchmark to assess the capability of multimodal retrieval systems. we also presented visdomrag, a multimodal retrieval-augmented generation approach that fuses visual and textual pipelines using consistency-constrained modality fusion. this method demonstrated a significant improvement over traditional long context, textual, and visual rag by 12-20%. while the current work focuses on rag in multimodal multi-doc settings, future work will extend this approach to include reasoning through end-to-end trained models, especially in low-resource settings.