Multimodal Retrieval-Augmented Generation: Unified Information Processing Across Text, Image, Table, and Video Modalities (2025)

Abstract: Retrieval-augmented generation (RAG) is a powerful paradigm for leveraging external data to enhance the capabilities of large language models (LLMs). However, most existing RAG solutions are tailored for single-modality or limited multimodal scenarios, restricting their applicability in real-world contexts where diverse data sources—including text, tables, images, and videos—must be integrated seamlessly. In this work proposes a unified Multimodal Retrieval-augmented generation (mRAG) system designed to unify information processing across all four modalities. Our pipeline ingests and indexes data from PDFs and videos using tools like Amazon Textract, Transcribe, Langfuse, and multimodal LLMs (e.g., Claude 3.5 Sonnet) for structured extraction and semantic enrichment. The dataset includes text queries, table lookups, image-based questions, and videos. Evaluation with the Deepeval framework shows improved retrieval accuracy and response quality, especially for structured text and tables. While performance on image and video queries is lower, the multimodal integration framework remains robust, underscoring the value of unified pipelines for diverse data.

## introduction the exponentially growing volume of digital content in various forms, including text, tables, images, and videos, has created new challenges. traditional information retrieval systems typically focus on a single modality, such as text or images, limiting their ability to process complex queries that require insight from multi-modal data sources. however, real-world applications, such as enterprise data analytics, troubleshooting equipment through video manuals, or processing product specifications, need a framework to manage various data types. retrieval-augmented generation (rag) systems have emerged as a powerful paradigm combining retrieval mechanisms with generative models to enhance information access and synthesis. however, conventional rag frameworks were not designed initially to handle multimodal data, restricting their utility in environments where diverse data forms must be unified and processed seamlessly. this limitation underscores the need for an evolved approach that extends the capabilities of rag systems to accommodate and integrate multiple modalities effectively. this paper presents an mrag system that unifies information across text, tables, images, and videos. using tools like aws, langchain, and multimodal llms, it provides a robust pipeline for data ingestion, retrieval, and response generation. ## background and related work the landscape of information retrieval has evolved significantly with the advent of large-scale digital data across diverse modalities. traditional information retrieval systems focus mainly on single modalities, such as text-based search engines @xcite @xcite or image retrieval systems @xcite , each optimized for their specific data type. multimodal information retrieval (mmir) aims to bridge the gap between different data types, facilitating comprehensive searches that span text, images, videos, and other formats @xcite . researchers have successfully applied deep learning techniques for multimodal information retrieval @xcite . rag systems represent a paradigm shift in combining retrieval mechanisms with generative models. introduced by @xcite , rag leverages llms to generate contextually relevant responses by retrieving pertinent information from extensive external knowledge bases. rag research has rapidly expanded, tackling efficiency bottlenecks @xcite , memory constraints @xcite , and self-reflection strategies @xcite . recent advances in rag have begun integrating multiple modalities to enhance retrieval and generation, as seen in murag @xcite . however, most work remains limited to small, domainspecific datasets (e.g., healthcare) and only two modalities @xcite . key challenges remain in the development of multimodal rag systems. most existing approaches lack unified frameworks capable of reasoning across more than two modalities, such as text, tables, images, and videos. scalability is also limited, as adding new modalities often requires separate training pipelines @xcite . furthermore, current evaluation benchmarks primarily focus on single-or dual-modality tasks, making it difficult to assess systems designed for more complex, fully multimodal scenarios @xcite @xcite . this work addresses these gaps by proposing a unified framework for building and evaluating multimodal rag systems. ## conclusion and future work this work presents a methodology for building an mrag system, focusing on pipelines for extracting and indexing text, tables, images, and videos. experimental results show improved contextual relevancy, llm evaluation scores, and reduced hallucinations, while performance variations highlight challenges with unstructured data. future work will focus on enhancing mrag with improved llm capabilities, fine-tuning embeddings for better domain understanding, incorporating user feedback, and adding visual modalities for input.