Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation (2025)

Abstract: Large Language Models (LLMs) suffer from hallucinations and outdated knowledge due to their reliance on static training data. Retrieval-Augmented Generation (RAG) mitigates these issues by integrating external dynamic information for improved factual grounding. With advances in multimodal learning, Multimodal RAG extends this approach by incorporating multiple modalities such as text, images, audio, and video to enhance the generated outputs. However, cross-modal alignment and reasoning introduce unique challenges beyond those in unimodal RAG. This survey offers a structured and comprehensive analysis of Multimodal RAG systems, covering datasets, benchmarks, metrics, evaluation, methodologies, and innovations in retrieval, fusion, augmentation, and generation. We review training strategies, robustness enhancements, loss functions, and agent-based approaches, while also exploring the diverse Multimodal RAG scenarios. In addition, we outline open challenges and future directions to guide research in this evolving field. This survey lays the foundation for developing more capable and reliable AI systems that effectively leverage multimodal dynamic external knowledge bases. All resources are publicly available at https://github.com/llm-lab-org/Multimodal-RAG-Survey.

## introduction & background recent advancements in transformer architectures @xcite , coupled with increased computational resources and the availability of largescale training datasets @xcite , have significantly accelerated progress in the development of language models. the emergence of foundational large language models (llms) @xcite @xcite @xcite , has revolutionized natural language processing (nlp), excelling in tasks such as instruction following @xcite , reasoning @xcite , in-context learning @xcite , and multilingual translation @xcite . despite these achievements, llms face challenges such as hallucinations, outdated knowledge, and a lack of verifiable reasoning @xcite . their reliance on parametric memory limits access to upto-date information, reducing their effectiveness in knowledge-intensive tasks. retrieval-augmented generation (rag) rag @xcite addresses these limitations by enabling llms to retrieve and incorporate external knowledge, improving factual accuracy and reducing hallucinations @xcite . by dynamically accessing external knowledge sources, rag enhances knowledgeintensive tasks while grounding responses in verifiable sources @xcite . in practice, rag systems follow a retriever-generator pipeline: the retriever uses embedding models @xcite to identify relevant passages from external knowledge bases and may apply re-ranking techniques to improve precision @xcite . the retrieved passages are then provided to the generator, which leverages this contextual information to produce more informed and coherent responses. recent advancements in rag frameworks, such as planning-guided retrieval @xcite , agentic rag @xcite , and feedback-driven iterative refinement @xcite , have further improved both the retrieval and generation components of these systems. ## datasets, evaluation, and applications we review diverse datasets and benchmarks supporting tasks such as multimodal summarization, visual qa, video understanding, and more. for full details, see appendix ( §b) and tables 1 and 2 . multimodal rag has been applied across various domains, including healthcare, software engineering, fashion, entertainment, and emerging fields. an overview of tasks and applications are detailed in appendix ( §e) and figure 3 . evaluating these systems requires multiple metrics, covering retrieval performance, generation quality, and modality alignment. the complete evaluation methods, metrics, and their definitions and formulations are in appendix ( §c). ## open problems and future directions additional challenges and future directions about long-context processing, scalability, efficiency, and personalization are discussed in appendix ( §f). ## conclusion this study provides a comprehensive review of multimodal rag, categorizing key advancements in retrieval, multimodal fusion, augmentation, generation, training strategies, and agents. we also examine task-specific applications, datasets, benchmarks, and evaluation methods while highlighting open challenges and promising future directions. we hope this work inspires future research, particularly in enhancing cross-modal reasoning and retrieval, developing agent-based interactive systems, and advancing unified multimodal embedding spaces.