Retrieving Multimodal Information for Augmented Generation: A Survey (2023)

Abstract: As Large Language Models (LLMs) become popular, there emerged an important trend of using multimodality to augment the LLMs’ generation ability, which enables LLMs to better interact with the world. However, there lacks a unified perception of at which stage and how to incorporate different modalities. In this survey, we review methods that assist and augment generative models by retrieving multimodal knowledge, whose formats range from images, codes, tables, graphs, to audio. Such methods offer a promising solution to important concerns such as factuality, reasoning, interpretability, and robustness. By providing an in-depth review, this survey is expected to provide scholars with a deeper understanding of the methods’ applications and encourage them to adapt existing techniques to the fast-growing field of LLMs.

## introduction generative artificial intelligence (gai) has demonstrated impressive performances in tasks such as text generation @xcite @xcite ) and text-to-image generation @xcite . the recent advancements in multimodal large language models (mllms) @xcite @xcite have further improved the models' capabilities to handle multi-format information, opening up possibilities for developing general-purpose learners. nevertheless, generative models are not exempt from inherent limitations, including the tendency for generating hallucinations @xcite , struggling with arithmetic tasks @xcite , and lacking interpretability . consequently, a promising solution for enhancing their capabilities lies in enabling them to interact with the external world and acquire knowledge in diverse formats and modalities, thereby improving the factuality and rationality of the generated content. recently, there have been emerging studies focusing on retrieval-augmented approaches @xcite , which aim to provide information that is more grounded and factually dependent. among them, most @xcite retrieves textual information, which matches the data format used during pre-training and offers a natural medium for interaction. however, there is more world knowledge stored in different structures and modalities, such as images and videos, which is often inaccessible, unavailable, or not describable in traditional textual corpora. therefore, there arises an important research intersection that retrieves multimodal knowledge to augment generative models. it offers a promising solution to current challenges such as factuality, reasoning, interpretability, and robustness. as this field is very recent, there lacks a unified understanding of recognizing these methods as a specific group, visualizing their innate connections, connecting their methodologies, and outlining their applications. therefore, we survey recent advancements in multimodal retrieval-augmented generation (rag). specifically, we discuss current research by grouping them into different modalities, including image, code, structured knowledge, audio, and video. for each modality, we systematically search the acl anthology and google scholar with relevant keywords and perform manual filtering to determine their relevance to the survey. as a result, we collect 146 papers for detailed analysis. in appendix a.1, we include search details, statistics, and a trend analysis figure, which shows that multimodal rag papers have indeed developed very fastly since the emergence of large-scale generalpurpose models. within each modality, we discuss relevant papers by grouping them under different applications. by providing an in-depth survey, we hope to help researchers recognize the importance of incorporating knowledge in different formats and encourage adaptation and advancements on existing techniques to the fast-growing field of llms. in summary, our contributions are as follows: • we establish retrieval augmented generation with multi-modality as an important group of methods that emerges with the recent advances in llms. ## definitions and background to better understand the state and advancements that inspired multimodal retrieval augmentation, we first define and discuss the background of two key concepts: multimodal learning and retrievalaugmented generation (rag). ## multimodal retrieval-augmented generation for each modality, there are different retrieval and synthesis procedures, targeted tasks, and challenges. therefore, we discuss relevant methods by grouping them in terms of modality, including image, code, structured knowledge, audio, and video. ## future directions with the development of multi-modal llms, retrieving multimodal information to augment text generation will be a promising direction to better ground textual generation in real-world contexts, contributing towards building a model that is fully aware and can better interact with the world. specifically, we describe some potential directions that can be of benefit to the community. ## conclusions this survey reviews research that augments generative models by retrieving multi-modal information. specifically, we categorize the current domain into enhancing with different modalities, including image, code, structured knowledge, speech, and video. with the emergence of large multi-modal models, we believe that this survey could serve as a comprehensive overview of an emerging and promising field. moreover, we hope it could encourage future research in the domain, including retrieval-augmented multimodal reasoning, building a multi-modal knowledge index, and combining retrieval with pretraining.