The Revolution of Multimodal Large Language Models: A Survey (2024)

Abstract: Connecting text and visual modalities plays an essential role in generative intelligence. For this reason, inspired by the success of large language models, significant research efforts are being devoted to the development of Multimodal Large Language Models (MLLMs). These models can seamlessly integrate visual and textual modalities, while providing a dialogue-based interface and instruction-following capabilities. In this paper, we provide a comprehensive review of recent visual-based MLLMs, analyzing their architectural choices, multimodal alignment strategies, and training techniques. We also conduct a detailed analysis of these models across a wide range of tasks, including visual grounding, image generation and editing, visual understanding, and domain-specific applications. Additionally, we compile and describe training datasets and evaluation benchmarks, conducting comparisons among existing models in terms of performance and computational requirements. Overall, this survey offers a comprehensive overview of the current state of the art, laying the groundwork for future MLLMs.

## introduction the introduction of the attention operation and the transformer architecture @xcite has enabled the creation of models capable of handling various modalities on an increasingly large scale. this advancement is largely attributed to the versatility of the operator and the adaptability of the architecture. initially, this breakthrough was leveraged for language-specific models @xcite but quickly extended to support diverse modalities @xcite and facilitate their integration within unified embedding spaces @xcite . the surge in sophisticated large language models (llms), particularly their capacity for the unusual aspect of this image is a man ironing clothes on the back of a minivan. this is not a typical place to iron … figure 1 : general architecture of multimodal large language models (mllms), composed of a visual encoder, a language model, and an adapter module that connects visual inputs to the textual space. in a context marked by the rapid release of new models, our goal is to offer an exhaustive overview of the mllm landscape, with a focus on models exploiting the visual modality. this overview serves as both an update on the current state and a source of inspiration for future developments. we identify three core aspects that define these models: their architecture, training methodologies, and the tasks they are designed to perform. we begin by detailing the prevalent choices for vision encoders and adapter modules that equip llms with crossmodal capabilities. following this, we delve into the training processes and data utilized. we then explore the range of tasks addressed by mllms. the review concludes with a discussion of the persisting challenges in the field and the promising directions for future research. further details on training data, evaluation datasets, performance and computational requirements are reported in the supplementary material. the motivation behind this survey stems from an emerging scientific interest in the field of mllms, as evidenced by the constant increase in published works. in comparison with existing surveys on the topic @xcite @xcite , our paper exhibits substantial differences. notably, it addresses several critical areas that were overlooked in prior works, including visual grounding, image generation, and editing. furthermore, our survey details the main components utilized by each discussed mllm, such as the visual encoders and the specific llm employed. additionally, our analysis offers a comparative perspective on the performance and hardware requirements of the discussed papers, incorporating both quantitative results and detailed information on benchmarks. through this comprehensive approach, our survey aims to fill the existing gaps and provide a more nuanced understanding of the current landscape in the field. ## conclusion and future directions in this survey, we have provided a comprehensive overview of the recent evolution of mllms, first focusing on how to equip llms with multimodal capabilities and then exploring the main tasks addressed by these models. based on the analysis presented, in the following, we outline important open challenges and promising future research directions to further empower mllms.