MM - LLM s: Recent Advances in M ulti M odal Large Language Models (2024)

Abstract: In the past year, MultiModal Large Language Models (MM-LLMs) have undergone substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or outputs via cost-effective training strategies. The resulting models not only preserve the inherent reasoning and decision-making capabilities of LLMs but also empower a diverse range of MM tasks. In this paper, we provide a comprehensive survey aimed at facilitating further research of MM-LLMs. Initially, we outline general design formulations for model architecture and training pipeline. Subsequently, we introduce a taxonomy encompassing 126 MM-LLMs, each characterized by its specific formulations. Furthermore, we review the performance of selected MM-LLMs on mainstream benchmarks and summarize key training recipes to enhance the potency of MM-LLMs. Finally, we explore promising directions for MM-LLMs while concurrently maintaining a [real-time tracking website](https://mm-llms.github.io/) for the latest developments in the field. We hope that this survey contributes to the ongoing advancement of the MM-LLMs domain.

## introduction multimodal (mm) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks @xcite @xcite @xcite @xcite @xcite @xcite . however, as the scale of models and datasets continues to expand, traditional mm models incur substantial computational costs, particularly when trained from scratch. recognizing that mm research operates at the intersection of various modalities, a logical approach is to capitalize on readily available pre-trained unimodal foundation models, with a special emphasis on powerful large language models (llms) @xcite . this strategy aims to mitigate computational expenses and enhance the efficacy of mm pre-training, leading to the emergence of a novel field: mm-llms. mm-llms harness llms as the cognitive powerhouse to empower various mm tasks. llms contribute desirable properties like robust language generation, zero-shot transfer capabilities, and in-context learning (icl). concurrently, foundation models in other modalities provide highquality representations. considering foundation models from different modalities are individually pre-trained, the core challenge facing mm-llms is how to effectively connect llms with models in other modalities to enable collaborative inference. the predominant focus within this field has been on refining alignment between modalities and aligning with human intent via an mm pre-training (pt) + mm instruction-tuning (it) pipeline. with the debut of gpt-4(vision) (openai, 2023) and gemini @xcite , showcasing impressive mm understanding and generation capabilities, a research fervor on mm-llms has been sparked. initial research primarily focuses on mm content comprehension and text generation, encompassing tasks such as image-text understanding, exemplified by projects like blip-2 @xcite , llava @xcite , minigpt-4 @xcite , and openflamingo @xcite ; video-text understanding, as demonstrated by initiatives such as videochat @xcite , video-chatgpt @xcite , and llama-vid @xcite ; and audio-text understanding, as seen in projects like qwen-audio @xcite . later, the capabilities of mm-llms have been expanded to support specific modality outputs. this includes tasks with image-text output, such as gill @xcite , kosmos-2 @xcite , emu @xcite , and minigpt-5 @xcite ; as well as speech/audio-text output, exemplified by projects like speechgpt @xcite and @xcite . recent research endeavors have focused on mimicking human-like any-to-any modality conversion, shedding light on the path to artificial general intelligence. some efforts aim to amalgamate llms with external tools to reach an approaching any-to-any mm comprehension and generation, such as visual-chatgpt @xcite , hugginggpt @xcite , and audiogpt @xcite . conversely, to mitigate propagated errors in the cascade system, initiatives like next-gpt @xcite , codi-2 @xcite , and modaverse @xcite have developed end-to-end mm-llms of arbitrary modalities. the timeline of mm-llms is depicted in figure 1 . in this paper, we present a comprehensive survey aimed at facilitating further research on mm-llms. to provide readers with a holistic understanding of mm-llms, we initially delineate general design formulations from model architecture (section 2) and training pipeline (section 3). we break down the general model architecture into five components: modality encoder (section 2.1), input projector (section 2.2), llm backbone (section 2.3), output projector (section 2.4), and modality generator (section 2.5). the training pipeline elu-cidates how to enhance a pre-trained text-only llm to support mm input or output, primarily consisting of two stages: mm pt (section 3.1) and mm it (section 3.2). in that section, we also provide a summary of mainstream datasets for mm pt and mm it. next, we establish a taxonomy encompassing 126 state-of-the-art (sota) mm-llms, each characterized by specific formulations, and summarize their development trends in section 4. in section 5, we comprehensively review the performance of major mm-llms on mainstream benchmarks and distill key training recipes to enhance the efficacy of mm-llms. in section 6, we offer promising directions for mm-llms research. moreover, we have established a website ( https://mm-llms.github.io ) to track the latest progress of mm-llms and facilitate crowdsourcing updates. finally, we summarize the entire paper in section 7 and discuss related surveys on mm-llms in appendix a. we aspire for our survey to aid researchers in gaining a deeper understanding of this field and to inspire the design of more effective mm-llms. ## model architecture in this section, we provide a detailed overview of the five components comprising the general model architecture, along with the implementation choices for each component, as illustrated in figure 2. mm-llms that emphasize mm understanding only include the first three components. during training, the modality encoder, llm backbone, and modality generator are generally maintained in a frozen state. the primary optimization emphasis is on input and output projectors. given that projectors are lightweight components, the proportion of trainable parameters in mm-llms is notably small compared to the total parameter count (typically around 2%). the overall parameter count is contingent on the scale of the core llm utilized in the mm-llms. as a result, mm-llms can be efficiently trained to empower various mm tasks. ## training pipeline mm-llms' training pipeline can be delineated into two principal stages: mm pt and mm it. ## sota mm-llms as shown in figure 3 , we classify the 126 sota mm-llms from both functional and design perspectives. in the design division, "tool-using" denotes treating the llm as a black box and provid- model @xmath0.pt #.it flamingo i+v+@xmath1: nfnet-f6 cross-attention chinchilla-1.4b/7b/70b ----blip-2 i+@xmath2: clip/eva-clip vit@224 q-former w/ linear projector flan-t5/opt --129m -llava i+@xmath3: clip vit-l/14 linear projector vicuna-7b/13b ----minigpt-4 i+@xmath4: eva-clip vit-g/14 q-former w/ linear projector vicuna-13b ----mplug-owl i+@xmath5: clip vit-l/14 cross-attention llama-7b ----otter i+@xmath6: clip vit-l/14 cross-attention llama-7b ----x-llm i+v+a+@xmath7: vit-g; a: c-former q-former w/ linear projector chatglm-6b ----videochat v+@xmath8: vit-g q-former w/ linear projector vicuna ----instructblip i+v+@xmath9: vit-g/14@224 q-former w/ linear projector flan-t5/vicuna --129m 1.2m pandagpt i+@xmath10: imagebind linear projector vicuna-13b ----gill i+@xmath11: clip vit-l linear projector opt-6.7b tiny transformer i: stable diffusion-1.5 --pali-x i+@xmath12: vit linear projector ul2-32b ----video-llama i+v+a+@xmath13: eva-clip vit-g/14; a: imagebind q-former w/ linear projector vicuna/llama ----video-chatgpt v+@xmath14: clip vit-l/14 linear projector vicuna-v1.1 ----shikra i+@xmath15: clip vit-l/14@224 linear projector vicuna-7b/13b --600k 5.5m llavar i+@xmath16: clip vit-l/14@224 & clip vit-l/14@336 linear projector vicuna-13b ----mplug-docowl id+@xmath17: clip vit-l/14 cross-attention llama-7b ----lynx i+v+@xmath18: eva-clip vit-1b cross-attention vicuna ----emu i+v+@xmath19: eva-clip-1b cross-attention llama-13b mlp i: stable diffusion-1.5 --dlp i+@xmath20: clip/eva-clip vit q-&p-former w/ linear projector opt/flan-t5 ----bubogpt i+a+@xmath21: clip/eva-clip vit; a: imagebind q-former w/ linear projector vicuna ----chatspot i+@xmath22: clip vit-l/14 linear projector vicuna-7b/llama ----idefics i+@xmath23: openclip cross-attention llama ----qwen-vl-(chat) i+@xmath24: vit@448 initialized from openclip's vit-bigg cross-attention qwen-7b --1.4b † 50m † lavit i+@xmath25: vit cross-attention llama-7b -i: stable diffusion --next-gpt i+v+a+@xmath26: imagebind linear projector vicuna-7b tiny transformer i: stable diffusion; v: zeroscope; a: audioldm --dreamllm i+@xmath27: clip vit-l linear projector vicuna mlp i: stable diffusion --anymal i+v+a+@xmath28: clip vit/l&vit-g&dinov2; v: intervideo; a: clap i/v: cross-attention; a: linear projector llama-2 ----minigpt-5 i+@xmath29: eva-clip vit-g/14 q-former w/ linear projector vicuna-7b tiny transformer i: stablediffusion-2 --llava-1.5 i+@xmath30: clip vit-l@336 mlp vicuna-v1.5-7b/13b --0.6m 0.7m minigpt-v2 i+@xmath31: eva-clip vit@448 linear projector llama-2-chat-7b ----cogvlm i+@xmath32: eva-2-clip vit mlp vicuna-v1.5-7b ----qwen-audio a+@xmath33: whisper-l-v2 linear projector qwen-7b ----dress i+@xmath34:eva-clip vit-g/14 linear projector vicuna-v1.5-13b ----x-instructblip i+v+a+3d+@xmath35: eva-clip vit-g/14; a: beats; 3d: ulip-2 q-former w/ linear projector vicuna-v1.1-7b/13b ----codi-2 i+v+a+@xmath36: imagebind mlp llama-2-chat-7b mlp i: stable diffusion-2.1; v: zeroscope-v2; a: audioldm-2 --rlhf-v i+@xmath37: beit-3 linear projector vicuna-v1-13b ----silkie i+@xmath38: vit initialized from openclip's vit-bigg cross-attention qwen-7b ----lyrics i+@xmath39: clip vit-l/14&grounding-dino-t &sam-hq&vit-h&ram++ mq-former w/ linear projection vicuna-13b ----vila i+@xmath40: vit@336 linear projector llama-2-7b/13b --50m 1m intrenvl i+v+@xmath41: internvit-6b; t: llama-7b cross-attention w/ mlp qllama-8b & vicuna-13b ----modaverse i+v+a+@xmath42: stable diffusion; v: videofusion; a: audioldm --mm-interleaved i+@xmath43: clip vit-l/14 cross-attention vicuna-13b tiny transformer i: stable diffusion-2.1 -- ## benchmarks and performance to provide a comprehensive performance comparison, we have compiled a table featuring major mm-llms across 18 vision-language (vl) benchmarks, as reported in various papers @xcite @xcite . this information is presented in following this, we will outline training recipes that enhance the effectiveness of mm-llms, drawing insights from sota models. training recipes firstly, higher image resolution can incorporate more visual details for the model, benefiting tasks that require fine-grained details. for example, llava-1.5 and vila employ a resolution of 336 × 336, while qwen-vl and minigpt-v2 utilize 448 × 448. however, higher resolutions lead to longer token sequences, incurring additional training and inference costs. minigpt-v2 addresses this by concatenating 4 adjacent visual tokens in the embedding space to reduce length. recently, monkey @xcite proposed a solution to enhance the resolution of input images without retraining a high-resolution visual encoder, utilizing only a low-resolution visual encoder, supporting resolutions up to 1300 × 800. to enhance the understanding of rich-text images, tables, and document content, docpedia @xcite introduced a method to increase the visual encoder resolution to 2560 × 2560, overcoming the limitations of poorly performing low resolutions in open-sourced vit. secondly, the incorporation of high-quality sft data can significantly improve performance in specific tasks, as evidenced by the addition of sharegpt4v data to llava-1.5 and vila-13b, as shown in ## conclusion in this paper, we have presented a comprehensive survey of mm-llms with a focus on recent advancements. initially, we categorize the model architecture into five components, providing a detailed overview of general design formulations and training pipelines. subsequently, we introduce various sota mm-llms, each distinguished by its specific formulations. our survey also sheds light on their capabilities across diverse mm benchmarks and envisions future developments in this rapidly evolving field. we hope this survey can provide insights for researchers and contribute to the ongoing advancements in the mm-llms domain.