From Specific- MLLM s to Omni- MLLM s: A Survey on MLLM s Aligned with Multi-modalities (2025)

Abstract: To tackle complex tasks in real-world scenarios, more researchers are focusing on Omni-MLLMs, which aim to achieve omni-modal understanding and generation. Beyond the constraints of any specific non-linguistic modality, Omni-MLLMs map various non-linguistic modalities into the embedding space of LLMs and enable the interaction and understanding of arbitrary combinations of modalities within a single model. In this paper, we systematically investigate relevant research and provide a comprehensive survey of Omni-MLLMs. Specifically, we first explain the four core components of Omni-MLLMs for unified multi-modal modeling with a meticulous taxonomy that offers novel perspectives. Then, we introduce the effective integration achieved through two-stage training and discuss the corresponding datasets as well as evaluation. Furthermore, we summarize the main challenges of current Omni-MLLMs and outline future directions. We hope this paper serves as an introduction for beginners and promotes the advancement of related research. Resources have been made publicly availableat https://github.com/threegold116/Awesome-Omni-MLLMs.

## introduction the remarkable performance of continuously evolving multi-modal large language models (mllms) has pointed to a possible direction for achieving general artificial intelligence @xcite . mllms extend large language models (llms) by integrating them with pre-trained models tailored to specific modalities, such as vision-mllms @xcite @xcite , audio-mllms @xcite , and 3d-mllms @xcite . however, these modality-specific mllms (specificmllms) are insufficient to tackle complex tasks in real-world scenarios that simultaneously involve multiple modalities. therefore, efforts are being made to expand the range of modalities for understanding and generation, giving rise to the omni-modality mllms (omni-mllms). by integrating multiple pre-trained models of more non-linguistic modalities @xcite @xcite @xcite , omni-mllms expand the modalities for understanding and even generation based on specific-mllms. omni-mllms leverage the emergent capabilities of llms to treat various non-linguistic modalities as different foreign languages, enabling the interaction and understanding of information across different modalities within a unified space @xcite . compared to specific-mllms, omni-mllms can perform multiple unimodal understanding and generation tasks, as well as cross-modal 1 tasks across two or more nonlinguistic modalities, allowing a single model to handle arbitrary combinations of modalities. a review of the development of omni-mllms reveals that it has been continuously expanding in three directions. on the one hand, the types of modalities processed by omni-mllm have been continuously increasing, from x-llms that handle vision and audio to x-instructblip @xcite which adds 3d modality capabilities, pandagpt @xcite that incorporates imu modality, and finally one-llm @xcite , which processes eight different modalities simultaneously. on the other hand, the ability to interact across modalities of omni-mllms has also expanded, from the joint 3d-image and audio-image cross-modal reasoning capability in imagebind-llm @xcite to the crossmodal generation capability of codi-2 that leverages interleaved audio and image contexts to generate both audio and images @xcite . the omni-mllm is thus trending towards an "any-to-any" model. besides, the application scenarios of omni-mllms have been broadened, encompassing real-time multimodal speech interaction like mini-omni2 and lyra @xcite , world simulation like @xcite , multi-sensor autonomous driving like drivemlm @xcite , etc. in addition to the open-source models, there are also some closed-source omni-mllms such as gpt-4o (openai), gemini @xcite , and reka @xcite . the timeline of omni-mllms is shown in figure 1 . despite the emergence of numerous omni-mllms, there is still a lack of systematic evaluation and analysis. to fill the gap, we propose this work to conduct a comprehensive and detailed analysis of omni-mllms. we first review the architecture of omni-mllms in four parts ( §2). next, we summarize how omni-mllms expand across multiple modalities through the two-stage training process ( §3); then present the training data construction and performance evaluation ( §4). furthermore, we highlight some key challenges and future directions ( §5). finally, we provide a brief summary ( §6) and discuss related surveys in the appendix a. our contributions can be summarized as follows: ## omni-mllm architecture as the extension of specific-mllms, omni-mllms inherit the architecture of encoding, alignment, interaction, and generation and broaden the types of non-linguistic modalities involved. this section introduces the implementation methods and functions of the four components in omni-mllm: multi-modalities encoding ( §2.1), multimodalities alignment ( §2.2), multi-modalities interaction ( §2.3), and multi-modalities generation ( §2.4). more details about the architecture of omni-mllms are shown in appendix b. ## omni-mllm training to achieve alignment across different vector spaces and improve instruction-following ability under arbitrary modality settings, omni-mllms extend the standard two-stage training pipeline of specific-mllms: multi-modalities alignment pre-training and multi-modalities instruction fine-tuning. ## data construction and evaluation this section summarizes the construction of modality alignment data and instruction data used in the omni-mllm training process ( §4.1), as well as the evaluation across four different capabilities ( §4.2). ## challenges and future directions despite omni-mllms having showcased remarkable performance on numerous tasks, there are still some challenges that necessitate further research. ## conclusion in this paper, we provide a comprehensive survey report on omni-mllm, offering a comprehensive review of the field. specifically, we break down omni-mllm into four key components and categorize them based on modal encoding and alignment methods. subsequently, we provide a detailed summary of the training process of omni-mllm and the related resources used. we also summarize the current challenges and the future development directions. this paper is the first systematic survey dedicated to omni-mllms. we hope this survey will facilitate further research in this area.