MME vol: Empowering Multimodal Large Language Models with Evol-Instruct (2025)

Abstract: The development of Multimodal Large Language Models (MLLMs) has seen significant progress, driven by increasing demands across various fields (e.g., multimodal agents, embodied intelligence). While model-driven approaches aim to enhance MLLM capabilities through diverse architectures, their performance gains have become increasingly marginal. In contrast, data-driven methods, which scale up image-text instruction datasets, have proven more effective but face challenges related to limited data diversity and complexity. The absence of high-quality instruction data remains a major bottleneck in MLLM development. To address this issue, we propose , a novel multimodal instruction data evolution framework. This framework iteratively enhances data quality through a refined combination of fine-grained perception, cognitive reasoning, and interaction evolution, generating a more complex and diverse image-text instruction dataset that significantly improves MLLM capabilities. Starting with an initial dataset, SEED-163K, we employ to systematically expand instruction diversity, extend visual reasoning steps to improve cognitive abilities, and extract fine-grained visual details to enhance understanding and robustness. To rigorously evaluate our approach, we conduct extensive qualitative analysis and quantitative experiments across 13 vision-language tasks. Compared to baseline models trained on the original seed dataset, our method achieves an average accuracy improvement of 3.1 percentage points. Moreover, our approach attains state-of-the-art (SOTA) performance in nine tasks while using significantly less data than existing state-of-the-art models.

## introduction multimodal large language models (mllms) @xcite @xcite have advanced rapidly over the past two years, emerging as the dominant paradigm for vision-language tasks @xcite @xcite . these models integrate visual encoders @xcite @xcite with large language models (llms) @xcite @xcite @xcite , leveraging large-scale coarse-grained image-text pre-training @xcite @xcite , followed by small-scale instruction-tuning @xcite . this approach has led to remarkable advancements in vision-language understanding, enabling applications across multimodal agents, embodied intelligence, and real-world reasoning tasks. existing model-driven approaches @xcite seek to enhance mllm performance by refining network architectures for more effective integration of textual and visual knowledge. however, the marginal benefits of these architectural improvements are diminishing due to the absence of high-quality training data. furthermore, redundant architectural modifications fail to push the boundaries of model intelligence. conversely, data-driven approaches @xcite @xcite @xcite have demonstrated greater effectiveness by expanding the scale of image-text instruction datasets. however, these methods often suffer from limited data diversity and insufficient complexity, constraining the models' ability to generalize and reason effectively. the lack of high-quality, diverse instruction data remains one of the most critical challenges in advancing mllms. thus, there is an urgent need to develop automated methods that can generate more complex, diverse, and high-quality instruction data at a low cost, thereby enhancing mllm capabilities. an analysis of existing data-driven methods for generating image-text instruction data reveals three major limitations. first, limited instruction diversity. manually annotated instructions are constrained by human cognitive limitations, while model-generated instructions rely on templatebased presets. this restricts the diversity of task formulations, making it difficult for mllms to generalize to real-world scenarios and limiting their instruction-following capabilities. second, limited instruction complexity. manually annotated instructions tend to be simple or moderately complex, while automatically generated instructions are often brief and lack multi-step visual reasoning. this prevents mllms from effectively handling complex vision-language tasks that require deeper cognitive reasoning. third, insufficient alignment granularity. both manually and model-generated instructions primarily focus on common objects, often neglecting rare or small visual elements. this results in coarse-grained image-text alignment, reducing the model's visual perception robustness and increasing susceptibility to hallucinations. to overcome these limitations, we propose mmevol, a novel multimodal instruction evolution framework that leverages advanced mllms for iterative instruction refinement. this framework automatically generates diverse, open-domain instructions at multiple difficulty levels, enhancing the overall instruction-following capabilities of mllms. given that visual-language instruction data must be tightly aligned with image content, existing evol-instruct @xcite methods often produce redundant restatements or instructions unrelated to images, making deep and broad instruction evolution difficult. to address this, we introduce a refined image-text instruction paradigm, which defines three key evolution directions. i) fine-grained perception evolution enhances visual detail extraction, generating instructions that capture more granular information from images. ii) cognitive reasoning evolution extends multi-step reasoning chains, improving the model's ability to process complex visual tasks. iii) interaction evolution increases instruction diversity by generating a wider variety of instruction formats. the mmevol mechanism is summarized in fig. 1 , where each evolution cycle consists of two main steps: instruction evolution and instruction elimination. during instruction evolution, one of the three evolution directions is randomly selected to transform simple instructions into more complex or diverse forms. since not all instruction transformations are successful, an instruction elimination step is applied to filter out ineffective or redundant evolutions. by iteratively refining the dataset through multiple rounds of evolution and elimination, mmevol produces a high-quality, complex instruction dataset that enhances mllm performance across a broad range of tasks. to validate the effectiveness of mmevol, we perform three rounds of evolutionary iterations on 163k seed data, leading to 447k evolved samples. we fine-tuned the open-source llava-next @xcite model with these evolved data and compared it with other advanced methods across 13 vision-language benchmarks. our method achieves state-of-the-art (sota) performance, demonstrating the effectiveness and efficiency of mmevol. additionally, we conduct detailed qualitative analysis and ablation experiments to showcase the contribution of each component of our method. we hope that the released evolutionary data and code will assist the community in understanding that using a small amount of high-quality image-text instruction data is far more critical than training mllms with large-scale low-quality image-text instruction data. our main contributions can be summarized as follows: (1) a image-text instruction evolution framework, mmevol, is designed to leverage advanced mllms, automating the generation of open-domain image-text instruction data across varying difficulty levels to enhance the diversity and complexity of existing datasets. ## method in this section, we first describe the curation process for the seed instruction data and then detail the methodological framework of mmevol. due to space constraints, we provide a concise overview of the seed data curation process and prompt templates, with additional details available in appendix f. ## conclusion in this work, we propose an image-text instruction evolution framework and explore the techniques, insights, and benefits of evol-instruct for enhancing the quality and quantity of image-text instruction data. we employ three distinct evolution methods to increase the complexity and diversity of instruction data based on a limited seed dataset while utilizing instruction elimination to filter out harmful data. the data evolved through three rounds of evolution is used to train a new model, demonstrating state-of-the-art (sota) performance across a comprehensive set of benchmarks. future directions include exploring integrating image generation models to synthesize new images and perform dual evolution of images and texts, aiming to train even more robust foundational models. 6 limitation despite the effectiveness of mmevol, certain limitations remain. due to computational constraints, we performed instruction evolution on 163k samples, which represents approximately 12% of the available data in existing instruction datasets. additionally, our experiments were conducted using an 8b-scale model, meaning that results could further improve with larger models and expanded training sets. another limitation stems from the use of the openai gpt-4o mini api in our evolution process. while gpt-4o mini provides high-quality instruction generation, its reliance on a proprietary model limits reproducibility. in future work, we plan to replace it with open-source alternatives, such as qwen2vl, to ensure greater accessibility and transparency in instruction evolution. furthermore, while mmevol effectively improves instruction complexity and diversity, future extensions could explore hierarchical evolution mechanisms that adaptively adjust instruction complexity based on model progression. expanding long-tail visual concepts and introducing multimodal active learning could further refine model training, helping mllms navigate real-world multimodal challenges with even greater accuracy and robustness.