On Domain-Adaptive Post-Training for Multimodal Large Language Models (2025)

Abstract: Adapting general multimodal large language models (MLLMs) to specific domains, such as scientific and industrial fields, is highly significant in promoting their practical applications. This paper systematically investigates domain adaptation of MLLMs via post-training, focusing on data synthesis, training pipeline, and task evaluation. (1) **Data Synthesis**: Using only open-source models, we develop a generate-then-filter pipeline that curates diverse visual instruction tasks based on domain-specific image-caption pairs. The resulting data surpass the data synthesized by manual rules or strong closed-source models in enhancing domain-specific performance. (2) **Training Pipeline**: Unlike general MLLMs that typically adopt a two-stage training paradigm, we find that a single-stage approach is more effective for domain adaptation. (3) **Task Evaluation**: We conduct extensive experiments in high-impact domains such as biomedicine, food, and remote sensing, by post-training a variety of MLLMs and then evaluating MLLM performance on various domain-specific tasks. Finally, we fully open-source our models, code, and data to encourage future research in this area.

## introduction general multimodal large language models (mllms; @xcite have shown impressive capabilities in general scenarios. however, their expertise plummets in specialized domains due to insufficient domainspecific training @xcite @xcite . for instance, scientific fields require learning from specialized images not commonly found in general scenarios @xcite ; and industrial applications face privacy constraints that limit data access for general training @xcite . domain-specific training for mllms requires diverse visual instruction tasks infused with domain knowledge @xcite . for domain-specific instruction synthesis, while using closed-or opensource models to synthesize data is common in training general mllms, challenges arise when applying this approach to domain-specific mllms due to privacy concerns with closed-source models @xcite and the lack of domain expertise in open-source models. for domain-specific training, the two-stage pipeline-first training on image-caption pairs, then on visual instruction tasks-is widely adopted for developing general mllms @xcite . however, tasks in specialized domains are often limited, and splitting them into two stages can further reduce task diversity within each stage. in this paper, we systematically investigate domain-specific instruction synthesis and training pipelines for mllm post-training, referred to as domain-adaptive training. specifically, we aim to ( 1 ) leverage only open-source models for data synthesis to avoid the privacy concerns of closedsource models and (2) maintain domain-specific task diversity throughout the post-training stage. for instruction synthesis with open-source models, we propose a generate-then-filter pipeline to address the lack of domain expertise. first, an open-source mllm is fine-tuned to output instruction-response pairs based on input domainspecific image-caption pairs. to ensure diversity, we curate a seed data collection encompassing various domains and tasks based on existing datasets, without the need for additional expert annotation. the fine-tuned mllm can effectively leverage domain knowledge in the image-caption source to generate diverse instruction-response pairs. then, to ensure synthetic response accuracy, instead of directly verifying each response based on the instruction-which requires significant expertise-we propose selecting tasks with inherently consistent responses. this significantly improves accuracy while reducing the need for expert annotation. although generated from open-source models, our synthetic tasks improve model performance more effectively than those generated by manual rules and strong closed-source models. for training pipeline, in the area of domain-adaptive post-training of mllms, two-stage training remains mainstream @xcite @xcite . however, we find that splitting the training data into two separate stages may hinder training task diversity and efficiency. therefore, we apply a single-stage training pipeline that combines the synthetic task with its corresponding image-caption pair. this simple method enriches task diversity during training and leads to better performance. in contrast to previous works @xcite @xcite which focus on a single domain or a single series of mllms per work, we conduct experiments across a variety of high-impact domains, such as biomedicine, food, and remote sensing, on general mllms of different sources and scales, such as qwen2-vl-2b @xcite , llava-v1.6-8b @xcite , and llama-3.2-vl-11b @xcite . as shown in figure 1 , our resulting model, adamllm (short for adapted multimodal large language model), consistently outperforms general mllms across various domain-specific tasks. in summary, our contributions include: • to the best of our knowledge, we present the first systematic investigation of domain-adaptive post-training of mllms through extensive experiments across diverse domains and mllms. our work is related to domain-specific mllm @xcite @xcite , instruction synthesis, and mllm training strategy. this section focuses on domain-specific mllm, and the other topics are discussed in appendix a. domain-specific data initially, @xcite utilized multimodal interleaved data. with the rise of visual instruction tuning, research has shifted to synthesizing visual instructions @xcite . the approaches fall into two categories: our work aligns with the second in synthesizing domain-specific data based on image-caption pairs, but we utilize open-source models to avoid privacy concerns and lead to even better performance. ## method we adapt mllms to domains via post-training. figure 2 provides the method overview: we begin by synthesizing domain-specific tasks using a unified visual instruction synthesizer, followed by a consistency-based data filter. the synthetic tasks are then combined with image-captioning tasks into a single stage for post-training. classify the image based on its domain. ## experiment settings we conduct experiments in high-impact domains including biomedicine, food, and remote sensing, because they are the only ones for which we can access public data without raising ethical concerns. for each domain, we perform post-training to adapt general mllms and evaluate model performance on various domain-specific tasks. note that our implementation is fully opensource, and the only required change for applying our method to a new domain is to replace the image-caption source with that of the new domain. image-caption data source for biomedicine domain, we use two sources: pmc raw in @xcite and pmc ref ined in @xcite . for food domain, we collect data from recipe1m @xcite . for remote sensing domain, we collect data from five imagecaptioning tasks. data details are in appendix c. ## main results overall performance as shown in tables 2 and 3 , ours consistently enhances mllm performance, outperforming baselines across various domain-specific tasks. although our synthesizer is based on llava-v1.6-8b, we observe consistent improvements on qwen2-vl-2b and llama-3.2-11b, demonstrating its effectiveness across different models and scales. among the evaluated tasks, vqa-rad, recipe1m and nwpu can be regarded as partially seen tasks, with vqa-rad included in our seed data for fine-tuning the synthesizer, recipe1m and nwpu included in the imagecaption source 1 . nevertheless, adamllm shows consistent gains on other unseen tasks, demonstrating strong task generalization in the target domain. ## ablations to evaluate the effectiveness of each component, we conduct ablations to post-train llava-v1.6-8b with different settings. we report the average task performance within each domain for the trained models in domain knowledge, task (cot) format & seed data our synthetic tasks are designed to integrate both (1) domain knowledge and (2) a visual instruction task format with chain-of-thought response. to assess the role of domain knowledge, we compare our method with general tasks that preserve the same cot response format but exclude importantly, "general cot task" is constructed from our seed data-originally used to finetune the synthesizer-and filtered using the same consistency-based method. this suggests that the performance gain of our method is not simply due to knowledge distillation from the seed data. moreover, our method outperforms "general cot task + domain caption", indicating that naively combining domain-specific image-caption pairs with general tasks is insufficient. in contrast, our synthesis pipeline effectively transforms the domain knowledge embedded in image-caption pairs into a format that general mllms can learn from more effectively. ## analysis ## conclusion this paper investigates adapting general mllms to specific domains via post-training. to synthesize domain-specific visual instruction tasks, we develop a unified visual instruction synthesizer that generates instruction-response pairs based on domain-specific image-caption data, and then apply a consistency-based filter to improve data accuracy. this enables us to effectively synthesize diverse tasks with high domain knowledge utilization. for the post-training pipeline, we propose combining the synthetic tasks with image-captioning tasks into a single training stage to enhance task diversity. in multiple high-impact domains, our resulting model, adamllm, consistently outperforms general mllms across various domain-specific tasks.