MLAN : Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models (2025)

Abstract: We present a novel visual instruction tuning strategy to improve the zero-shot task generalization of multimodal large language models by building a firm text-only knowledge base. Existing work lacks sufficient experimentation on the importance of each modality in the instruction tuning stage, often using a majority of vision-language data while keeping text-only data limited and fixing mixtures of modalities. By incorporating diverse text-only data in the visual instruction tuning stage, we vary vision-language data in various controlled experiments to investigate the importance of modality in visual instruction tuning. Our comprehensive evaluation shows that the text-heavy instruction tuning approach is able to perform on par with traditional vision-heavy mixtures on both modalities across 12 general datasets while using as low as half the total training tokens. We find that simply increasing sufficiently diverse text-only data enables transfer of instruction following ability and domain knowledge across modalities while being more efficient than the vision-language approach.

## introduction multimodal large language models (mllms) have advanced and enabled a wide range of vision-language tasks such as visual question answering and image captioning @xcite @xcite @xcite . their zero-shot generalization ability to unseen tasks has the potential to further revolutionize broader real-world applications @xcite @xcite . to construct mllms, vision-language pretraining is performed on a large scale with imagetext data, aligning the modalities before visual instruction tuning aligns the model with human pref-* equal contribution † corresponding author erences @xcite @xcite . the importance of strong vision-language pretraining is established, with more data resulting in greater improvements in instruction-following abilities and downstream performance @xcite . however, current visual instruction tuning practices overwhelmingly rely on image-text pairs and large-scale visionlanguage datasets. this emphasis introduces a significant distributional shift from the language-rich corpora used during pretraining, often degrading the model's general language understanding and leading to catastrophic forgetting of core knowledge @xcite . given the similarity in instruction tuning data across modalities and the strong modality alignment achieved with visionlanguage pretraining, we believe text-only data is underutilized in existing training mixtures. additionally, various design choices regarding the instruction tuning dataset composition with respect to modalities are underexplored. in this work, we introduce mlan (multimodal language-based instruction tuning), a new perspective in vision instruction tuning that treats language as the primary way to unlock knowledge during instruction tuning (figure 1 ). our key insight is that instruction-following abilities and domain knowledge, once acquired through diverse language-only tasks, can generalize across modalities with minimal vision-language supervision. by grounding vision capabilities in a small number of targeted image-text examples, we maintain high performance across both vision and text tasks while significantly reducing training costs. specifically, with mlan we unlock vision instruction following abilities by teaching a pretrained model to execute text-only instructions and then complementing the dataset with a relatively small portion of vision-language examples in a domain adaptation fashion. to demonstrate mlan's effectiveness, we preafter vision-language pretraining, we include diverse text-only data in our instruction tuning mixture spanning many tasks. we emphasize including text-only data to show the transferability of instruction tuning across modalities. for evaluation, we select ample text-only and vision-language datasets, allowing us to compare performance changes across modalities. (b) we evaluate mlan on two pretrained multimodal models based on llama-3.2-3b and llama-3.1-8b across unseen language and vision benchmarks, achieving comparable performance at higher training efficiency (up to almost 2x as efficient compared to standard vision-heavy instruction tuning) with our language-based approach. ## approach mlan views vision instruction following abilities as a natural extension of text-only abilities, a transfer that can occur due to the extensive multimodal pretraining used in mllms. we begin by motivating our method through an empirical analysis of similarities in text-only and vision-language instruction tuning data, which leads to our hypothesis that text-only data can largely replace vision-language data to improve performance on general tasks. then, we detail our training, following the standard design of existing instruction tuning methods @xcite @xcite in four stages: selecting training data, format-(a) distribution of cross-modal similarity scores between modalities with a non-negative mean by z-test (@xmath0.001). ## experiments in this section, we show that mlan is both more effective and training efficient compared to the pretrained mllms as well as state of the art multimodal instruction tuning mixtures across all the tasks we evaluate. additional details of the training and experimental setup are described in appendix b. ## related work multimodal large language models (mllm) are language models endowed with the ability to use multiple modalities, such as images, videos, and audio @xcite @xcite @xcite @xcite @xcite @xcite @xcite . the most widely adopted are vision enhanced llms, where many design choices are already extensively studied @xcite @xcite @xcite @xcite . a prevalent approach to building such mllms links pretrained visual encoders @xcite to llms @xcite @xcite via an adapter, thus transforming deep image features into soft prompts for the base llm. in our work, we focus on one of the simplest yet high-performing and widely adopted mllms, using only a multi-layer perceptron as the adapter @xcite @xcite @xcite . inspired by the success of instruction tuning in llms in zero-shot generalization @xcite @xcite , following a pretraining step for vision-language feature alignment, there is a multimodal instruction tuning step to improve zeroshot performance on multimodal tasks @xcite . notably, instructblip @xcite and llava @xcite transform existing datasets into multimodal instructions using manual templates and synthetic data, a practice expanded upon in subsequent work @xcite @xcite . further work investigates how instruction tuning varies under different settings, e.g., how different components of the mllm should learn differently during instruction tuning @xcite and how instruction tuning works in a continual learning setting with many new tasks @xcite . however, there lacks a comprehensive set of experiments that varies the composition of each modality in instruction tuning. though the primary goal of multimodal instruction tuning is to improve vision-language performance, text-only data is often included in both pretraining @xcite and finetuning @xcite @xcite @xcite @xcite @xcite @xcite to prevent catastrophic forgetting and improve language performance. many such papers disregard the impact of finetuning with text-only data on vision performance, focusing solely on language performance when ablating text-only data away, though there are notable exceptions @xcite @xcite @xcite . in these cases, there is modest evidence of transferability between modalities, where finetuning on both language and vision data exhibits about equal or better performance than training on one modality alone. however, in each of the existing work that finetune with text-only data alongside vision data, this performance boost is achieved by increasing the dataset size without consideration of how such data will increase the training cost (with the exception of @xcite , which only tests with a low amount of text-only data). hence, even though better performance is obtained when increasing the dataset size to train on text-only data, the instruction tuning step is more costly. due to the general cost of instruction tuning a mllm, many approaches aims to decrease the cost of instruction tuning in the multimodal setting. these primarily include using lightweight adapters to decrease the number of parameters @xcite @xcite and choosing a subset of the training data using the mllm itself or other methods @xcite @xcite @xcite . a simpler way to decrease the cost is to instruction tune with a focus on text-only data. since training on language instruction data is cheaper than training on the same number of vision instances, and language is foundational to the functioning of mllms, we focus on such a language-based approach. ## conclusion we present mlan, a language-based multimodal instruction tuning strategy for mllms that enhances zero-shot generalization and promotes effective knowledge transfer across modalities. we demonstrate-through controlled ablations under fixed training budgets-that language-based tuning establishes a robust knowledge foundation, even for tasks requiring visual understanding. crucially, mlan achieves strong performance on both language and vision benchmarks while significantly reducing reliance on image supervision. our results show that language is not only sufficient but essential for efficient and generalizable multimodal learning. with mlan, we hope to bring attention to the importance of language in mllms in visual instruction tuning, which we believe can be used in future work to improve training efficiency and performance.