V i FT : Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models (2025)

Abstract: Visual instruction tuning has become the predominant technology in eliciting the multimodal task-solving capabilities of large vision-language models (LVLMs). Despite the success, as visual instructions require images as the input, it would leave the gap in inheriting the task-solving capabilities from the backbone LLMs, and make it costly to collect a large-scale high-quality dataset. To address it, we propose ViFT, a visual instruction-free fine-tuning framework for LVLMs. In ViFT, we only require the text-only instructions and image caption data during training, to separately learn the task-solving and visual perception abilities. During inference, we extract and combine the representations of the text and image inputs, for fusing the two abilities to fulfill multimodal tasks. Experimental results demonstrate that ViFT can achieve state-of-the-art performance on several downstream benchmarks, with rather less training data. Our code and data will be publicly released.

## introduction recently, large vision-language models (lvlms), built upon existing visual encoders @xcite and large language models (llms) @xcite , have gained widespread attention by demonstrating superior performance across diverse multimodal tasks @xcite . to empower lvlms with multimodal task-solving capabilities, a fundamental problem is to inherit and transfer the task-solving ability of llms into multimodal tasks (with image inputs). recently, visual instruction tuning @xcite has emerged as the predominant framework to achieve this goal. through fine-tuning on a variety of vision-language instruction-following data from different sources, lvlms can directly learn the corresponding knowledge and generalize into other related tasks. despite its success, it is still necessary to continue scaling up the number of visual instruction data for fully learning multimodal advanced capabilities (e.g., visual reasoning). however, there are two bottlenecks that greatly limit the scaling of visual instructions. first, due to the multimodal nature, visual instructions foot_0 need to incorporate both visual contents (e.g., images or videos) and related language instructions, which makes the creation of large-scale visual instructions much more challenging compared to unimodal language instructions. second, although existing work @xcite has adopted the data synthesis strategy for visual instructions, the synthesized instructions might include unreliable information regarding the visual inputs, leading to the risks of potential performance decline. considering the above challenges, we rethink whether it is feasible to reduce the reliance on visual instruction data during training lvlms. existing lvlms typically map visual inputs into the llm's token space and then generate the text output based on it. if the visual inputs have been well perceived and aligned with text tokens, the llm can comprehend the visual contents and leverage its inherent task-solving ability for tackling multimodal tasks. therefore, lvlm's multimodal tasksolving capability should be the combination of (1) the visual perception ability (for alignment) and (2) the task-solving ability from llms. although it is hard and costly to synthesize extensive amount of high-quality visual instructions for learning the multimodal capabilities, it is promising to sufficiently learn the two individual abilities separately, thanks to the rich resources of natural language instructions @xcite and image caption data @xcite . therefore, our goal is to disentangle and separately strengthen the two individual abilities during training, then combine them during inference to enhance lvlms. in this work, we propose a visual instruction-free fine-tuning framework (vift) for training lvlms. concretely, we ensure that the two key abilities are separately optimized after extracting from the lvlm, and then combined during inference. in this way, we can take advantage of using available image caption and natural language instruction data, to better learn the two individual abilities. during inference, we extract the hidden states of the lvlm by using only the image and text parts from the input visual instruction, which are the disentangled representation vectors @xcite corresponding to the two individual abilities. through the addition of two vectors, the lvlm can benefit from the improvement on the individual abilities and well fulfill multimodal tasks. note that as vift does not require any visual instruction data for finetuning, it can better inherit the original abilities from llms, and avoid the high costs for synthesizing high-quality visual instructions, especially in new visual domains. to study the effectiveness of our approach, we conduct extensive experiments on a series of benchmarks. our approach outperforms the visual instruction tuning baseline across all seven evaluation tasks, achieving an average performance gain of 6.3% (47.5 vs. 41.2). when compared with the state-of-the-art lvlm, llava-onevision, our vift framework attains superior average performance (47.5 vs. 45.5), while using less than 30% amount of the training data, as shown in figure 1 . the primary contributions of this work can be summarized as followed: ## related work large vision-language models. large visionlanguage models (lvlms) @xcite are capable of processing visual and textual inputs and tackling a variety of multimodal tasks. currently, visual instruction tuning is the predominant framework for training lvlms. by training on a large number of visual instructions, lvlms can directly learn the task-solving capabilities for the corresponding multimodal tasks. early studies @xcite leverage llms to synthesize image-related gpt-style visual instructions. subsequent studies leverage more advanced lvlms (e.g., gpt-4v) for higher-quality instruction synthesis @xcite and quantity scaling @xcite . in addition to general instruction following, another line of works focus on the lvlm's visual reasoning capability @xcite @xcite and the performance in other visual domains (e.g., geometry @xcite , scientific @xcite , and medical @xcite ). despite its success, it's costly to synthesize high-quality visual instructions, particularly when adapting to diverse new visual domains and visual tasks. representation engineering for llms. our approach is closely related to studies of the representation engineering for llms @xcite , which aims to extract a compact vector from the llm's intermediate representation (e.g., hidden states). the extracted representations can be leveraged to manipulate the llm's behaviour. an application of representation vectors is task arithmetic @xcite . through feature engineering (e.g., addition) of the representations, the llm's behaviour on target tasks can be effectively controlled. representation engineering is successfully implemented across various tasks, including style transfer @xcite , knowledge editing @xcite , and sentiment control @xcite . recent researches @xcite extend their application to in-context learning, where they are referred to as task vectors. in our study, we leverage the representations extracted from the lvlm to combine the individual abilities for solving multimodal tasks. ## preliminary existing lvlms @xcite generally consist of a pretrained visual encoder f (•) to process visual inputs (e.g., images or videos), a connection layer g(•) for feature projection, and an llm p(•) for autoregressive generation. during inference, given a visual instruction including an image input v and a text instruction query q, the image is first processed through visual encoder f (•) and connection layer g(•), producing visual tokens these tokens are then prepended to the tokens of the text input x q to compose the input of the llm for autoregressively generating the target text. to train the lvlm for integrating the visual encoder and llm, existing methods mainly incorporate two training stages: alignment pre-training and visual instruction tuning. the first stage only requires caption data and the second stage requires visual instructions. in this work, we aim to skip the visual instruction tuning stage, and only train the model with captions and text instructions for disentangling and improving the task-solving and visual perception abilities. for model architecture, we follow llava's design. concretely, we adopt siglip as the visual encoder according to its suggestion @xcite , and select qwen2.5-7b-instruct @xcite as our base llm due to its remarkable performance. for connection layer, we follow the widely-used setting in current lvlms @xcite ) that implement a simple 2-layer mlp. ## approach in this section, we introduce vift, a visual instruction-free fine-tuning framework for lvlms. our main motivation is that the multimodal tasksolving capability of lvlms can be split into the task-solving ability of llms and the visual perception ability, which can be separately learned through text-only instructions and image caption data. in vift, we first collect the above data to fine-tune the lvlm for learning the two individual abilities, and then extract their corresponding representation to integrate the individual abilities during inference to tackle multimodal tasks. we show the overall framework in figure 2 . ## experiment 5.1 evaluation benchmarks to evaluate the performance of vift, we conduct experiments on seven public benchmark datasets: ## further analysis transferability evaluation. we evaluate the transferability of vift across different model architectures. specifically, we modify our base vift-7b model by replacing the backbone llm (qwen2.5-7b-instruct) with llama-3-8b-instruct and qwen2.5-14b-instruct, and the visual encoder (siglip) with clip, while keeping all other experimental configurations unchanged. for all variant models, we train both vit-based and vift-based models based on the same training data used in our main experiment. then, under the same setting, we evaluate the performance of vift and vit for each model variant. we present the results in computation complexity. we examine the additional time overhead of ability-fused inference compared to standard inference. the results are presented in figure 3a . as we can observe, when generating short responses (e.g., 25 tokens), utilizing ability-fused inference may introduce an 8% increase in computational overhead. however, as the generation length increases, the generation speeds for standard inference and ability-fused inference gradually converge. when generating more than 400 tokens, ability-fused inference almost doesn't introduce any additional computational overhead. this aligns with our expectations. for ability-fused inference, we merely introduce one additional forward pass during the entire generation process. thus, while there is some discrepancy when generating short responses, such differences become negligible as generation length increases. ## conclusion in this paper, we proposed an instruction-free finetuning framework vift, for enhancing the multimodal task-solving capabilities of lvlms. concretely, instead of using visual instructions, we only leveraged text instructions and image caption data, to separately learn the individual task-solving and visual perception abilities for the lvlm. after that, we extracted the representation vectors by using the model's hidden space for the disentangled abilities via different modality inputs, and combined them to guide the inference of the lvlm in multimodal tasks. with rather less training data, our trained model, vift-7b, achieved state-of-the-art performance among competitive lvlms across various downstream benchmarks. furthermore, based on our proposed framework, we can efficiently scale the vision data and text data to enhance the model's performance, which facilitates further advancements in this field.