Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning (2024)

Abstract: Despite vision-language models’ (VLMs) remarkable capabilities as versatile visual assistants, two substantial challenges persist within the existing VLM frameworks: (1) lacking task diversity in pretraining and visual instruction tuning, and (2) annotation error and bias in GPT-4 synthesized instruction tuning data. Both challenges lead to issues such as poor generalizability, hallucination, and catastrophic forgetting. To address these challenges, we construct Vision-Flan, the most diverse publicly available visual instruction tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances sourced from academic datasets, and each task is accompanied by an expert-written instruction. In addition, we propose a two-stage instruction tuning framework, in which VLMs are firstly finetuned on Vision-Flan and further tuned on GPT-4 synthesized data. We find this two-stage tuning framework significantly outperforms the traditional single-stage visual instruction tuning framework and achieves the state-of-the-art performance across a wide range of multi-modal evaluation benchmarks. Finally, we conduct in-depth analyses to understand visual instruction tuning and our findings reveal that: (1) GPT-4 synthesized data does not substantially enhance VLMs’ capabilities but rather modulates the model’s responses to human-preferred formats; (2) A minimal quantity (e.g., 1,000) of GPT-4 synthesized data can effectively align VLM responses with human-preference; (3) Visual instruction tuning mainly helps large-language models (LLMs) to understand visual features.

## introduction recent vision-language models (vlms) @xcite @xcite , built upon pre-trained large-language models (llms) @xcite and pretrained image encoders @xcite , have shown impressive capabilities as general visual assistants. besides the unimodal encoders, the main ingredients of these vlm frameworks encompass: (1) a bridging module, such as the mlp layers in the llava model @xcite , that establishes connections between the pretrained image encoders and llms, (2) large-scale textimage pairs @xcite used for pre-training the bridging module, and (3) gpt-4 synthesized visual instruction tuning datasets @xcite to align the responses of vlms with human preferences (i.e., following users' instruction to generate detailed and helpful responses). despite their notable successes, we identify two remaining challenges that merit further investigation. firstly, the data used in the pre-training stage is dominated by the image captioning task, which lacks diversity, resulting in limited generalizability of vlms @xcite . for instance, the llava model @xcite performs poorly on the optical character recognition (ocr) task due to the absence of instances related to text detection during pre-training @xcite . several recent studies address this problem by further fine-tuning vlms on instruction tuning datasets covering more tasks @xcite @xcite ) such as visual question answering and ocr but the coverage of the tasks is still limited. secondly, most of the existing visual instruction tuning datasets @xcite @xcite are synthetically generated via gpt-4 by repurposing text annotations such as captions or dense captions from existing computervision datasets to generate new tasks, such as visual dialogue, complex vqa and detail captions. while they enable vlms to generate fluent and detailed responses aligned with human preferences, the lack of task diversity, spurious co-occurring patterns between objects, and long-form outputs may cause severe hallucination @xcite @xcite , and catastrophic forgetting -vlms fail to maintain a similar classification performance on basic detection tasks, such as mnist @xcite and @xcite , compared to the zero-shot performance of their vision encoders @xcite . to address both challenges, we introduce vision-flan, the most diverse public-available visual instruction tuning dataset consisting of 187 tasks drawn from academic datasets, covering perception tasks such as object detection and ocr, domain-specific tasks such as image-quality classification and image-style classification, complex reasoning tasks such as graph understanding and geometric question answering, and many more. each task in vision-flan is accompanied by an expertwritten instruction. we show some sample tasks from vision-flan in figure 1 and provide the full list of tasks in appendix j. in addition, we introduce a two-stage instruction tuning framework. in the first stage, we utilize the pre-trained llava model @xcite ) as our initial model, and finetune it on vision-flan to gain diverse capabilities, resulting in the vision-flan base model. however, due to the concise nature of target outputs in academic datasets, the responses generated by vision-flan base tend to be brief and not aligned with human preferences. therefore, in the second stage, we further finetune vision-flan base using a minimal amount of gpt-4 synthesized data. this step aims to adjust the model's outputs to be more in line with human preferences, resulting in the vision-flan chat model. our experimental results demonstrate that highquality human annotations from vision-flan significantly enhance the capabilities of both vision-flan base and vision-flan chat while reducing the risk of hallucination and catastrophic forgetting. the two-stage instruction tuning framework enables vision-flan chat to achieve better human preference alignment using much less gpt-4 synthesized data compared to the state-of-the-art vlms. finally, we perform extensive analysis to understand visual instruction tuning including the roles of human-labeled and gpt-4 synthesized data, and the impacts of various training strategies. our investigation yields several key insights: ## vision-flan finetuning model architecture we adopt the same vlm architecture as llava @xcite and denote it as llava-architecture. as shown in figure 3 , it consists of a pre-trained vision encoder, a pre-trained large language model, and two layers of mlps to connect them. in the vision-language pre-training phase of the llava-architecture, both the pre-trained vision encoder and large language model remain frozen, and only the mlp layers are trained on a large-scale image captioning dataset @xcite . we leverage this pre-trained llava model , without any visual instruction tuning, as our initial model and finetune it on vision-flan. during visual instruction tuning, we finetune both the mlp layers and the language model while keeping the vision encoder frozen. implementation details we leverage llava-architecture with vicuna-13b v1.5 @xcite , clip-vit-l-336px @xcite and two layers of mlp as our vlm. for the firststage instruction tuning, we finetune the mlp layers and the language model on vision-flan for 1 epoch with a learning rate 2e-5 and per device batch size 16 on 8 a100 gpus. for the second-stage instruction tuning, we further finetune the mlp layers and the language model on 1,000 instances randomly sampled from the llava dataset @xcite with learning rate 1e-5 and per device batch size 8 on 8 gpus for 128 steps. in the following sections, we use llava dataset and gpt-4 synthesized data interchangeably. ## experiment setup evaluation datasets we evaluate the models on several widely adopted multimodal evaluation benchmark datasets including multiple-choice benchmarks: mmbench @xcite , mme @xcite , and mmmu; free-form generation benchmarks: mm-vet @xcite and llava-bench; the hallucination benchmark: pope @xcite , and catastrophic forgetting benchmarks: cifar-10 and cifar-100 @xcite , mnist (lecun, 1998), and miniimagenet @xcite . more details of the evaluation datasets can be found in appendix b. evaluation protocols for mmbench, mme, mm-vet, llava-bench, pope and mmmu, we strictly follow their official implementations of evaluation code to evaluate the performance of each model. for datasets that do not have official evaluation codes including cifar-10, cifar-100, mnist, and miniimagenet, we leverage the stateof-the-art open-source llm, vicuna 1.5 13b, to perform the evaluation and report the averaged performance on these four datasets in the cf column in ## related work instruction tuning @xcite is first introduced in nlp and has been adapted to the visuallanguage domain. multiinstruct @xcite propose the first human-label multi-modal instruction tuning dataset for improving the zero-shot performance of pre-trained vlms. llava @xcite leverage gpt-4 to repurpose text annotations such as captions or dense captions from existing computer-vision datasets to generate visual dialogues, complex vqa and detail captions for visual instruction tuning. following llava, mplug-owl @xcite , lamm @xcite , mimic-it @xcite and macaw-llm @xcite leverage proprietary llms such as gpt-4 and chatgpt to further extend the instruction tuning tasks into 3d-domain, multipleimages and videos, and increase the amount of training instances. minigpt-4 @xcite utilizes chatgpt to refine output from the pretrained vlm itself. instructblip @xcite and llava-1.5 @xcite ) mix the humanannotated and gpt4 synthesized datasets to enhance visual instruction tuning. several recent work explores different strategies to improve visual instruction tuning. sta-blellava @xcite and vpg-c @xcite generate both images and texts using stable diffusion @xcite or blended diffusion @xcite to alleviate domain bias and encourage vlms attend to visual details. @xcite demonstrate the bias introduced by positive instructions and introduce negative instruction examples for improving robustness. shikra @xcite incorporate visual grounding tasks in visual instruction tuning to improve the vlm's referential capability. llavar @xcite and bliva @xcite leverage ocr tools and gpt-4 to generate tasks helping vlms to understand text in images. @xcite and svit @xcite empirically study the effect of scaling the size of vlms and the size of gpt-4 synthesized dataset. two concurrent works @xcite directly prompt gpt-4v with images as input to generate visual instruction tuning data and achieve superior performance. additional related work can be found in appendix g. unlike all prior work, our work mainly focuses on scaling human-labeled tasks in visual instruction tuning to improve vlms' capabilities. additionally, we perform extensive analysis to understand the characteristics of human-labeled and gpt-4 synthesized data and draw meaningful conclusions. ## conclusion we construct vision-flan, the most diverse public-available visual instruction tuning dataset, consisting of 187 diverse tasks and 1,664,261 instances collected from academic datasets, and each task is accompanied by an expert-written instruction. we demonstrate that vlms trained on vision-flan with proposed two-stage tuning framework achieve state-of-the-art performance on comprehensive evaluation benchmarks. additionally, we perform extensive analysis and reveal the distinct contributions of human-labeled and gpt-4 synthesized data in visual instruction tuning.