Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection (2024)

Abstract: Data selection in instruction tuning emerges as a pivotal process for acquiring high-quality data and training instruction-following large language models (LLMs), but it is still a new and unexplored research area for vision-language models (VLMs). Existing data selection approaches on LLMs either rely on single unreliable scores, or use downstream tasks for selection, which is time-consuming and can lead to potential over-fitting on the chosen evaluation datasets. To address this challenge, we introduce a novel dataset selection method, Self-Filter, that utilizes the VLM itself as a filter. This approach is inspired by the observation that VLMs benefit from training with the most challenging instructions. Self-Filter operates in two stages. In the first stage, we devise a scoring network to evaluate the difficulty of training instructions, which is co-trained with the VLM. In the second stage, we use the trained score net to measure the difficulty of each instruction, select the most challenging samples, and penalize similar samples to encourage diversity. Comprehensive experiments on LLaVA and MiniGPT-4 show that Self-Filter can reach better results compared to full data settings with merely about 15% samples, and can achieve superior performance against competitive baselines.

## introduction instruction-following foundation models, such as gemini @xcite , @xcite , have shown exceptional performance in multi-modality tasks. their capabilities are primarily derived from the reinforcement learning with human feedback (rlhf) @xcite paradigm and the instruction fine-tuning (ift) @xcite framework. lima @xcite indicates that while foundation models primarily acquire their knowledge during pre-training phases, only a small amount of instruction is needed for fine-tuning these models to produce the desired outputs during interactions. however, the low-quality instructions can significantly degrade the models' performance. to address this challenge, dataset selection tasks have been proposed, which aim to select high-quality instruction-tuning data to enhance the performance of instruction-following foundation models. existing data selection methods have many limitations in selecting high-quality instruction-tuning data. traditional data pruning metrics @xcite @xcite usually use the influence on model accuracy to filter samples, i.e., dependent on the classification tasks, and are unsuitable for the generative models. @xcite manually crafted highquality instruction dataset, which is costly and timeconsuming. @xcite query chatgpt to rank instructions, but the performance is unstable and there is no information from images in the multimodal setting. other works @xcite use pre-defined evaluation datasets to construct metrics, thus highly relying on the quality and the distribution of the selected evaluation datasets, which can potentially cause over-fitting problems and harm the models' generalization ability. these limitations prevent existing data selection approaches from being applied to vlms. to our best knowledge, instructiongpt-4 @xcite is the only dataset selection method focusing on vlms. they also introduce additional tasks to train a data selector, and their experiments are only limited to minigpt-4 on a small dataset with 3.4k samples. to address the aforementioned issues, we propose self-filter by leveraging vlm itself as an effective data filter. specifically, we first generate score embeddings for each training instruction through feature extractors, such as clip @xcite and gpt-4 vision, with the training instruction itself as inputs. after obtaining score embeddings, we use them to train a score net together with the target vlm. the score net takes the score embeddings as input and produces the weight of corresponding training instructions in vlm training losses. we expect that these weights can indicate the "difficulty" of the samples to be learned by vlms during instruction tuning. then we choose the instruction data with the highest difficulty to construct the filtered dataset. moreover, we introduce a penalty mechanism on similar training samples to enhance the diversity of instructions. compared with instructiongpt-4 @xcite , our method can easily transfer to any visionlanguage model with large-scale datasets. through comprehensive experiments on the widely studied vision-language models (e.g., llava and minigpt-4), we show that with only around 15% samples of the raw instruction tuning dataset, self-filter can surpass the model trained on the original instruction data across various evaluation datasets and benchmarks. self-filter also outperforms several competitive baselines and achieves state-of-the-art performance. our contributions are summarized as follows: ## methodology in this section, we briefly review the related background knowledge of vision-language models and elaborate on the task formulation. then we thoroughly introduce our proposed self-filter. ## experiments in this section, we first detail our settings and the chosen hyperparameters. then we introduce the evaluation benchmarks and datasets used in our experiments and the baseline methods. finally, we show that our proposed self-filter can get state-of-the-art performance through extensive experiments. ## analysis in this section, we first show the importance of the diversity module and choosing harder samples through ablation studies. then we show that our method can indeed select the difficult samples in the case study. finally, we analyze the influence of pruning size and batch size. ## conclusion in this paper, we propose self-filter to select high-quality instructions from the noisy dataset. our method leverages the vision-language model itself as a filter, and does not need any supplemental evaluation datasets, enhancing the model's generalization ability. we design a score net framework to effectively rank all the samples based on the predicted difficulty and discard low-quality instructions to help representation learning. our comprehensive experiments on llava and minigpt-4 demonstrate the effectiveness of our approach, and verify that large vision-language models learn most of their knowledge from the backbones and the pretraining stage, while a small amount of high-quality samples is sufficient for instruction tuning.