Concept-skill Transferability-based Data Selection for Large Vision-Language Models (2024)

Abstract: Instruction tuning, or supervised finetuning on extensive task-specific data, is necessary for Large Vision-Language Models (LVLMs) to generalize well across a broad range of vision-language (VL) tasks. However, training on large VL datasets can become prohibitively expensive. In this work, we introduce COINCIDE, an effective and scalable data selection technique that uses a small model as a reference model to select visual instruction tuning data for efficient finetuning of a target LVLM, focusing on diversity and transferability. Specifically, we cluster the training data using internal activations from a small model, which identifies VL concept-skill compositions needed by a target LVLM. We then sample data from these diverse clusters by considering their density and transferability, or the ability to transfer well to other concept-skill compositions. This approach ensures the diversity of these compositions, which is vital for LVLM generalization. Extensive experiments demonstrate that COINCIDE achieves superior performance and data selection efficiency against 8 strong baselines on two distinct datasets: LLaVA-1.5 and Vision-Flan. Using only 20% of the LLaVA-1.5 dataset, COINCIDE achieves performance comparable to the LVLM finetuned on the whole dataset, with 70% reduction of the wall-clock running time. On the Vision-Flan dataset, our method achieves superior results with only 16.7% of the training data.

## introduction large vision-language models (lvlms) @xcite @xcite are often built by (1) pretraining on paired image-caption datasets and (2) subsequent finetuning on image-instruction data on diverse vision-language (vl) tasks. the second † equal advising sharegpt ocr-vqa gqa vqav2 a-okvqa okvqa textcaps refcoco vg llava-conv llava-reason llava-detail self-filter score distribution density density el2n score distribution score mostly from -a-okvqa -llava-reason -ocr-vqa mid 20% top 20% mostly from -sharegpt -gqa ## related work coreset selection coreset selection attempts to extract a subset of training data that functions comparably to the full training set. this technique is adopted for problems like active learning @xcite , continual learning @xcite , and data pruning @xcite . recent works @xcite investigate coreset selection for instruction tuning of llms. alpagasus @xcite uses @xcite to rate the quality of instruction samples. s2l @xcite leverages the training loss trajectory of smaller models to find optimal samples for training larger llms. diverseevol @xcite utilizes the target model itself to iteratively choose beneficial data for the current training episode. ## method we start by introducing the framework that utilizes neuron activations from a small lvlm to group vit data into clusters, where each cluster comprises samples exhibiting a similar concept-skill composition (section 3.2). next, we conduct experiments to examine the correlation between the similarity of a cluster centroid to other centroids and the transferability of that cluster to others (section 3.3). based on our findings, we describe our data selection strategy, which performs cluster-wise sample selection by selecting different numbers of samples from clusters depending on their transferability and diversity (section 3.4). the overall framework of our approach is illustrated in figure 3 . ## conclusion in this paper, we introduce coincide, a clusterlevel data selection technique for efficient visual instruction tuning of large vision-language models. we demonstrate that clustering based on internal activations from a small model can represent visuallinguistic concept-skill compositions shared among diverse tasks in visual instruction tuning datasets. additionally, our empirical investigation validates a strong positive correlation between cosine similarity and transferability among clusters. based on the transferability and density of clusters, coin-cide selects more samples from more transferable and less dense clusters to enhance training efficacy, while preserving the diversity of concept-skill compositions within the coreset to ensure better model generalization ability. comprehensive experiments on the llava-1.5 and vision-flan datasets demonstrate that our method outperforms baselines across several benchmarks with the lowest data selection cost, showcasing its effectiveness and efficiency. the success of coincide suggests redundancy in popular vit datasets and underscores the importance of a thorough understanding of data in training lvlms.