Paper: InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4 (ICLR-2024)

Reviewer comment (Experiments):
The experiments in the article are relatively limited and the dataset used has a small sample size, which raises doubts about the effectiveness of the proposed method when applied to larger datasets.

Author response:
Thanks for your questions. In particular, we want to find out **if there exists a subset making InstructionGPT-4 achieve better performance**. As a result, our proposed method indeed shows the existence of the subset. Using only 6% of the data, we manage to make InstructionGPT-4 surpass MiniGPT-4 in most tasks, which is quite surprising. 

To address your concern about scalability and generalizability, we also extend our investigation to include additional models and datasets. **Notably, we have applied our method to Qwen_VL [1] and the detail_10k [2] dataset without retraining the data selector**.
Here are our experimental results:

| detail_10k   | Qwen_VL (1k) | Qwen_VL (2k) | Qwen_VL (10k) | MiniGPT-4 (1k) | MiniGPT-4 (2k) | MiniGPT-4 (10k) |
|:------------:|:------------:|:------------:|:-------------:|:--------------:|:--------------:|:---------------:|
| MME          |    1802.50   |   **1806.81**|    1769.28    |   **614.04**   |      608.54    |      604.47     |
| Perception   |    1423.57   |   **1426.10**|    1398.92    |   **434.04**   |      431.04    |      428.54     |
| Cognition    |    378.93    |   **380.71** |    370.36     |   **180.00**    |      177.50    |      173.93     |

By comparing the performance between models tuned from selected subsets and the whole dataset, we observe that less instruction data for better performance still work for various MLLMs. Our findings indicate promising results, suggesting that our method is not only effective with MiniGPT-4's datasets but also exhibits potential for broader applicability. 

[1] Bai, et al. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023.

[2] Liu, Li, et al. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.

Reviewer comment (Experiments):
Regarding the data selector, although it appears to be concise, it can be time-consuming as it requires continuous training of the model on the validation dataset for testing. This becomes particularly challenging when dealing with large amounts of data.

Author response:
Thank you for your comment. In fact, we have designed our method with efficiency in mind, balancing the need for accurate data selection against the constraints of resource usage.

First of all, based on our data selection method mentioned in Section 2.3, **the time and resource can be lowered by reducing the number of divided subsets, which means training fewer models to gain quality labels**. 

Secondly, **the cost of our data selector's training is acceptable and reasonable compared to constructing new high-quality datasets**. Inspired by LIMA [1]'s approach of manually curating 1000 high-quality instruction data for fine-tuning, we take this concept further by **selecting only 200 high-quality multimodal data points from raw datasets**. Our data selection method avoid the need of curating high-quality instruction data from scratch, but select high-quality data from exisiting datasets. This finding is particularly significant given the complexity and resource demands of constructing new datasets.

Most importantly, once our data selector is trained on one dataset, **it can be applied to other datasets without the need for retraining** as mentioned in A1 above. This reusability makes the initial investment in training the selector more cost-effective over time. When dealing the large amounts of data, the data selector can directly identify high-quality data points based on the criteria learned during its initial training.

Thus, our data selector is practical and resource-efficient when dealing with new datasets.

[1] Zhou, Liu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023.

Reviewer comment (Experiments):
The article suggests that the data selector can be directly applied to other multimodality datasets. It would be beneficial if the authors could provide evidence or supporting experiments.

Author response:
Thanks for your questions. In particular, we want to find out **if there exists a subset making InstructionGPT-4 achieve better performance**. As a result, our proposed method indeed shows the existence of the subset. Using only 6% of the data, we manage to make InstructionGPT-4 surpass MiniGPT-4 in most tasks, which is quite surprising. 

To address your concern about scalability and generalizability, we also extend our investigation to include additional models and datasets. **Notably, we have applied our method to Qwen_VL [1] and the detail_10k [2] dataset without retraining the data selector**.
Here are our experimental results:

| detail_10k   | Qwen_VL (1k) | Qwen_VL (2k) | Qwen_VL (10k) | MiniGPT-4 (1k) | MiniGPT-4 (2k) | MiniGPT-4 (10k) |
|:------------:|:------------:|:------------:|:-------------:|:--------------:|:--------------:|:---------------:|
| MME          |    1802.50   |   **1806.81**|    1769.28    |   **614.04**   |      608.54    |      604.47     |
| Perception   |    1423.57   |   **1426.10**|    1398.92    |   **434.04**   |      431.04    |      428.54     |
| Cognition    |    378.93    |   **380.71** |    370.36     |   **180.00**    |      177.50    |      173.93     |

By comparing the performance between models tuned from selected subsets and the whole dataset, we observe that less instruction data for better performance still work for various MLLMs. Our findings indicate promising results, suggesting that our method is not only effective with MiniGPT-4's datasets but also exhibits potential for broader applicability. 

[1] Bai, et al. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023.

[2] Liu, Li, et al. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.

Reviewer comment (Experiments):
The dataset size of 3.4k in MiniGPT-4 is relatively small in terms of instruction data. It would be interesting to investigate whether the proposed method remains effective on larger instruction datasets.

Author response:
Thanks for your questions. In particular, we want to find out **if there exists a subset making InstructionGPT-4 achieve better performance**. As a result, our proposed method indeed shows the existence of the subset. Using only 6% of the data, we manage to make InstructionGPT-4 surpass MiniGPT-4 in most tasks, which is quite surprising. 

To address your concern about scalability and generalizability, we also extend our investigation to include additional models and datasets. **Notably, we have applied our method to Qwen_VL [1] and the detail_10k [2] dataset without retraining the data selector**.
Here are our experimental results:

| detail_10k   | Qwen_VL (1k) | Qwen_VL (2k) | Qwen_VL (10k) | MiniGPT-4 (1k) | MiniGPT-4 (2k) | MiniGPT-4 (10k) |
|:------------:|:------------:|:------------:|:-------------:|:--------------:|:--------------:|:---------------:|
| MME          |    1802.50   |   **1806.81**|    1769.28    |   **614.04**   |      608.54    |      604.47     |
| Perception   |    1423.57   |   **1426.10**|    1398.92    |   **434.04**   |      431.04    |      428.54     |
| Cognition    |    378.93    |   **380.71** |    370.36     |   **180.00**    |      177.50    |      173.93     |

By comparing the performance between models tuned from selected subsets and the whole dataset, we observe that less instruction data for better performance still work for various MLLMs. Our findings indicate promising results, suggesting that our method is not only effective with MiniGPT-4's datasets but also exhibits potential for broader applicability. 

[1] Bai, et al. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023.

[2] Liu, Li, et al. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.

Reviewer comment (Experiments):
The authors did not demonstrate the generality of the method. There are many nuances in the method and the experiments are done on only one dataset "cc_sbu_align". It is unclear whether the method will work on another dataset or another MLLM.

Author response:
Thank you for your comments. Following LIMA [1] that proposes "less is more for alignment" only using LLaMA in textual modal, our main motivation is to prove that less but high-quality instruction data can outperform the whole dataset for multimodal LLMs. In particular, we want to find out **if there exists a subset making InstructionGPT-4 achieve better performance**. As a result, our proposed method indeed shows the existence of the subset. Besides, we want to focus on MiniGPT-4 because it's the first and most popular multimodal model utilizing LLM. 
To address your concern about scalability and generalizability, we also extend our investigation to include additional models and datasets. **Notably, we have applied our method to Qwen_VL [2] and the detail_10k [3] dataset without retraining the data selector**.
Here are our experimental results:

| detail_10k   | Qwen_VL (1k) | Qwen_VL (2k) | Qwen_VL (10k) | MiniGPT-4 (1k) | MiniGPT-4 (2k) | MiniGPT-4 (10k) |
|:------------:|:------------:|:------------:|:-------------:|:--------------:|:--------------:|:---------------:|
| MME          |    1802.50   |   **1806.81**|    1769.28    |   **614.04**   |      608.54    |      604.47     |
| Perception   |    1423.57   |   **1426.10**|    1398.92    |   **434.04**   |      431.04    |      428.54     |
| Cognition    |    378.93    |   **380.71** |    370.36     |   **180.00**    |      177.50    |      173.93     |

By comparing the performance between models tuned from selected subsets and the whole dataset, we observe that less instruction data for better performance still work for various MLLMs. Our findings indicate promising results, suggesting that our method is not only effective with MiniGPT-4 but also exhibits potential for broader applicability. 

[1] Zhou, Liu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023.

[2] Bai, et al. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023.

[3] Liu, Li, et al. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.

Reviewer comment (Experiments):
There are many systematic methods in the literature that do something similar to select a subset from a dataset based on the model performance on a validation set. A notable one is Meta-Weight-Net [1] based on meta learning. How does this InstructionGPT-4 compare with the Meta-Weight-Net?

Author response:
Thanks for your suggestion. It's important to clarify that while both our method and Meta-Weight-Net aim to improve model performance through selective data, **they are based on different learning mechanisms**.
Meta-Weight-Net, as you mentioned, is grounded in meta-learning. It typically involves generating weights for training samples in a classification task, which are then used to adjust the loss function during the training of a model like ResNet. This approach is particularly suited for classification tasks where the output of a MLP can effectively influence the training process of the model.

In contrast, **our work focuses on subset selection for generative tasks within the multimodal LLM domain**. Our approach is not about adjusting weights during training but rather about selecting a subset of data that improves the efficacy of fine-tuning for generative tasks. This involves developing a multimodal data selector that identifies the most impactful data for model alignment, which is a different challenge compared to what Meta-Weight-Net addresses.

Given the generative nature of our task and the multimodal aspects of our model, applying a meta-learning approach like that of Meta-Weight-Net directly to our context would be challenging. The complexity of generative tasks in multimodal settings requires a different strategy, which is what we have developed and focused on.

We have included citations for Meta-Weight-Net in the revised version along with the above detailed discussion.

Besides, we conduct additional experiments on several data pruning methods including EL2N [1] and prototypicality [2], which were proposed in vision modal. We represent our experiment results below:
      
| Benchmark | InstructionGPT-4 |  EL2N  | Prototypicality | Random |
|:---------:|:----------------:|:------:|:---------------:|:------:|
| MME       |      **648.26**      | 627.27 |     569.46      | 527.26 |
| VQA       |      **22.30**       |  20.62 |     21.83       |  19.87 |


We observe that these previous systematic methods for single modal can't achieve competitve performance for multimodal LLM compared to our data selector. It showcases that our novel data selection method for multimodal datasets is quite necessary. We also add these experiments to the revised version along with the above detailed discussion.

Reviewer comment (Experiments):
If the data selector $F$ takes a data sample as input (see questions below for this), then in the paper the authors actually assign the same label to every data sample in the same subset $D_i$, which does not make sense because the label is due to the contribution of the whole subset (the model performance after being trained on the subset).

Author response:
Thanks for your comment. **The decision to assign the same label to every data sample in the same subset $D_i$ is indeed a tradeoff, and I'd like to clarify the rationale behind this approach**. 

If we were to fine-tune our pre-trained model on each individual data sample and then evaluate the performance of the fine-tuned model to determine quality labels, the process would require training and evaluating thousands of models. This approach would be prohibitively expensive and time-consuming, making it impractical for our purposes. At the same time, for the second-stage fine-tuning of multimodal LLMs, the core purpose is to align the instructions, responses and images. If the model is only fine-tuned on a very small dataset, the model training is not sufficient to obtain good alignment results, and the evaluation will be meaningless at this situation. 

Thus, we need to divide the original dataset into several groups. The size of each group cannot be too large or too small. On the one hand, if each group contains too much data, then there will be too few groups to fit an appropriate mapping relationship when training the data selector. On the other hand, if each group contains too little data, then the MLLM cannot be fully fine-tuned for alignment. So in this tradeoff, we choose to divide the data into 30 groups to ensure that there are enough samples during fitting and that the MLLM can achieve good alignment results when fine-tuned on each group.

Reviewer comment (Writing):
The description in Section 2.3 is confusing. In the "Training" paragraph, it seems that the data selector $F$ maps from the feature of a __whole__ subset to a score, because the index $i$ is for the subsets and it seems to mean $y_i = F(e_i)$. But in the "Testing" paragraph, it seems that the data selector $F$ maps from the feature of a single data sample to a score. Can the authors clarify it?

Author response:
Thank you for your inquiry regarding the features used to train the data selector F. Assume that each data point in our dataset is represented by a D-dimensional feature vector. When we form a cluster of K data points, these D-dimensional vectors are concatenated to create a feature vector with a shape of (K, D). **In training the data selector F, we utilize these (K, D) shape vectors**. Each cluster, represented by its (K, D) vector, is associated with a "genius quality label" that reflects the collective quality of all K data points within the cluster. This approach allows us to train the data selector on a rich and detailed representation of data clusters.

Regarding the application of F on a single data point as discussed in Section 2.3, it's important to clarify that F is trained on clusters but is capable of evaluating individual data points as well. **When applied to a single data point, F operates on its (1, D) feature vector**. This flexibility is a key aspect of F, allowing it to function effectively both at the cluster level during training and at the individual data point level during testing.

To address the potential confusion this might cause, we have revised the relevant sections of our paper to more clearly articulate how F is trained on clusters but can also evaluate individual data points. This explanation should make it clearer how F operates across different levels of data granularity.

Reviewer comment (Experiments):
Why did the authors not simply use the quality scores to select a subset for finetuning the final MLLM? It seems that the training data and the testing data for the data selector are the same. What is the meaning of performing the testing if we already have the labels on the data?

Author response:
Thank you for your question. We do not use the "genuine quality label" as the criterion for data selection because **different data points in the same subset divided by K-Means++ share the same "genuine quality label"**. We need to distinguish these data points in same subsets by conducting a testing stage. 

By employing spectral clustering after the data selector has been trained, it helps to prevent the homogenization of selected data and ensures that our model can handle a variety of data patterns. This diversity is crucial for capturing a wide range of data distribution patterns in the multimodal dataset, which can not be adequately represented if we solely rely on "genuine quality labels" for data selection.

We have made efforts to clarify this distinction and the importance of each clustering step in our revised paper.

Reviewer comment (Evaluation):
What is the relationship between the validation data used to generate the quality labels and the evaluation data? Why could the model performance on validation data be used as an indicator for model performance on the evaluation (test) data?

Author response:
Thank you for your question. In our study, **the validation data consists of four widely recognized VQA datasets**. We choose these four VQA datasets to produce "genuine quality labels" because they are sufficiently diverse and contain various question-answer pairs.

**The evaluation data, on the other hand, include additional VQA datasets and multiple-choice questions specifically designed for multimodal large language models**. A key dataset we use for evaluation is MME, a standard benchmark in the field. Unlike the validation datasets, MME is not split into training and test sets; it's used entirely for evaluation purposes.

The relationship between the validation and evaluation datasets is founded on the principle of **generalization**. In real-world applications of MLLMs, the specific downstream tasks may not always be known in advance. Therefore, it's crucial to train and validate models on a range of datasets that collectively encompass a broad scope of possible scenarios and tasks. By ensuring our model performs well on the diverse validation datasets, we can reasonably infer that it will also perform effectively on the evaluation datasets, including MME.

Although dividing MME into a training and test set might potentially yield better results due to more targeted training, our approach aims to make multimodal LLM generalized to different tasks. In fact, our validation strategy is designed to strike a balance between specificity and generalizability, ensuring that our model is well-equipped to handle a variety of multimodal tasks.

Reviewer comment (Experiments):
As far as I can tell, results with the selected data are from single runs. I worry that when selecting a small number of examples, the variance of the results might be quite high. Seeing results over multiple random seeds with regards to the data selection (training and selection itself) would be useful. However, considering the variance over the random selection is not massive, I think it's clear the selection method is effective in most cases.

Author response:
Thanks for your suggestion. Our main motivation is to prove that less but high-quality instruction data can outperform the whole dataset. In particular, we want to find out **if there exists a subset making InstructionGPT-4 achieve better performance**. As a result, our proposed method indeed shows the existence of the subset. 

Moreover, in response to your suggestion, we conduct additional experiments to try different random seeds. These experiments help us evaluate the stability and generalizability of our selection method under different initialization conditions. Our experimental results are shown below:

| Benchmark |  Seed1  |  Seed2  |  Seed3  | MiniGPT-4 |
|:---------:|:-------:|:-------:|:-------:|:---------:|
| MME       |  648.26 |  637.58 |  628.99 |   625.20  |
| VQA       |  22.30  |  21.78  |  22.14  |   20.54   |


We observe that the performance variance across different seeds is not obvious. This consistency of the performance underscores the effectiveness of our data selection method, suggesting that it is not overly sensitive to the initial random seed and can reliably identify high-quality data subsets across different scenarios. We also add these experiments to the revised version.

Reviewer comment (Experiments):
The authors only explore one model/architecture, so it is unclear how well their method would apply to larger/different models. In particular, as models scale, they benefit less from instruction data, so it is unclear if the rules for selecting quality data derived from smaller models will transfer to larger models.

Author response:
Thank you for your comments. Following LIMA [1] that proposes "less is more for alignment" only using LLaMA in textual modal, our main motivation is to prove that less but high-quality instruction data can outperform the whole dataset for multimodal LLMs, which is also mentioned in A1 above. Besides, we want to focus on MiniGPT-4 because it's the first and most popular multimodal model utilizing LLM. 
To address your concern about scalability and generalizability, we also extend our investigation to include additional models and datasets. **Notably, we have applied our method to Qwen_VL [2] and the detail_10k [3] dataset without retraining the data selector**.
Here are our experimental results:

| detail_10k   | Qwen_VL (1k) | Qwen_VL (2k) | Qwen_VL (10k) | MiniGPT-4 (1k) | MiniGPT-4 (2k) | MiniGPT-4 (10k) |
|:------------:|:------------:|:------------:|:-------------:|:--------------:|:--------------:|:---------------:|
| MME          |    1802.50   |   **1806.81**|    1769.28    |   **614.04**   |      608.54    |      604.47     |
| Perception   |    1423.57   |   **1426.10**|    1398.92    |   **434.04**   |      431.04    |      428.54     |
| Cognition    |    378.93    |   **380.71** |    370.36     |   **180.00**    |      177.50    |      173.93     |

By comparing the performance between models tuned from selected subsets and the whole dataset, we observe that less instruction data for better performance still work for various MLLMs. Our findings indicate promising results, suggesting that our method is not only effective with MiniGPT-4 but also exhibits potential for broader applicability. 

[1] Zhou, Liu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023.

[2] Bai, et al. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023.

[3] Liu, Li, et al. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.

Reviewer comment (Experiments):
The GPT-4 evaluation is performed over only 60 questions, which seems quite small. In particular, I worry that 60 questions is not enough to thoroughly test the model over a diverse set of queries that may expose gaps in performance due to the small number of examples seen in finetuning compared to a fully-finetuned model. This can be seen e.g. in Table 2, where InstructionGPT-4 performs significantly worse than MiniGPT-4 in commonsense reasoning and OCR tasks.

Author response:
Thank you for your comments. Firstly, in our evaluation, we want to cover both close-ended and open-ended benchmarks. For open-ended benchmarks, **LLaVA-Bench is the only one requiring GPT-4 for evaluation proposed during our work to our best knowledge**. Secondly, while LLaVA-Bench only contains 60 questions, it includes suffcient senarios such as indoor and outdoor scenes, memes, paintings, and sketches, and each image is associated with a carefully designed selection of high-quality questions. Thirdly, LLaVA-Bench is widely adopted by most recent related works, such as [1] and [2]. In addition, besides LLaVA-Bench, we also conduct sufficient closed-ended experiments, showing the effectiveness of our proposed method. We will try other multimodal open-ended benchmarks as future work. 

[1] Sun, Shen, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023.

[2] Liu, Li, et al. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.

Reviewer comment (Evaluation):
Why do you think there is a big drop in MME performance when moving from two layers to three layers in table 5?

Author response:
Thanks for your question. Given a fixed size of training set, **more layers containing more parameters tend to overfit**. Therefore, there will be a drop in MME performance when the number of layers increases. In our specific case, we cluster the datasets into 30 groups for training so that the training set is extremely small, and thus a drop in MME performance was observed from two layers to three layers.

Reviewer comment (Experiments):
It would be useful and interesting to see when using the data selection method becomes no better than random selection (if this happens) when increasing the number of data points selected (i.e. extending Figure 7 and adding a line for random data selection).

Author response:
Thank you for this suggestion. We have incorporated this experiment into our revised paper, extending the analysis in Figure 7 to include a comparison with random data selection. We also display the results below:

| Data Size             |    50   |   100  |   150  |   200  |   250  |   300  |   400  |   600  |   800  |  1000  |
|:---------------------:|:-------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|
| Data Selector (MME)   |  515.98 | 599.59 | 617.17 | 648.26 | 648.97 | 649.68 | 651.83 | 643.00 | 622.21 | 598.44 |
| Random Selection (MME)|  435.94 | 535.93 | 523.63 | 527.26 | 602.12 | 615.82 | 608.07 | 612.10 | 622.19 | 616.75 |
| Data Selector (VQA)   |  22.75  |  22.03 |  21.99 |  22.30 |  22.17 |  21.88 |  21.92 |  21.67 |  21.00 |  20.70 |
| Random Selection (VQA)|  20.30  |  20.62 |  19.83 |  19.87 |  20.21 |  20.23 |  21.36 |  20.73 |  20.38 |  20.68 |


Our findings reveal that our data selection method outperforms random selection in all cases except the size of 1000.
Through our observation, there is an interesting trend as the size of the selected subset increases: the performance gap between our method and random selection begins to narrow.

This decrease in the performance differential can be attributed to the nature of our data selection process. As the size of the subset selected by our method increases, it inevitably starts to include more data points of lower quality. This inclusion of lower-quality data diminishes the overall effectiveness of the selected subset, thereby reducing the gap in performance compared to a randomly selected subset.

Despite this trend, it is important to note that our data selection method continually maintains a performance advantage over random selection in most cases. This finding underscores the efficacy of our approach, particularly when working with smaller subsets. It also highlights a key insight: the quality of data, not just the quantity, is crucial for improving model performance.

We believe these results add a valuable dimension to our analysis, providing a clearer understanding of the scalability and efficiency of our data selection method. The revised paper includes a detailed discussion of these findings and their implications for the use of data selection in training large multimodal models.

Reviewer comment (Experiments):
The most important weakness of the paper is the lack of baselines. Many data quality and data pruning metrics have been proposed over the years. The authors should consider comparing against Influence Functions, Data Shapley, HyDRA, Diverse Ensembles [1], prototypicality [2], etc. A random baseline is way too weak.

Author response:
Thanks for your suggestion. We would like to highlight that **we are the first to propose a data selection method working on multimodal LLM for generative tasks**. Since there are no other data seletion methods for multimodal LLMs, we have to compare our method with single-modal baselines in our ablation study. In particular, **we compare our proposed selector with the recent methods such as Alpagasus (GPT Score only) in Table 5 and Instruction Mining (Linear Selector) in Figure 5**, demonstating that data selected by our multimodal selector works much better than these two methods.  
While our data selection method and data pruning methods both aim to reduce the size of training data, we would like to emphasize several important differences.
Our data selection method is used for multimodal LLMs in generative tasks. In contrast, data pruning methods are mostly used for vision models (e.g., ResNet, ViT) in classification tasks.
Moreover, in response to your suggestion, we conduct additional experiments on data pruning methods including EL2N [1] and prototypicality [2]. We represent our experiment results below:
      
| Benchmark | InstructionGPT-4 |  EL2N  | Prototypicality | Random |
|:---------:|:----------------:|:------:|:---------------:|:------:|
| MME       |      **648.26**      | 627.27 |     569.46      | 527.26 |
| VQA       |      **22.30**       |  20.62 |     21.83       |  19.87 |



We observe that these previous methods can't achieve competitve performance for multimodal LLM compared to our data selector. It showcases that our novel data selection method for multimodal datasets is quite necessary. We also add these experiments to the revised version along with the above detailed discussion.

[1] Paul, Ganguli, Dziugaite G K. Deep learning on a data diet: Finding important examples early in training. NeurIPS, 2021.

[2] Sorscher, Ben, et al. Beyond neural scaling laws: beating power law scaling via data pruning. NeurIPS, 2022.

Reviewer comment (Experiments):
It is not clear how the proposed method could scale or generalize to different datasets and models, as it is only tested on a single dataset and a single MLLM.

Author response:
Thank you for your comments. Following LIMA [1] that proposes "less is more for alignment" only using LLaMA in textual modal, our main motivation is to prove that less but high-quality instruction data can outperform the whole dataset for multimodal LLMs. In particular, we want to **find out if there exists a subset making InstructionGPT-4 achieve better performance**. As a result, our proposed method indeed shows the existence of the subset. Besides, we want to focus on MiniGPT-4 because it's the first and most popular multimodal model utilizing LLM. 
To address your concern about scalability and generalizability, we also extend our investigation to include additional models and datasets. **Notably, we have applied our method to Qwen_VL [2] and the detail_10k dataset [3] without retraining the data selector**.
Here are our experimental results:

| detail_10k   | Qwen_VL (1k) | Qwen_VL (2k) | Qwen_VL (10k) | MiniGPT-4 (1k) | MiniGPT-4 (2k) | MiniGPT-4 (10k) |
|:------------:|:------------:|:------------:|:-------------:|:--------------:|:--------------:|:---------------:|
| MME          |    1802.50   |   **1806.81**|    1769.28    |   **614.04**   |      608.54    |      604.47     |
| Perception   |    1423.57   |   **1426.10**|    1398.92    |   **434.04**   |      431.04    |      428.54     |
| Cognition    |    378.93    |   **380.71** |    370.36     |   **180.00**    |      177.50    |      173.93     |


By comparing the performance between models tuned from selected subsets and the whole dataset, we observe that less instruction data for better performance still work for various MLLMs. Our findings indicate promising results, suggesting that our method is not only effective with MiniGPT-4 but also exhibits potential for broader applicability. 

[1] Zhou, Liu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023.

[2] Bai, et al. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023.

[3] Liu, Li, et al. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.

Reviewer comment (Experiments):
It is not clear why only four VQA datasets are selected to produce "genuine quality labels", whereas the model is evaluated on bigger benchmarks.

Author response:
Thanks for your question. We choose these four VQA datasets to produce "genuine quality labels" because **they are sufficiently diverse and contain various question-answer pairs**. In particular, GQA focuses on reasoning skills and combined language understanding skills; IconQA requires perceptual skills such as object recognition and text understanding; OKVQA is a large-scale dataset requiring external knowledge; ScienceQA can well diagnose whether the multimodal LLM has multi-step reasoning capabilities and interpretability. These four datasets are wide enough to cover multiple aspects of multimodal tasks. Thus, we choose to select these four datasets to produce "genuine quality labels".

Reviewer comment (Writing):
The writing contains a number of vague descriptions and logic disconnects. The authors seem to be in a habit of fancy word choices, which may be the result of some large language model. However, word choices, however literary, cannot compensate for logic issues.

Author response:
Thanks for your comment. We have undertaken a thorough review of our manuscript, focusing specifically on areas where the descriptions were vague or the logic was unclear. We also pay particular attention to the sections and points you've highlighted, ensuring that our arguments are presented clearly and logically.  We believe these revisions will address the issues you've raised and enhance the overall readability and coherence of our paper.

Reviewer comment (Writing):
Page 3: "... steer multimodal language models in learning to generate responses in a particular manner" → what manner is that?

Author response:
Thanks for your question. In our paper, **the "manner" refers to the stylistic and structural characteristics of the generated content**, which are significantly influenced by the the instruction data. Instruction data can greatly affect models' generation during instruction tuning. Thus, **the manner of the generated content of the multimodal model is closely related to the manner of the instruction data**. To illustrate, if the instruction data mostly consist of lengthy and complex sentences, the model is more possible to generate responses that follow this style -- extensive and intricate. Conversely, if the instruction data are composed of concise, straightforward sentences, the model tends to produce briefer and more direct responses.  We have edited this to a clearer description in the revised version.

Reviewer comment (Writing):
Page 4: "... can more completely cover the various aspects of multimodal data quality" → More than what? What exactly are the aspects of multimodal data quality?

Author response:
Thanks for your question. In our paper,  it means that **more than using only one quality method to define multimodal data quality**, which is mentioned behind this sentence: "Using a single score to filter data can be useful, but it may not provide a comprehensive measure of data quality. Therefore, it is necessary to combine multiple indicators as an embedding to assess data quality collectively". When we state that our method "more completely covers the various aspects of multimodal data quality", we are comparing it to approaches that rely on a single metric for data quality assessment. Specifically, some existing methods, like Alpagasus [1], use the GPT score to evaluate textual data quality. However, we argue that multimodal data quality is multi-faceted and cannot be accurately captured by a single metric. To address this problem, our approach incorporates a combination of metrics: CLIP score, length score, reward score, and GPT score. This multi-metric method allows for a more complete evaluation of multimodal data quality. 

[1] Chen, Li, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701, 2023.

Reviewer comment (Writing):
Page 5: It is unclear what x represents in the unnumbered equations of Section 2.3. Is x a data point or a cluster? Additionally, it would be a good idea to number all equations, in case reviewers want to refer to them.

Author response:
Thanks for your question. From our paper in Section 2.3 Testing: "Given a multimodal dataset D of triplets x = (image, instruction, answer)", **x refers to an individual data point in our multimodal dataset D**. Specifically, each data point x is a triplet containing an image, an instruction, and an answer.

Reviewer comment (Reproducibility):
Page 5: It is unclear what features the data selector F is trained with. The text says "we concatenate these embeddings into a single composite embedding". If a data point has D-dimensional features, and a cluster contains K data points, do we have a KD-dimensional feature vector? If that's the case, how can we apply F on a single data point, as implied in the later part of Section 2.3?

Author response:
Thank you for your inquiry regarding the features used to train the data selector F. Each data point in our dataset is represented by a D-dimensional feature vector. When we form a cluster of K data points, these D-dimensional vectors are concatenated to create a feature vector with a shape of (K, D). **In training the data selector F, we utilize these (K, D) shape vectors**. Each cluster, represented by its (K, D) vector, is associated with a "genius quality label" that reflects the collective quality of all K data points within the cluster. This approach allows us to train the data selector on a rich and detailed representation of data clusters.
Regarding the application of F on a single data point as discussed in Section 2.3, it's important to clarify that F is trained on clusters but is capable of evaluating individual data points as well. **When applied to a single data point, F operates on its (1, D) feature vector**. This flexibility is a key aspect of F, allowing it to function effectively both at the cluster level during training and at the individual data point level during testing.

To address the potential confusion this might cause, we have revised the relevant sections of our paper to more clearly articulate how F is trained on clusters but can also evaluate individual data points. This explanation should make it clearer how F operates across different levels of data granularity.

Reviewer comment (Reproducibility):
How do you make sure K-means and spectral clustering return clusters of equal size?

Author response:
Thanks for your question. In the selector's training stage, we observe that the data quantity in each group from K-means clustering is roughly the same, while other clustering algorithms results in more significant differences in group sizes. This is a characteristic of K-means, as it partitions data into clusters that minimize the variance within each cluster. To acquire genuine quality labels (which necessitate training a multimodal model on each cluster group for evaluation) without letting the quantity of data impact the capabilities of the trained multimodal models, we employ a simple post-processing technique. This involves identifying clusters with either excessively high or low sample counts, and **redistributing some samples from the larger clusters to the smaller ones based on their distance to different cluster centroids**, thereby equalizing the number of samples in each cluster.

In the testing stage, we use spectral clustering to select a diverse set of data points. Unlike in the training stage, the requirement here is not to have clusters of equal size but rather to select the top 200 data points that best represent the diversity of the dataset. Hence, in this stage, the equal size of clusters is not a concern as our focus shifts to ensuring the diversity and representativeness of the selected data.

We have revised our paper to clearly explain this differentiation in the use of clustering techniques between the training and testing stages, and how we ensure appropriate cluster sizes and data diversity in each stage.

Reviewer comment (Experiments):
What is the purpose of reclustering the data points? Why not use the "genuine quality label" as the criterion for data selection?

Author response:
Thank you for your question, which we address in Appendix A.3 of our paper. The primary purpose of reclustering data points using **spectral clustering** (denoted as Λ) in the data selection stage is to **ensure the diversity of the selected data**. This diversity is crucial for capturing a wide range of data distribution patterns in the multimodal dataset, which can not be adequately represented if we solely rely on "genuine quality labels" for data selection.

To elaborate, the initial clustering using **K-Means++** (denoted as Γ) serves to **divide the data into subsets for obtaining "genuine quality labels"**. These labels are assigned based on the collective characteristics of data points within each subset. While these labels are valuable for assessing overall data quality, they may not fully capture the diversity within the dataset. This is where spectral clustering comes into play.

We do not use the "genuine quality label" as the criterion for data selection because **different data points in the same subset divided by K-Means++ share the same "genuine quality label"**. We need to distinguish these data points in same subsets by conducting a testing stage. By employing spectral clustering after the data selector has been trained, it helps to prevent the homogenization of selected data and ensures that our model can handle a variety of data patterns. 

In addition, **we have conducted detailed experiments to study the effectiveness of applying spectral clustering within the data selector mechanism in Figure 5**. These experiments demonstrate the significance of maintaining diverse vision-language instruction data and validate the necessity of the reclustering process.
We have made efforts to clarify this distinction and the importance of each clustering step in our revised paper, ensuring a clearer understanding of our methodology and the rationale behind the use of different clustering techniques.