Paper: Instruction Mining: Instruction Data Selection for Tuning Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
The method is computed over loss, but it is unclear how this translates to eval task performance. Appendix C presents a somewhat weak argument - it would be good to e.g. strongly correlate the inference loss with MMLU/MT-Bench/etc results. It’s non-obvious that this is true in instruction tuning setups due to the lack of hard gold labels. There may be many potential outputs that are equally valid, and only one is generated as the gold label by GPT-4. This also makes it unclear to determine what differences in loss are significant - for example, in Table 3, Random 150(Train) gets .0085 higher loss than Selected 22(Rule) + 150(Train). Is this difference significant? Does it actually translate to downstream task gains?

Author response:
**Q.1 Difference in loss:**

It’s hard to say the numerical relevance between loss and performance difference, here is the OpenLLM leaderboard result for random and selected 10k data of dolly.

|  Model Name  | Datasize | loss@self-instruct | loss@mt-bench | lm-eval-avg | ARC    | Hellaswag | MMLU   | TruthfulQA |
| -------------- | -------- | ------------------ | ------------- | ----------- | ------ | --------- | ------ | ---------- |
| dolly random   | 10000    | 1.0356             | 0.8086        | 0.5486      | 0.5733 | 0.8116    | 0.4266 | 0.3828     |
| dolly Selected | 10000    | 1.0371             | 0.8001        | 0.5495      | 0.5682 | 0.8123    | 0.4392 | 0.3782     |

However, we would like to point out that, during our experimentation, we feel that loss on mt-bench, also LLM-as-a-judge, is only able to evaluate a model’s conversational ability, while a language model still needs other abilities, e.g., question answering. And this is why we add OpenLLM benchmarks. From our point of view, Mt-bench and MMLU are two separate evaluation benchmarks, which does not correlate much. For example, a response below could yield high LLM-as-a-judge score and low loss, however, it is not a correct answer to this question[1]. 

```
Instruction: Sort the following list into alphabetical order. apple, banana, orange, grape.

Wrong Answer: No problem! Here’s the sorted list. Grape, apple, banana, orange.

Correct Answer: apple, banana, grape, orange.
```
- **W.1 The correlation between LLM-as-a-judge scores and loss.**

In table4, we bring loss and LLM-Leaderboard together:

| Model                   | Data size | Loss        | Loss(MT-BENCH) | Avg. Metric | ARC   | HellaSwag | MMLU  | TruthfulQA |
| ----------------------- | --------- | ----------- | -------------- | ----------- | ----- | --------- | ----- | ---------- |
| INSTRUCTMINING-Selected | 10,000    | 0.9767      | 0.7177         | 58.65       | 56.66 | 79.77     | 49.89 | 48.26      |
| INSTRUCTMINING-Selected | 40,000    | 1.006       | 0.7462         | 59.25       | 54.44 | 80.11     | 52.6  | 49.83      |
| INSTRUCTMINING-Random   | 10,000    | 1.012±0.008 | 0.75±0.007     | 58.74       | 54.78 | 79.58     | 49.02 | 51.58      |
| INSTRUCTMINING-Random   | 40,000    | 1.015±0.004 | 0.74±0.035     | 58.95       | 54.78 | 79.89     | 51.16 | 49.95      |

We would like to kindly point out that, since our optimization goal is to minimize the distance between the finetuned model and the target model, hence, GPT-4 in our paper, it is possible that MMLU does not correlate much with the loss. A straightforward intuition is that,  MMLU is more like a question answering task, while our loss minimization goal aims to improve the model’s ability in conversation. In this case, we choose to relate loss with mt-bench, which is also LLM-as-a-judge scores to prove that loss can serve as a suitable indicator for model performance.

We are still downloading model weights and running experiments on 7B open-source models. We will provide the experiment results as soon as it is available.

Reviewer comment (Evaluation):
The lack of multiple random seeds/error bars with the experiments makes it unclear how well InstructMining is performing over randomly selected data. Looking at the results in Table 4 especially, it seems that randomly selecting data outperforms InstructMining-Selected at the 10k size, and only underperforms the Selected model by .7 at the 40k size, which might not be a significant difference. While it’s expensive to run multiple trials, especially for larger data sizes, I think it’s important to make sure that the claimed performance improvements are actually significant.

Author response:
**Q.2 Results on random selection experiments.**

Thanks for pointing out. We will update more random results. These results and analysis will be added to our paper.
Below is the detail for the appended random experiment results.

| Model Name | Data size | Loss@self-instruct | Loss@mt-bench |
| ------------------- | ----- | ----- | ----- |
| figure1 orca Random | 1000  | 1.001 | 0.746 |
| orca Random         | 1000  | 0.974 | 0.715 |
| orca Random         | 1000  | 0.970 | 0.715 |
| figure1 orca Random | 2000  | 0.966 | 0.711 |
| orca Random         | 2000  | 0.980 | 0.713 |
| orca Random         | 2000  | 0.990 | 0.713 |
| figure1 orca Random | 3000  | 0.993 | 0.727 |
| orca Random         | 3000  | 1.013 | 0.728 |
| orca Random         | 3000  | 1.000 | 0.736 |
| figure1 orca Random | 4000  | 1.014 | 0.752 |
| orca Random         | 4000  | 0.999 | 0.751 |
| orca Random         | 4000  | 1.015 | 0.728 |
| figure1 orca Random | 5000  | 0.991 | 0.751 |
| orca Random         | 5000  | 0.999 | 0.738 |
| orca Random         | 5000  | 0.998 | 0.734 |
| table4 orca Random  | 10000 | 1.020 | 0.750 |
| orca Random         | 10000 | 1.012 | 0.757 |
| orca Random         | 10000 | 1.003 | 0.742 |
| table4 orca Random  | 40000 | 1.015 | 0.734 |
| orca Random         | 40000 | 1.019 | 0.732 |
| orca Random         | 40000 | 1.011 | 0.754 |

We summarize this table in to the analysis below:

| Datasize | Metric                    | Average | Std.           | Value       |
| -------- | ------------------------- | ------- | -------------- | ----------- |
| 1000     | avg/std of loss           | 0.981   | 0.0169  | 0.981±0.017 |
| 1000     | avg/std of loss(mt bench) | 0.725   | 0.0178  | 0.725±0.018 |
| 2000     | avg/std of loss           | 0.979   | 0.0117  | 0.979±0.012 |
| 2000     | avg/std of loss(mt bench) | 0.713   | 0.0014 | 0.713±0.001 |
| 3000     | avg/std of loss           | 1.002   | 0.0104  | 1.002±0.01  |
| 3000     | avg/std of loss(mt bench) | 0.731   | 0.0048  | 0.731±0.005 |
| 4000     | avg/std of loss           | 1.009   | 0.0091 | 1.009±0.009 |
| 4000     | avg/std of loss(mt bench) | 0.744   | 0.0135   | 0.744±0.014 |
| 5000     | avg/std of loss           | 0.996   | 0.0043 | 0.996±0.004 |
| 5000     | avg/std of loss(mt bench) | 0.741   | 0.0089 | 0.741±0.009 |
| 10000    | avg/std of loss           | 1.012   | 0.0083 | 1.012±0.008 |
| 10000    | avg/std of loss(mt bench) | 0.750   | 0.0072 | 0.75±0.007  |
| 40000    | avg/std of loss           | 1.015   | 0.0039 | 1.015±0.004 |
| 40000    | avg/std of loss(mt bench) | 0.740   | 0.0351  | 0.74±0.035  |
We will also update the figure in our paper with this result.

Reviewer comment (Experiments):
Missing baselines: it would be good to compare against models trained on all data from the different base datasets: just open orca, just dolly, and the two combined. This would allow us to examine how the InstructMining models compare to models trained on all available data while keeping the finetuning setup consistent. Currently, the main baselines are Vicuna, Llama-chat, and StableBeluga, none of which were trained on OpenOrca or Dolly.

Author response:
**W.3 & Q.4 Models trained on full dataset.**

Orca full is too large for us to train at this moment, here we provide our updated result on full dolly(15k) dataset.

|Model | Datasize | Loss@self-instruct | Loss@mt-bench | OpenLLM Avg| ARC| Hellaswag| MMLU|TruthfulQA|
|---|---|---|---|---|---|---|---|---|
|Dolly-full| 15,000| 1.047 | 0.807 | 0.546 | 0.566 | 0.808 | 0.437 | 0.371 |
This result will also be added into our paper.

Reviewer comment (Presentation):
This is minor, but the names used in Table 4 could be explained better. I’m assuming InstructMining-Random is randomly selecting over OpenOrca and Dolly.

Author response:
**W.4 Name explanation.**

Thanks for pointing out the ambiguity in our paper. In Table 4, both InstructionMining-Selected and InstructionMining-Random are over Orca.

Reviewer comment (Novelty):
I think the method in this paper may not be novel and solid. Whether selecting another dataset as D-eval is a good choice is not sufficiently justified. The distributions of D-eval and the target task are different. The paper does not touch the core problem, i.e., how to select and estimate the effect of some instructions with a biased/smaller dataset.

Author response:
Thanks for pointing this out. Indeed, we recognize that we didn’t explain in detail why we selected another D-eval, instead of just choosing an evaluation subset from the training data. When selecting D-eval, We want 
1. D-eval to have high-quality instruction following examples in it. 
2. D-eval to be diverse enough, hence, it should have:
(a). various instructions to test the model’s generalizability; 
(b). High-quality responses.
3. D-eval to be relatively small, so that we can conduct the experiments efficiently.

At the time of submission, we only found two suitable, human labeled instruction datasets, self-instruct and mt-bench. To make sure that our selected D-eval satisfies these requirements, we also re-annotated self-instruct dataset using GPT-4 to get high-quality responses. 

- **Q.1.2 Distribution difference between D-eval, target tasks, and D**

We agree that if we directly apply this rule to select data of other target tasks, our rule will probably not work as expected. However, we would like to point out that our paper mainly focuses on instruction tuning, hence, teaching the model to follow instructions. The datasets we choose to train the model and fitting the rule and mostly instruction-following data, and we mainly focus on tuning the model to make it better at following instructions. In this case, our task is general instruction following, which aims to tune an LLM to follow instructions. So there should not be much difference in the distribution of D-eval and target tasks.

- **Q.1.2 How to select and estimate the effect of instructions with a biased/smaller dataset.**

We agree that if the D-eval is biased or too small, our method might not work under these circumstances. We want to kindly point out that our method is highly dependent on a suitable, high-quality D-eval dataset. A strong hypothesis in this paper is that D-eval is of relatively high quality. In this case, selecting and estimating instruction quality using a biased D-eval is out of our current scope. However, selecting data when D-eval is biased or too small is of course an important research topic, and hopefully in our future research, we can keep on improving our method to make it more robust, towards different D-eval sets.

Reviewer comment (Evaluation):
The paper posits that the instruction quality of D can be estimated through the inference loss of M-ft on a evaluation dataset D-eval. However, the negative log-likelihood (NLL) reflects the extent to which a language model trained on D can explain the data in D-eval, essentially measuring the textual similarity between D and D-eval. If the quality of D-eval is not high enough, this method of assessing instruction quality could be biased. Furthermore, how can we ensure the quality of D-eval? Does a higher similarity to D-eval necessarily indicate a higher quality of D?

Author response:
We acknowledge that using D-eval for fitting the rule and then using the rule to select data will possibly result in overfitting on the D-eval. This could harm performance if D-eval is of low-quality. However, if D-eval is of high-quality, the selected D should be of similar quality. Hence, our method is strongly dependent on the hypothesis that D-eval is of high-quality. To ensure its quality, which is the self-instruct dataset in our paper, we re-annotated self-instruct dataset using GPT-4, to ensure that the label of self-instruct is of high-quality.

Reviewer comment (Evaluation):
The paper associates the optimal point with the lowest loss, selecting the data that can achieve the lowest loss as the optimal data. However, it is important to note that there is no direct correlation between the training loss and the quality of data. An output with a longer length often results in a higher training loss, but this does not necessarily indicate lower quality. The paper does not take this factor into account.

Author response:
Instead of using training loss, we use model inference loss on D-eval, so D-eval is an unseen dataset to the model. We recognize that inference loss cannot accurately represent a generative model’s performance. However, inference loss is still an indicator of the textual similarity between responses generated by GPT-4 and the trained model, and it is able to indicate data quality to some extent, as shown in Table 8. Choosing a suitable metric to evaluate a model’s performance or data quality requires us to consider both representability and efficiency, which still remains an unsolved problem in the research area. Inference loss is like a “compromise” that it is able to represent a model’s ability in following instructions to some extent and also save inference time.

Also, we would like to clarify that our analysis in appendix section D is only a descriptive analysis over the linear similarity. For example, output length in appendix D shows a positive linear correlation with loss, however, this does not mean that output length is strongly correlated with loss in other spaces. During rule fitting, under the combined influence of multiple indicators, output length becomes less significant, which results in its absence in the final estimated law.

We also would like to note that, during experimentation, we found finding a suitable metric quite hard since a generative language model cannot be simply evaluated only from its conversation ability or from its question answering ability. [Recent works](https://arxiv.org/abs/2310.07641) also discover that even GPT-4 could be “tricked” by superficial expressions[1]. However, expanding our optimization objective to multiple metrics is out of our current scope.

Reviewer comment (Experiments):
Each time D-eval changes (for instance, when a higher quality D-eval is chosen), the multivariate linear function must be relearned. In other words, the method proposed in the paper lacks scalability.

Author response:
- **Q.4 The expense of changing D-eval.**

Normally, you don’t need to redo the experiments. If a D-eval with significantly higher quality is out, e.g. GPT-5 labeled D-eval, and we want a better law to select examples which are close to this D-eval set, then we need to redo the fitting experiments. However, the expense is still affordable. During our experimentation, every data collection experiment takes ~15mins to complete, and the total fitting experiments takes around 32 hours to complete.

Reviewer comment (Experiments):
It would be great to see experiments conducted using the same volume of data for model training on both LIMA and the INSTRUCTION MINING method proposed in the paper, followed by a comparison of their respective performances on MT-BENCH.

Author response:
We just got the result of LIMA-7b model (LLaMA-2-7B finetuned with LIMA data). Its OpenLLM benchmark scores are shown in the table below.
|Model | Datasize | loss@self-instruct| loss@mt-bench| OpenLLM Avg| ARC| Hellaswag| MMLU|TruthfulQA|
|---|---|---|---|---|---|---|---|---|
|LIMA-7b|1000|0.9947|0.7604|0.5533|0.5563|0.8009|0.4371|0.419|
|InstructMining-7b|1000| 0.9576 | 0.7114 | 0.562525  | 0.5589 | 0.7877 | 0.4299 | 0.4736 |

We also compared its performance with our finetuned InstructMining model using LLM-as-a-judge. Results are listed in the table below.
| Model | Win Rate | Lose Rate | Tie Rate | Adjusted Win Rate |
|---|---|---|---|---|
|InstructMining-7b(vs LIMA-7b)| 0.4875 | 0.1063 | 0.4062 | 0.6906 |

This result will also be updated into our paper . If you have further questions, please feel free to reply. Thank you again for your review and kind advice!

Reviewer comment (Theory):
Why can the learned Eq.4 work well on unseen instruction datasets? Is there any intuitive explanations or any insight from the perspective of theory for example any theory guarantee of the proposed method?

Author response:
Thanks for pointing out the weakness in our paper. We don’t have a theory guarantee of our method currently, which is a little bit out of our scope. However, a possible reason of the effectiveness of our method is that, based on our current hypothesis, which are:
1. The rule-fitting test set is a high-quality dataset at the time of experiments.
2. Inference loss can indicate a model’s performance to some extent.

Our method would possibly select data examples which are of similar quality to the selected rule-fitting set. Hence, for an unseen dataset, our proposed framework can:
1. Label data quality. Our proposed rule can help quantify and label example quality.
2. Search in an effective way and converge towards a certain direction. After we get the labels, we will rank the data according to their quality scores, and set a quantile point to choose the top high-quality examples. To do this, we use Flaml blendsearch to ensure that the final chosen dataset converges on a relatively low inference loss.

We also would like to point out that the effectiveness of our method is based on the superficial alignment hypothesis[2] and mainly focuses on instruction tuning LLMs. We believe that base LLMs already have the knowledge to answer certain problems, while instruction tuning teaches the LLM to learn to understand and follow the instructions. Hence, we believe instruction tuning does not always require large scale training data. Through small amount of high-quality examples, an LLM should be able to learn to follow instructions.[1,2]

Reviewer comment (Experiments):
This work only compared with random data selection methods as baselines. Are there any simple or straight-forward data selection methods for comparison?

Author response:
Currently, instruction-following natural language data selection methods can be divided into three types:
1. Human selection. This method requires human efforts to label and select examples from datasets, which always results in large expenses.
2. Strong LLM labeling. This method requires serving or connecting very strong LLMs through apis, e.g., GPT-4. This method might lack explainability and also result in relatively large expense if the LLM api is not free and have high rate limits.
3. Machine selection. This method requires a model e.g., reward model, to mimic human or strong LLM to label data and set selection quantiles. This method can save efforts and work under very low expense.

At the time of our submission, only (a) and (c) is available. For (b), since our resource is limited at this point, leveraging strong LLMs like GPT-4 to label over 100k data examples is too expensive for us. We will try our best to find other ways to implement (b).
For (a), we are currently conducting experiments using LIMA data. We will provide the result as soon as possible and will update it into our paper and comment here. And for (c), please refer to our ablation study section.

Reviewer comment (Experiments):
I'm concerned about the expense of training the linear function. It requires training 129 LLMs on 129 subsets. My questions are how 129 is chosen and how long it takes to train the 129 LLMs?

Author response:
- **Q.1 & W.1: Expense of training.**

In implementing regression with ten variables, it is imperative to maintain an Event Per Variable (EPV) ratio exceeding 10. In pursuit of analytical robustness, we intentionally surpassed this threshold by collecting over 100 experiments. Each subset comprising 1,000 samples was trained with 8 A100 GPUs within 15 minutes. As a result, the cumulative time for these 129 experiments totals approximately 32 hours.

Reviewer comment (Experiments):
The generalization ability of this method is unclear. There are two dimensions: the model used and the rule-fitting set for the combination function. I suppose the coefficients of indicators are model-dependent. The authors only tested on the Llama-2-7B model, I wonder whether using the same set of coefficients is possible for different models. Also, how sensitive the coefficients are towards the choice of the meta-training set, i.e., the rule-fitting set. Do you have numbers to demonstrate the stability of it?

Author response:
- **Q.1 & W.2: generalization of the rule**

In table 6 we also show the generalization ability for our rule across model (LLaMA-1-7B), size(LLaMA-2-13B) and LoRA(LLaMA-2-7B with LoRA).

- **Q.1 & W.2: coefficients sensitivity for rule-fitting set**

Certainly, the coefficient has sensitivity to the test set for rule-fitting. However, it's crucial to note that such changes do not compromise the validity of our methodology. Selecting an ideal set of rule fits presents a challenge, and for our study, we employed a dataset generated through human-labeled GPT-4, representing the best available option at the time of publication.

Reviewer comment (Evaluation):
I'm not sure whether using the inference loss as the y-axis of the double descent figure is the correct choice.

Author response:
- **W.3: loss in double descent**

Since loss is the main optimization objective in our paper, we use loss in y-axis in figure 1. We believe that loss is highly correlated with PPL, which used to be a commonly used metric for evaluating traditional generative language models. Hence, inference loss could be an indicator of a model’s performance. We’ve also done some experiments on other metrics. Please refer to Figure 4 for more details.