Paper: What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (ICLR-2024)

Reviewer comment (Reproducibility):
Reproducibility is limited because code, data, model checkpoints, or data selection tools are not provided during the reviewing phase.

Author response:
### Q3: Reproducibility: Code, data, model checkpoints, or data selection tools are not provided during the reviewing phase.

We have promised to release these resources after the review period in the footnote of the first page, we surely will do when we deanonymize this paper – it is not common practice to release model checkpoints in an anonymous way during the review period as far as we know.

Reviewer comment (Presentation):
The appendix is not cut from the main paper, and the PDF provided for the main paper is a 20-page document.

Author response:
### Q4: Appendix is not cut from the main paper. Appendix should not be submitted under the main paper.

> Authors may use as many pages of appendices (*after the bibliography*) as they wish   

quoted from ICLR 2024 Call for Papers (https://iclr.cc/Conferences/2024/CallForPapers).

> Q: Should the appendices be added as a separate PDF or in the same PDF as the main paper?  
Either is allowed: you can include the appendices at the end of the main pdf after the references, or you can include it as a separate file for the supplementary materials.

quoted from ICLR 2024 Author Guide (https://iclr.cc/Conferences/2024/AuthorGuide)

Reviewer comment (Novelty):
The authors should spend significant effort discussing the current research landscape and identifying a valid (important and essential) research gap that this work can make a substantial and unique contribution to, as this is the main concern for this line of work.

Author response:
### Q1: How much “research gap” remains in instruction mining or SFT data selection? What is the research gap that this work contributes to? 

We respectfully disagree with the reviewer on these comments about this research line in general, as well as our work’s contribution and significance. While we have no doubt that the reviewer understands these “simple” methods from previous works and our study, we believe there may be an oversight from the reviewer regarding the overall progress and current state of this research direction.

To provide context, we would like to briefly summarize recent advancements in SFT data selection, which will illustrate the "research gap" and the broader picture. Initial studies like Self-instruct [1] and Alpaca [2] showed that medium-sized datasets (ranging from 50k to 100k samples) could endow pretrained models with effective instruction-following capabilities, though Alpaca was later identified as a weak baseline upon more rigorous evaluations. LIMA [3] highlighted the significance of data selection in SFT by using just 1,000 carefully chosen or human-crafted samples to surpass Alpaca's performance. However, these samples were not selected automatically.

Alpagasus [4] and Instruction Mining [5], as the reviewer mentioned, began exploring automatic data selection in SFT. Yet, their comparisons were limited to random baselines or the already weak model Alpaca at the time. They did not conduct standard evaluations on challenging benchmarks like MT-bench/AlpacaEval nor did they compete with strong models such as Vicuna and WizardLM. This indicates a clear gap: **no study in automatic data selection has demonstrated practical effectiveness through evaluations on challenging benchmarks and against SOTA open-source SFT models -- we do not know whether [4] and [5] really work or not from their weak evaluations**. This gap is significant as it leaves unanswered questions about the potential and practicality of automatic data selection and whether it can lead to SOTA open-source SFT models.

We note that [4] and [5] have updated their papers with more evaluations and baselines *after* the ICLR submission deadline, which we hope the reviewer realizes. However, even in their latest versions, [4] does not compare with any other state-of-the-art open-source SFT model. Our approach has significantly outperformed it (as shown in Table 5 of our paper). Meanwhile, [5] does not report standard MT-bench and AlpacaEval scores.

Reviewer comment (Novelty):
The technical approach should be improved. The current methods do not seem particularly novel compared to existing works and appear capped by heavy reliance on GPTs. Designing original technical approaches (new metrics, new evaluation methods, novel ways of combining different metrics) could make the paper more attractive.

Author response:
### Q2: Our contribution and significance over previous/recent works on instruction mining.

Following the response above, our research stands out as the *first* to propose an automatic data selection approach that consistently demonstrates strong performance across multiple challenging benchmarks, including MT-bench, AlpacaEval, and the Open LLM Leaderboard. Our approach outperforms other data selection methods by a large margin (Table 5), yielding results that are comparable to, or even surpass, those of SOTA open-source SFT models such as Vicuna, WizardLM, and Mistral-Instruct. These empirical results – being the first to achieve SOTA open-source performance with 10x less data selected automatically –  is particularly noteworthy because it suggests that our approach, the datasets selected, and the data selection tools we have developed can be practically and beneficially applied by a broad spectrum of researchers and practitioners.

In terms of technical innovation, our approach diverges from previous methodologies. We utilize an evolving strategy to elicit more faithful complexity and quality scores from ChatGPT, as opposed to a direct scoring method, which our results (shown in Tables 2 and 3) indicate is less effective. While we recognize that the combination of metrics and reliance on GPT models are not novel, we have no reason to complicate things when we find such simple or trivial combinations of our proposed metrics are highly effective. Methodologically, exploring alternatives to GPT reliance is an intriguing future direction, yet it was not deemed essential for the current scope of this paper, particularly given that our data selection tool does not require ChatGPT at the inference stage.


Furthermore, we would like to remind the reviewer that both Alpagasus and InstructionMining were first arxived within 3 months prior to the ICLR submission deadline. According to the ICLR official reviewer guide (https://iclr.cc/Conferences/2024/ReviewerGuide#Reviewing%20instructions), such works are regarded as concurrent. While we still believe our work represents significant advancements beyond these studies as described above, we urge the reviewer to consider this timing factor as well when evaluating our research.

Reviewer comment (Presentation):
The appendix should not be submitted as part of the main paper.

Author response:
### Q4: Appendix is not cut from the main paper. Appendix should not be submitted under the main paper.

> Authors may use as many pages of appendices (*after the bibliography*) as they wish   

quoted from ICLR 2024 Call for Papers (https://iclr.cc/Conferences/2024/CallForPapers).

> Q: Should the appendices be added as a separate PDF or in the same PDF as the main paper?  
Either is allowed: you can include the appendices at the end of the main pdf after the references, or you can include it as a separate file for the supplementary materials.

quoted from ICLR 2024 Author Guide (https://iclr.cc/Conferences/2024/AuthorGuide)

Reviewer comment (Evaluation):
No evaluation result on benchmarks like MMLU and Big-Bench-Hard, which allows verification of commonsense knowledge and reasoning with ground-truth answers. GPT4-based evaluation often includes errors or bias, so not enough.

Author response:
Thanks for the comments. We have added more evaluation results on the Open LLM Leaderboard benchmark [1] that consists of four tasks, ARC, HellaSwag, MMLU, and TruthfulQA.  Additionally, we have enriched our evaluations by training more pre-trained LLMs on our selected data, such as  LLaMA-2-13B and Mistral-7B besides LLaMA-1-13B in the original submission. We have summarized these results in the table of the general response, while the detailed results are in Table 3,4,5 of the paper. On the Open LLM Leaderboard, our approach outperforms the random baseline and other strong models across three different backbone models. However, we would like to note that the performance on benchmarks such as MMLU is largely determined by the underlying base model, thus the gains after alignment tuning is typically not large, unless the training data is specifically crafted for the benchmark. This is why recent works on alignment put more emphasis on MT-bench and AlpacaEval [2]. 

Notably, on MT-bench and AlpacaEval, our added results show that our method with the LLaMA-2-13B model surpasses Vicuna-13B-1.5 that trains on 10x more data, and LLaMA-2-13B Chat that has undergone extensive RLHF training. With Mistral-7B, our approach attains an MT-bench score of 7.29 with only 10k training data, marking a state-of-the-art performance on MT-bench *for all open-source SFT models at 7B and 13B sizes*. We also observe that, while the gains of the Mistral DEITA model on AlpacaEval are slightly lower than those on MT-bench, a deeper analysis (as depicted in Figure 2) indicates that the Mistral DEITA model excels particularly in mathematical and complex reasoning tasks, areas not primarily covered by AlpacaEval.

Reviewer comment (Evaluation):
Marginal performance improvement. In Table 3, DEITA is worse than Vicuna-13B on AlpacaEval dataset. Also, in Table 4 with human evaluation, Vicuna performs almost similar to DEITA.

Author response:
We emphasize that the purpose of this work is not to set up a new record on the performance, but to achieve top results with much less data, for example, the training of DEITA is far more efficient than Vicuna by using 10x less training data. Furthermore, we note that Vicuna is a SOTA SFT model, and DEITA outperforms it on most of the cases across three different benchmarks and various backbone models (see the table in General Response), except for just one setting on AlpacaEval –  **DEITA is the first automatic data selection approach that achieves comparable performance to SOTA models such as Vicuna and WizardLM with 10x less data**. Therefore, we view comparable performance to Vicuna as the strength of our approach, rather than weaknesses.

Reviewer comment (Evaluation):
Why the complexity measurement is obtained from “instruction”, and the quality measurement is obtained from “response”? Is there any intuition of this?

Author response:
We measure complexity from instruction only since the evolution procedure of complexity directly follows WizardLM, which takes only the instruction as input and evolves it step by step to increase the instruction complexity.  For quality, our intuition is to measure how effectively a response addresses the query, thus both the instruction and the response are inputs to the scorer. We recognize that in Figure 1, the instruction input was inadvertently omitted in the depiction of the quality scorer. This was our oversight, we have slightly revised Figure 1 to be more clear on this.

Reviewer comment (Evaluation):
The main evaluation is done in terms of AlpacaEval and MT-Bench alone. Since these are relatively small evaluation sets and it has been shown that GPT-4 evaluation can be biased (Wang et al., 2023; https://arxiv.org/abs/2306.04751), one wonders if the data selection does better than random only because it is aligned with those biases. Including further evaluation, possibly on targeted benchmarks covering abilities like reading comprehension, complex reasoning etc. can be helpful.

Author response:
Thanks for the comments. We have added more evaluation results on the Open LLM Leaderboard benchmark [1] that consists of four tasks, ARC, HellaSwag, MMLU, and TruthfulQA.  Additionally, we have enriched our evaluations by training more pre-trained LLMs on our selected data, such as  LLaMA-2-13B and Mistral-7B besides LLaMA-1-13B in the original submission. We have summarized these results in the table of the general response, while the detailed results are in Table 5,6,7 of the paper. On the Open LLM Leaderboard, our approach outperforms the random baseline and other strong SOTA models across three different backbone models in terms of the average results. In the meanwhile, we would like to note that the performance on benchmarks such as MMLU is largely determined by the underlying base model, thus the gains after alignment tuning is typically not large, unless the training data is specifically crafted for the benchmark. This is why recent works on alignment put more emphasis on MT-bench and AlpacaEval [2]. 

Notably, on MT-bench and AlpacaEval, our added results show that our method with the LLaMA-2-13B model surpasses Vicuna-13B-1.5 that trains on 10x more data, and LLaMA-2-13B Chat that has undergone extensive RLHF training. With Mistral-7B, our approach attains an MT-bench score of 7.29 with only 10k training data, marking a state-of-the-art performance on MT-bench *for all open-source SFT models at 7B and 13B sizes*. We also observe that, while the gains of the Mistral DEITA model on AlpacaEval are slightly lower than those on MT-bench, a deeper analysis (as depicted in Figure 4) indicates that the Mistral DEITA model excels particularly in mathematical and complex reasoning tasks, areas not primarily covered by AlpacaEval.

Reviewer comment (Evaluation):
Related to the above point, since human evaluation was done on only 100 instances, it would be helpful to quantify the reliability of this assessment, e.g.: using inter-annotator agreement scores and significance, and perform a larger scale evaluation if needed.

Author response:
Thanks for the advice! We would like to note that we tried hard to control the quality of human evaluation in the original submission – as described in Appendix D, we adopted MTurk in the beginning but abandoned their annotations due to low agreement scores among them. To justify our human evaluation, we follow LIMA [3] to randomly sample 50 generation pairs from our human evaluation dataset and compute the author-annotator agreement score – the resulting agreement score is 77%, which is reasonably high and aligns closely with the agreement levels reported in [3]. We have added the inter-annotator agreement score to Appendix D of the paper.

Reviewer comment (Experiments):
Section 2.3 states that ChatGPT is shown multiple samples evolved from the same seed example are shown to ChatGPT at a time for scoring their complexity (and this is possibly true for quality as well). Are these scores comparable across evalved sets from different seed examples? This is necessary because all these data points are used together as training data for training the Llama-based scorer. If they are not comparable, it might help to randomize the sets shown to ChatGPT for scoring.

Author response:
This is a good point. To answer this question, we conduct human evaluation on randomly selected 50 sample pairs from the ChatGPT complexity score pool – any example pair is not from the same evolved set. The authors then score each pair of examples from five categories: “A is significantly more complex than B”, “A is slightly more complex than B”, “neither is more complex”, etc. Then we compute the agreement score between the authors and ChatGPT – first, we transform the ChatGPT scores to pairwise judgment. We count it as “significantly better/worse” when the difference between ChatGPT scores is larger than 2, “slightly better/worse” for difference of 1, and tie in the case of 0. If ChatGPT has a high agreement score with the authors, that means the ChatGPT scores are comparable across evolved sets. 

In the “significantly better/worse” sample pairs, the ChatGPT scores are highly consistent with the authors, producing 83% agreement rate, which means the ChatGPT scores are comparable across evolved sets when the scores are not very close. In the “slightly better/worse” or tie categories, the authors also assign “slightly better/worse” or tie labels to 96% of such samples, suggesting high coarse-grained consistency. However, the fine-grained agreement on the three labels (slightly better/worse, tie) is only 43% – we find that it is hard even for humans to be consistent on comparing the slightly different complexity of instructions, especially when the two instructions are not relevant.

We tried to repeat the above experiments for quality scores as well, but we found that it is really difficult for humans to compare the quality of two responses that correspond to two very different query instructions.

Reviewer comment (Theory):
Is it possible for the diversity criterion to end up selecting outliers in the datasets? How can this issue be fixed?

Author response:
We agree with the reviewer that the diversity criterion may select outliers in the dataset. . However, we think the outliers may not be always bad – being diverse and infrequent, the outliers may actually increase the diversity of the selected SFT dataset. Although outliers in a dataset are often considered as low-quality samples, our score-first data selection approach should have filtered out the low-quality samples in the first step.

Reviewer comment (Evaluation):
DEITA-6K loses more to the random selection baseline than to the Vicuna model according to the human evaluation results in Table 4. This seems surprising. How does the random selection model perform compared to the Vicuna model in this case?

Author response:
We agree that this is a bit surprising, and we think this may be attributed to the simplicity of the LIMA test set, which fails to effectively distinguish between models that generally perform well. It is worth noting that the random baseline is not weak, as it is derived from a SOTA data pool. This situation highlights a broader challenge in human evaluation: for complex queries like those in MT-bench, human evaluation becomes difficult and costly, even for expert annotators, due to the complex examples which often include extensive mathematical and coding content. Conversely, with simpler queries like the LIMA test, human evaluators struggle to differentiate between strong models that excel in simple scenarios. To provide more clarity, we have included a detailed breakdown of the MT-bench results (based on GPT-4 evaluations) in Figure 4 of our paper. This breakdown clearly shows that the random baseline based on Mistral-7B lag significantly in areas such as coding, mathematics, and reasoning, aspects that are less prominent in the LIMA test and challenging for human evaluators to assess.