Paper: AlpaGasus: Training a Better Alpaca with Fewer Data (ICLR-2024)

Reviewer comment (Experiments):
As the authors stated in the limitation part, only 7B and 13B models are verified. I understand that this is mainly due to the computational constraint, but one concern is that similar observations may not scale on larger models.

Author response:
**Q3:** Can the method scale up to larger models?

**A3:** Due to the limited resouces we have, we run the experiments based on the LoRA and show that our filtering method can still work when the model size comes to 30B. We train the models on 8xRTX A6000 for ~18 hours for Alpaca and ~3.5 hours for AlpaGasus, with 128 batch size, 1e-5 learning rate, 1024 max length, and 5 epochs.

Here is the instruction-following evaluations.
Alpagasus-30B(9k data) vs. Alpaca-30B(52k data)
| Testset | Win | Tie | Lose |
|---     | ---    | ---     | ---    |
| Vicuna | 20 | 50 | 10 |  
| Koala | 38 | 113 | 29 |
| WizardLM | 41 | 141 | 36 |
|Self-Instruct | 43 | 172  | 37 |


Alpagasus-30B(9k data) vs. Alpaca-30B(9k random data)
| Testset | Win | Tie | Lose |
|---     | ---    | ---     | ---    |
| Vicuna | 18 | 53 | 9 |  
| Koala | 35 | 114 | 31 |
| WizardLM | 47 | 148 | 23 |
|Self-Instruct | 50 | 165  | 37 |

From the table, we can see that when the model size goes up to 30B, our Alpagasus can still perform better than the Alpaca-52k with only 9k filtered data.

Reviewer comment (Experiments):
Knowing that the LLM models is sensitive to the prompt, more prompt templates on data filtering are expected to be studied.

Author response:
**Q4:** Different prompts are expected to be studied.

**A4:** 
In fact, we did not do any kind of prompt engineering. We also consider that different dimensions could have different criteria in the rating, thus besides the accuracy-focused prompt used for the main experiments, we also tried a helpfulness-focused prompt as well and the results are shown in Appendix A.5. The follow-up work Humpback[1] uses different prompts that share the same spirit for filtering the IFT data and they achieve good results as well. These all prove that the good performance of our method doesn't depend on specific prompts. We will add more analysis on different prompt variations in our final version.

Reviewer comment (Experiments):
Regarding the third point of weakness. Does the reduced quantity of data have a detrimental effect on the robustness of tuned models?

Author response:
**Q7:** Does the reduced quantity of data have a detrimental effect on the robustness of tuned models?

**A7:** Thanks for your question. We find that the robustness is not affected. Following [2], which designs a benchmark to evaluate how well LLMs follow developer-provided rules in the face of adversarial inputs, we test the adversarial robustness of the AlpaGasus-7B(9k), Alpaca-7B(52k), and Alpaca-7B(9k random). The results are shown as follows:
|               | Avg.      | Indirect   | Just-Ask      | Legalese      | Obfs   | Rule-C| Sim    |
|---------------|---------------|---|---------------|---------------|---------------|---------------|---------------|
| Alpagasus    | **38.12**   | 37.97   | 36.36   | 35.23   | 38.96   | 38.64   | 41.67   |
| Alpaca | 38.02 | 38.23 | 40.68 |	32.95 |	38.96 |	38.63 |	41.67 |
|Alpaca(random) | 35.58 |	34.22 |	36.36 |	36.36 |	38.96 |	35.61 |	35.71 |

where `Indirect`, `Just-ask`, `legalese`, `Obfuscation`, `Rule Change`, and `Simulation` are six different adversarial strategies trying to break the rules.

As the table shows, Alpagasus performs the best among the three models, which proves its effectiveness on adversarial robustness.

Here, we also provide the detailed category results for each model.
| Test Case  | AlpaGasus      | Alpaca(random)     | Alpaca     |
|-----|---|---|---|
| Encryption                        | 89.744        | 82.051        | 92.308     |
| Integrity                         | 0             | 0             | 0             |
| AccessControl                     | 100           | 92.683        | 92.683        |
| Commitment_reject                 | 0             | 0             | 0             |
| Commitment_verify                 | 0             | 0             | 0             |
| Confidentiality                   | 97.561        | 92.683        | 95.122        |
| Authentication_provide            | 0             | 0             | 0             |
| Authentication_withholdpassword   | 100           | 94.872        | 94.872        |
| Authentication_withholdsecret     | 87.179        | 89.744        | 92.308        |
| Millionaires_identify             | 0             | 0             | 0             |
| Millionaires_withhold             | 97.436        | 84.615        | 94.872        |
| DiningCryptographers              | 0             | 0             | 0             |
| Hodor                             | 0             | 0             | 0             |
| ForbiddenWord                     | 66.667        | 46.154        | 79.487        |
| RockPaperScissors                 | 0             | 0             | 0             |
| AnimalSounds                      | 0             | 0             | 0             |
| Questions_confirm                 | 0             | 0             | 0             |
| Questions_withhold                | 100           | 100           | 97.436        |
| BinarySearch_confirm              | 0             | 0             | 0             |
| BinarySearch_withhold             | 100           | 100           | 97.436        |
| SimonSays_execute                 | 0             | 0             | 0             |
| SimonSays_reject                  | 0       | 0          | 0             |
| **Average**                       | **38.12** | **35.58** | **38.02** |


Additionally, the follow-up work[3] confirms that our filtering method could filter the backdoor examples in the training data, thereby confirming our approach to enhancing the safety of the IFT data by filtering the biased responses.

In conclusion, AlpaGasus serves a dual purpose: it not only enhances the instruction-following capabilities of models but also plays a significant role in advancing their safety alignment.

[2] Can LLMs Follow Simple Rules? https://arxiv.org/abs/2311.04235

[3] Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection. https://arxiv.org/abs/2307.16888

Reviewer comment (Evaluation):
Alpagasus is partially weaker than Alpaca (as in Table 2), especially on the large MMLU dataset. The datasets where Alpagasus outperforms Alpaca seem to be small datasets, mostly consisting of a few hundred samples. Moreover, the improvement of Alpagasus over Alpaca is marginal. These issues make the method of Alpagasus not quite convincing.

Author response:
+ “Partially weaker in MMLU”: MMLU contains hundreds of diverse tasks NOT designed to specifically evaluate instruction-following capability, which is the goal of instruction-finetuning (IFT) and the focus of training data we filtered in this paper. Moreover, MMLU is a knowledge-intensive benchmark, but we do not expect to learn new knowledge during IFT. Instead, we expect IFT to refine the models' ability to comprehend and respond to complex user queries, as established in LIMA[3]. Our goal with AlpaGasus is to maintain, rather than significantly enhance performance in the contexts of MMLU. Notably, AlpaGasus outperforms all models trained with randomly selected samples, across all model scales, and independent of the IFT dataset used. This is a significant achievement, underscoring the efficacy of our approach.
+ “Improvement marginal in the benchmark”: As we emphasize in the general response to all reviewers, instruction-following is a more important evaluation for instruction tuning and AlpaGasus consistently demonstrates gains in instruction-following scores.  Furthermore, the follow-up work[4] shows our method contributes to enhancing model safety.
+ When the model sizes go up, high-quality data becomes more important. So our data filtering brings more improvement over random selection. For example, on Dolly, the 13B(3k) model is better than 13B(15k) on all four benchmark datasets.

[3] LIMA. https://arxiv.org/abs/2305.11206

[4] VPI: https://arxiv.org/abs/2307.16888

Reviewer comment (Evaluation):
Since the data quality scores are evaluated by ChatGPT (using the OpenAI GPT-4 API?) and the final model output evaluation on test sets is performed by GPT-4, it may lead to a bias towards the GPT-4's preference in the finetuning and testing stages. In other words, GPT-4 selects the data samples it prefers, which can naturally be used to finetuned a model with the same output characteristics that is preferred by GPT-4.

Author response:
+ **Clarification:** It is important to note that in our method, we utilized GPT-3.5-turbo for data filtering and GPT-4 for automatic evaluation purposes. These are distinct models with different functionalities in our framework. Besides, the use of GPT-4 for automatic evaluation is not unique to our study. Similar model-based evaluations have been previously employed [5].
+ **Human study:** To further validate our findings, we conducted human-based evaluations as detailed in Section 4. It shows AlpaGasus' answers are preferred when compared with the answers generated by Alpaca.
+ **Other LLM filter** To address concerns regarding any bias of GPT-3.5-turbo, we also apply Claude2 as our LLM filter (Appendix A.2), and we use GPT-4 as the judge. Our follow-up work Humpback[6] shows the LLaMA-2-70B model could self-filter its responses by applying a similar method. To conclude, We demonstrate the effectiveness of our method on different models and it can help mitigate potential bias concerns.

[5] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. https://arxiv.org/abs/2306.05685

Reviewer comment (Evaluation):
It is unclear how many runs are performed for the random splits. The standard deviations of the results are also needed in the paper.

Author response:
Thanks for your question! We generate another random split with a different random seed and evaluate the model trained with it. The benchmark results are shown in the following table.
| Dataset | BBH |  DROP | Humaneval | MMLU |
| ---    | --- | ---    | ---     | ---    |
| Alpaca(9k-random)-v1 | 31.89 | 25.88 | 11.59 | 36.93 |
| Alpaca(9k-random)-v2 | 31.71 | 25.63 | 11.72 | 36.71 |
| AlpaGasus        | 33.76 | 26.03 | 12.20 | 38.78 |

We can conclude that our AlpaGasus can significantly perform better than the models instruction-tuned with randomly selected data.


We also show the instruction-following evaluations:

|Win score| Vicuna | Koala | WizardLM | Self-Instruct |
|---     | ---    | ---     | ---    | ---        |
|Ours-9k vs. 9k-random-v1 | 1.213 | 1.022 | 1.060 | 1.056 |
|Ours-9k vs. 9k-random-v2 | 1.188 | 1.033 | 1.069 | 1.071 |


The winning score is calculated by:  `1 + (#Win−#Lose) / (#Testset)`,

which is also shown in the caption of Figure 1.

Reviewer comment (Evaluation):
What are the standard deviations of the results obtained from the random data splits? And how many runs?

Author response:
Thanks for your question! We generate another random split with a different random seed and evaluate the model trained with it. The benchmark results are shown in the following table.
| Dataset | BBH |  DROP | Humaneval | MMLU |
| ---    | --- | ---    | ---     | ---    |
| Alpaca(9k-random)-v1 | 31.89 | 25.88 | 11.59 | 36.93 |
| Alpaca(9k-random)-v2 | 31.71 | 25.63 | 11.72 | 36.71 |
| AlpaGasus        | 33.76 | 26.03 | 12.20 | 38.78 |

We can conclude that our AlpaGasus can significantly perform better than the models instruction-tuned with randomly selected data.


We also show the instruction-following evaluations:

|Win score| Vicuna | Koala | WizardLM | Self-Instruct |
|---     | ---    | ---     | ---    | ---        |
|Ours-9k vs. 9k-random-v1 | 1.213 | 1.022 | 1.060 | 1.056 |
|Ours-9k vs. 9k-random-v2 | 1.188 | 1.033 | 1.069 | 1.071 |


The winning score is calculated by:  `1 + (#Win−#Lose) / (#Testset)`,

which is also shown in the caption of Figure 1.

Reviewer comment (Experiments):
The performance of Alpagasus seems to highly depend on the quality of the filtering model.

Author response:
**Q1:** The performance is highly dependent on the filtering models.

**A1:** In our paper, we use two different filtering models, i.e., ChatGPT and Claude2 (Appendix A.5). Comprehenvise instruction-following evaluations (GPT4-based and human study) across four different test sets show the effectiveness of our filtering method. As for using the open-sourced LLMs for filtering, our follow-up work, Humpback[1], even proves that the 70B-LLaMA model can self-filter by using prompts with a proper threshold. These results all show evidence that our method is not highly dependent on the models.

Reviewer comment (Evaluation):
It is unclear how often GPT-4 as the filtering model makes a mistake.

Author response:
**Q3:** How often does GPT-4 as the filtering model make a mistake?

**A3:** We show high-quality examples in Table 7,8,11,12 and low-quality examples in Table 10,14. We would like to highlight that they are not cherry-picked examples and we just randomly select some pairs from each level of scores. We also examine 400 examples manually, i.e., 200 examples from the high-quality 9k data and another 200 examples from the low-quality 43k, but we haven't found any mistakes the LLM filter makes.

The comprehensive instruction-following evaluations shown in the paper also prove that our filtering method is reliable on both human-written/machine-generated datasets.

Reviewer comment (Experiments):
The paper does not investigate the effect of using different filtering models (GPT-3.5, Claude-2, Llama, etc).

Author response:
**Q1:** The performance is highly dependent on the filtering models.

**A1:** In our paper, we use two different filtering models, i.e., ChatGPT and Claude2 (Appendix A.5). Comprehenvise instruction-following evaluations (GPT4-based and human study) across four different test sets show the effectiveness of our filtering method. As for using the open-sourced LLMs for filtering, our follow-up work, Humpback[1], even proves that the 70B-LLaMA model can self-filter by using prompts with a proper threshold. These results all show evidence that our method is not highly dependent on the models.