Paper: CITING: Large Language Models Create Curriculum for Instruction Tuning (ICLR-2024)

Reviewer comment (Evaluation):
Because of this formulation the student model requires a rubric and a version of the response to rewrite at inference time. The dependence on additional inputs might impact the generalizability of the student model as obtaining. Particularly, the rubrics generated for the training data may not generalize to the new instructions at inference time.

Author response:
Since the queries in the Alpaca data set are diverse and cover a wide range of types, we consider selecting a certain amount of queries so that the criteria obtained by GPT-4 can have better generalization. In addition, we agree with what you said about continuing to train the student LLM to be a rewriter might affect the original instruction following capabilities. However, our experimental results show that this method still shows good performance. To train a separate rewriter  is indeed a good way to solve the above problems, but this method requires a lot of cost of  teacher LLM, and training a separate rewriter independently also requires additional network cost. Thank you very much for your suggestions. We will compare your proposed solutions in the final version and comprehensively analyze the advantages and disadvantages of our methods.

Reviewer comment (Evaluation):
Since the student is initialized using SFT on the instruction tuning dataset, the initial draft responses produced by the model may be of good quality, but continuing to train it to be a rewriter might affect the original instruction following capabilities. An evaluation of the original drafts produced by the student after the iterative training might be helpful.

Author response:
Since the queries in the Alpaca data set are diverse and cover a wide range of types, we consider selecting a certain amount of queries so that the criteria obtained by GPT-4 can have better generalization. In addition, we agree with what you said about continuing to train the student LLM to be a rewriter might affect the original instruction following capabilities. However, our experimental results show that this method still shows good performance. To train a separate rewriter  is indeed a good way to solve the above problems, but this method requires a lot of cost of  teacher LLM, and training a separate rewriter independently also requires additional network cost. Thank you very much for your suggestions. We will compare your proposed solutions in the final version and comprehensively analyze the advantages and disadvantages of our methods.

Reviewer comment (Experiments):
One solution to address the concerns above could be to train a separate rewriter (conditioned on the rubric and the previous responses), and iteratively finetune the student model only on the instruction following task (i.e., not condition its outputs on the rubric and previous responses).

Author response:
Since the queries in the Alpaca data set are diverse and cover a wide range of types, we consider selecting a certain amount of queries so that the criteria obtained by GPT-4 can have better generalization. In addition, we agree with what you said about continuing to train the student LLM to be a rewriter might affect the original instruction following capabilities. However, our experimental results show that this method still shows good performance. To train a separate rewriter  is indeed a good way to solve the above problems, but this method requires a lot of cost of  teacher LLM, and training a separate rewriter independently also requires additional network cost. Thank you very much for your suggestions. We will compare your proposed solutions in the final version and comprehensively analyze the advantages and disadvantages of our methods.

Reviewer comment (Experiments):
Effect of continued training: Simply train the student model with additional SFT steps, and do not use a teacher model at all. The current comparison between CITING and SFT is not entirely fair because CITING uses additional training steps.

Author response:
Thank you very much for your suggestions. We will include comparisons with these baselines in the final version.

Reviewer comment (Experiments):
Effect of the rubrics: Use the teacher model to generate a rubric and train the student model with additional SFT steps also conditioned on the rubric.

Author response:
Thank you very much for your suggestions. We will include comparisons with these baselines in the final version.

Reviewer comment (Experiments):
Fairer comparison with RLHF: RLHF might perform better with better pairwise preference data. A good baseline would be to generate rewrites from a teacher model and use them to train a reward model and perform RLHF.

Author response:
Thank you very much for your suggestions. We will include comparisons with these baselines in the final version.

Reviewer comment (Experiments):
The details of the baselines are missing. How exactly are the RLHF models trained? Do they use preferences on the same 1000 instances used for CITING?

Author response:
Thanks for noticing the missing implementation details in the paper. Specifically, to ensure a fair comparison of all baselines, we first use the original llama-1 model to generate two different responses for 1,000 queries from the training dataset. We then use GPT-3.5 to rank the quality of these two responses for each query. We then utilize these data to  implement RLHF. We will add these explanations in Appendix G.

Reviewer comment (Evaluation):
How do you ensure that the teacher minimally edits the student responses in the CITING algorithm? Do you actually see that the edits are minimal?

Author response:
This is a good point. Our method does not guarantee that teachers can minimally edit the student responses. In fact, the CITING algorithm is to build a curricularly revision mechanism so that student LLM can learn how to modify its initial answer step by step with the help of teacher LLM. Of course, such a mechanism does not guarantee that every teacher's modification to the student's response will be minimal. This modification may be large or small. We have tried to add an additional prompt to the teacher model that "it should make minimal edits to the student model" and it can work for some cases. But when the student model's outputs are totally wrong are meaningless, the edits can still be equal to rewriting.

Reviewer comment (Experiments):
How do you select the number of iterations at training time and inference time in CITING?

Author response:
We set up a validation set during both the training and testing stages to help us choose the number of iterations. We select the number of iterations with the best performance on the validation set as the final result.

Reviewer comment (Novelty):
The technical novelty may be limited. The method is also complicated.

Author response:
We thanks for the reviewer to propose the limitations of this work. Though we are not clear which parts the reivewer refers to, we hope to clarify more on the novelty and complexity of the proposed method below.

* Novelty: Compared with the previous learning-from-AI-feedback (LAIF) method, CITING can more efficiently utilize the information fed back by teacher LLM. The traditional LAIF method often only provides the query response to the student LLM. On this basis, CITING allows student LLM to provide the response to the query and also provides the criteria to answer the query. Therefore, CITING not only teaches student LLM the answer to the question, but also teaches it how to answer and correct the question than the traditional LAIF. Compared with the previous synthetic instruction tuning method (i.e., ask the teacher LLM to generate synthetic instruction data), CITING makes full use of human written answers and prompts the teacher LLM to supervise the student model to improve their outputs. This alignment process is free of reinforcement learning as used in RLHF thus being easy to implement and more robust.

* Complexity：We add little complexity in CITING compared with the classic instruction tuning process. There is one more step that we need to call a teacher model to provide feedback to the student model to improve the outputs of the student. Compared to other alignment approaches, such as RLHF, CITING does not involve an additional reward model and reinforcement learning steps, thus being much easier to implement and computationally friendly.

Reviewer comment (Novelty):
My main concern about this work is about novelty and clarity. Even though the method proposes using criteria as a guidance to augment instruction tuning data, the overall method, still, can be viewed as a complex version of data distillation. Recently there have been tons of works proposing pretty similar ideas, such as Orca [1], WizardLM [2], MAmmoTH [3], etc., which all leverage data augmentation (guided by certain criteria/score function, etc.). It might be good to compare CITING with these methods, or at least discuss why the contribution of this work is significant.

Author response:
**Response:**  Thanks for the reviewer to propose those relevant papers from the literature! However, after taking a deep look into these papers, we believe CITING is fundamentally different from them. 

First, these methods essentially try to distill the capability from the teacher model by training the student model on the synthetic instruction data from the teacher. In this process, no human written answers are involved, and there is limited control of hallucinations and errors in the synthetic instruction datasets. By contrast, CITING leverages the teacher model to generate the revisions from the student model referring to the human written answers, alleviating the hallucination issues. Also, the student model in CITING is a distilled teacher model and specialized in a specific scope of tasks, thus being possible to be better than the teacher while the other synthetic instruction tuning method cannot yield such a student model but causes the model to deteriorate (mode collapse) over time, which has been proposed by recent papers **[R1]**.

Second, there is no alignment for the student model in the other synthetic instruction tuning method but CITING focuses on how to optimize the student LLM step by step with the feedback from teacher LLM. During this process, the teacher LLM will receive the last revision of the student LLM and conduct revisions and guides based on the criteria, thus establishing communication and feedback between the two.

**[R1]** Shumailov I, Shumaylov Z, Zhao Y, et al. The Curse of Recursion: Training on Generated Data Makes Models Forget[J]. arXiv preprint arxiv:2305.17493, 2023.

Reviewer comment (Theory):
I'm not fully convinced about the method of how you assign quality scores to the generated data. In Section 3.2, why does the fact that the candidate instructions have their embeddings near those of rubrics mean that the candidates are legit?

Author response:
**Response:**  Thanks for pointing out the parts that cause confusion. We hope to add the following clarifications.

1) In order to verify the fact that the embedding of candidate instructions is close to the title means that the candidate instructions are legal, we designed experiments for 1000 queries to verify. We first let GPT-4 determine its criteria for these 1000 queries. Then take out 100 of them as data for known criteria, and the remaining 900 as test data for unknown criteria. Our experiments found that the method based on embedding similarity matching can achieve a criterion prediction accuracy of 72% on 900 test data sets, which proves the effectiveness of our method.

2) Although simply using rubrics as "system message" in the few-shot examples can simplify CITING's pipeline, it has two problems: 1. Since the query exists in various forms in the data set, the "system message" needs to be very complex. To achieve a good effect, a large number of examples need to be given, and the limitations of the context window of student LLM make this approach impossible to implement; 2. The corresponding information of query and criteria contained in the example in "system message" also requires student LLM to learn, which increases the difficulty of student LLM learning compared to directly matching the criteria of the corresponding query. We have added this part of the discussion to Appendix D of the paper.

Reviewer comment (Experiments):
In terms of the Appendix, the so-called rubrics are basically some descriptions or requirements of a certain task — do we really need a complex pipeline like CITING or can we simply include them as the "system message" in the few-shot examples (for augmentation)?

Author response:
**Response:**  Thanks for pointing out the parts that cause confusion. We hope to add the following clarifications.

1) In order to verify the fact that the embedding of candidate instructions is close to the title means that the candidate instructions are legal, we designed experiments for 1000 queries to verify. We first let GPT-4 determine its criteria for these 1000 queries. Then take out 100 of them as data for known criteria, and the remaining 900 as test data for unknown criteria. Our experiments found that the method based on embedding similarity matching can achieve a criterion prediction accuracy of 72% on 900 test data sets, which proves the effectiveness of our method.

2) Although simply using rubrics as "system message" in the few-shot examples can simplify CITING's pipeline, it has two problems: 1. Since the query exists in various forms in the data set, the "system message" needs to be very complex. To achieve a good effect, a large number of examples need to be given, and the limitations of the context window of student LLM make this approach impossible to implement; 2. The corresponding information of query and criteria contained in the example in "system message" also requires student LLM to learn, which increases the difficulty of student LLM learning compared to directly matching the criteria of the corresponding query. We have added this part of the discussion to Appendix D of the paper.

Reviewer comment (Reproducibility):
There is no code and data uploaded as supplementary materials, which causes some difficulties for me to fully understand your method.

Author response:
**1.** We appreciate your interest in our work. To provide further clarity, we have included 1000 samples along with the corresponding criteria similarity matching code in the supplementary material. We kindly invite you to review these for more detailed information.

Reviewer comment (Reproducibility):
How did you implement RLHF? There is no official implementation of that and your method has shown significant improvement over RLHF. Could you upload your code if possible (or point to some open-sourced implementation you are using)?

Author response:
**Response:**  We have upload the  implementation of our RLHF to the supplementary material. Specifically, to ensure a fair comparison of all baselines, we first we first use the original llama-1 model generate two different responses for 1,000 queries from the training dataset. We then use GPT-3.5 to rank the quality of these two responses for each query. Next, we train a reward model based on this data and implement RLHF. We have added this part of the discussion to Appendix E of the paper.

Reviewer comment (Evaluation):
Over-reliance on Teacher LLM: There's a potential risk if the teacher LLM possesses biases or inaccuracies, as it could transfer these shortcomings to the student LLM. Consequently, the effectiveness of CITING is largely contingent on the quality and resilience of the teacher LLM.

Author response:
Thanks for pointing out this concern. The current CITING does not have a dedicated module to consider potential biases or inaccuracies if the teacher LLM possesses. We will consider adding some feedback mechanisms based on CITING to solve this problem. Nonetheless, we conjecture the bias and inaccuracies should be minor since we are not asking the teacher LLM to make the revision from scratch. Instead, the teacher LLM's input also includes the groundtruth answers, so it is prompted to improve the student's input referring to the groundtruth.

Reviewer comment (Evaluation):
Evaluation Metrics: The paper predominantly emphasizes the winning rate for comparing with other techniques. Yet, in scenarios like QA or RC, shouldn't the method also be evaluated using standard metrics?

Author response:
We thanks the reviewer to propose more evaluation metrics. There are indeed many NLP standard metrics that can be used to evaluate CITING, such as PPL, BLEU and Precision. However, these standard metrics are evaluated by measuring the similarity between the response generated by LLM and the ground truth. However, they are very limited in actual QA and RC tasks, because LLM may generate responses that are equally reasonable but very different from the ground truth. There are now many recent works that introduce GPT-4 to conduct a more comprehensive evaluation of the responses generated by LLM from various perspectives **[R1,R2]**. Therefore, in this paper, we follow this setting and utilize GPT-4 as our evaluation metric.

Reviewer comment (Theory):
How does CITING address potential biases or inaccuracies if the teacher LLM possesses them?

Author response:
Thanks for pointing out this concern. The current CITING does not have a dedicated module to consider potential biases or inaccuracies if the teacher LLM possesses. We will consider adding some feedback mechanisms based on CITING to solve this problem. Nonetheless, we conjecture the bias and inaccuracies should be minor since we are not asking the teacher LLM to make the revision from scratch. Instead, the teacher LLM's input also includes the groundtruth answers, so it is prompted to improve the student's input referring to the groundtruth.

Reviewer comment (Evaluation):
Beyond the winning rate, have other standard metrics been considered for evaluating CITING, especially in QA or RC scenarios?

Author response:
We thanks the reviewer to propose more evaluation metrics. There are indeed many NLP standard metrics that can be used to evaluate CITING, such as PPL, BLEU and Precision. However, these standard metrics are evaluated by measuring the similarity between the response generated by LLM and the ground truth. However, they are very limited in actual QA and RC tasks, because LLM may generate responses that are equally reasonable but very different from the ground truth. There are now many recent works that introduce GPT-4 to conduct a more comprehensive evaluation of the responses generated by LLM from various perspectives **[R1,R2]**. Therefore, in this paper, we follow this setting and utilize GPT-4 as our evaluation metric.