Paper: Ada-Instruct: Adapting Instruction Generators For Complex Reasoning (ICLR-2024)

Reviewer comment (Experiments):
Why is the example number 10? Does more/less examples increase performance or lower the generalization ability?

Author response:
**Weakness 1: Why is the example number 10? Does more/less examples increase performance or lower the generalization ability?**

**Response:** Certainly, increasing the number of initial samples typically implies a higher quality of generated instructions and a closer distribution. We use 10 initial samples because we believe this setting represents a typical and challenging few-shot setting.

Reviewer comment (Experiments):
Can the finetuned LLM generalize to other datasets? For example, from math to GSM8K or even HumanEval? How do you know the training boundary given the randomly selected ten examples?

Author response:
**Weakness 2: Can the finetuned LLM generalize to other datasets? For example, from math to GSM8K or even HumanEval?**

**Response:** Although the ability of cross-dataset transfer learning is beyond this paper, we show that Ada-Instruct easily generalizes to a domain with multiple datasets. Please refer to our response to General Response - Question 1.

Reviewer comment (Experiments):
What are the properties of the selected ten examples?

Author response:
**Question 2: What are the properties of the selected ten examples?**

**Response:** As we have already stated in the experimental setup section, the initial samples for MBPP/MATH/GSM8k/CSQA are randomly selected, rather than deliberately chosen by humans. The effectiveness of this random strategy suggests the robustness of Ada-Instruct.

Reviewer comment (Experiments):
Since the proposed method focuses on improving the data generation procedure, it is interesting to know whether it is effective for the pretrained LLM in the general domain. For example, if 10-shot FT is better than 10-shot ICL for directly solving tasks instead of generating training data.

Author response:
**Question 3: Since the proposed method focuses on improving the data generation procedure, it is interesting to know whether it is effective for the pretrained LLM in the general domain. For example, if 10-shot FT is better than 10-shot ICL for directly solving tasks instead of generating training data.**

**Response:** This is substantiated. As illustrated in our related work section:
>  In more equitable experimental setups, researchers found that FT outperforms ICL (Mosbach et al., 2023), thereby lending support to our strategy of using FT models for instruction generation.

Reviewer comment (Experiments):
The exploration of which instructions are useful for SFT is not sufficiently clear. The paper initially points out that the issue with self-instruction is the limited length of generated instructions. However, later experiments show that Evol-Instruct with longer instructions does not perform well. The authors attribute this to "unnatural" instructions that do not align with downstream task distributions, but lack experimental validation. The authors can rewrite these instructions with open-source to make them more natural. Visually demonstrating the distribution relationships between training, Ada-Instruct, and Evol-Instruct can also support the argument.

Author response:
We highlight that Ada-Instruct not only creates long and complex instructions but also aligns with the target instruction distribution, a capability Evol-Instruct lacks.

To provide a straightforward understanding of the difference between Ada-Instruct and Evol-Instruct, we have calculated the average token length for their instructions on HumanEval. The average token length for the original instructions is 65.4 (±40.0), for Ada-Instruct it is 62.6 (±31.4), while for Evol-Instruct it is 138.1 (±122.7). It is clear that the distribution of Ada-Instruct is much closer to that of the original instructions.

Ada-Instruct's idea is simple: its instructions' distribution is more aligned with the target instructions. This idea has been validated from multiple perspectives, including length consistency (Figure 1 and previous experiments) and semantic consistency (Figure 3).

**We understand your concern that the gaps in Evol-Instruct's length and natural expression could be simply addressed with further prompting**. To address this, we updated Figure 3 (a) to visualize Evol-Instruct's semantic distribution (see paper's updated PDF). We switched from MPNet to OpenAI's text-embedding-ada-002 for more accurate semantic embeddings. It's evident that **Evol-Instruct significantly lags behind Ada-Instruct in semantic consistency, a gap that is hard to bridge with further prompting**.

Reviewer comment (Evaluation):
Ada-Instruct performs significantly better than self-instruct on math and commonsense reasoning tasks. The reviewer suggests using Figure 3 or other methods to demonstrate the improved alignment of Ada-Instruct with downstream tasks.

Author response:
Ada-Instruct's idea is simple: its instructions' distribution is more aligned with the target instructions. This idea has been validated from multiple perspectives, including length consistency (Figure 1 and previous experiments) and semantic consistency (Figure 3).

**We understand your concern that the gaps in Evol-Instruct's length and natural expression could be simply addressed with further prompting**. To address this, we updated Figure 3 (a) to visualize Evol-Instruct's semantic distribution (see paper's updated PDF). We switched from MPNet to OpenAI's text-embedding-ada-002 for more accurate semantic embeddings. It's evident that **Evol-Instruct significantly lags behind Ada-Instruct in semantic consistency, a gap that is hard to bridge with further prompting**.

Reviewer comment (Experiments):
Is Ada-Instruct sensitive to initial data? The paper asserts that Ada-Instruct performs better than self-instruct because data generated by tuning models with task data fits downstream task distributions better than in-context learning. Analyzing the impact of different initial samples on the final results would provide meaningful insights. Is Ada-Instruct robust when using initial samples that are not as scattered as shown in Figure 3 and do not align well with the training data?

Author response:
As we have already stated in the experimental setup section, the initial samples for MBPP/MATH/GSM8k/CSQA are randomly selected, rather than deliberately chosen by humans. The effectiveness of this random strategy suggests the robustness of Ada-Instruct.

Reviewer comment (Evaluation):
In Table 1, Ada-Instruct-HumanEval and Ada-Instruct-MBPP outperform GPT-3.5, but their labels are derived from GPT-3.5. Do the authors have further explanations?

Author response:
It is likely that different versions of GPT-3.5 exhibit performance variations. The results of GPT-3.5 we cited are from the "GPT-4 Technical Report" published in March 2023. In contrast, Ada-Instruct employs ChatGPT as of August 3, 2023.

Reviewer comment (Reproducibility):
Would the authors consider open-sourcing their code? This would facilitate verification and follow-up work.

Author response:
Certainly. We will release the code for verification and follow-up work.

Reviewer comment (Experiments):
**Comparison to Evo-instruct is lacking**: Though Evo-instruct seems to generat unnatural prompt, it has shown significant improvement over normal prompting. It’s necessary to directly compare Ada-instruct with Evo-instruct.

Author response:
We have already compared our Ada-Instruct with Evo-Instruct-based WizardCoder in Table 1. This baseline was implemented under higher resource conditions: it utilized 20,000 initial samples, significantly more than the 10 used by Ada-Instruct. Despite this, its performance was only on par with Ada-Instruct. Therefore, we believe that under identical conditions, Ada-Instruct performs better. Further evidence of this is shown in Table 2, where Ada-Instruct is closer to the target distribution compared to Evo-Instruct.

Moreover, Evo-Instruct cannot address the few-shot setting that Ada-Instruct solves. Taking the specific implementation of WizardCoder as an example, it requires a large-scale initial sample pool (e.g., 20,000 samples) to evolve diverse instructions. This requirement is in conflict with the few-shot setting presented in our paper.

Reviewer comment (Experiments):
**Lack of comparison to self-instruct in Table 4**: There is no comparison to self-instruct in Table 4, and it’s unclear if the proposed method outperforms simply prompting a close-source model.

Author response:
We have added results for Self-Instruct using the same initial samples as Ada-Instruct, and a baseline by prompting ChatGPT, as shown in the table below. Ada-Instruct still performs the best.

| Model| Initial Data  |  SFT Data  |  CommonsenseQA  |
|  :----  | :----:  | :----:  |  :----:  |
|Self-Instruct| 10 | 10k | 71.4 |
|ChatGPT [1]| - | - | 74.0 |
|Ada-Instruct| 10 | 10k | 75.5 |

[1] Chen, Jiuhai, et al. "When do you need Chain-of-Thought Prompting for ChatGPT?." arXiv preprint arXiv:2304.03262 (2023).

Reviewer comment (Reproducibility):
The SFT data generation process for the code llama-instruct model described in Rozière et al., 2023 (i.e., generate 62,000 questions with Llama 2-70b, and then extrapolate) is very different from what the authors described in the paper (i.e., using 10 initial data points to extrapolate with self-instruct), why does there exist such a discrepancy?

Author response:
The instruction generation technique used by Code Llama-instruct is similar to that of Self-Instruct. It involves prompting Llama 2-70b with in-context prompts and examples to generate instructions. As demonstrated in Figure 1, in-context examples are not well-suited to adapt to the target instruction distribution, especially in generating longer and  complex instructions.

Reviewer comment (Novelty):
In terms of innovation, the authors seem to have some misconceptions. Specifically, there have been previous works that used open-source models to generate instructions, such as the use of the open-sourced LLM Llama in [1], rather than ChatGPT or GPT-4. So what the authors mentioned in the introduction is not true: 
> A prevalent approach is called “self-instruct” (Wang et al., 2022), which involves having ChatGPT sequentially generate both instructions and answers (Sun et al., 2023; Peng et al., 2023; Taori et al., 2023; Schick & Schutze, 2021; Honovich et al., 2022; Ye et al., 2022; Meng et al., 2022; 2023).
And it leads to the comparison between this work and "previous work" in Figure 2 being inappropriate.

Author response:
**Response:** We believe there is a misunderstanding by the reviewer about the novelty of our work. Dromedary [1] employs Self-Instruct technology for generating instructions, which is one of our main baselines. As shown in Figure 1, Self-Instruct struggles with creating long, complex instructions. Our key innovation is demonstrating that fine-tuning, rather than in-context prompting, overcomes this issue, producing instructions that better match downstream task distributions.

Reviewer comment (Experiments):
I think one model for all reasoning tasks is needed, instead of training a separate model for each benchmark. I wonder if training a model for all benchmarks will increase the required instruction overhead. Can the authors show some evidence against this?

Author response:
**Response:** The instruction overhead of Ada-Instruct is, in fact, lower than that of the baselines. We will explain this below.

As addressed in our response to the Weakness 2, we can merge few-shot initial samples from multiple benchmarks to enhance a specific domain model. Subsequently, due to Ada-Instruct's superior ability to fit data distributions (as seen in Figure 1 and Table 2), it incurs lower instruction overhead when enhancing a domain model. For instance, in Table A in General Response, Ada-Instruct with 10k SFT data significantly surpasses Self-Instruct-Alpaca with 20k SFT data.