Paper: Agent Instructs Large Language Models to be General Zero-Shot Reasoners (ICLR-2024)

Reviewer comment (Reproducibility):
Although the methodology promotes a higher degree of autonomy in the question-answer process, its implementation might be deemed intricate by some, which could be a potential shortcoming (on the contrary, the implementation of simple methods such as CoT is easier).

Author response:
**Implementation**: We acknowledge that there are more steps in the zero-shot AgentInstruct pipeline than chain-of-thought. But one of the motivations to develop an agent-based approach is that the recent language agents (ReAct [1], LangChain [2], AutoGPT [3], GPTs [4], AutoGen [5]) are able to automate task planning and process. So, we have made it as easy to use as possible, which is one of our major internal goals. For now, AgentInstruct only requires the input consisting of basic dataset information such as the name of the dataset (Sec. 2). To facilitate the following research (as you have also mentioned), we already released our code and plan on further increasing the simplicity of our method.

Reviewer comment (Reproducibility):
Implementing the method could demand higher computational throughput from the LLM, leading to increased computational costs or API fees.

Author response:
**Computational costs**: One goal of zero-shot AgentInstruct is to improve the small models (like Llama 2) to perform competitively with large models (e.g., GPT 3.5). The result is shown in Figure 6. Since we only generate instructions once per dataset and the instructions do not add much to the context window, we believe the slightly increased cost is worthwhile for the increased performance. Notably, we observe more performance improvement with AgentInstruct on smaller models like Vicuna-13b.

Reviewer comment (Experiments):
While there's a notable enhancement in the performance of cutting-edge LLMs — Vicuna-13b by 13.3%, Llama-2-70b-chat by 23.2%, and GPT-3.5 Turbo by 17.0% — the performance increment isn't as pronounced with larger models like GPT-3.5 Turbo as it is with Llama-2-70b-chat. This raises concerns about the method's scalability with even larger models. The absence of data on its application to GPT-4, which is larger than GPT-3.5, further intensifies this curiosity about the method's scalability.

Author response:
**Performance on larger models**: Because zero-shot AgentInstruct uses GPT-4 (within an agent) to generate the instructions, we would expect better performance where the difference between that and the inference model’s size is larger. This is because we’d expect more 
information to be passed to the smaller model that it doesn’t inherently have access to, so the instructions would be more helpful in this situation. To address your review, we run an additional dataset, IMDB, with the following settings: Zero-Shot GPT-4, Zero-Shot CoT, and Zero-Shot AgentInstruct using GPT-4 for both instruction generation and inference, and the results are 87.4, 96.1, and 96.6 respectively. We will add the results to the paper. These experimental results seem to match up with this intuition, suggesting that as the performance gap between the agent model and inference model narrows, so does the impact of zero-shot AgentInstruct. This aligns with the conclusion from knowledge distillation work in LLMs like Alpaca [6], Vicuna [7], Self-Instruct [8].

Reviewer comment (Evaluation):
Evaluation vs “In-the-wild” generalization: While the paper shows strong improvements across various benchmarks, I am not convinced of the fundamental assumption made in the paper. To improve a performance on the task, the authors assume that the task is one of the “standard” tasks LLMs are typically evaluated on. While this is great for evaluating, it’s unclear how the proposed approach will work “in-the-wild”. In realistic use cases, the task might be completely different than one of the standard use cases. Additionally, we might not know beforehand which benchmark the task belongs to. In that case, it’s not going to always be possible to generate task-specific instructions.

Author response:
**In-the-wild generalization**: Thanks for pointing this out. In fact, in-the-wild tasks are among 29 tasks (53 sub-datasets) that our approach was evaluated on. For example, CivilComments and GSM8K are in-the-wild benchmark datasets as evidenced by HELM [1] and WILDS [2]. Zero-shot AgentInstruct outperforms standard zero-shot and CoT on these datasets. As shown in Table 6, when a task is completely new or out of domain (e.g., AQuA and SingleEq), the performance basically degrades to zero-shot CoT given there is no high-quality task knowledge on the web (Table 9). Of course, we agree that there are directions to enhance our method’s generalization ability. First, zero-shot AgentInstruct would be also suitable for in-the-wild tasks, if, for example, a set of many instances were seen on-the-fly that were drawn from a particular task. Second, given minimal task information, zero-shot AgentInstruct can be expanded to identify which task an instance belongs to, then choose from its previously generated instructions which instruction to use similar to [3]. Third, we can expand our method to use less information if it is not available, e.g., the labels. The requirement for the dataset name could be removed and replaced with more advanced prompting and querying methods. For example, we could imagine solving AddSub; instead of searching for information for “AddSub” explicitly, we could query for help on solving basic arithmetic problems using addition and subtraction and incorporate that general information into solving a specific task. This approach will allow for the utilization of our method and great expansion to more datasets.

Reviewer comment (Evaluation):
I also believe that cost comparisons done in Figure 6 are a bit misleading. Here, the authors show that Zero-shot AgentInstruct costs ~2$ vs 20$ (ZS-COT GPT4). This is true when running evaluation on standard benchmarks because we have to generate task-specific instructions once per benchmark. But like I said before, this is not going to be true in-the-wild. In that setup, a new instruction will have to be generated for every unique user query. For completeness, and to be consistent with a realistic use case, it would be more accurate to include the costs of creating task-specific instruction to each instance of the task, and then show the overall cost.

Author response:
**Cost of instructions**: We will add the suggested cost to Figure 6. So if the agent is called to generate instructions for each data instance instead of for the AddSub dataset once, the total cost will be approximately 50 cents per instance for 395 instances, costing about $197 overall. Currently, our methodology is the necessity of a repeated application of LLM reasoning steps on instances that fall under the same task. This is because instructions are only generated once per dataset, so the cost of using a large model is small, at most 1 dollar per dataset. When the number of tasks equals the number of instances, the costs would be similar. Still, as the number of instances increases, there will be more savings from using a smaller model for reasoning with zero-shot AgentInstruct regardless of the number of tasks.

Reviewer comment (Experiments):
How will the approach work in-the-wild, when the source of task is not known a priori? That is, we don’t know which dataset the task belongs to, or if the task belongs to any dataset at all.

Author response:
**In-the-wild generalization**: Thanks for pointing this out. In fact, in-the-wild tasks are among 29 tasks (53 sub-datasets) that our approach was evaluated on. For example, CivilComments and GSM8K are in-the-wild benchmark datasets as evidenced by HELM [1] and WILDS [2]. Zero-shot AgentInstruct outperforms standard zero-shot and CoT on these datasets. As shown in Table 6, when a task is completely new or out of domain (e.g., AQuA and SingleEq), the performance basically degrades to zero-shot CoT given there is no high-quality task knowledge on the web (Table 9). Of course, we agree that there are directions to enhance our method’s generalization ability. First, zero-shot AgentInstruct would be also suitable for in-the-wild tasks, if, for example, a set of many instances were seen on-the-fly that were drawn from a particular task. Second, given minimal task information, zero-shot AgentInstruct can be expanded to identify which task an instance belongs to, then choose from its previously generated instructions which instruction to use similar to [3]. Third, we can expand our method to use less information if it is not available, e.g., the labels. The requirement for the dataset name could be removed and replaced with more advanced prompting and querying methods. For example, we could imagine solving AddSub; instead of searching for information for “AddSub” explicitly, we could query for help on solving basic arithmetic problems using addition and subtraction and incorporate that general information into solving a specific task. This approach will allow for the utilization of our method and great expansion to more datasets.

Reviewer comment (Evaluation):
What is the cost of creating the task-specific instruction? Can you add a column to Figure 6, which adds this cost multiplied by the number of instances in the dataset plus the additional cost of reasoning using GPT-3.5 Turbo?

Author response:
**Cost of instructions**: We will add the suggested cost to Figure 6. So if the agent is called to generate instructions for each data instance instead of for the AddSub dataset once, the total cost will be approximately 50 cents per instance for 395 instances, costing about $197 overall. Currently, our methodology is the necessity of a repeated application of LLM reasoning steps on instances that fall under the same task. This is because instructions are only generated once per dataset, so the cost of using a large model is small, at most 1 dollar per dataset. When the number of tasks equals the number of instances, the costs would be similar. Still, as the number of instances increases, there will be more savings from using a smaller model for reasoning with zero-shot AgentInstruct regardless of the number of tasks.

Reviewer comment (Evaluation):
I didn’t understand the point of experiments in Section 3.3. What is the insight from that experiment?

Author response:
**Section 3.3 clarification**: Section 3.3 details the drop in performance when context length is expanded, which is largely based on the lengths of instances in the dataset. This is relevant for users who may be choosing which reasoning model to use; since the instructions take up space in the context window, they may decrease performance. Recent LLMs with longer context support will help further improve the performance.

Reviewer comment (Reproducibility):
I didn’t fully understand how the agent uses the question-answering API (Section A.3.1) to add context about the task. I think it’s one of the most crucial steps of the pipeline, and it’d be great to explain how it’s implemented in more detail (and perhaps in the main manuscript). Concretely, how are the retrieved documents used to generate the instruction? Are the retrieved documents added as prompt to the agent to generate task-specific instruction? Is that in addition to the name of the dataset, task information, and few input examples? Why are they added to a vector database?

Author response:
**Question answering API**: We will move some technical details (currently in Appendix A.3.1) to the main body based on your suggestion. For the question answering API, it is used as a tool for the agent. Since the documents are long, they are added to a vector database so they do not expand past the context length of the language model powering the agent. During the agent instruction generation step, another instance of the powerful language model receives questions about the document and interfaces with the vector database to provide an answer. Further information can be seen in our code.