Paper: FireAct: Toward Language Agent Finetuning (ICLR-2024)

Reviewer comment (Novelty):
Generally, I think the main contribution of this paper can be considered as a data augmentation, which aims to utilize strong LLMs to generate trajectories for language models and mix multiple training tasks for fine-tuning. The idea of mixing multiple training tasks is not interesting, and previous works like T5 or instruction tuning also adopted such an experience for tuning.

Author response:
**On the Contribution of FiReAct:**
While we acknowledge that the concept of data augmentation and multi-task fine-tuning is not novel in itself, the unique contribution of FiReAct lies in its application to language agent fine-tuning. We are among the first to systematically explore how agents can acquire generalizable skills through carefully designed fine-tuning trajectories. Our findings offer preliminary yet promising insights into the potential of language agent fine-tuning, paving the way for future research in this area.

Reviewer comment (Experiments):
Although the authors have provided massive experiments to verify different fine-tuning factors in this paper, most of the conclusions are obvious and not inspired. For example like the experiments in sec 5.3, many conclusions are straightforward and it is evident that full-model training is better than LoRA. How to design experiments to prove the innovation of the proposed method (FireAct) is more important, rather than analyzing these factors.

Author response:
**Regarding Experiment Design and Conclusions:**
We understand your concern about the apparent straightforwardness of some conclusions, such as the comparison between full-model training and LoRA. However, these results are crucial for practitioners in the field, providing concrete, quantitative insights into the performance-efficiency tradeoff in agent fine-tuning. For instance, our findings on the comparative efficiency of LoRA versus full-model training on different hardware setups (detailed in Appendix A.5) are valuable for those implementing these methods in practice.

Reviewer comment (Experiments):
Too many fine-tuning details and settings are given in the appendix, not the main paper. And many useless experimental results cost too many pages in the whole paper. However, these experiments do not bring any insights. On the contrary, some fine-tuning details are more important for us to know the contribution of this paper.

Author response:
**Concerning the Detailedness of the Appendix:**
We appreciate your feedback on the balance between the main paper and the appendix. We invite specific suggestions on which experimental results you find less insightful and which fine-tuning details you believe are crucial but missing. Such feedback will be invaluable in refining our paper.

Reviewer comment (Presentation):
What is the content of Appendix A.2?

Author response:
**Content of Appendix A.2:**
Appendix A.2 presents a detailed comparison of results across different tasks and methods, focusing on robustness analysis. We have made efforts to enhance the clarity of this section to avoid any ambiguity.

Reviewer comment (Novelty):
The FiReACT method is not novel today. Several works have proposed fine-tuning Language Model Models (LLMs) on ReACT, COT, or Reflection data. Examples include "Large Language Models Are Reasoning Teachers" (Ho et al., 2022) and toolLLaMA (Qin et al., 2023).

Author response:
**On the Novelty of FiReAct:**
We acknowledge the existence of concurrent works such as those by Ho et al. (2022) and Qin et al. (2023), which we have duly cited and discussed in our related work section. FiReAct, however, distinguishes itself by exploring the multi-task, multi-method fine-tuning approach for language agents. Our work delves into new dimensions such as generalization to novel tasks, robustness against noise, and efficiency in agent operation, thereby contributing unique insights to the field.

Reviewer comment (Experiments):
The authors have not conducted enough experiments on the setting of full fine-tuning. I wonder whether the conclusions of this paper still hold when the method is applied to fully fine-tune an agent.

Author response:
**Experiments on Full Fine-Tuning:**
Our experimental setup primarily utilized LoRA fine-tuning due to computational constraints. However, we posit that our key findings, particularly regarding the enhanced robustness of fine-tuned agents to tool noise and the efficiency benefits during inference, would likely remain valid in a full fine-tuning scenario.

Reviewer comment (Evaluation):
What is the reason behind the observation that the fine-tuned agents exhibit more robustness compared to the zero-shot ones, and that the LLMs fine-tuned on the multi-method training data can implicitly select reasoning methods?

Author response:
**Robustness and Implicit Method Selection:**
The observed robustness of fine-tuned agents, especially in comparison to zero-shot counterparts, can be attributed to the inclusion of noisy data in the fine-tuning trajectories. This exposure enables the agents to better handle imperfect tool outputs, a scenario less effectively addressed by few-shot prompting, which often relies on idealized examples (refer to Table 3 for details).

The phenomenon of implicit method selection in multi-method fine-tuned agents is indeed intriguing. Our current hypothesis is that the fine-tuning process, which selectively focuses on successful trajectories, results in a varied distribution of training questions associated with different methods. This could enable the fine-tuned agents to learn associations between question features and the most effective method for addressing them. We believe this presents an exciting avenue for future research to explore in depth.

Reviewer comment (Theory):
It is unclear the advantage of FiReAc compared to ReAct considering FiReAc is fine-tuned based on ReAct trajectories. The authors claimed that “FiReAct agents benefit from the diversity of learning support, thus become more robust to external noises, more generalizable to novel tasks, and more flexible to choose various agent methods and task solving strategies adaptive to the problem at hand.” However, intuitively, it is hard to understand the reasons why fine-tuning can make the agents more robust and generalizable. It is suggested that authors could provide more explanations.

Author response:
**Advantages of FiReAct over ReAct:**
Your query about the benefits of FiReAct compared to ReAct is crucial. FiReAct's fine-tuning process, unlike ReAct's few-shot prompting, involves learning from a broader range of trajectories that encompass diverse task-solving strategies. This diversity is key to FiReAct's enhanced robustness and generalizability. Specifically:
- FiReAct shows improved performance on the test sets of training tasks (refer to Table 2).
- When applied to novel tasks, FiReAct maintains or enhances performance, as evidenced in Table 3 and our General Response. This is a notable departure from traditional NLP fine-tuning, where the focus is often on specific facts or task formats. FiReAct, by teaching LLMs how to reason and utilize tools like Google search, imparts skills that are transferable across various tasks.
- FiReAct demonstrates greater resilience to noise in tool outputs. This is because the fine-tuning process includes trajectories with noise, teaching the model to handle imperfect data. In contrast, few-shot prompting often relies on 'ideal' examples, which may not prepare the model for real-world variability (see Table 3).

Reviewer comment (Evaluation):
It is known that fine-tuning can hurt the generalization ability of LLMs. But the authors claimed that the proposed “agent fine-tuning” method is “more generalizable to novel tasks”, which is hard to understand.

Author response:
**Advantages of FiReAct over ReAct:**
Your query about the benefits of FiReAct compared to ReAct is crucial. FiReAct's fine-tuning process, unlike ReAct's few-shot prompting, involves learning from a broader range of trajectories that encompass diverse task-solving strategies. This diversity is key to FiReAct's enhanced robustness and generalizability. Specifically:
- FiReAct shows improved performance on the test sets of training tasks (refer to Table 2).
- When applied to novel tasks, FiReAct maintains or enhances performance, as evidenced in Table 3 and our General Response. This is a notable departure from traditional NLP fine-tuning, where the focus is often on specific facts or task formats. FiReAct, by teaching LLMs how to reason and utilize tools like Google search, imparts skills that are transferable across various tasks.
- FiReAct demonstrates greater resilience to noise in tool outputs. This is because the fine-tuning process includes trajectories with noise, teaching the model to handle imperfect data. In contrast, few-shot prompting often relies on 'ideal' examples, which may not prepare the model for real-world variability (see Table 3).

Reviewer comment (Novelty):
The method seems to be very trivial. It seems this paper just renames the conventional “fine-tuning” as “agent fine-tuning”.

Author response:
**On the Alleged Triviality of Agent Fine-Tuning:**
We appreciate your concern regarding the perceived simplicity of our approach. However, the novelty of FiReAct lies in:
- Addressing the gap in research: Prior work has largely overlooked fine-tuning LLMs specifically for agent roles, with most studies either fine-tuning LLMs for non-agent tasks or using prompting for agent tasks.
- Demonstrating multi-faceted empirical benefits: FiReAct, despite its simplicity, showcases significant benefits in terms of robustness, generalization, efficiency, and cost, which are crucial for practitioners in the field of agent development.
- Exploring new technical depths: FiReAct pioneers the approach of multi-method, multi-task fine-tuning for agents. It demonstrates that even smaller LLMs can effectively acquire generalizable agent skills through supervised fine-tuning, presenting a viable alternative to more complex and less controllable methods like reinforcement learning.

Reviewer comment (Novelty):
The technical contribution is limited because it is unclear what is technical challenges of “agent fine-tuning”. It is suggested the authors provide more explanations.

Author response:
**On the Alleged Triviality of Agent Fine-Tuning:**
We appreciate your concern regarding the perceived simplicity of our approach. However, the novelty of FiReAct lies in:
- Addressing the gap in research: Prior work has largely overlooked fine-tuning LLMs specifically for agent roles, with most studies either fine-tuning LLMs for non-agent tasks or using prompting for agent tasks.
- Demonstrating multi-faceted empirical benefits: FiReAct, despite its simplicity, showcases significant benefits in terms of robustness, generalization, efficiency, and cost, which are crucial for practitioners in the field of agent development.
- Exploring new technical depths: FiReAct pioneers the approach of multi-method, multi-task fine-tuning for agents. It demonstrates that even smaller LLMs can effectively acquire generalizable agent skills through supervised fine-tuning, presenting a viable alternative to more complex and less controllable methods like reinforcement learning.

Reviewer comment (Writing):
It is suggested the authors summarize their contributions more clearly. It is very vague to say “To sum up our contributions, we advocate for the overlooked direction of language agent fine-tuning, and propose novel methodologies, systematic experiment designs, practical insights and guidelines, as well as new research questions for this direction."

Author response:
**Clarification of Contributions:**
In response to your suggestion, we have revised our introduction to more clearly articulate our contributions. We now explicitly state how FiReAct advances the field of language agent fine-tuning, detailing the novel methodologies, systematic experiment designs, practical insights, and new research questions that our work introduces.

Reviewer comment (Experiments):
Task and Tool Limitation: The research is limited to question-answering (QA) tasks and relies on a single tool (Google search) for evaluation. This limitation questions the generalizability of FiReAct to other types of tasks and tools, which is a critical aspect for language agents intended for broader applications. To make the conclusions more convincing, it would be great if you could evaluate on other types of interactive tasks such as ScienceWorld, Mind2Web, InterCode, etc.

Author response:
**Task and Tool Limitation:** 
We acknowledge your concern regarding the generalizability of FiReAct. In response, we have expanded our experiments to include additional tasks from AgentBench, which extends beyond QA and incorporates a variety of tools. These new experiments are designed to demonstrate FiReAct's adaptability to a broader range of tasks and tools.

Reviewer comment (Experiments):
Methodology Limitations: The paper focuses on three methods that maintain a single autoregressive trajectory context, which might not be representative of more complex agent behaviors. The ability to handle multiple prompts, roles, and contexts is a significant aspect of agent design that is not addressed in the current research.

Author response:
**Methodology Limitations:** 
We understand the importance of handling multiple prompts, roles, and contexts in agent design. Our current focus on methods like CoT, ReAct, and Reflexion, which align with the ReAct autoregressive format, was a strategic choice due to their compatibility with multi-method fine-tuning. However, we recognize this as a limitation and an exciting avenue for future research, as noted in our revised manuscript.

Reviewer comment (Experiments):
How do the authors envision the application of FiReAct to tasks beyond QA and the use of tools other than Google search? Can FiReAct work on other types of interactive tasks such as ScienceWorld, Mind2Web, InterCode, etc? Why do you choose to focus on QA only?

Author response:
**Application of FiReAct Beyond QA and Advanced Agents:** 
As detailed in our General Response, FiReAct's application extends to various interactive tasks, including those mentioned. Our initial focus on QA was due to its well-defined structure and data availability, making it an ideal starting point for demonstrating FiReAct's capabilities.

Reviewer comment (Experiments):
How might prompting and fine-tuning be optimally combined in a complex agent system?

Author response:
**Combining Prompting and Fine-Tuning:** 
As discussed in Section 8 of our paper, we advocate for a balanced approach. Fine-tuning is preferable for well-defined, frequently used tasks with sufficient data, while prompting offers flexibility and convenience for less defined or zero-shot scenarios.