Paper: ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR-2024)

Reviewer comment (Evaluation):
This paper introduces TaskEval, which encompasses two metrics, *Pass Rate* and *Win Rate*. Specifically, *Pass Rate* detects whether LLM can successfully execute user instructions, and *Win Rate* is designed to judge which solution path is better for a given instruction. However, to some degree, I think these metrics still have some deficiencies and cannot efficiently [measure] the capability of LLMs in tool utilization. For example, *Pass Rate* can only reflect whether LLM can execute user instruction. Sometimes, powerful LLMs (e.g., ChatGPT) can always generate answers for any user instructions without any tool use, and besides, the hallucination of LLMs can also let it generate some executed but counterfactual answers. Besides, *Win Rate* can only reflect the capability of different LLMs, not their ability of tool utilization. Even two LLMs without tool utilization (e.g., ChatGPT v.s. LLaMA-7b), will also demonstrate differences in performance. Therefore, I think the proposed metrics can only reflect the capability of LLMs but not their tool-use ability. Of course, I also admit that it is challenging to build such a metric since there have been no metrics in this area before.

Author response:
Thank you for your insightful comments regarding our ToolEval metrics, including Pass Rate and Win Rate. We appreciate your concerns about the potential limitations of these metrics in fully capturing the tool utilization capabilities of LLMs. Below are our detailed responses:

+ **Purpose and Scope of Pass Rate**: The Pass Rate metric is designed to assess whether an LLM can successfully execute user instructions. While it is true that this metric primarily focuses on the execution aspect, it is an essential baseline measure. It helps to filter out responses that are non-executable or irrelevant, which is a fundamental requirement for effective tool use. The concern about powerful LLMs like ChatGPT generating answers without tool use is valid. However, in our experimental setup, the emphasis is on the execution of tasks involving specific tools, thereby directly relating to tool utilization.

+ **Role of Win Rate in Evaluating Tool Utilization**: The Win Rate metric, designed to judge the quality of solution paths, indirectly assesses the tool utilization capabilities of LLMs. It compares different models based on the effectiveness of their solutions, which inherently involves the utilization of tools. While it’s true that Win Rate also reflects general LLM capabilities, in the context of tasks that require tool use, it becomes a measure of how well these tools are utilized in achieving the desired outcome. Win rate not only considers the final answer but also the solution path. The solution path also reflects LLM’s ability to use tools to solve problems.

+ **Pioneering Effort in a New Metric Space**: It is important to recognize that the development of metrics for evaluating LLMs in tool utilization is a relatively new and evolving area. Our ToolEval represents one of the first forays into this space. While not exhaustive, these metrics provide a foundational framework for assessing LLMs in this context. We are open to evolving these metrics and incorporating additional measures as the field progresses.

+ **Continuous Improvement and Future Work**: We are committed to refining our evaluation methods. Future iterations of our work will focus on developing more nuanced metrics that can better capture the specific nuances of tool utilization, possibly incorporating aspects like tool relevance, efficiency of use, and the complexity of tasks executed.

In conclusion, while the ToolEval metrics have certain limitations, they represent a significant step forward in assessing LLMs' tool-use capabilities. As pioneers in this space, we acknowledge the need for continuous improvement and welcome further research to develop more comprehensive metrics. Our current metrics provide a valuable starting point for this evolving area of study.

Reviewer comment (Experiments):
I appreciate that the paper releases a high-quality dataset in this area. However, it will be better to provide a detailed statistical of this dataset (e.g., distribution in domain, error analysis).

Author response:
Thank you for acknowledging the quality of the dataset we have released in our paper. We also appreciate your suggestion to provide a more detailed statistical analysis of the dataset, including domain distribution and error analysis. We understand the importance of offering comprehensive insights into our dataset, as detailed statistics can significantly enhance its utility and applicability for future research. Therefore, we are committed to including a thorough statistical analysis in our revised paper.

Reviewer comment (Experiments):
In the design of solution path annotation, authors prefer to use DFS instead of BFS. However, DFS could also possibly backtrack and search multiple answers if it does not obtain a valid path. So, any experiments to prove the cost of DFS to find a valid path when compared with BFS?

Author response:
Thank you for your question concerning the use of Depth First Search (DFS) over Breadth First Search (BFS) in our solution path annotation approach. We appreciate the opportunity to address this query with reference to empirical evidence.

In our research, the preference for DFS over BFS was primarily driven by the goal of maximizing efficiency in finding valid solution paths. While it is true that DFS may require backtracking when an invalid path is encountered, our implementation of DFS in the DFSDT method is specifically designed to minimize the cost and time associated with such backtracking. DFS is more easily to reach a plausible solution path than BFS, and is more efficient than the latter.

To validate the efficiency of DFS in comparison with BFS, we refer to the results from a follow-up study to our work [1]. This independent study conducted a comprehensive comparison between two versions of Tree of Thoughts (ToT) - one using BFS and the other using DFS - and our DFSDT method, which is an improved version of DFS. Below is the results on three different benchmarks (Game of 24, WebShop, and our ToolBench):

| model                   | Game of 24 | WebShop | IToolBench |
|-------------------------|----------|----------|---------|
| CoT@3 (self-consistency)  | 7.00       | 56.45       | 31.20      |
| ToT (BFS)  | 11.00       | 50.20       | 38.00      |
| ToT (DFS) | 14.00       | 55.60       | 45.58      |
| DFSDT (Ours)               | 29.00       | 57.25       | 50.20      |

The results from this comparison, as shown in the table provided, clearly demonstrate that **DFSDT outperforms both BFS and DFS implementations of ToT across multiple benchmarks**, including Game of 24, WebShop, and our own ToolBench. Specifically, DFSDT not only achieves higher accuracy but also does so with fewer API calls, indicating a more efficient path to finding valid solutions.

Besides the performance, the authors of work [1] also show that our **DFSDT is far more efficient than ToT (BFS)** within limited API calls (Figure 2 in work [1]), demonstrating DFSDT’s excellent efficiency in finding a good solution path. For more detailed discussion on the difference between these works, please kindly refer to paper [1].

[1] Anonymous: Rational Decision-Making Agent with Internalized Utility Judgment   https://openreview.net/forum?id=l1pNNQSzZv

Reviewer comment (Experiments):
This paper mainly uses RapidAPIs for training. So is the fine-tuned ToolLLaMa suitable or adapted for APIs in other sources/formats (e.g., Pytorch/Tensorflow/HuggingFace in Gorilla, ChatGPT plugins)?

Author response:
Thanks for your suggestions!  As shown in Table 5, to empirically validate ToolLLaMa's generalization capabilities, **we conducted extensive out-of-distribution (OOD) experiments using the APIBench dataset**, which includes diverse APIs from Pytorch, Tensorflow, and HuggingFace. These platforms, as mentioned, are different from the ones in our training set and present a robust testbed for evaluating our model's adaptability. The results of these experiments, as detailed in Table 5, demonstrate remarkable OOD generalization performance by ToolLLaMa. Specifically, ToolLLaMa, equipped with our API retriever, consistently outperforms the Gorilla model (fine-tuned on APIBench) in AST accuracy across different datasets like HuggingFace and TorchHub. This is a strong indicator of ToolLLaMa's ability to adapt to and perform well with APIs not present in its training data.

Furthermore, when comparing ToolLLaMa with the same oracle retriever as Gorilla, our model consistently shows superior performance, particularly in zero-shot (ZS) scenarios. This further underscores ToolLLaMa's robustness and flexibility in handling diverse APIs, even those it was not explicitly trained on.

In conclusion, the findings from our OOD experiments firmly establish **ToolLLaMa's generalization capabilities across various API sources and formats**. The model's strong performance in these tests reflects its adaptability and suitability for a wide range of APIs, extending well beyond the scope of its initial training set on RapidAPIs. This versatility is a key strength of ToolLLaMa, addressing your concerns about its applicability to different API platforms.

Reviewer comment (Novelty):
Most of the technical ideas in this work are from past works, with the exception of DFSDT, which is a simple application of DFS to prompting. In this sense, the current work's technical contribution is low.

Author response:
Thank you very much for your insightful comments. Below we wish to highlight several key aspects that distinguish our work:

+ **Limitation of Previous Works**: As mentioned in the introduction, existing works fail to fully stimulate the tool-use capabilities within LLMs and have inherent limitations: (1) **limited APIs**: they either fail to involve real-world APIs (e.g., RESTAPI) or consider only a small scope of APIs with poor diversity; (2) **constrained scenario**: existing works are confined to instructions that only involve one single tool. Besides, they often assume that users manually specify the ideal API set for a given instruction in advance, which is infeasible in practice; In addition, some works do not even execute APIs to obtain real responses, which serve as important information for subsequent model planning.

+ **Our Unique Setting and Rich Resources**: Different from exsiting works, our ToolBench, stands out due to its comprehensive and diverse collection of **over 16,000** REST APIs from RapidAPI. For each API, we crawl detailed API documents from RapidAPI, including the functionality descriptions, required parameters, code snippets for API calls, etc. **This level of diversity in dataset construction is unprecedented in previous studies**. We believe the resource we provide represents a significant advancement in the field of tool learning research. Its diversity and comprehensiveness offer a robust platform for future studies focused on enhancing tool-use capabilities in LLMs.

+ **Innovative Instruction Crafting Methodology**: Our methodology for crafting instructions is notably different from prior works. Instead of brainstorming instructions from scratch and then searching for relevant APIs, we adopt a reverse engineering approach. We sample different combinations of APIs and craft various instructions that involve them. This strategy ensures **comprehensive coverage for all collected APIs** and leads to the generation of instructions that are more aligned with real-world application scenarios. 

+ **Leveraging RapidAPI Hierarchy for Multi-tool Instructions**: To address the challenge of creating realistic multi-tool instructions, we use the RapidAPI hierarchy to identify tool relationships. This aids in generating multi-tool instructions that are not only diverse but also functionally coherent. Our approach to generating intra-category and intra-collection multi-tool instructions is a novel strategy that effectively tackles the sparsity issue often encountered in random sampling of tool combinations.

+ **Function Call Feature Utilization**: We innovatively leverage the function call feature of ChatGPT by treating each API as a special function and feeding its documentation into ChatGPT's function field. To the best of our knowledge, we are **the first** to introduce function calling feature into the filed of tool learning. This approach fully exploits ChatGPT’s capability of tool use.

+ **Depth First Search-based Decision Tree (DFSDT)**: Existing works adopted either CoT or ReACT for model reasoning, which cannot fully elicit the capabilities stored in LLMs and thus fail to handle complex instructions in tool learning. To overcome the limitations of existing methods like CoT and ReACT, we propose a novel Depth First Search-based Decision Tree. This method allows ChatGPT to assess different reasoning paths, choose promising directions, or abandon ineffective ones. Our DFSDT approach significantly enhances the exploration of the action space, increasing the likelihood of finding valid solution paths. In experiments, DFSDT significantly improves the annotation efficiency and successfully completes those complex instructions that cannot be fulfilled using ReACT.

+ **Benchmarking Against Established Models**: To assess the tool-use capabilities of LLMs, we develop an automatic evaluator, ToolEval. We demonstrate that ToolEval achieves a high correlation with human evaluation and provides a robust, scalable, and reliable assessment for machine tool use. Our comparative analysis using ToolEval offers a new perspective on how LLaMA, fine-tuned on ToolBench, measures up against well-established models (Claude, Davinci, ChatGPT, GPT-4). This benchmarking is essential for understanding the relative progress in the field and provides a practical reference point for future research.

+ **A Capable Open-source Model Released**: We provide a capable model ToolLLaMA, which demonstrates a compelling capability to handle both single-tool and complex multi-tool instructions. ToolLLaMA outperforms Text-Davinci-003 and Claude-2, achieves comparable performance to the ChatGPT, and is only slightly inferior to GPT4. Also, ToolLLaMA exhibits robust generalization to previously unseen APIs.

Reviewer comment (Novelty):
The author proposed "Depth First Search-based Decision Tree" (DFSDT) that seems to outperform ReAct prompting. However, the technique is closely related to self-consistency + CoT and Tree-of-thoughts. It is also unclear how DFSDT will perform for other types of LLMs not in the category of LLaMA / GPT / Claude.

Author response:
Thank you for your insightful comments regarding our DFSDT method. Below are our responses to your concerns:

+ **Difference with Tree-of-thoughts (ToT)**: As mentioned in section 4, ToT is our concurrent work and there is a huge difference between ToT and DFSDT. Our DFSDT targets general decision-making problems where the decision space is infinite, compared to ToT's relatively simple tasks that can be addressed by brute-force search, such as Game of 24 and Crosswords. The distinct target between DFSDT and ToT determines the significant difference in the implementation details.

+ **Difference with self-consistency**: Self-consistency can be seen as performing multiple times of ReACT and performing a majority vote. This method is similar to one of our baseline in Table 3, i.e., ReACT@N. As mentioned in Section 3.1, ReACT@N conducts multiple times of ReACT until the total costs reach the same level of DFSDT. Once a valid solution is found by ReACT@N, we deem it a pass. Therefore, it resembles a form of self-consistency under our experimental setup. From Table 4, we can see that DFSDT outperforms this baseline by a large margin, which shows that our DFSDT performs much better than the self-consistency baseline. The reason for DFSDT's superiority lies in its ability to judge multiple reasoning paths during exploration, instead of after the exploration (i.e., the case of self-consistency). Hence DFSDT can detect and rectify errors earlier in the decision-making process, avoiding unnecessary exploration of mistaken path.

+ **Performance Comparison**: For performance comparison between DFSDT, ToT, and self-consistency, we borrow the results from a follow-up paper of our work [1]. They compared two versions of ToT (BFS and DFS) with our DFSDT. They implemented ToT the same as the original ToT paper. Below is the results on three different benchmarks (Game of 24, WebShop, and our ToolBench):

| model                   | Game of 24 | WebShop | IToolBench |
|-------------------------|----------|----------|---------|
| CoT@3 (self-consistency)  | 7.00       | 56.45       | 31.20      |
| ToT (BFS)  | 11.00       | 50.20       | 38.00      |
| ToT (DFS) | 14.00       | 55.60       | 45.58      |
| DFSDT (Ours)               | 29.00       | 57.25       | 50.20      |

From the above results, we can see that our DFSDT outperforms the baseline of self-consistency and ToT in three benchmarks, demonstrating DFSDT’s superiority. Besides the performance, the authors of work [1] also show that our DFSDT is far more efficient than ToT and CoT within limited API calls (Figure 2 in work [1]), demonstrating DFSDT’s excellent efficiency in finding a good solution path. For more detailed discussion on the difference between these works, please kindly refer to paper [1].

Besides elaborating on the difference between DFSDT and previous methods, we would also like to highlight the unique novelty of our DFSDT:

+ **Innovative Approach to Overcome Limitations of Existing Methods**: Our DFSDT was developed in response to identified limitations in CoT and ReACT methods, notably the issues of error propagation and limited exploration. Unlike these methods, which explore a single direction and can be trapped in faulty loops, DFSDT expands the search space, assessing multiple reasoning paths. This allows for a more comprehensive exploration of potential solutions, reducing the risk of error propagation and increasing the likelihood of finding valid solution paths.

+ **Explicit Encouragement for Diverse Node Generation**: During the node expansion process in our DFSDT, we prompt ChatGPT with information from previously generated nodes and explicitly encourage the generation of distinct nodes. This strategy ensures a diverse exploration of potential solutions, further expanding the search space.

+ **Flexibility and Versatility**: The design of our DFSDT allows it to degrade to ReACT for simpler instructions, making it as efficient as ReACT in these cases. For more complex instructions, it retains the full capabilities of a classical DFS search. This versatility ensures that our approach is applicable across a wide range of instruction complexities, enhancing the model's utility.

In conclusion, our DFSDT presents significant advancements and distinct features. DFSDT is specifically designed to address complex, general decision-making problems with an infinite decision space, setting it apart from the more limited scopes of ToT and self-consistency approaches.

[1] Anonymous: Rational Decision-Making Agent with Internalized Utility Judgment   https://openreview.net/forum?id=l1pNNQSzZv

Reviewer comment (Evaluation):
The evaluation metrics are based on ChatGPT, which is highly unreliable and may favor models tuned with ChatGPT-prompted datasets. The author mentioned in Appendix A.5 (Paragraph "Comparing Human Evaluation and ToolEval") that `Our ChatGPT evaluator demonstrates a high agreement of 87.1% in pass rate and 80.3% in win rate with human annotators.` This shows that there is **close to 20% disagreement** between human evaluation and ChatGPT evaluation. This is a large discrepancy and should not be ignored.

Author response:
Thank you for your observations regarding the use of ChatGPT-based evaluation metrics in our study. We understand your concerns about the reliability of these metrics and the noted discrepancy between our ChatGPT evaluator and human annotators. Below are our responses to this concern:

+ **ToolEval Achieves Comparable Performance to Humans**: According to our statistics, ChatGPT achieves a high agreement rate of 80.3% in win rate with human annotators, similar to the level among human experts (83.54% on average). This high level of agreement rate indicates that ChatGPT’s annotations **are in line with human judgment and are a reliable indicator of performance**. It is important to note that achieving complete alignment between human and automated evaluations in tool learning evaluation is challenging due to the subjective nature of certain judgments and the inherent variability in human evaluation. In addition, the fact that under our evaluation, GPT4 > GPT3.5 > Davinci (both pass rate and win rate) also reflects the reliability of our ToolEval.

+ **ChatGPT-based Evaluation as the Common Practice**: Employing ChatGPT for automatic evaluation is a common practice in the field of instruction-tuned LLMs [1,2,3,4]. This approach is not unique to our research but is **widely adopted** due to its effectiveness in providing quick and reasonably accurate assessments of model performance. The use of ChatGPT for evaluation in our research is driven by a need for efficiency and scalability in the evaluation process.

+ **We also Apply Evaluation other than ChatGPT-based Evaluation**: It should be noted that we do not rely solely on ChatGPT-based evaluation in this paper. For the out-of-distribution generalization experiments on APIBench, we use the **ground truth** to calculate the hallucination and ast accuracy. The promising results reflect the strong capabilities of our ToolLLaMA.

In addition, when performing ChatGPT-based evaluation, we have performed a **rigorous analysis** of the evaluation ability of ChatGPT in our setting. Specifically, we have implemented specific strategies to mitigate some of these biases: 

+ **Multi-Sample Strategy**: As mentioned in appendix A.5, to reduce individual biases, we employ a multi-sample strategy where multiple instances of evaluations (at least four) are aggregated. This approach ensures that the final output is not overly influenced by the peculiarities of a single model's response, providing a more balanced and representative result.

+ **Order-Switching Mechanism**: To counteract order bias, we systematically vary the order of options presented to the evaluator in different runs. This order-switching mechanism ensures that the sequence in which options are presented does not unduly influence the evaluation outcomes.

In conclusion, while acknowledging the limitations of ChatGPT-based evaluation metrics, we believe that our approach provides a reliable, efficient, and scalable method for assessing LLM performance. The substantial agreement with human annotators underscores the validity of our findings, and we remain open to integrating more advanced evaluation techniques as they become available.

[1] Liu, Yang, et al. "Gpteval: Nlg evaluation using gpt-4 with better human alignment." arXiv preprint arXiv:2303.16634 (2023).

[2] Dubois, Yann, et al. "Alpacafarm: A simulation framework for methods that learn from human feedback." arXiv preprint arXiv:2305.14387 (2023).

[3] Chiang, Wei-Lin, et al. "Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality." See https://vicuna. lmsys. org (accessed 14 April 2023) (2023).

[4] Chiang, Cheng-Han, and Hung-yi Lee. "Can Large Language Models Be an Alternative to Human Evaluations?." arXiv preprint arXiv:2305.01937 (2023).

Reviewer comment (Evaluation):
How does the models in Table 4 compare with each other if we adopt human evaluation?

Author response:
Recognizing the importance of human judgment in validating model evaluations, we commit to including corresponding human evaluations in our revised paper. This will provide an additional layer of verification to our results, ensuring that they are not only efficient but also accurately reflect the true capabilities of the evaluated models. Due to the time limit of the rebuttal period, we have provided part of the results in the following:

## Win Rate: ChatGPT-DFSDT vs. ChatGPT-ReACT
| Evaluation Method           | I1-instruction | I1-Tool | I1-Category | I2-Instruction | I2-Category | I3-Instruction | Average    |
|-----------------------------|----------------|---------|-------------|----------------|-------------|----------------|------------|
| Human Evaluation            | 55             | 58      | 61          | 75             | 61.5        | 69             | 63.3 |
| ChatGPT Evaluation          | 60.5           | 62      | 57.3        | 72             | 64.8        | 69             | 64.3       |

## Pass Rate (Evaluate by ChatGPT)
| Model           | I1-instruction | I1-Tool | I1-Category | I2-Instruction | I2-Category | I3-Instruction | Average |
|-----------------|----------------|---------|-------------|----------------|-------------|----------------|---------|
| ChatGPT-DFSDT   | 54.5           | 65      | 60.5        | 75             | 71.5        | 62             | 64.8    |
| ToolLLaMA-DFSDT | 57             | 61      | 62          | 77             | 77          | 66             | 66.7    |
| GPT4-DFSDT      | 60             | 71.5    | 67          | 79.5           | 77.5        | 71             | 71.1    |

## Pass Rate (Evaluate by Human)
| Model           | I1-instruction | I1-Tool | I1-Category | I2-Instruction | I2-Category | I3-Instruction | Average      |
|-----------------|----------------|---------|-------------|----------------|-------------|----------------|--------------|
| ChatGPT-DFSDT   | 50             | 60      | 63          | 70             | 68          | 52             | 60.5         |
| ToolLLaMA-DFSDT | 60             | 64      | 60          | 76             | 70          | 56             | 64.33333333  |
| GPT4-DFSDT      | 64             | 66      | 64          | 76             | 76          | 62             | 68           |

**Consistency Between Human and ChatGPT Evaluations**: Our results indicate a reasonable level of consistency between human evaluations and ChatGPT evaluations, particularly in terms of Win Rate. This consistency lends credibility to our automated evaluation methods while also providing the nuanced understanding that only human judgment can offer.

We promise to include these results in the camera-ready version. The inclusion of human evaluation in our revised paper emphasizes our commitment to a comprehensive assessment of model capabilities. By combining automated and human evaluations, we can provide a more balanced and thorough understanding of each model's strengths and weaknesses. Thanks again for your suggestive comments!

Reviewer comment (Evaluation):
The current annotation/evaluation pipeline is not rigorous and may lead to false impressions about model’s performance. ToolBench is entirely annotated by ChatGPT. While the annotations of the instruction-tuning data don’t have to be perfect, as a benchmark for evaluation it should be held to higher standards. ToolEval solely relies on ChatGPT to evaluate whether a solution path “passes” and whether it “wins” the solution path from ChatGPT. Although the authors compare ChatGPT’s evaluation with human subjects’, the current evaluation pipeline is problematic for several reasons: (a) Many tool-related tasks have definite correct answers or solution paths, so real correct answers should be used to judge correctness. Relying on ChatGPT’s annotation of win rates and pass rates can lead to overconfidence about wrong answers that look satisfactory. (b) ChatGPT and other language models as evaluators have known biases (order bias, egocentric bias, length bias, selection bias, preference for style over substance). There should be more discussion on if and how ToolEval mitigates these biases. (c) The evaluation rules for pass rates and win rates are too complex even for human annotators to reliably follow, undermining confidence in human evaluation results. Appendix A.5 shows these rules form a complex 3-layer decision tree requiring non-trivial examination of instructions, APIs, and solution paths. Questions include whether annotators recorded only final decisions or also intermediate reasoning, and whether there is evidence they strictly followed rules instead of cognitive biases. Unreliable human annotations would also undermine reported correlation with ChatGPT evaluations. (d) Win rates are computed by comparing model generations with annotations from ChatGPT+ReAct, raising the question of whether inference algorithms more similar to the annotation pipeline might achieve better results without actually being better.

Author response:
Q1: Many tool-related tasks can have a definite set of correct answers or correct solution paths. For these tasks, the real correct answers should be used to judge the correctness of generated answers.

A1: Most of our instructions (tool-related tasks) **do not** have a definite set of correct answers or solution paths. We conducted human evaluation on 1000 instructions, and found that **86.7%** of them do not have a definite ground truth answer or solution path. The reasons are listed in the following:

+ **Temporal Variability of API Responses**: Many of our tool-related tasks are designed to interact with real-world data, which is inherently dynamic. For instance, an instruction about today's weather will yield different answers at different times, reflecting the ever-changing nature of weather conditions. **This temporal variability is a fundamental characteristic of the real-world APIs we are evaluating, making it impractical to define a single 'correct' answer or solution path for such tasks**.

+ **Infinite Solution Paths Due to API Diversity**: The vast array of available APIs (over 16,000) and their potential combinations contribute to **a multitude of valid solution paths for a given query**. Different APIs can interact in varied and sometimes unexpected ways, leading to multiple valid approaches to a given problem. **This complexity is reflective of real-world scenarios** where different tools and data sources are combined to solve complex tasks. As such, **our evaluation process is closely aligned with practical applications**, where there is often more than one way to achieve a desired outcome. Besides, in real scenarios, numerous APIs available offer similar functionalities or data (e.g., Bing translation and Google translation). This substitutability significantly increases the number of potential combinations and solution paths for a given task.

+ **Infinite Final Answers in Information Synthesis**: Even if a standard 'solution path' of API calls were to be established, the synthesis of the information could still vary. For example, in response to an instruction like "find the latest trends in technology," different APIs might offer varying perspectives—ranging from tech news to emerging products, to usage statistics. The LLM's role is to synthesize this diverse information into a coherent response that aligns with the user's query. This synthesis involves judgment and contextual understanding, which naturally leads to **multiple correct interpretations or presentations of the data**.

In conclusion, while the idea of using a set of definite correct answers is appealing for simplicity, **it does not align with the reality of the tasks and APIs we are examining**. Our evaluation framework is designed to reflect real-world conditions, where answers are often not static but are influenced by temporal changes, the diversity of data sources, and the need for information synthesis. We believe that our approach offers a more accurate and practical assessment of LLM performance in real-world scenarios.

Reviewer comment (Reproducibility):
The evaluation results are constantly changing over time, making it very expensive and difficult to compare new methods and older baselines, and harming the reproducibility of the evaluation pipeline. While the authors acknowledge temporal variability on RapidAPI and the infeasibility of annotating fixed ground-truth solution paths, the problem remains. First, reported pass rates and win rates can only be compared with evaluations from the same time period, making it hard to judge method quality. Second, temporal variability makes evaluation more difficult, as the same version of APIs must be used for fair comparison. This forces every new method evaluation to rerun all baselines within a short time window, which might be as short as a few hours for highly variable APIs (e.g., restaurant availability, real-time weather). This creates a heavy burden for developers and researchers, who would need to run evaluations multiple times and in parallel. At the very least, qualitative analysis of how many task instances involve temporally variable tools and their impact on results should be provided. A potential solution is creating and releasing a “snapshot” of API call results at a certain time with the evaluation suite, to enable reproducible results.

Author response:
Q6: The evaluation results are constantly changing over time, making it very expensive and difficult to compare new methods and older baselines. This also ruins the reproducibility of the evaluation pipeline. First, people are not able to know roughly how good a method is by looking at the reported pass rates and win rates, because the reported results can only be compared with the results evaluated during the same time period.

A6: We appreciate your concern regarding the temporal variability of evaluation results and its potential impact on reproducibility and comparability. We understand that the dynamic nature of APIs can lead to fluctuations in pass rates and win rates over time. However, we assert that **this variability does not undermine the reproducibility or relevance of our evaluation pipeline**. Our approach is designed to provide a realistic assessment of model performance in real-world scenarios, where such variability is inherent:

+ **Real-World Relevance**: Our evaluation framework is tailored to reflect real-world conditions, where APIs and data sources are inherently dynamic. The varying nature of APIs is a critical aspect of real-world applications, and our evaluation setup aims to capture this dynamism. Thus, the changing evaluation results offer a more accurate reflection of how models would perform in practical settings.

+ **Framework for Comparative Analysis**: Followers of this paper **do not need to refer to the specific numbers reported in this paper**. Instead, we encourage researchers and developers to use our released codes and evaluator to conduct their evaluations in the same temporal context as their models. This ensures that comparisons are made under similar conditions, providing a fair and relevant benchmarking of model performances. **The focus should be on the relative performance of different methods rather than the absolute numbers**, which are expected to vary over time.

+ **Guidance for Future Research**: In our revised paper, we will provide detailed guidelines on how to use our evaluation framework effectively. This will include instructions on how to compare new methods with existing baselines, such as ToolLLaMA, by conducting simultaneous evaluations to obtain comparable pass rates and win rates. This approach allows for a meaningful comparison of different methods, even as specific numbers change over time.

+ **Adding Contextual Information**: To further aid in reproducibility and comparability, we plan to include contextual information about the API versions and the temporal conditions under which the evaluations were conducted. This will provide future researchers with a reference point for understanding the conditions of our initial evaluations and for making informed comparisons with their own results.

Reviewer comment (Evaluation):
The comparison between DFSDT and ReAct is unclear. It is expected that DFSDT should always perform better than ReAct since ReAct appears to be a special case of DFSDT with a branching factor of 1. Therefore, in most cases DFSDT should outperform ReAct, and the win rate for ChatGPT+DFSDT against ChatGPT+ReAct should approach 100%. However, Table 4 shows that this is not the case in the first two rows. An explanation is needed for why DFSDT does not consistently outperform ReAct.

Author response:
Q9:  Why isn’t DFSDT always better than ReAct? According to your description, it seems that ReAct is a special case of DFSDT where the branching factor is 1. Therefore, I expected DFSDT to be better than ReAct in most cases. In other words, the win rate for ChatGPT+DFSDT against ChatGPT+ReAct should be close to 100%. According to the first two rows in Table 4, that is clearly not the case. Is there some explanation on why that happened?

A9: We appreciate your question regarding the performance comparison between DFSDT and ReAct, and why DFSDT does not consistently outperform ReAct as might be expected. There are several factors that contribute to this observation:

+ **Inherent Randomness in LLMs**: One of the primary factors is the inherent randomness in large language models (LLMs), such as the temperature setting in generation. This randomness means that the first branch generated by DFSDT can vary significantly across different instances. As we have indicated in our paper, multiple iterations of ReAct can sometimes yield better results compared to a single iteration of DFSDT. This variation is due to the stochastic nature of LLM outputs, which can affect the initial branching and subsequent solution path development in DFSDT.

+ **LLM as Imperfect Judges**: The second factor is the reliance of DFSDT on the LLM to judge and select the best solution path. While DFSDT theoretically expands the search space for potential solutions, it ultimately depends on the LLM's judgment to select the most promising path. LLMs, despite their advanced capabilities, are not infallible judges. There are cases where the LLM might fail to identify the optimal solution path, leading to a scenario where DFSDT does not outperform ReAct.

For more detailed analysis, discussion, and experiments on the performance of different searching algorithms (e.g., DFSDT, ToT, ReACT) and their difference, we refer you to one follow-up work of our paper: [1].

[1] Ye, Yining, et al. "Large language model as autonomous decision maker." arXiv preprint arXiv:2308.12519 (2023).