Paper: MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use (ICLR-2024)

Reviewer comment (Novelty):
I think the comparison with existing benchmarks still needs polishing. To the best of my knowledge, I have found these places to be potentially incorrect: (1) API-Bank (https://arxiv.org/pdf/2304.08244v1.pdf) has included the task ① of determining whether LLMs need to leverage external tools or not (level-1 description in Section 1 Page 2). Thus, it might be incorrect to describe API-Bank as only focusing on tasks ③④; (2) ToolBench (https://arxiv.org/abs/2305.16504) also involves the task ②, requiring the retriever to retrieve the most relevant tools. Thus, it is also incorrect to describe the ToolBench as a benchmark that only focuses on tasks ③④; (3) There are also some benchmarks that are missing in the table, like ToolQA (https://arxiv.org/pdf/2306.13304.pdf). In summary, the authors better clarify the differences compared with these existing benchmarks more clearly.

Author response:
Thank you very much for your suggestion! We apologize for the confusion caused, and next, we will gradually explain your concerns, to give you a clearer understanding of your contributions to the work:

Q: I think the comparison with existing benchmarks still needs polishing.

A: First, we acknowledge that there are some deficiencies in the comparison with benchmarks, which may be due to differences in understanding from various perspectives of different persons. Based on the issues identified after your careful review, we have revisited the related literature and engaged in in-depth discussions. We have made modifications to the comparison as per your request.

Reviewer comment (Evaluation):
Most statements in the experiments are not well explained. I think the authors should focus more on the potential reasons and analysis behind these observations.

Author response:
Q: Most statements in the experiments are not well explained.

A: To address your concern, we have carefully reviewed the analysis of our experimental results and found that we indeed did not effectively analyze the experimental outcomes. We have implemented the following improvements:

1. Additional Experiments: We added **few-shot prompt experiments** for three subtasks of the tool usage awareness task and the tool selection task; we introduced **a new prompt template in multi-tool selection** to specify the number of tools returned by LLMs.
2. More In-Depth Analysis of Experimental Results: For both the original and the additional experiments mentioned above, we conducted a comprehensive and profound analysis of our experimental results. We have reached the following conclusions: 

(1) **Even under few-shot prompts, the majority of LLMs still perform poorly in tool usage awareness.** In Table 2, we observe that under the zero-shot prompt, only ChatGPT has both accuracy and F1 score exceeding 70\%, while the performance of other models is relatively poor, with the F1 score of llama2-13b being only 11.53%. Under the five-shot prompt, some models show significant improvement in F1 scores, for example, llama2-13b increased by 42.79%, and vicuna-7b by 42.28\%. This indicates that though few-shot learning generally improves the performance of LLMs in tool usage awareness, they still lack sufficient tool usage awareness.


(2) **When selecting similar tools, there is a significant performance disparity among existing LLMs, and the improvement brought by few-shot prompts is limited.** Table 3 shows that under the zero-shot prompts, the best-performing LLM is Vicuna-7b, with nearly a 30\% difference compared to the worst-performing Llama2-13b. The gap between the best-performing ChatGPT and the worst-performing Llama2-13b still exceeds 20% under 5-shot prompts. Additionally, the maximum improvement brought by 5-shot prompts does not exceed 7%. Moreover, the performance of Vicuna-7b even declined by 10\% under the five-shot condition, suggesting a potential bias in its 0-shot performance, which reflects either a lack of robustness or over-sensitivity of the model.

(3) **LLMs still face serious challenges in dealing with reliability issues, such as reducing hallucinations.** As seen from Table 3, although few-shot prompts improve the performance of all LLMs, the CSR of most LLMs remains below 20\%. We find that LLMs sometimes fabricate non-existent tools, a severe hallucination issue that has a significant impact on LLM-based agents. Additionally, the potential sycophancy of LLMs may lead them to avoid returning a "none" answer, instead choosing irrelevant tools to respond to users.

(4) **LLMs perform poorly in processing long texts.** From Figure 4, we can see that the CSR of almost all LLMs decreases as the length of the tool list increases, especially in the range from the top 5 to the top 10. This indicates that LLMs still need improvement in understanding long texts. **LLMs exhibit imbalances and biases in tool selection across different scenarios.** For example, in Figure 5, LLMs generally have a higher CSR in tool selections related to the elderly and artists & designers, while their CSR is lowest for tools related to students. This means that developers still need to enhance the generalization capabilities of LLMs. At the same time, for downstream applications, it is best to choose suitable LLMs based on different applied fields.

(5) **There are significant performance differences among LLMs in multi-tool selection.** As shown in Table 4, ChatGPT, the top-performing model, outperforms ChatGLM2, the worst-performing model, by nearly 70%, highlighting the variability in the capabilities of different language models for this task. Furthermore, the most common error made by the models is omitting tool selection, such as in the case of Vicuna-33b, which only selected one tool in 48.49% of cases. Moreover, **several LLMs overly rely on the explicitly specified number of tools they should select in the prompts.** When explicitly instructed to return two tools, Vicuna-33b's correct selection rate increased to over 90%, and Vicuna-7b also improved by over 20%. This indicates that these LLMs still possess good multi-tool selection capabilities but require prior knowledge, which makes it challenging to apply in LLM-based agents.

**The detailed analysis for the above conclusions is updated in the PDF document, marked in blue font.**

Reviewer comment (Presentation):
Section 2 is rich in substance but the presentation is disorderly. There's probably at least 2 different sections that should be there, like "Task Formulation & Desiderata" first then "Dataset Generation" second, but these two considerations are mixed up repeatedly in the current presentation. I'm happy to change my mind on this if the authors would like to defend the current organization.

Author response:
Thanks for your feedback. We acknowledge that we did not organize the structure of this section well. However, swapping the positions of dataset generation and task formulation might not be suitable. This is mainly because many definitions in task formulation rely on the premise that the reader is already clear about the structure of the dataset. Therefore, placing dataset generation after task formulation might cause more confusion for the reader.

However, we appreciate your expression of confusion, so we have added a brief introduction at the beginning of Section 2 (already updated in the PDF in blue font) to help readers better understand the structure of the section.

Reviewer comment (Evaluation):
It's unclear (and seemingly undiscussed) how fundamentally the generation of the test queries with LLMs biases the evaluation for/against particular models, or reduce the true diversity of these queries.

Author response:
Thank you for pointing out this issue! We acknowledge this is a concern and still exists in the current LLMs-generated dataset.

Specifically, we attempted to use various models to generate a mixed dataset, such as Llama2-70b. However, in the actual generation process, we found that the diversity of samples generated by Llama2-70b was lacking, and the quality of generated samples was low (e.g., not relevant to the tool's description information). However, MT-Bench [1] indicated that ChatGPT has a low bias on self-enhancement. Therefore, we believe ChatGPT does not strongly favor its own generated content, making it the least biased choice for generating data while ensuring data quality.

Reviewer comment (Experiments):
What kind of dev vs. test guidance do you recommend for this benchmark? Is it meant to be a truly "blind zero-shot, no tuning" dataset? What did the authors do for tuning? Did they use the prompt unchanged for each LLM? (Having no dev set is completely reasonable in principle, but practical considerations tell us that much of the community will, in effect, reuse the test set so often it becomes a dev set. I recommend having at least a few examples explicitly for tuning.)

Author response:
Your suggestion is forward-thinking. Inspired by your advice, we have started constructing dev. Specifically, we are going to create a fine-tune dataset for tool usage awareness. We plan to use all positive samples (all LLMs must use the tool to answer questions) as input to GPT-4 and obtain answers (output should be a refusal to answer, e.g., “Sorry, I cannot answer this…”). We intend to use positive samples and their corresponding answers as the fine-tune dataset to improve LLMs' tool usage awareness. This ensures LLMs can accurately identify inputs beyond their ability and refuse to answer, allowing external tools to assist in answering rather than generating illusory or low-quality answers. Once the dataset is completed, we will release it together.

Reviewer comment (Reproducibility):
Which Llama2 models were used, vanilla or chat? I presume it's chat. It may be good to include more explicit information in the main text. You find that "the worst-performing model, Llama2-13b, has an F1 Score of only 11.53%". This is surprisingly low. Is this some kind of unintended interaction between the chat format or some other minor detail and the task?

Author response:
We used the chat version, and we have added details in the experiment setting. Thanks for your careful consideration.

We admit that we were surprised when we first saw this data, so we manually reviewed the results. Since we require LLMs to explain their answers, we consider this answer relatively reliable. We believe the main reason for this result is the lack of capability awareness, meaning the model fails to have a clear understanding of its capability boundaries. The drawback of Llama2-13b lies in its overconfidence; for questions it cannot solve, it still believes it can.

Here is an example of Llama2-13b’s answer:

Question: `Can you check if there are any trending discussions related to the Sakura festival occurring in Japan on Google Trends or Twitter?`

Answer: `No, I do not need to use external tools to address the user's query. The information the user is seeking can be found through a simple search on Google Trends or Twitter, and I have the ability to perform such searches within my language model capabilities. I can provide the user with relevant information related to the Sakura festival in Japan, such as recent discussions or trends, without the need for external tools.`

Reviewer comment (Writing):
Is the word "task" is overloaded in Sec 2.3, where it refers both to the two major tasks "2.3.1" and "2.3.2", and to the four sub-tasks of "2"? Maybe rename the four tool selection tests to "sub-tasks".

Author response:
Thank you for your suggestion; we have made the modification.

Reviewer comment (Novelty):
Related work is very important given the number of papers in this domain

Author response:
Q: No Related work

A: Compared to other datasets, we believe ToolE has two advantages: (1) Our dataset exhibits greater diversity, and this diversity is tailored to real user scenarios, such as expression style, mood, and level of detail. As mentioned in Q1, we employed various prompt methods to induce LLMs to generate more diverse user inputs, ensuring that ToolE covers a wide range of inputs representative of real users. (2) By employing a pipeline process to address the overlapped issue, we can ensure the rigor of the data. As outlined in section 2.2.1 of the paper, we adopted multiple steps to resolve the overlapped issue, ensuring that there is no overlap between tools, which is crucial for ensuring the quality of the dataset and task evaluation. We provide additional information on dataset comparison in A.5 and Table 8 to better illustrate the distinctions between the datasets.

Reviewer comment (Experiments):
The experimental setup for some of the tasks does seem unnatural

Author response:
Q: There are concerns about how this experiment is set up or the motivation behind tool selection with possible reliability issues.

A: I apologize for not making it clear in the paper. We designed this subtask because we wanted to test the reliability of LLMs in tool selection. That is, when there are no suitable tools to solve the user's problem in the tool list, it should return "none" instead of fabricating a non-existent tool (hallucination), thus spreading misinformation, which is a realistic issue. This is crucial for trustworthy LLMs and trustworthy LLM-based agents. 

To ensure the impact of other overlapped tools in the tool list, we not only addressed overlapped issues at the beginning (details can be found in 2.2.1) but also removed the top $t$ tools with the highest semantic similarity to the ground-label tool. This was done to ensure that the tool list is clean (this operation is explained in the last sentence of our description of subtask 3).