Paper: On the Tool Manipulation Capability of Open-sourced Large Language Models (ICLR-2024)

Reviewer comment (Novelty):
Missing comparison with closely related works. I would like to inquire about the absence of a comparative analysis with Toolbench, ToolQA and Gorilla (see [1,2,3] for references). An exploration of potential distinctions between the two approaches would provide valuable insights.

Author response:
1. **Comparison with other similar works** - Thanks for pointing them out. We will definitely include those concurrent work in the literature review in the following version of the paper. Although those works also study different ways of tools and API use, the tasks they are focusing on are very similar to certain subsets from our ToolBench, since the principal of the *Toolbench* design is to cover tasks with different levels and types of difficulty. Nevertheless, we are more than happy to extend our *Toolbench* to include their tasks if this can further benefit the community.

- The Toolbench from ToolLLaMA paper includes a large collection of APIs, but all of them are only REST APIs, which are pretty similar to the OpenWeather and the CAT API from our ToolBench collection. In order to generalize the tool using capability of LLMs, a good benchmark should include different types of APIs - REST API, Python API, human text command API, etc. This is also one of the main focus of our ToolBench design.

- Gorilla’s APIBench mainly focuses on loading models from 3 major model hubs. Their tasks can also be categorized into the ToolBench evaluation framework. Those tasks have very limited number of API functions in total, and the major difficult lies in populating the proper arguments (e.g. model names). This is also very similar to our WebShop task, where each function call is simply to click a certain button on the webpage, but the difficulty lies in for example selecting the correct product among tens of others to click. 

- ToolQA is designed for a different purpose. It focuses only on question answering scenario, and has an additional layer checking whether or not tool uses needs to be invoked and what tool to use. While the other works, including ours, only focusing on evaluating the model capability and accuracy in using certain tools.

Reviewer comment (Evaluation):
The retrieval quality in this study appears to be not very good. In practice, I have observed that the retrieved demonstrations often lack meaningful relevance. Such limitations can significantly impact overall system performance.

Author response:
2. **Demonstration examples concerns**

- In practice, for each given tool or a set of API functions, we found it is usually easy to collect a bunch of demonstration examples for most of the typical uses cases. However, the demonstration example may not 100% cover all the potential use cases that need new combinations of API functions to achieve certain goals. To validate this, we conduct experiments in section 4.2, where we made sure that the test cases all require completely different API function combinations than the ones in the 10 demonstration examples. As shown in Figure 4, the models are still able to reach good accuracy with only several examples retrieved, indicating that the models can indeed generalize to unseen API combinations from limited retrieved examples.   

- We agree that improving the retriever capability is definitely a promising research direction. But since we mainly revisited those three techniques in our work as proof-of-concepts to show that they are effective to improve model tool using capabilities with limited human supervision, we decided to leave retriever improving as future work.

Reviewer comment (Presentation):
The template of the paper seems to be incorrect for ICLR. Please fix it in the camera-ready version.

Author response:
4. **Paper formatting** - Thanks for the suggestions! We will double check to make sure the format is correct in the next version of the paper. We will also stick with anonymous github link in the future submissions.

Reviewer comment (Experiments):
Since the curated data are involved, are there any experiments to validate the connection between the quality of curated data and final results (e.g., model alignment)? We need to confirm how curated data affect model performance to what extent. The value of the proposed method could be reduced if it heavily relied on human-curated data.

Author response:
1. **Finetuning Transferability** - We agree that this is a topic we didn’t discuss in our paper. We include the finetuning technique in our work only as a proof-of-concept to show that it is also effective to improve model tool using capabilities with limited human supervision.  We wanted to keep involving as minimum human supervision as possible in our work. With the training data details in appendix section C.1, the dataset is just prepared by 1 author of us in just one day using the programmatic way described in section 4.1. Since making a perfect model is not the main goal of this paper, we leave the finetuning scaling law study as well as the transferrablity across tasks as future work, where we would really love to work with the community together to explore more robust algorithms and more efficient data collecting methods to fundamentally improve model performance on the ToolBench.

Reviewer comment (Novelty):
Recently, there also have some benchmarks like another ToolBench [1]. Can authors compare or describe the differences among these works?

Author response:
3. **Comparison with other similar works** - Thanks for pointing it out. We will definitely include all the concurrent work, including ToolLLM, in the literature review in the following version of the paper. Although those works also study different ways of tools and API use, the tasks they are focusing on are very similar to certain subsets from our *ToolBench*, since the principal of the *Toolbench* design is to cover tasks with different levels and types of difficulties. Nevertheless, we are more than happy to extend our *Toolbench* to include their tasks if this can further benefit the community.

- The Toolbench from ToolLLaMA paper includes a large collection of APIs, but all of them are only REST APIs, which are pretty similar to the OpenWeather and the CAT API from our ToolBench collection. In order to generalize the tool using capability of LLMs, a good benchmark should include different types of APIs - REST API, Python API, human text command API, etc. This is also one of the main focus of our ToolBench design.

Reviewer comment (Experiments):
Improving model performance by fine-tuning LMs is not surprising. It is necessary to validate the generalization of tuning LLMs. For example, tuning LLMs on a subset of the dataset (e.g., Open weather) and then evaluating it on the remaining parts (e.g., Virtual Home).

Author response:
1. **Finetuning Transferability** - We agree that this is a topic we didn’t discuss in our paper. We include the finetuning technique in our work only as a proof-of-concept to show that it is also effective to improve model tool using capabilities with limited human supervision.  We wanted to keep involving as minimum human supervision as possible in our work. With the training data details in appendix section C.1, the dataset is just prepared by 1 author of us in just one day using the programmatic way described in section 4.1. Since making a perfect model is not the main goal of this paper, we leave the finetuning scaling law study as well as the transferrablity across tasks as future work, where we would really love to work with the community together to explore more robust algorithms and more efficient data collecting methods to fundamentally improve model performance on the ToolBench.

Reviewer comment (Presentation):
Can authors check the LaTeX format of the submitted version? It seems that the font style is different from the standard version of the ICLR submission.

Author response:
4. **Latex format** - Thanks for capturing this. We will double check to make sure the format is correct in the next version of the paper.

Reviewer comment (Novelty):
The novelty is limited because prior works (such as [NeurIPS2023] GPT4Tool) have already shown that instruction-tuning can boost the tool-use capability of open-source LLMs.

Author response:
**Regarding the novelty and motivation**

Rather than completely solving the tool using issues for LLMs by proposing new algorithms or models, our main motivation of this paper is more like opening the door for the community to explore and improve models/algorithms on tool using. As you mentioned in the summary and strengths sections, we sincerely hope our work can be published at ICLR so that the researchers and practitioners can 1) see and use our *ToolBench* to guide and evaluate their further research, 2) take the techniques we revisited as teasers to encourage further innovation that can fundamentally improve model tool using capability, and 3) exploit those techniques to enhance model performance on their own tools and tasks in practice.

Reviewer comment (Experiments):
Coding models seem to perform better at API tasks. However, the author has not compared with CodeLLaMA (https://huggingface.co/codellama).

Author response:
2. **Code LLaMA results** - Thanks for checking the details in our Appendix. Code models are indeed more robust in tool using, CodeLLaMA is not an exception. CodeLLaMA 34b is even better than LLaMA2-70b on most of the tasks. We attached the results below, please compare them with the other results in Appendix Table 8, we will add them in the paper in the next version. 

| Model | Open weather | Cat API | Home Search | Trip Booking | Google Sheet | Virtual Home | WebShop (long) | WebShop (short) | TableTop |
| ----------------------------- | ----------- | ----------- | --------- | ---------- | ----------- | ---------------------- | -------- | ---------- | ----------- |
| CodeLlama-34b-hf | 96.39 | 85.0 | 88.0 | 88.33 | 64.29 | 34.0 / 24.65 | 0.0 | 5.53 | 51.32 |
| CodeLlama-34b-Instruct-hf | 94.28 | 86.0 | 90.0 | 89.17 | 61.11 | 31.0 / 24.34 | 0.0 | 25.99 | 47.46 |
| CodeLlama-34b-Python-hf | 91.11 | 88.42 | 91.0 | 85.83 | 55.87 | 28.0 / 21.24 | 0.0 | 6.47 | 33.33 |
| CodeLlama-13b-hf | 96.0 | 86.83 | 87.0 | 70.83 | 51.26 | 35.0 / 22.28 | 0.0 | 0.0 | 40.95 |
| CodeLlama-13b-Instruct-hf | 98.0 | 89.58 | 85.0 | 72.5 | 48.97 | 31.0 / 22.56 | 0.0 | 9.7 | 57.62 |
| CodeLlama-13b-Python-hf | 93.0 | 92.35 | 86.0 | 62.5 | 50.79 | 37.0 / 21.94 | 0.0 | 2.4 | 41.53 |
| CodeLlama-7b-hf | 86.0 | 92.0 | 74.0 | 63.33 | 38.08 | 35.0 / 21.97 | 0.0 | 0.0 | 11.16 |
| CodeLlama-7b-Instruct-hf | 90.0 | 94.0 | 78.0 | 61.67 | 41.27 | 32.0 / 21.95 | 0.0 | 0.0 | 16.98 |
| CodeLlama-7b-Python-hf | 83.0 | 88.0 | 83.0 | 68.33 | 49.13 | 31.0 / 21.33 | 0.0 | 1.58 | 22.86 |

Reviewer comment (Reproducibility):
The author gave an example about advanced reasoning in Table-4. In this example, the model needs to understand that in a Google Sheet, the cell that contains the beef's price is "C2". However, what if the column that contains Price is column "D" rather than column "C"? Did you give additional context to the LLM?

Author response:
3. **About Google Sheet Columns** - We present the meta data in markdown format to the model in prompt during inference. So the model always has access to all the column names as well as the rows of the content.