Paper: Large Language Models as Tool Makers (ICLR-2024)

Reviewer comment (Experiments):
Tool making is an interesting idea. However, one weakness in this direction is that we could have many external tools in real-world scenarios and the capability of tool-making should be able to explore new tools that do not have before. Therefore, to further improve the feasibility of the proposed method, it will be better to explore the capability of tool-making to make new tools when also considering the existence of external tools.

Author response:
Regarding weakness 1, we agree exploring the tool maker's ability to create entirely new tools beyond existing APIs would be an exciting direction for future work. Our initial study focused on a simpler closed-loop setting, but we plan to investigate more open-ended tool making scenarios as well.

Reviewer comment (Novelty):
Compared with Tool making, the design of Tool use is relatively trivial. Many works (e.g., AutoGPT, HuggingGPT) have explored the feasibility of tool use by using LLMs.

Author response:
For weakness 2, we agree that tool use with LLMs has been explored in prior work, as cited in our paper. Our contribution is showing tool making can further enhance the performance and efficiency of tool use.

Reviewer comment (Reproducibility):
In Tool verification, the authors utilize 3 validation samples. So how to obtain these validation samples? Are these samples manually created and Are these validation samples fixed or different when meeting different user instructions? What are the generalizations of these examples in different scenarios?

Author response:
Q2) In our evaluation, we use the examples from a dev set as validation examples. However, in a streaming setting, validation examples can be collected by using the first few results generated by powerful LLM such as GPT-4 or human-authored.

Reviewer comment (Experiments):
Based on my observations, the current LATM can only make tools in language form and cannot support multimodal ability, right?

Author response:
Q3) Our current implementation focuses on textual tools, yet the method is general. Extending to multimodal abilities is an interesting direction we hope to explore.

Reviewer comment (Experiments):
The paper states that they use a self-verification process with a few labeled instances. However, it is not clear whether the performance is stable with a different selection of few-shot validation examples.

Author response:
Regarding the stability with different validation examples (W1), we clarify that the self-verification serves only as a simple sanity check. We tried three different sets of validation examples, they can all filter out apparent mistakes and produce correct tools. As the reviewer noted, studying the self-verification process itself is not the focus of this work, and we kindly refer the reviewer to the cited papers in our paper for further discussion.

Reviewer comment (Experiments):
Is it possible to evaluate other tool-making benchmarks such as ToolBench[1], and APIBank[2]? These datasets will be useful in faithfully evaluating the tool-making ability of LLMs.

Author response:
The suggested ToolBench and APIBank benchmarks (W3) do seem promising for evaluating tool use. However, as we noted, benchmarks focused on workflow customization may be better suited for assessing tool-making abilities. We appreciate these references and will consider evaluating on these datasets in follow-on work once tool-making benchmarks are available.