Planning, Creation, Usage: Benchmarking LLM s for Comprehensive Tool Utilization in Real-World Complex Scenarios (2024)

Abstract: The recent trend of using Large Language Models (LLMs) as tool agents in real-world applications underscores the necessity for comprehensive evaluations of their capabilities, particularly in complex scenarios involving planning, creating, and using tools. However, existing benchmarks typically focus on simple synthesized queries that do not reflect real-world complexity, thereby offering limited perspectives in evaluating tool utilization. To address this issue, we present UltraTool, a novel benchmark designed to improve and evaluate LLMs’ ability in tool utilization within real-world scenarios. UltraTool focuses on the entire process of using tools - from planning and creating to applying them in complex tasks. It emphasizes real-world complexities, demanding accurate, multi-step planning for effective problem-solving. A key feature of UltraTool is its independent evaluation of planning with natural language, which happens before tool usage and simplifies the task solving by mapping out the intermediate steps. Thus, unlike previous work, it eliminates the restriction of pre-defined toolset. Through extensive experiments on various LLMs, we offer novel insights into the evaluation of capabilities of LLMs in tool utilization, thereby contributing a fresh perspective to this rapidly evolving field. The benchmark is publicly available at https://github.com/JoeYing1019/UltraTool.

## introduction recent advancements in equipping large language models (llms) @xcite @xcite with external tools @xcite @xcite have markedly improved capability of ai systems in solving complex realworld tasks. as this field evolves, it becomes crucial to conduct a comprehensive evaluation covering full aspects of tool utilization, particularly within complex real-world contexts. as depicted in figure 1 (a), addressing real-world tasks often necessitates not only the planning and usage of multiple tools but also the creation of new tools, when the existing tools are not enough to meet all specific requirements. however, existing benchmarks @xcite @xcite @xcite often focus only on limited dimensions of this entire process. besides, the user queries in most existing benchmarks exhibit limitations in mirroring the complexity of real-world tasks and unreasonable dependency on pre-defined toolsets. to tackle these challenges, we introduce ultratool, a benchmark covering a wide range of evaluation dimensions. it is constructed on complex, real-world queries and involves evaluation of tool-independent natural-language planning and advanced tool creation capabilities. ultratool comprises 5,824 examples, spanning 22 diverse domains and incorporating 2,032 tools, and it comprehensively evaluates the tool utilization process including six dimensions covering three aspects: ① planning: decomposing the complex goal into logical sequence of simpler sub-tasks for effective problem-solving. tool creation comprises two dimensions: ② awareness -assessing whether existing tools suffice, and ③ creation -developing the necessary tools if existing ones are inadequate. tool usage involves three dimensions: ④ awareness -determining which sub-tasks require tools, ⑤ selection -choosing the appropriate tools, and ⑥ usage -specifying the input parameters for these tools. as demonstrated in moreover, the queries in ultratool have better realism and complexity. the natural workflow of tool-augmented task solving includes: "user queries @xmath0. importantly, the query and the plan should not be constrained by pre-existing tools. in contrast, existing benchmarks @xcite @xcite are typically constructed by collecting toolset, generating simulated queries with tools randomly selected from toolset, and then annotating the solution as shown in figure 1 (b) . despite the collecting efficiency, the generated queries may not accurately mirror real user demands. furthermore, the randomly selected tools may lack a coherent logical relationship, potentially leading to skewed influences on the subsequent process. to ensure the complexity and reality of queries, we collect real-world multidomain user queries with high complexity. specifically, we engage experts from various domains to craft complex queries that reflect real-world needs and potentially incorporate the use of diverse tools. a distinct feature of ultratool is it explicitly evaluates natural language (nl)-based plan, which simplifies task solving by decomposing a complex goal into several simpler sub-tasks described in nl. each sub-task is then solved by tool creation and usage. therefore, prior planning overcomes the restriction brought by limited pre-existing tools. we conduct experiments on various llms and make in-depth analyses about strengths and challenges in the tool utilization of llms. contributions of this work are: (1) ultratool is a comprehensive evaluation benchmark derived from complex real-world queries, covering six key dimensions in tool utilization. (2) ultratool explicitly evaluates nl-based planning and advanced tool creation capabilities. (3) extensive experiments uncover the limitations and inspire the future direction of llms in tool utilization. ## ultratool construction ultratool begins by gathering real-world user queries and encompasses a comprehensive range of evaluation dimensions, including planning, tool creation and tool usage. as depicted in figure 4 , the construction process of ultratool includes: (1) query collection ( §2.2); (2) solution annotation ( §2.3); and (3) manual refinement ( §2.4). for the detailed prompts utilized in the ultratool construction process, please refer to appendix a.5. ## ultratool evaluation in this section, we introduce the metrics ( §3.1), and the definition and measurement of each evaluation aspect, including: planning( §3.2), tool creation( §3.3) and tool usage( §3.4). ## conclusion in this paper, we introduce ultratool, a comprehensive evaluation benchmark derived from realworld complex queries, aimed at evaluating the tool utilization capabilities of llms across six dimensions that encompass three critical aspects: planning, tool creation, and tool usage. ultratool excavates the necessary evaluation dimensions from actual tool utilization processes and pioneers in explicitly evaluating the nl-based planning and tool creation abilities. our extensive analyses reveal that many current llms still have significant potential for enhancing their tool utilization abilities. we hope that ultratool, coupled with our detailed experiments, will offer valuable insights and stimulate further research into the real-world application of llms in tool utilization. tools in ultratool are not executable ; rather, they are tool skeletons that represent the tools' functionalities, as many real-world demands cannot be fully addressed by existing real-world tools. despite these tool skeletons are not specific implementations, they serve as simulated representations that accurately depict the tool's functionality and offer guidance for the future development of actual tools. in the future, we plan to explore simulating the execution of our designed tools or gathering real-world data involving only executable tools.