A pp B ench: Planning of Multiple API s from Various APP s for Complex User Instruction (2024)

Abstract: Large Language Models (LLMs) can interact with the real world by connecting with versatile external APIs, resulting in better problem-solving and task automation capabilities. Previous research primarily either focuses on APIs with limited arguments from a single source or overlooks the complex dependency relationship between different APIs. However, it is essential to utilize multiple APIs collaboratively from various sources, especially for complex user instructions. In this paper, we introduce MetaBench, the first benchmark to evaluate LLMs’ ability to plan and execute multiple APIs from various sources in order to complete the user’s task. Specifically, we consider two significant challenges in multiple APIs: 1) graph structures: some APIs can be executed independently while others need to be executed one by one, resulting in graph-like execution order; and 2) permission constraints: which source is authorized to execute the API call. We have experimental results on 9 distinct LLMs; e.g., GPT-4o achieves only a 2.0% success rate at the most complex instruction, revealing that the existing state-of-the-art LLMs still cannot perform well in this situation even with the help of in-context learning and finetuning. Our code and data are publicly available at https://github.com/ruleGreen/AppBench .

## introduction empowering large language models (llms) @xcite with versatile tools such as retrievers @xcite , models @xcite , and even physical robots @xcite , holds significant promise in overcoming inherent limitations, such as hallucination @xcite and outdated information @xcite , and unveils the immense † equal contributions. work done when first author visits edinburghnlp. ## related work tool benchmarks. the complexity of real-world tasks necessitates the integration of diverse tools and services, consisting of three types of tools @xcite ): 1) physical interaction-based tools @xcite ; 2) gui-based tools @xcite ; and 3) program-based tools @xcite . on the one hand, some work focuses on models, retrievers, or calculators to address the intrinsic limitations of llms, such as toolqa @xcite and tool-bench @xcite . on the other hand, another line of work targets apis since they are particularly crucial for bridging smooth interaction between humans and the digital realm @xcite @xcite . most previous works formulate this as an api selection task given all related information about each api and current input, which overlooks the nuanced dependencies and permission constraints between different apis, such as apibench @xcite and api-bank @xcite . nevertheless, the successful execution of apis in the real world necessitates meeting requirements fulfilled (either the value is provided by the user or previous apis) and obtaining permission from trusted agents beyond just knowing api names and a few arguments. more details can refer to latest survey @xcite and tutorial @xcite . language agent. existing frameworks for language agents have made notable strides in facilitating interaction with external tools @xcite @xcite and environment @xcite . they usually follow the single-agent paradigm to access different tools or services sequentially @xcite , or multi-agent framework by assigning different agents different roles to call different cognitive tools @xcite . for example, @xcite propose chameleon which utilizes one agent to plan the execution order of different services by outputs a sequence of names of tools, which assume that the agents to call these tools are already known, and lots of works follow @xcite . furthermore, various benchmarks are proposed to evaluate the abilities of llms serving as agents in different situations @xcite @xcite . for instance, @xcite proposes webshop to evaluate whether llms are capable of interacting with the web. similarly, @xcite simulates household activities through programs, and many works use this as a testbed for embodied agents @xcite . latest work focus on using apis or functions to control the whole planning processing of agents @xcite . ## appbench construction in this section, we start with a formal task definition and then provide a detailed explanation of how we efficiently and effectively built our appbench by leveraging existing datasets. ## analysis in this section, we conduct a comprehensive analysis, aiming to answer three research questions. rq1: how do the parallel and sequential dependencies influence the model performance? (sec 5.1) rq2: is it necessary to identify app first to reduce the context window? (sec 5.2) and rq3: what is the major bottleneck of current llms (sec 5.3), and can fine-tuning or in-context learning alleviate it? (sec 5.4, 5.5). ## conclusion in this paper, we introduce a new benchmark, appbench, addressing the challenge of complex user instructions that require the involvement of multiple apis. these scenarios demand advanced planning capabilities from llms to effectively handle graph structures and ensure permission isolation in practical applications. we left the selfevolving or more effective fine-tuning framework in our future work.