API -Bank: A Comprehensive Benchmark for Tool-Augmented LLM s (2023)

Abstract: Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs’ ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs’ capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains. Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca’s tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error analysis, we highlight the key challenges for future research in this field to answer the third question.

## introduction over the past several years, significant progress has been made in the development of large language models (llms), including gpt-3 @xcite , @xcite , chatgpt, and impressive gpt-4 @xcite . these models exhibit increasingly human-like capabilities, such as powerful conversation, in-context learning, and code generation across a wide range of open-domain tasks @xcite . despite their utility, llms are still constrained by training data @xcite @xcite . this data can quickly become outdated and may not cover all scenarios @xcite . consequently, there has been a surge of research focused on enhancing llms by enabling them to leverage external api tools, such as accessing up-to-date information @xcite and interacting with third-party services @xcite . traditionally, tool usage has been viewed as a uniquely human behavior, with its emergence considered a significant milestone in primate evolution @xcite . drawing an analogy to the timeline of human evolution, we argue that it is currently imperative to address three pivotal questions: (1) how effective are current llms at utilizing tools? (2) how can we enhance llms' ability to utilize tools? (3) what obstacles still need to be overcome for llms to effectively leverage tools? to tackle these inquiries, we present api-bank, a groundbreaking benchmark specifically designed for tool-augmented llms. in order to determine the users' needs regarding the utilization of toolaugmented llms, we initially conducted interviews with 500 users. taking into account their feedback, we establish the design principles for api-bank. the evaluation scope of api-bank must encompass three essential capabilities: planning, retrieving, and calling api tools, in order to fulfill the users' requirements. additionally, while constructing the benchmark, it is imperative to consider various aspects such as domain diversity, api diversity, api authenticity, and evaluation authenticity. subsequently, to answer the first research question, we implement the evaluation system of api-bank, adhering to design principles. the sys-tem comprises 73 commonly used apis, along with 314 tool-use dialogues containing 753 api calls, all manually annotated, thus forming the first executable and authentic system capable of evaluating the effectiveness of llm utilizing tools. for the second question, we develop a comprehensive tool-augmented llm training dataset within api-bank. this dataset comprises 2,138 distinct apis and includes 1,888 dialogues with a total of 4,149 api calls, reflecting three different levels of api usage abilities. however, annotating thousands of apis and the corresponding dialogues, adhering to our design principles, is not only prohibitively expensive but also timeconsuming. in this work, we introduce multi-agent, a novel method using llms instead of humans to automatically mass-produce tool-augmented llm training data. multi-agent consists of five collaborative agents that step-by-step generate domains, apis, user queries, api calls & responses, while also ensuring the quality of the generated data aligns with our design principles. multi-agent remarkably reduces the annotation cost by 98% compared to human annotation. to verify the effectiveness of our training set, subsequently, we fine-tune a popular llm alpaca-7b @xcite , resulting in our own tool-augmented llm lynx. we conduct extensive experiments on api-bank and obtained insightful results regarding the api tool usage capabilities of llms. our findings reveal that even smaller models like alpaca-7b and chatglm-6.2b possess a certain level of api call accuracy, about 20%. however, their proficiency in api retrieval and planning is negligible. in contrast, the larger-scale gpt-3 davinci exhibits almost no capability in api usage, indicating that api usage might not be an inherent feature of llms. our lynx, demonstrates an average improvement of 24% in the three api usage capabilities compared to alpaca-7b. while it approaches the performance of gpt-3.5, there remains a 21% gap from gpt-4. additionally, we present a comprehensive experimental analysis that sheds light on the primary challenges encountered by gpt-4 and lynx in utilizing apis, addressing the third research question. ## design principles of api-bank due to the absence of authoritative ability definitions and benchmarks about tool-augmented llm, we conducted an extensive questionnaire survey in the initial phase. through interviews with over 500 users who expressed interest in incorporating additional tools into llm, we collected their requirements. based on this foundation, we provide, for the first time, a definition to measure the abilities of tool-augmented llm and the data standards for training and evaluating its abilities. we believe that these design principles, which reflect the actual needs of users, can assist researchers in the field and contribute to the future development of tool-augmented llm benchmarks. ## evaluation system of api-bank based on the aforementioned principles, we first introduce how to construct the evaluation system of api-bank, including the system implementation, the data annotation, and the evaluation metrics. ## training set of api-bank we hope api-bank not only evaluate the effectiveness of existing llms in utilizing tools but also to enhance their performance in utilizing such tools. the most direct approach to achieve this goal is creating a high-quality training dataset tailored to tool-augmented llms. however, it is challenging to construct a large-scale training dataset at a low cost while meeting the our design principles of domain diversity and api authenticity. the manual annotation cost for each dialogue in our evaluation set reaches @xmath00.1 per dialogue and saving 98% compared to manual annotation. ## benchmark analysis statistics. in the end, we construct a benchmark consisting of 1,008 domains, 2,211 apis, 2,202 dialogues, and 6,135 turns. among them, comparison. we have conducted a comparative analysis of api-bank in relation to recently released benchmarks, as presented in ## related work recent research in language modeling has explored the use of external tools to supplement the knowledge stored in the model's weights @xcite . this approach allows for tasks such as exact computation or information retrieval to be offloaded to external modules such as a python interpreter or a search engine @xcite . these tools can include other neural networks or even the language model itself. socratic models @xcite is a modular framework that allows for the composition of different pre-trained models on various modalities. alternatively, natural language knowledge can be retrieved from external sources, as demonstrated by webgpt @xcite and react @xcite through the use of search apis. other approaches, such as toolformer @xcite , art @xcite , toolkengpt @xcite and trice @xcite leverage a combination of search apis, question-answering apis, machine translation apis, calculators, and other tools to solve various nlp tasks. chatgpt plugins 1 and taskmatrix.ai @xcite demonstrate the potential for language models to integrate with thousands to millions of apis. latm @xcite and creator (qian et al., 2023) leverage gpt-4 to make api tools. despite the promising demonstrations of these approaches, researchers have limited knowledge regarding three key issues: ## experiments we proceed to fine-tune lynx, a model based on llama-7b @xcite , using our api-bank training dataset. the fine-tuning process consists of three epochs, with a batch size of 256 and a learning rate of 2e-5. subsequently, we conduct a series of experiments on our api-bank evaluation system, wherein we benchmark our model lynx against other llms. the prompts used for evaluation are given in the appendix. through these experiments, we aimed to identify the remaining challenges that hinder the effective utilization of 1 https://openai.com/blog/chatgpt-plugins llms in conjunction with tools. ## conclusion this paper presents api-bank, the pioneering benchmark for tool-augmented llms comprising 2,202 dialogues involving 2,211 apis from 1,008 domains, and provides three key findings: (1) current llms demonstrate some ability to use apis, but there is still room for improvement in providing reliable services to users consistently. (2) building diverse and high-quality training data is a promising approach to enhance llms' ability to use tools. (3) proper api calls and strict adherence to api documentation pose significant challenges for llms in utilizing api tools. overall, the api-bank benchmark provides a groundbreaking resource for evaluating and advancing the state-ofthe-art in tool-augmented llms. we believe that this research will inspire future studies and pave the way for developing more advanced ai systems that can intelligently integrate external resources to fulfill human requirements.