An Evaluation Mechanism of LLM -based Agents on Manipulating API s (2024)

Abstract: LLM-based agents can greatly extend the abilities of LLMs and thus attract sharply increased studies. An ambitious vision – serving users by manipulating massive API-based tools – has been proposed and explored. However, we find a widely accepted evaluation mechanism for generic agents is still missing. This work aims to fill this gap. We decompose tool use capability into seven aspects and form a thorough evaluation schema. In addition, we design and release an instruction dataset and a toolset – the two sides that the agents bridge between – following the principle of reflecting real-world challenges. Furthermore, we evaluate multiple generic agents. Our findings can inspire future research in improving LLM-based agents and rethink the philosophy of API design.

## introduction large language models (llms) exhibit remarkable capabilities across a variety of tasks, such as language, mathematics, coding, and etc @xcite . however, they still face some limitations, such as having frozen knowledge, being bad at some specialized tasks like calculation, and not being able to ground their generated solution outlines to the real world. meanwhile, there are existing systems or models that can perform very well on domain-specific tasks. therefore, a mechanism that links llms with the existing ecosystem of tools can bring the ability of llm-based ai to another level. to stretch the ability of llms, a sharply increasing number of works have studied llm-based agents 1 which can manipulate api-based tools. a very ambitious vision is to build a new ai ecosystem that connects llms with millions of apis, assessed via an api platform, for task completion. as shown in fig. 1 , the agent acts like a super-app. it eases user interaction with language rather than gui, manipulates massive api-based tools and thus supports a mass of functionalities. this requires the agent to have, on one hand, rich knowledge to deal with different user needs, and on the other hand developer's skills for manipulating apis given documentation and understanding the results of api execution. though the research community has made an effort to build up generic llm-based agents, we found that a widely accepted evaluation mechanism for llm-based agents on tool use is still lacking. this prevents researchers from making fair comparisons between different agent systems, as well as gaining insights into the challenges of designing agents. in this work, we aim to narrow this gap by providing an evaluation mechanism, putting special emphasis on discovering the limitations of existing agents as well as the problems of current api design philosophy. it is very challenging to evaluate an agent because of the complex process of an agent performing user tasks. once a user issues an instruction, the agent first decomposes it into solvable subtasks with available tools, depending on the complexity of the task. then, it may need to collect user needs by interacting with the user and call multiple apis. eventually, it responds to the user as per the results of apis. the problems this complex process brings to agent evaluation are: (1) the possible involvement of users makes the evaluation very hard. (2) the agent may fail in any step of the sequential process, making the samples in the later stage rare. (3) executing a complex process may involve multiple aspects of capability. however, an end-to-end performance cannot help locate the weaknesses of agents and gain more insights. to solve these problems, we dissect the whole process into intermediate decision behaviors to get a thorough view of the involved capabilities. according to this anatomy, we include 7 aspects of tool use capability in our evaluation schema. each aspect corresponds to one separate evaluation subtask without involving human users. under the guide of our evaluation schema, we build a toolset and a dataset of instructions, following the principle of reflecting real-world challenges. we construct our toolset by addressing a series of concerns as demonstrated below rather than assembling some random tools. (1) to ensure reflection of real-world challenges, we collect tools from the real api platform. it matters to reflect the properties of api design and documentation. (2) we intentionally control the diversity of tools regarding functionalities and api structures. according to our observation, a tool may have a single api, a list of apis, or several collections of apis. these organizational structures can indicate the variety of functionality and expose different difficulties in calling. (3) we find some apis depend on other apis foot_0 . we include this kind of dependency relationship in our toolset. (4) to enable high-level tasks applied to the toolset, we take measures to increase the coherence of tools regarding application scenarios. (5) we noticed that the affordability of toolset can be one potential problem for individual researchers. to avoid this problem, we devoted lots of engineering work to make the toolset usable at low or even no cost. on top of our toolset, we construct a set of user instructions that may use the contained tools to solve. we analyze how humans ask questions and summarize five types of instructions, varying in user intentions and complexities. these different types of instructions can be used to produce evaluation data required by our evaluation schema. additionally, we emphasize that the instructions should be in the real way of user expression. only by this can the evaluation data imply the mismatch between user expression and the form of information required by apis. to conclude, we contribute an evaluation mechanism of llm-based agents on api manipulation. it is composed of several evaluation subtasks supported by one toolset and one dataset. we make these resources publicly available at https: //github.com/oppo-mente-lab/agent_eval . ## methodology we make our evaluation schema first to make clear our targets. then, we take measures to construct a toolset inheriting real-world challenges and supporting tasks with varying complexities. extra engineering work makes its usage affordable. following this, we design five types of instruction data and align them to the evaluation schema. finally, we determine the metrics of each evaluation sub-task. ## experiment in this section, we first demonstrate more details of our dataset and toolset. then, we apply our evaluation mechanism to examine several well-known llms equipped with generic tool use ability, including chatgpt series -gpt-3.5-turbo and gpt-4-8k (abbreviated as gpt-3.5 and gpt-4 below), and qwen1.5 series with sizes 7b, 14b and 72b (abbreviated as qwen-7b, qwen-14b and qwen-72b) @xcite . the new findings can show the value of our evaluation mechanism. ## conclusion the llm community is driving towards an ambitious vision: building a new ai ecosystem in which llm-based agents serve users by manipulating millions of apis. however, an evaluation mechanism for such agents is still missing, preventing studies from proceeding in this area. in this work, we narrow this gap by proposing a new evaluation mechanism for generic llm-based agents. we designed a thorough evaluation mechanism schema aiming to examine seven different aspects of tool use capability. also, we release one dataset and one toolset, both designed to reflect real-world challenges. these resources can support the studies on improving llm-based agents as well as a new philosophy of api design. we evaluated five influential llms and shared findings and insights into their tool use capability. the found weaknesses of llms can indicate the future directions to go.