CT ool E val: A C hinese Benchmark for LLM -Powered Agent Evaluation in Real-World API Interactions (2024)

Abstract: Assessing the capabilities of large language models (LLMs) as agents in decision making and operational tasks is crucial for the development of LLM-as-agent service. We propose CToolEval, a benchmark designed to evaluate LLMs in the context of Chinese societal applications, featuring 398 APIs across 27 widely-used Apps (e.g., Apps for shopping, map, music, travel, etc.) that cover 14 domains. We further present an evaluation framework that simulates real-life scenarios, to facilitate the assessment of tool invocation ability of LLMs for tool learning and task completion ability for user interation. Our extensive experiments with CToolEval evaluate 11 LLMs, revealing that while GPT-3.5-turbo excels in tool invocation, Chinese LLMs usually struggle with issues like hallucination and a lack of comprehensive tool understanding. Our findings highlight the need for further refinement in decision-making capabilities of LLMs, offering insights into bridging the gap between current functionalities and agent-level performance. To promote further research for LLMs to fully act as reliable agents in complex, real-world situations, we release our data and codes at https://github.com/tjunlp-lab/CToolEval.

## introduction recent years have witnessed that large language models achieve remarkable progress in planning and decision making. this promotes the emergence of numerous llm-as-agent applications, such as chatgpt plugins 1 , autogpt 2 , and generative agents @xcite , even enabling llmbased plugin control with natural language interface and provoking the development of llm-based os. in view of such rapid progresses, pressing issues are urgently emerging and soliciting attention and solutions: (i) how should we assess the capability of llms as agents? (ii) which llms have reached agent level, and which have not? (iii)what challenges do llms still face when used as agents? to address these urgent and unresolved issues, we curate a benchmark, ctooleval, for tool learning in chinese scenarios. this benchmark is of very high quality, characterized by that (i) all apis come from a wide and rigorous collection of open apis from widely-used real-world apps, making the benchmark akin to a testbed of natural language driven mobile operation system. it assesses whether an llm-powered agent can understand and accurately respond to various functionalities within apps, offering a high degree of real-world interactions. (ii) the generation of instructions is meticulously designed, with the use of certain tools not being completely unrelated and abrupt, but rather closely simulating all-around scenarios in our daily lives. (iii) each instruction is annotated with a detailed reference answer and has undergone a two-stage rigorous manual review to ensure its quality. to facilitate the agent assessment with ctool-eval, we futher propose a fine-grained evaluation framework. initially, we categorize the data from single-tool and multi-tool scenarios into four types: fixed-answer, open-ended, real-time, and operational types. in the first stage of evaluation, we assess whether an llm-powered agent has invoked an appropriate tool, including understanding and correctly calling the api in a single-tool scenario, or whether the agent can decompose a complex task into multiple subtasks, then plan to call apis for each subtask and understand the use of api parameters to make the correct calls in a multi-tool scenario. in the second stage of evaluation, we start from the perspective of accuracy rather than having humans @xcite @xcite benchmark statistics language tool type evaluation type # tools # apis # instances real-world multi-tool api call response apibench (patil et al., 2023) 3 1,645 17,002 english ✘ ✘ ✔ ✘ restbench @xcite 2 94 157 english ✔ ✔ ✔ ✘ api-bank @xcite 53 53 274 english ✔ ✘ ✔ ✔ toolalpaca @xcite 400 400 3,938 english ✘ ✘ ✔ ✔ toolqa @xcite 13 13 1,600 english ✔ ✔ ✔ ✔ toolbench1 @xcite 3 , 451 16 , 464 12,657 english ✔ ✔ ✔ ✘ toolbench2 @xcite 8 232 2,746 english ✔ ✘ ✔ ✘ tptu @xcite 12 12 120 english ✔ ✔ ✔ ✘ tooleyes @xcite with the proposed evaluation framework, we evaluate 11 llms with chinese language capabilities. we find that openai models generally have superior tool invocation capabilities compared to chinese llms, but models with rich internal knowledge like gpt-3.5-turbo 1 , when faced with conflicts between internal and external knowledge, tend to rely on internal knowledge rather than toolinvoked knowledge to answer questions. for instance, in multi-tool scenarios requiring the acquisition of a location's coordinates, it might provide coordinates it already knows internally, which are usually close but still have discrepancies, rather than those obtained through tool invocation. and we also find chinese llms, like chatglm3-6b 2 , have severe hallucination issues. they can decompose instructions into subtasks and choose the correct tool names. however, due to a lack of complete understanding of how to use the api, they cannot pass in the correct parameters to obtain the correct output from the api. nevertheless, they still pretend to have received output from the api server and reply with fabricated answers. overall, our contributions are threefold: ## related work research on evaluating llms for tool manipulation and api interaction has led to benchmarks like toolbench @xcite , with its extensive collection of 16,464 apis across 3,451 tools and 12,657 instances. the emulation of real-world scenarios is a pivotal aspect of benchmark evaluation for llms, as it significantly affects the relevance and applicability of evaluation findings. api-bank @xcite , provides an interface with 53 apis, asserting its utility with 274 instances drawn from real-world applications. toolqa @xcite highlights multi-tool interactions, emphasizing their significance in llm evaluation. the dimension of evaluation type in benchmarking llms is crucial as it directly influences the interpretability and applicability of evaluation results. this dimension encompasses the accuracy @xcite @xcite et al., 2023) and execution success rate of api calls @xcite @xcite , the quality and relevance of the responses @xcite @xcite , and the model's ability to avoid hallucination -generating responses based on false or nonexistent information @xcite . specifically, the metrics used in restbench @xcite and toolalpaca @xcite require manual evaluation, while the win rate indicator in toolbench1 @xcite relies on scoring by gpt-4, all of which inevitably introduce elements of subjective judgment and assessment unfairness caused by the biases of human and llms. a concurent work has a motivation very similar to ours. tooleyes @xcite creates a finegrained evaluation of llm's tool learning capabilities. particularly in the data dimension, both of our works include real-time and operational data types. however, tooleyes @xcite still relies mostly on scoring by advanced models like gpt-4 to deliver evaluation results, inevitably encountering issues such as model bias. our contribution, ctooleval, seeks to address these shortcomings by prioritizing accuracy, solving for the dynamic evaluation of answers that may change over time, aiming to minimize reliance on model scoring for better reuse and scalability. ## benchmark curation our data creation process, as illustrated in figure 1 , involves sourcing open apis from real-world applications. ## evaluation framework in a comprehensive process where an llm acts as an agent to solve problems using tools, the model initially extracts the query and the list of available tools along with their descriptions from the prompts. it then decomposes the query into smaller sub-tasks and sequentially calls the corresponding tools, using the knowledge returned by each tool to progressively deduce the correct answers. however, prior research @xcite @xcite ) reveals the inadequacies of most llms in tool utilization, highlighting their failure in task decomposition, tool invocation, and knowledge application, with errors frequently emerging at various stages. thus, to thoroughly assess llms' tool utilization proficiency, we scrutinize both tool invocation and task completion capabilities. through an indepth analysis shown in appendix a, we evaluate llms' ability to apply tool-generated knowledge. these competencies are interrelated, marking ascending levels of tool usage mastery. additionally, leveraging actual user inputs, ctooleval's instructions are classified into four categories: fixed-answer, open-ended, operational, and realtime, enabling a nuanced, multi-faceted evaluation of accuracy in these dimensions. ## experiments we evaluated a variety of llms, including 1 closedsource llms and 10 open-source chinese llms. as our evaluation requires llms to have a certain level of compliance with instructions, we selected those that have been fine-tuned with sft and possess chat capabilities. additionally, due to the large number of tools, and some multi-tool queries even containing more than 10 tools, the prompts fed into the llms can be very long, necessitating llms with a relatively large context window, at least above 8k. the evaluated closedsource llms is gpt-3.5-turbo. 1 the open-source llms include qwen-7b-chat, foot_2 qwen-14b-chat, foot_3 qwen-72b-chat, 4 internlm-chat-7b, 5 internlmchat-20b, foot_6 baichuan2-13b-chat, foot_7 baichuan2-13b-chat, 8 chinese-alpaca-2-7b-16k, 9 chinese-alpaca-2-7b-64k foot_10 and chinese-alpaca-2-13b-16k. foot_11 ## conclusion in this paper, we have presented ctooleval, a benchmark for assessing llms as agents to interact with chinese apps, which covers 398 realworld apis. through our fine-grained evaluation framework, we identify critical insights into the capabilities of llms, specifically in the domains of tool invocation and task completion. our analysis indicates that while gpt-3.5-turbo demonstrates a higher proficiency in tool utilization, there remains a notable gap in the ability of chinese llms to apply tools effectively. this gap is often marked by a reliance on internal knowledge and the occurrence of hallucinations, which leads to a discrepancy between the output provided and the information available through tool invocation. by dissecting the errors and pinpointing the nuances in the operational use of apis, our work contributes to a more nuanced understanding and evaluation of the agent-level capabilities of llms and opens up avenues for future research and development of llm-powered agents in real-world scenarios.