ACEB ench: A Comprehensive Evaluation of LLM Tool Usage (2025)

Abstract: Large Language Models (LLMs) have demonstrated significant potential in decision-making and reasoning, particularly when integrated with various tools to effectively solve complex problems. However, existing benchmarks for evaluating LLMs’ tool usage face several limitations: (1) limited evaluation scenarios, often lacking assessments in real multi-turn dialogue contexts; (2) narrow evaluation dimensions, with insufficient detailed assessments of how LLMs use tools; and (3) reliance on LLMs or real API executions for evaluation, which introduces significant overhead. To address these challenges, we introduce ACEBench, a comprehensive benchmark for assessing tool usage in LLMs. ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. “Normal” evaluates tool usage in basic scenarios; “Special” evaluates tool usage in situations with ambiguous or incomplete instructions; “Agent” evaluates tool usage through multi-agent interactions to simulate real-world, multi-turn dialogues. We conducted extensive experiments using ACEBench, analyzing various LLMs in-depth and providing a more granular examination of error causes across different data types.

## introduction large language models (llms), such as gpt-4 @xcite , have demonstrated exceptional performance across numerous natural language processing tasks @xcite @xcite . studies have shown that incorporating tools can significantly expand llm capabilities, particularly in specialized domains such as mathematics @xcite @xcite , programming @xcite , and reasoning @xcite @xcite . on one hand, integrating tools into llms can enhance capabilities in multiple domains, for example, tooltransformer @xcite enhances the ability of llms to solve complex problems by utilizing tools. on the other hand, adopting a tool usage paradigm can improve the robustness of the response and the transparency of the generation, thus increasing the explainability and trust of users @xcite , as well as improving the system's adaptability. as this field continues to evolve, it is essential to comprehensively evaluate all aspects of tool usage, particularly in complex scenarios. while several studies have focused on evaluating tool usage @xcite @xcite @xcite @xcite , there are still some shortcomings in the existing tool-use benchmarks. firstly, existing benchmarks lack multi-turn dialogue evaluation in real-world scenarios. for example, the multi-turn dialogues in bfcl @xcite and hammerbench @xcite are composed of predefined fixed content combinations. secondly, current tool-use benchmarks @xcite @xcite lack fine-grained evaluation and personalized data assessment. additionally, existing benchmarks @xcite @xcite ignore the assessment of special cases, or the evaluation methods are simplistic @xcite , as user instructions in real life are not always perfect @xcite . the model's ability to recognize and handle these issues is also crucial for evaluation. lastly, evaluation costs are high @xcite , as many studies rely on advanced large models for evaluation. to address these shortcomings, we propose ## related works the emerging trend of leveraging llms' tool-use capabilities in real-world applications underscores the need for comprehensive evaluations of their performance and effectiveness. despite recent advancements, existing benchmarks for evaluating the tool-use capabilities of llms still have significant limitations stable toolbench @xcite addresses the issue of unstable external apis by employing a virtual api server, but its dependence on large models for evaluation results in high costs and scalability challenges. bfcl @xcite introduces a benchmark for tool use in multi-turn dialogue scenarios. yet, it assembles dialogues from fixed content, failing to capture the dynamic and adaptive nature of real-world interactions. similarly, τ -bench @xcite evaluates language agents' ability to engage with human users while adhering to domain-specific rules. still, its narrow focus on just two scenarios limits its generalizability across diverse tasks. hammerbench @xcite improves upon this by incorporating datasets derived from popular mobile applications and merging dialogues to simulate typical question-answer trajectories. however, like bfcl, its multi-turn dialogues are simplistic concatenations of pre-defined content, which do not reflect the complexities of real-world conversational dynamics. in addition, some benchmarks @xcite rely on large language models (llms) for result evaluation, leading to high costs and unstable operations. in contrast, our work addresses these limitations by expanding the scope of evaluation to encompass a broader range of tool usage scenarios. we user tool assistant multi-agent interactive generation apis single turn single function single turn parallel function …… advanced model generation dialogue formats real-rorld scenarios. add_food_order get_products login_platform food platform class food_platform: def __init__(self): self.@xmath0, username: str, merchant_name: str, items:list ) …… dialogue rule eg before ordering takeout, you need to obtain …… questions designed by experts. environment construction scenario construction api examples api context tree llm generate apis self-evolving api synthesis dialogue construction (b) agent data construction (a) normal and special data construction propose a framework that simulates realistic multiturn dialogue processes and enables end-to-end automated assessment, thereby reducing evaluation costs and improving scalability. a comparative analysis of acebench against recent benchmarks, as shown in ## conclusion this paper introduces acebench, a comprehensive tool-use benchmark designed to evaluate the tool-use capabilities of large language models (llms), including data from three types: normal, special, and agent. it addresses key limitations of existing evaluation benchmarks, such as the lack of multi-turn dialogue assessments in real-world scenarios, the absence of fine-grained evaluations for parameter-type function calls, and the high costs associated with using large models for evaluation. the experimental results indicate that models finetuned on specific tool-use datasets to struggle with generalization when faced with complex or imperfect instructions, and code capabilities enhance the tool-use performance of large models. through extensive experiments, we demonstrate the effectiveness of acebench in providing deeper insights into the tool-use abilities of various models.