S table T ool B ench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models (2024)

Abstract: Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.

## introduction with the rapid developments of large language models @xcite @xcite @xcite , tool learning which leverage llms to schedule a variety of external tools has attracted enormous attention @xcite @xcite . previous studies @xcite @xcite aim to augment llms with tools to enhance performance on conventional natural language processing (nlp) downstream tasks, while recent work @xcite @xcite primarily focus on solving realworld scenarios that require the use of tools. in project: zhichengg.github.io/stb.github.io/ github: thunlp-mt/stabletoolbench general, tool learning complements the capabilities of vanilla llms and bridges the gap to real-world applications. to assess the capability of llms to use tools, a series of tool learning benchmarks have been introduced. several pioneering studies have heavily relied on human-crafted offline tools @xcite or hand-selected online tools @xcite . while these tools are high-quality, their scale remains relatively small, thereby limiting their ability to accurately reflect real-world scenarios. to address this limitation, subsequent studies @xcite @xcite have advocated for leveraging extensive collections of online tools that span across various domains. owing to the increased scale, the automatic evaluation of tool learning has moved closer to real-world scenarios. however, concerns have been raised regarding the stability of these online tools, which has implications for the reproducibility and comparability of benchmark performance over time foot_0 . for instance, the well-recognised toolbench foot_1 @xcite has shown performance discrepancies that cannot be reproduced months after its release, as analysed in section 2.1. this is even more important when faced with a complex environment, where apis and tools keep changing while the evaluation should maintain its consistency across time. existing large-scale benchmarks may struggle to provide stable evaluations for various reasons. we propose several hypotheses for this issue. firstly, the complexity of tasks involving tool usage makes it challenging for the common automatic evaluator, gpt-3.5, to function effectively as a discriminator. as discussed in section 2.2, the evaluator cannot reliably determine whether a task is solvable or unsolvable, leading to variability in model performance due to this capability limitation. secondly, the stability of api status for a significant portion of online tools (55.6% in toolbench) is inconsistent. users may be required to authorise the use of these tools or apis, and tools provided by developers may be accessible during the initial construction of the benchmark but become unavailable later. this fluctuation further undermines the reliability and reproducibility of model performance assessments over time. this situation results in a problem where the constructed queries in the benchmarks may no longer be completed with their originally referenced tools. consequently, it is crucial to strike a balance between enhancing the stability of these benchmarks and maintaining their diversity and scope. to address these issues, we propose a new benchmark named stabletoolbench, which incorporates a virtual api system and a stable evaluation system. we first build a virtual api system to replace the real one. as a start, we build a caching system to store the outputs of api calls. this approach ensures the stability and reproducibility of api behaviours. given the limited number of benchmark questions, our caching system can cover a significant number of api call scenarios. however, relying solely on a cache is insufficient because many apis remain unavailable. to resolve this problem, we use large language models (llms) to simulate the behaviours of these apis. specifically, we feed the documentation and few-shot real api calls if available in the cache to llms and ask llms to mock the behaviour of the apis given a request. as a result, users can always get responses from apis in an indistinguishable way as long as the llms are accessible. on the whole, our system first tries to find a hit in the cache. unless there is a cache miss and a real api call is not received, the simulated server will be used. we then improve the evaluation system to make it more stable. we design two metrics (i.e., sopr and sowr) after judging solvable tasks and replace all the automatic evaluators with gpt-4 to mitigate the randomness and indistinguishability during evaluation. experiments demonstrate that our virtual api system, when combined with the improved evaluation system, can provide stable evaluation against api modifications. furthermore, our system exhibits significant reliability in terms of realism, diversity, and documentation following accuracy. the main contributions of our work are summarised as follows: ## stability analysis on toolbench in this section, we initiate a comprehensive analysis to reveal the stability of established tool benchmarks, using toolbench @xcite as a case study. we examine the stability of tool-bench by investigating three key dimensions: performance, evaluation, and api status. ## stabletoolbench considering that stability is a crucial feature of benchmarking, in this paper, we specifically design a virtual api server and stable evaluation system to improve the stability based on toolbench, and propose a new benchmark, named stabletoolbench. ## 11148 ## conclusion in this paper, we propose stabletoolbench, a benchmark developed to enhance the stability of toolbench. our analysis identified instability issues in the evaluation processes of toolbench and api status, causing variability in model performance assessments. to address this, we implement a caching system for consistent data availability. we also replace the real api server with an llm-simulated virtual server for reliable api behaviour simulation. experiments show that sta-bletoolbench significantly improves the stability of model performance evaluations, with the simulated apis offering realism and the caching system contributing greatly to the enhanced stability of the benchmark.