T ool B e H onest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models (2024)

Abstract: Tool-augmented large language models (LLMs) are rapidly being integrated into real-world applications. Due to the lack of benchmarks, the community has yet to fully understand the hallucination issues within these models. To address this challenge, we introduce a comprehensive diagnostic benchmark, ToolBH. Specifically, we assess the LLM’s hallucinations through two perspectives: depth and breadth. In terms of depth, we propose a multi-level diagnostic process, including (1) solvability detection, (2) solution planning, and (3) missing-tool analysis. For breadth, we consider three scenarios based on the characteristics of the toolset: missing necessary tools, potential tools, and limited functionality tools. Furthermore, we developed seven tasks and collected 700 evaluation samples through multiple rounds of manual annotation. The results show the significant challenges presented by the ToolBH benchmark. The current advanced models Gemini-1.5-Pro and GPT-4o only achieve total scores of 45.3 and 37.0, respectively, on a scale of 100. In this benchmark, larger model parameters do not guarantee better performance; the training data and response strategies also play crucial roles in tool-enhanced LLM scenarios. Our diagnostic analysis indicates that the primary reason for model errors lies in assessing task solvability. Additionally, open-weight models suffer from performance drops with verbose replies, whereas proprietary models excel with longer reasoning.

## introduction to replicate human intelligence in artificial general intelligence (agi), recent studies suggest that a promising solution is enabling large language models (llms) to use tools to handle diverse complex scenarios @xcite . for instance, figure 1 : hallucinations (red) and expectations (blue) of the llm's response to the use of the task for tools that do not have a correct answer. the problem example is taken from the agentboard @xcite tool-query dataset, tested with chatgpt @xcite . wrong tool reflects a common situation where llm uses a provided wrong tool as the final answer; non-existent tool, on the other hand, is an example of hallucination. however, current benchmarks for tool-based tasks presuppose the provision of complete and usable tools by users, primarily focusing on evaluating how llms utilize these tools @xcite @xcite . for example, agentbench @xcite and toolbench @xcite aim to introduce a broad array of tasks, and agentboard @xcite evaluate the multi-step tool planning abilities of these models. although metatool @xcite considers incomplete lists of tools provided by users, it still focuses solely on determining whether a task can be completed. therefore, to ensure a comprehensive evaluation of the various hallucinatory phenomena in tool usage, we develop the toolbh benchmark. it is a multi-level hallucination diagnostic benchmark for tool-augmented llms. inspired by medical diagnostic procedures @xcite and existing hallucination benchmarks @xcite , toolbh systematically assesses hallucinations from perspectives of depth and breadth (details in sec. 3). in the assessment of hallucination depth, we introduce a multi-level diagnostic procedure framework that incorporates (1) solvability detection, (2) solution planning, and (3) missing-tool analysis. the solvability detection task assesses whether the user query can be addressed with the given toolset. subsequently, the solution planning task requires the llm to propose a plan of tools tailored to address the task sequentially. in the end, the missing-tool analysis task builds upon its predecessor by requiring the llm to delineate how the tools address the specific demands in each step. for tasks considered unsolvable, the llm is tasked with identifying and elaborating on the functionalities of the missing tools, thereby offering insights to aid in completing the toolkit. on the side of breadth, we consider three critical scenarios from the perspective of toolset features: missing necessary tools, potential tools, and limited functionality tools. furthermore, we design seven tasks across these scenarios, including iterative calls and optimal tool selection tasks. finally, we collect 700 samples (user query, tool set, solution) across 7 tasks through multiple rounds of human annotations and establish an evaluation metric for each diagnostic task (details in sec. 3.2). we conduct experiments on 14 llms using the toolbh benchmark, which includes 7 proprietary and 7 open-weight models. our analysis reveals that despite some open-weight models achieving competitive performance compared to proprietary counterparts in general tasks @xcite , they exhibit substantial gaps in unsolvable scenarios presented by the toolbh benchmark. notably, the best-performing open-weight model, llama-3-70b, still trails behind the proprietary models, with only 32% of the total score of gemini-1.5-pro @xcite and 40% of gpt-4o (openai, 2024), indicating considerable room for improvement in open-weight models. a detailed error analysis shows a variety of error patterns. proprietary models tend to exhibit fewer instances of using non-existent tools, suggesting better tool recognition capabilities. however, they also showed higher rates of instrumental reasoning errors, reflecting problems comprehending the task's solution sequence and reasoning logic. on the other hand, open-weight models suffer more from solvability hallucinations, often misjudging the complexity and feasibility of tasks. this highlights a critical area for enhancement in model reasoning. our findings demonstrate the significance of the toolbh benchmark in identifying and addressing deficiencies in llms. the benchmark provides detailed analysis and valuable insights into specific error patterns under unsolvable llm tool usage scenarios, offering guidance for future research. ## conclusions this paper introduces the toolbh benchmark for diagnosing hallucinations in tool-augmented llms. specifically, we evaluate hallucination from the perspective of depth (multi-level evaluation framework) and breadth (three scenarios to induce hallucinations). our results indicate that llms are prone to hallucinations even in straightforward tool-using tasks under unsolvable conditions. the impact of the quantity and quality of the training data, response strategy, and long-context reasoning consistency on the performance should be addressed. error analysis reveals that llms primarily struggle with solvability hallucination, characterized by their inability to comprehend user queries and accurately interpret the tools and their functions. this often leads these models to mistakenly conclude that they can solve tasks. our toolbh marks a pivotal advancement in diagnosing hallucinations, providing resources for exploring the real-world performance of tool-augmented llms.