S afe T ool B ench: Pioneering a Prospective Benchmark to Evaluating Tool Utilization Safety in LLM s (2025)

Abstract: Large language models (LLMs) have exhibited great performance in autonomously calling various tools in external environments, leading to better problems solving and task automation capabilities. However, these external tools also amplify potential risks such as financial loss or privacy leaking with ambiguous or malicious user instructions. Compared to previous studies, which mainly assess the safety awareness of LLMs after obtaining the tool execution results (i.e., retrospective evaluation), this paper focuses on prospective ways to assess the safety of LLM tool utilization, aiming to avoid irreversible harm caused by directly executing tools. To this end, we propose SafeToolBench, the first benchmark to comprehensively assess tool utilization security in a prospective manner, covering malicious user instructions and diverse practical toolsets. Additionally, we propose a novel framework, SafeInstructTool, which aims to enhance LLMs’ awareness of tool utilization security through three perspectives (i.e., User Instruction, Tool Itself, and Joint Instruction-Tool), leading to nine detailed dimensions in total. We experiment with four LLMs using different methods, revealing that existing approaches fail to fully capture all risks in tool utilization. In contrast, our framework significantly enhances LLMs’ self-awareness, enabling a more safer and trustworthy tool utilization.

## introduction recently, tool learning @xcite has emerged as a crucial and effective way to empower large language models (llms) to interact with the external physical world. this capability allows llms to overcome their own inherent limitations such as hallucination (huang † equal contribution. ‡ corresponding author. et al., 2025) and outdated information @xcite while also providing domain-specific services through function calls @xcite @xcite . therefore, many studies focus on evaluating and enhancing the capabilities of llms to plan and utilize tools in different contexts such as api-bank @xcite , appbench @xcite , toolbench @xcite and dialogtool @xcite , which simulate hundreds and even thousands of real-world api calls to complete user instructions. however, it is worth noting that the more tools from the external environment integrated into llms, the more risks it brings @xcite . on the one hand, most of the existing methods usually directly follow user instructions and execute corresponding tools as required without any verification and risk management. thus, some high-risk tools, such as money transfers or email, maybe maliciously attacked, resulting in users' money lost or privacy leakage @xcite . on the other hand, there is another line of work investigating the security risk awareness of llms at the multiple actions with interactive environment @xcite @xcite . such as toolemu @xcite is proposed to emulate the tool execution and testing against a diverse range of tools and scenarios, followed by r-judge @xcite and agentsafetybench @xcite . despite these works examining tool failures and quantifying associated risks, it is usually conducted in the retrospective way by assessing subsequent consequences after the tool execution. nevertheless, most high-risk tool executions are irreversible such as money transfers, making previous studies impractical and still risky when applied in real-world. to overcome these limitations, we first present safetoolbench, a benchmark encompassing 1,200 adversarial user instructions spanning 16 realworld domains (i.e., healthcare, finance, social media) and categorizing risks into four critical types (i.e., privacy leak, property damage, physical injury, and bias & offensiveness). unlike retrospective evaluation methods, safetoolbench simulates scenarios where unsafe tool execution could trigger irreversible harm (e.g., unauthorized fund transfers), enabling prospective safety assessment before actions are taken. furthermore, while existing work predominantly focuses on risks within user instructions, we argue that risk vulnerabilities also stem from tools themselves and their interactions with instructions. as illustrated in figure 1 , given the user instruction "please download my medical report from january 26 and send it to my colleague group." and corresponding two api calls, there are three distinct aspects of security risks: (1) user instructions, (2) tool calls, and (3) their joint interactions (as illustrated in the bottom part). to holistically address these gaps, we further propose safeinstructtool, the first framework to evaluate risks across these three perspectives from nine dimensions: user instruction perspective (data sensitivity, harmfulness of the instruction, urgency of the instruction, frequency of tool utilization in the instruction), tool itself perspective (key sensitivity, type of operation, impact scope of the operation) and joint instruction-tool perspective (alignment between instruction and tool, value sensitivity). thus, it can enhance llms' awareness of tool utilization safety, leading to more safer and trustworthy language agents. in summary, our contributions are as follows: ## safetoolbench construction in this section, we begin by defining the task and then provide a detailed pipeline to collect safetool-bench efficiently and effectively. ## framework of safeinstructtool to prospectively identify all risks from different perspectives, i.e., user instructions, tool utilization, and joint instruction-tool, we introduce safe-instructtool, beginning with a comprehensive definition of each perspective, followed by a detailed explanation of our safety awareness scoring mechanism. ## analysis in this section, we provide a comprehensive analysis designed to address both research questions. rq1: is it necessary to evaluate the risks of agents executing user instructions from three perspectives (user instruction perspective, tool itself perspective, combined user instruction and tool perspective)? (sec 6.1) rq2: what is the major bottleneck of current llms? (sec 6.2, 6.3). ## conclusion in this paper, we introduce safetoolbench, which comprehensively evaluates the safety of tool utiliza-tion in a prospective manner, covering malicious user instructions and diverse practical toolsets. furthermore, we present safeinstructtool, a novel framework that aims to enhance llms' awareness of tool utilization security from three key perspectives -user instruction, tool itself, and joint instruction-tool -covering nine detailed risk dimensions. we conduct comprehensive experiments and analysis, and the results show existing llms still have a poor self-awareness towards more safer and trustworthy tool utilization.