T ool S word: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages (2024)

Abstract: Tool learning is widely acknowledged as a foundational approach or deploying large language models (LLMs) in real-world scenarios. While current research primarily emphasizes leveraging tools to augment LLMs, it frequently neglects emerging safety considerations tied to their application. To fill this gap, we present ToolSword , a comprehensive framework dedicated to meticulously investigating safety issues linked to LLMs in tool learning. Specifically, ToolSword delineates six safety scenarios for LLMs in tool learning, encompassing malicious queries and jailbreak attacks in the input stage, noisy misdirection and risky cues in the execution stage, and harmful feedback and error conflicts in the output stage. Experiments conducted on 11 open-source and closed-source LLMs reveal enduring safety challenges in tool learning, such as handling harmful queries, employing risky tools, and delivering detrimental feedback, which even GPT-4 is susceptible to. Moreover, we conduct further studies with the aim of fostering research on tool learning safety. The data will be released upon acceptance of the paper.

## introduction recently, tool learning has garnered significant attention as a potent approach for seamlessly integrating large language models (llms) @xcite @xcite into real-world applications @xcite @xcite . the tool learning process for llms can be delineated into three distinct stages: input, execution, and output. more precisely, when a user submits a request, llms scrutinize the user's intent, choose appropriate tools to engage with the external environment. upon receiving feedback from the environment, llms structure the pertinent information to provide a response to the user's initial query. existing research primarily concentrates on enhancing llms capabilities through tool utilization. one proposed approach involves fine-tuning the base model by generating numerous tool usage trajectories for a specific set of tools @xcite @xcite . this approach aids llms in swiftly grasping the functionality of various tools and mastering their utilization for problem-solving. another strategy aims to bolster the model's generalization skills by devising prompts that instruct llms to read tool descriptions and employ external tools as necessary @xcite @xcite . however, these studies overlook the fact that tool learning also introduces new safety concerns. as illustrated in figure 1 , in standard dialogues, llms can recognize and refuse to provide assistance when users enter unsafe queries. in contrast, in the context of tool learning, the safety alignment mechanism may be compromised. consequently, llms may provide corresponding answers to unsafe queries by utilizing relevant tools. furthermore, the selection of tools by llms may be influenced by malicious noise @xcite . therefore, there is an urgent need for a comprehensive analysis of the current safety challenges faced by llms in the realm of tool learning to facilitate research aimed at their development. to fill this gap, we introduce toolsword, a comprehensive framework crafted for unveiling the safety issues of llms throughout the tool learning process. toolsword encompasses six safety scenarios that llms encounter in tool learning, encompassing malicious queries and jailbreak attacks in the input stage, noisy misdirection and risky cues in the execution stage, as well as harmful feedback and error conflicts in the output stage. through an analysis of llms performance within these safety scenarios, we can gain insight into how they manage various safety challenges in tool learning at a granular level. leveraging toolsword, we analyze 11 opensource and closed-source llms equipped with robust tool learning capabilities. our findings reveal that current llms frequently encounter safety issues across all stages of tool learning, leading to significant safety risks such as responding to harmful queries, invoking risky tools, and providing detrimental feedback, despite these issues being easily discernible by humans. even the most advanced llms, such as @xcite , are not immune to these challenges. moreover, our futher studies indicate that llms can demonstrate performance comparable to humans in tool learning environments devoid of safety concerns. hence, enhancing safety measures is essential to drive the practical application of llms. we hope that our findings will contribute to advancing research in the domain of tool learning safety. the main contributions of our work are summarized as follows: ## toolsword as depicted in figure 2 , toolsword offers a comprehensive analysis of the safety challenges encountered by llms during tool learning, spanning three distinct stages: input, execution, and output. within each stage, we have devised two safety scenarios, providing a thorough exploration of the real-world situations llms may encounter while utilizing the tool, which are elaborated upon in the subsequent sections. foot_0 ## further studies upon synthesizing the performance of individual llms across various scenarios throughout the three stages, several additional points merit attention. increasing the size of a model doesn't necessarily enhance its safety in tool learning. although llms typically demonstrate enhanced capability as their size increases @xcite , our experiments indicate that this trend does not extend to safety in tool learning. for instance, within the qwen-chat family of models, while safety might improve with larger sizes in certain input scenarios, this improvement isn't consistent across all scenarios, including execution and output stages. there's even a possibility of safety diminishing with larger models. in the context of the rc scenario, transitioning from 7b to 72b increases the percentage of llms opting for unsafe tools from 54.55% to 87.27%. this underscores that current safety mechanisms primarily address rejecting unsafe inputs without adequate consideration for other factors. the limitation in tool learning capability within llms can partially mitigate non-safe behaviors. interestingly, in our experiments, we observe that the smallest chatglm-3-6b demonstrates better safety across most scenarios. upon conducting a thorough analysis of its behavior, we discover that this unexpected improvement is attributed to its limited tool learning ability. for instance, as shown in figure 6 , in the hf scenario, although its ratio of unsafe output is only 65.45%, a significant 27.28% of instances involve the model halting output after receiving tool feedback. additionally, in 5.45% of cases, the final result notifies the user of execution success without providing a summary, while only 1.82% of cases recognize tool feedback as dangerous content. consequently, while the reduced capability of llms may decrease their unsafe behavior to some extent, it comes at the expense of their usefulness to users. in tool learning scenarios devoid of safety concerns, llms exhibit the potential to outperform humans. while various llms may perform worse than humans in safety scenarios, it is noteworthy that in noise-free tool environments, as illustrated in figure 4 , the tool selection error rate of the gpt family of models consistently matches the human level at 0. furthermore, we conduct assessments on the accuracy of llms when directly presented with various common sense questions in the ec scenario. the results outlined in ## conclusion and future works in this paper, we present toolsword, a framework that thoroughly analyzes the safety issues faced by llms in tool learning across three stages. we evaluate llms at a granular level by crafting two safety scenarios with varying degrees of complexity at each stage. this examination underscores the imperative for future research to bolster the safety alignment mechanisms within llms. we envision several avenues for future research: