From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection (2025)

Abstract: Tool-calling has changed Large Language Model (LLM) applications by integrating external tools, significantly enhancing their functionality across diverse tasks. However, this integration also introduces new security vulnerabilities, particularly in the tool scheduling mechanisms of LLM, which have not been extensively studied. To fill this gap, we present ToolCommander, a novel framework designed to exploit vulnerabilities in LLM tool-calling systems through adversarial tool injection. Our framework employs a well-designed two-stage attack strategy. Firstly, it injects malicious tools to collect user queries, then dynamically updates the injected tools based on the stolen information to enhance subsequent attacks. These stages enable ToolCommander to execute privacy theft, launch denial-of-service attacks, and even manipulate business competition by triggering unscheduled tool-calling. Notably, the ASR reaches 91.67% for privacy theft and hits 100% for denial-of-service and unscheduled tool calling in certain cases. Our work demonstrates that these vulnerabilities can lead to severe consequences beyond simple misuse of tool-calling systems, underscoring the urgent need for robust defensive strategies to secure LLM Tool-calling systems.

## introduction recent advancements in large language models, such as gpt-4 @xcite , llama3 @xcite , and qwen2 @xcite , have dramatically changed ai applications by enabling seamless integration with external tools @xcite . this integration, commonly referred to as toolcalling or function calling, allows llm to extend their capabilities beyond text generation, making them more versatile for real-world tasks (jana et al., * these authors contributed equally to this work. however, such flexibility also introduces significant security risks, as malicious actors can inject adversarial tools into these systems, exploiting the tool-calling process in previously unanticipated ways. as illustrated in figure 1 , typical llm toolcalling systems consist of three main components: tool platform: a collection of external tools, each with specific functionalities, input/output formats, descriptions, etc. these tools are designed to handle tasks or queries the language model cannot process independently. the platform is typically flexible, allowing tools to be added or removed over time. retriever: a retrieval mechanism that selects the most relevant tools from the tool platform based on a given user query. the retriever operates by computing a relevance score and returns an ordered subset of tools most related to the query. llm: the core system processes the user query with relevant tools from the retriever and invokes the appropriate tools. in tool-calling systems, the react (reasoning and acting) paradigm plays a crucial role, as the llm must reason about the user query and then act by invoking the tool and incorporating its output into the final response @xcite . such tool integration introduces new and practical attack surfaces, as malicious tools can be injected into the tool platform, exploiting the tool selection process. in addition, unlike traditional rag-based systems @xcite , where the focus is on retrieving relevant documents and generating a single response, tool-calling systems dynamically reason and invoke tools based on an evolving context. this introduces an additional layer of complexity, as the attack must influence the tool-calling decisions during the reasoning process. in particular, existing studies, such as toolsword @xcite , primarily evaluate the general robustness of llm tool-calling systems under benign safety scenarios but do not explore targeted attack strategies that manipulate the tool selection and execution process. our focus is not on problems inherent to the tools themselves, such as noisy descriptions or incorrect outputs-issues not exclusive to tool-calling systems-but on how malicious behavior can undermine the decision-making process of the tool-calling system. additionally, prior works on adversarial attacks, such as jailbreaking or prompt injection attacks @xcite @xcite , primarily focus on general adversarial attacks on llms rather than llm applications. moreover, unlike previous triggerword attacks @xcite , which focus on specific categories and target fixed queries, we develop methods to extend target queries dynamically, thereby enhancing attack performance and adaptability to various contexts. in this work, we propose toolcommander foot_0 , a novel framework that targets security vulnerabilities in llm tool-calling systems through injecting adversarial tools(hereinafter referred to as manipulator tool). toolcommander operates in two stages. at first, it injects privacy theft manipulator tools to gather actual user queries, which are then used to refine the subsequent attacks. in the second stage, manipulator tools manipulate the tool scheduling process by exploiting entry points to interfere with legitimate tools, allowing attackers to control which tools the llm selects. this manipulation enables attacks such as denial-of-service and unscheduled tool-calling, posing risks to users and skewing commercial competition by biasing the llm towards specific tools, disrupting fair marketplace dynamics. toolcommander uniquely targets vulnerabilities specific to tool integration, where tools are retrieved and executed based on user intent. instead of assuming that tools are retrieved or invoked successfully, toolcommander focuses on attacking the entire tool-calling system end-to-end. our approach targets the complete process, from tool retrieval to final output. this strategy offers a thorough understanding of how adversarial tools can disrupt the integrity and functioning of llm tool-calling systems. by introducing toolcommander, we contribute to ensuring the reliability of llm tool-calling systems under adversarial conditions. our research uncovers critical vulnerabilities and offers guidance for developing robust defense strategies, ultimately enhancing the security and trustworthiness of ai applications. the main contributions of our work are summarized as follows: our results demonstrate that toolcommander achieves superior performance, outperforming existing baselines. ## threat model we outline our threat model for the tool-calling system by focusing on the following key aspects: the attacker's objectives, knowledge, capabilities, conditions for a successful attack, and constraints on attack. attacker's objectives the attacker's primary objective is to exploit the llm's decision-making process, compelling it to select and invoke a designated tool for specific target queries. this can result in privacy breaches, denial-of-service (dos), or unscheduled tool calling. attacker's knowledge and capabilities our scenario encompasses three primary components: tool platform, retriever, and llm. we posit the following assumptions: ## conclusion and future works in this work, we explored the vulnerabilities of llm tool-calling systems to malicious tool injection attacks using the toolcommander framework. through comprehensive experiments, we demonstrated that even sophisticated models like gpt and llama3 are susceptible to privacy theft, denial-ofservice, and unscheduled tool-calling attacks when paired with general-purpose retrieval mechanisms. the demonstrated risks have severe implications for real-world llm deployments, particularly in domains that handle sensitive data or missioncritical operations. although integrating tools enhances capabilities, our research indicates that existing implementations, unfortunately, render systems vulnerable to weaponization by malicious actors. future work must prioritize security as a firstclass design principle in tool-augmented llms to enable robust, trustworthy human-ai collaboration. research should explore methods to improve the stealthiness of attacks, such as optimizing valid fields in the tool json schema or designing specific triggers to activate malicious content. additionally, developing advanced optimization techniques to enhance retrieval and invocation success rates could further refine the effectiveness of such attacks. investigating how llms' instruction-following capabilities may inadvertently increase their vulnerability to injection manipulation could provide valuable insights into the underlying mechanisms of this threat. this understanding can help inform the development of robust countermeasures and safeguard ai systems against exploitation.