I njec A gent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024)

Abstract: Recent work has embodied LLMs as agents, allowing them to access tools, perform actions, and interact with external content (e.g., emails or websites). However, external content introduces the risk of indirect prompt injection (IPI) attacks, where malicious instructions are embedded within the content processed by LLMs, aiming to manipulate these agents into executing detrimental actions against users. Given the potentially severe consequences of such attacks, establishing benchmarks to assess and mitigate these risks is imperative.In this work, we introduce InjecAgent, a benchmark designed to assess the vulnerability of tool-integrated LLM agents to IPI attacks. InjecAgent comprises 1,054 test cases covering 17 different user tools and 62 attacker tools. We categorize attack intentions into two primary types: direct harm to users and exfiltration of private data. We conduct a comprehensive evaluation of 30 different LLM agents and show that agents are vulnerable to IPI attacks, with ReAct-prompted GPT-4 vulnerable to attacks 24% of the time. Further investigation into an enhanced setting, where the attacker instructions are reinforced with a hacking prompt, shows additional increases in success rates. Our findings raise questions about the widespread deployment of LLM Agents.

## introduction large language models (llms) @xcite @xcite are increasingly being incorporated into agent frameworks (significant gravitas; openai, 2023a; nvidia, 2024), where they can perform actions via tools. increasingly, these agents are being deployed in settings where they access users' personal data @xcite and perform actions in the realworld @xcite . however, these features introduce security risks. attackers can steal sensitive information through messaging tools and cause direct financial and physical harm by executing unauthorized transactions or manipulating smart home devices. they can achieve this by injecting malicious content into the information retrieved by agents @xcite @xcite , which is often fed back to the llm as part of the context. such attacks are known as indirect prompt injection (ipi) attacks @xcite . due to the low technical requirements needed to carry out such attacks and the significant consequences they can cause, it is important to systematically evaluate the vulnerabilities of llm agents to these types of attacks. in this paper, we present the first benchmark for assessing indirect prompt injection in tool-integrated llm agents, injeca-gent. the benchmark comprises 1,054 test cases spanning multiple domains such as finance, smart home devices, email, and more. one example of a test case is a user requesting doctor reviews through a health application, where an attacker's review tries to schedule an appointment without user consent, risking privacy violations and financial losses (figure 1 ). in this example, the user first initiates the instruction to the agent and the agent performs an action to retrieve the review. the tool then returns a review written by an attacker, which is actually a malicious instruction to schedule an appointment with a doctor. if the agent proceeds to execute the tool to fulfill the attacker instruction, the attack succeeds, resulting in an unauthorized appointment. conversely, if the agent responds to the user without executing the malicious command, the attack fails. our dataset includes 17 different user instructions, each using a distinct tool to retrieve external content that is susceptible to modification by attackers. examples of this content include product reviews, shared notes, websites, emails, among others. the dataset also covers 62 different attacker instructions, with each using distinct tools to perform harmful actions towards users. we categorize these attacks into two main types: direct harm attacks, which involve executing tools that can cause immediate harm to the user, such as money transactions and home device manipulation; and data stealing attacks, which entail stealing the user's personal data and sending it to the attacker. we summarize these attack categories in we quantitatively evaluate various types of llm agents, including prompted agents which incorporate an llm prompted by react prompts @xcite , and fine-tuned agents which are finetuned llms over tool-calling examples. our results show that the prompted agents are vulnerable to attacks. for example, an agent based on gpt-4 has an attack success rate of 24%. the incorporation of the "hacking prompt" further increases its success rate to 47%. additionally, we observed that fine-tuned agents are more resilient to such attacks compared to prompted agents. the fine-tuned gpt-4, for example, showed a significantly lower attack success rate of just 7.1%. in summary, our paper makes the following contributions: @xcite we are the first to formalize ipi attacks on tool-integrated llm agents; @xcite we introduce injecagent, a novel and realistic benchmark that covers various domains and can serve as the standard for evaluating an agent's resilience to ipi attacks; @xcite we evaluate 30 llm agents us-ing injecagent and reveal that most agents are vulnerable to ipi attacks. ## benchmark construction to build the injecagent benchmark, we first formalize ipi attacks of llm agents (section 2.1). we then introduce the process of generating test cases with the assistance of gpt-4 (section 2.2). finally, we discuss our approach for evaluating different llm agents using these test cases (section 2.3). ## experiments we quantitatively evaluated various llm agents using our benchmark injecagent to assess their resilience against ipi attacks. ## analysis in this section, we investigate the following questions: (1) does the user case or attacker case exhibit a stronger correlation with the success of an attack? (section 4.1) ( 2 ) what kinds of user cases are more vulnerable? (section 4.2) (3) how does the enhanced setting affect the agents' sensitivity to the attacker instructions? (section 4.3) to ensure the validity of the conclusions, our analysis is limited to agents demonstrating a valid rate above 50%, as shown in ## related work llm agents. a central aim in artificial intelligence has been the creation of intelligent agents @xcite @xcite . the advent of llm agents, which combine powerful llms with a range of tools, represents a significant stride in this direction @xcite . these agents are enabled through two primary methodologies: (1) using in-context learning to equip llms with the capability to use various tools, like react (yao et al., 2022), mindact @xcite and autogpt (significant gravitas); (2) finetuning llms with function calling examples, seen in toolformer @xcite , gorilla @xcite , toolllm @xcite , and ope-nai models that support function calling @xcite . in this paper, we assess the security of both types of llm agents when confronted with indirect prompt injection attacks. prompt injection. prompt injection (pi) is the attack of maliciously inserting text with the intent of misaligning an llm @xcite @xcite . currently, pi attacks can be categorized into two main types: direct prompt injection (dpi) and indirect prompt injection (ipi). dpi @xcite @xcite @xcite ) involves a malicious user injecting harmful prompts directly into the inputs of a language model, with the aim of goal hijacking or prompt leaking. on the other hand, ipi @xcite ) entails an attacker injecting harmful prompts into external content that an llm is expected to retrieve, with the goal of diverting benign user instructions. notably, real-world instances of ipi attacks on ope-nai plugins have been documented, leading to various consequences, such as phishing link insertion, conversational history exfiltration, github code theft, and more @xcite @xcite . however, a comprehensive analysis of ipi attacks on llm agents has remained unexplored. concurrent research @xcite benchmarks ipi attacks on llms but primarily focuses on simulated llm-integrated applications under limited scenarios. it includes only five application types: email qa, web qa, table qa, summarization, and code qa, with the most detrimental text attack intentions being scams/malware distribution. in contrast, our work delves into tool-integrated llm agents, examining their behavior and vulnerabilities in real-world scenarios. our benchmark covers a wide range of scenarios, with 17 different user cases and 62 different attacker cases. ## conclusion in this work, we introduce the first benchmark for indirect prompt injection attacks targeting toolintegrated llm agents, named injecagent. we evaluate 30 different llm agents and conduct a comprehensive analysis. our findings demonstrate the feasibility of manipulating these agents into performing harmful actions toward users by merely injecting malicious instructions into external content. this work underscores the risks associated with such attacks, given their ease of deployment and the severity of potential outcomes. furthermore, it emphasizes the urgent need for and offers guidance on implementing strategies to safeguard against these attacks.