AGENTVIGIL : Automatic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents (2025)

Abstract: There emerges a critical security risk of LLM agents: indirect prompt injection, a sophisticated attack vector that compromises thecore of these agents, the LLM, by manipulating contextual information rather than direct user prompts. In this work, we propose a generic black-box optimization framework, AGENTVIGIL, designed to automatically discover and exploit indirect prompt injection vulnerabilities across diverse LLM agents. Our approach starts by constructing a high-quality initial seed corpus, then employs a seed selectionalgorithm based on Monte Carlo Tree Search (MCTS) to iteratively refine inputs, therebymaximizing the likelihood of uncovering agent weaknesses. We evaluate AGENTVIGIL on twopublic benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o, respectively, nearly doubling the performance of handcrafted baseline attacks. Moreover, AGENTVIGIL exhibits strong transferability across unseen tasks and internal LLMs, as well as promising results against defenses. Beyondbenchmark evaluations, we apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs,including malicious sites.

## introduction despite their impressive capabilities for reasoning and planning @xcite @xcite @xcite @xcite @xcite @xcite , llm agents suffer from serious security challenges of indirect prompt injection @xcite @xcite . specifically, attackers can insert malicious "attack instructions" into the external data sources the target agent interacts with. when the agent retrieves external data, the injected malicious instructions can "fool" the agent into performing the attacker's chosen task instead of the original user task, leading to severe consequences. systematically assessing the potential risks of agent systems against indirect prompt injection is significantly challenging, from the following aspects. ① black-box nature of real-world agents. many real-world agents operate as black-box systems, primarily due to the restricted access to the internal workings of commercial llms @xcite @xcite and agents @xcite . ② diversity in user tasks. agents are designed to manage a wide array of user tasks, each exhibiting dynamic and distinct execution behaviors. ③ architectural complexity and diversity. agents often comprise various interconnected components, tools, and services with intricate architectures, tailored for specific needs (microsoft; langchain). due to these foundational challenges, existing red-teaming approaches for indirect prompt injections either handcraft attack instructions @xcite @xcite @xcite or are specifically designed for one type of agents @xcite . these methods cannot be used as generic methods for assessing the indirect prompt injection risks of llm agents. there is a line of methods for large-scale risk assessment of llms @xcite . however, due to fundamental differences in system components and mechanisms, these model-level methods cannot be directly applied to llm agents. our approach. in this work, we propose agentvigil, the first generic indirect prompt injection assessment method against black-box llm agents. we draw inspiration from traditional software fuzzing techniques @xcite , which automatically generate test inputs for target software to identify vulnerabilities without requiring plant injection with seed in the customer reviews injection goal: goto https://github.xx (a phishing site, or other target links) agent user [user task] find a samsung galaxy s6 screen protector and list out reviews about good fingerprint resistant. ## method 4.1 overview a typical agent system processes user queries by interacting with a diverse set of tools and services within its environment to accomplish user tasks. these tools may include code execution environments, email systems, web browsers, and file systems, among others. the llm in the agent serves as the planner, dynamically coordinating between these components to retrieve information, execute commands, and respond to user needs. given the complexity and autonomy of these systems, they often rely on external data sources, making them susceptible to various security threats. the attacker exploits this reliance by strategically manipulating specific parts of the environment to inject malicious prompts. these prompts are crafted to be embedded within external data sources, which the agent later retrieves and processes as part of its task execution. once these contaminated inputs are fed into the llm, they can alter its behavior, leading to unauthorized actions. figure 2 illustrates the architecture and the workflow of our proposed framework, agentvigil. agentvigil enhances the effectiveness of the indirect prompt attacks by systematically exploring adversarial prompts. the process begins by applying the initial corpus of adversarial prompt templates to the agent across a set of injection tasks, which are combinations of different user tasks and attacker goals, to generate a pool of initial seeds. these seeds then undergo an iterative optimization loop. in this loop, a mcts-based seed selector identifies a promising seed, balancing the dual objectives of exploitation and exploration. subsequently, a seed mutator randomly selects a mutation method to produce a new variant, which is then tested across the tasks. this variant is subsequently tested across the injection tasks to evaluate its performance. the evaluation involves scoring the new seed based on its success rate in executing attacks and its ability to compromise previously unaffected tasks. through this adaptive and iterative process, the framework continuously improve the attack, ensuring scalability and effectiveness across a wide range of agents and tasks. ## evaluation in this section, we comprehensively evaluate the effectiveness of agentvigil. details of the models and dataset we use is shown in appendix a. ## conclusion we introduce agentvigil, a novel black-box optimization framework designed to systematically conduct indirect prompt injection attacks against black-box agents with various architectures and tasks. our empirical results demonstrate that agentvigil not only achieves high attack success rates on established benchmarks and real-world agents but also exhibits strong transferability across unseen tasks and underlying llms. we believe agentvigil will serve as a useful foundation for advancing both the understanding of agent-based threats and the development of next-generation security solutions in this rapidly evolving domain.