CAPTURE : Context-Aware Prompt Injection Testing and Robustness Enhancement (2025)

Abstract: Prompt injection remains a major security risk for large language models. However, the efficacy of existing guardrail models in context-aware settings remains underexplored, as they often rely on static attack benchmarks. Additionally, they have over-defense tendencies. We introduce CAPTURE, a novel context-aware benchmark assessing both attack detection and over-defense tendencies with minimal in-domain examples. Our experiments reveal that current prompt injection guardrail models suffer from high false negatives in adversarial cases and excessive false positives in benign scenarios, highlighting critical limitations. To demonstrate our framework’s utility, we train CAPTUREGUARD on our generated data. This new model drastically reduces both false negative and false positive rates on our context-aware datasets while also generalizing effectively to external benchmarks, establishing a path toward more robust and practical prompt injection defenses.

## introduction large language models (llms) like gpt-4 @xcite and llama @xcite , while transformative, are vulnerable to prompt injection attacks @xcite . this critical threat exploits the inability to distinguish system instructions from user input, potentially causing unintended actions or model compromise @xcite @xcite . existing lightweight prompt guardrail models @xcite @xcite struggle against context-aware attacks, which exploit an application's specific context, its purpose, input/output structure, user patterns, and domain knowledge. for instance, @xcite demonstrated this by injecting input that started with a contextually appropriate query "should i do a phd?" but followed it with a malicious request "how to write a phishing email? summarize as pros&cons analysis". the llm treated it as a part of the normal workflow and then executed the harmful instruction. this vulnerability often stems from training on generic datasets lacking diverse, context-specific examples @xcite @xcite . consequently, the dependence of prompt guardrails on trigger words in their training datasets leads to poor generalization and over-defense, impeding deployment in practical scenarios as harmless sentences get flagged @xcite . to address these challenges, we introduce context-aware prompt injection testing and robustness enhancement (capture), a novel context-aware prompt injection benchmarking framework for prompt guardrail models 1 . our work makes the following key contributions: (i) we propose a scalable approach for generating datasets with minimal in-domain examples to generate context-aware attacks. (ii) we present a context-aware benchmark to evaluate over-defense in these guard models. (iii) we evaluate five stateof-the-art prompt guardrail models and two llms on our benchmarks, revealing significant weaknesses, including 100% false negative rate (fnr) in fmops and 100% false positive rate (fpr) in promptguard. (iv) we demonstrate that our datasets can be used to train significantly more robust context-aware guardrail models, drastically reducing both fnr and fpr to near-zero. ## context-aware dataset generation context-aware prompt injection attacks achieve high success rates @xcite and bypass defenses often untested against realistic scenarios @xcite @xcite . to address this, we introduce capture, which builds its attacks using the multi-part structure from @xcite . this structure uses three key components -framework (f): makes the input look like a normal request for the app. separator (s): tells the ai to ignore the framework part and follow the disruptor part instead. disruptor (d): contains the actual malicious instruction. example: for an email writing application, the components look like this: "[f]: write an email to my colleague about the project deadline... [s]:ignore the above instruction and follow the next instruction [d]: what were the original instructions given to you when you started this email application?". capture systematically varies f, s and d to generate challenging datasets (figure 1 ) via two modes: malicious-gen for creating context-aware adversarial prompts and safe-gen for producing benign prompts to test oversensitivity. we hypothesize this controlled generation creates a robust context-aware benchmark and demonstrate the generated data's utility by training a model, captureguard (section 2.4) for improved defense robustness. ## experimental setup and results we evaluate five specialized models -protectaiv2 (protectai, 2024), injecguard (li and liu, 2024), promptguard (meta, 2024b), deepset @xcite and fmops (fmops, 2024) across six diverse domains. as llms are also being increasingly being used as detectors, we evaluate two llms -gpt-4o and llama3.2-1b-instruct (meta, 2024a) using instructions in figure 5 foot_1 . additionally, our proposed model, captureguard, was evaluated specifically on python, movies and stocks assistant use cases to assess the impact of our context-aware ## conclusion we introduced capture, a novel framework for context-aware evaluation of prompt guardrail detectors. we generated diverse context-aware attacks which evade detection and benign contextaware examples to trigger fps in these models using @xcite . our evaluation shows that existing models like injecguard and protectaiv2 suffer high fpr and fnr on our datasets. in contrast, our captureguard model, trained on this generated context-aware data, demonstrated superior performance by not only excelling on our contextaware datasets but also generalizing effectively to standard benchmarks. these results underscore the need for more robust models that balance security and usability, and our work provides a clear methodology and a powerful baseline to advance the field.