Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack (2025)

Abstract: Batch prompting, which combines a batch of multiple queries sharing the same context in one inference, has emerged as a promising solution to reduce inference costs. However, our study reveals a significant security vulnerability in batch prompting: malicious users can inject attack instructions into a batch, leading to unwanted interference across all queries, which can result in the inclusion of harmful content, such as phishing links, or the disruption of logical reasoning. In this paper, we construct BatchSafeBench, a comprehensive benchmark comprising 150 attack instructions of two types and 8k batch instances, to study the batch prompting vulnerability systematically. Our evaluation of both closed-source and open-weight LLMs demonstrates that all LLMs are susceptible to batch prompting attacks. We then explore multiple defending approaches. While the prompting-based defense shows limited effectiveness for smaller LLMs, the probing-based approach achieves about 95% accuracy in detecting attacks. Additionally, we perform a mechanistic analysis to understand the attack and identify attention heads that are responsible for it.

## introduction the increasing complexity of large language models (llms) has made efficient and affordable inference a critical requirement for their large-scale deployment. batch prompting has emerged recently as a promising solution to meet this need @xcite . by combining queries sharing the same prefix (e.g., task demonstrations, conditional context, etc.) into a single batch and feeding them to an llm in one inference, batch prompting saves the compute from repetitive infer- ## defending llms against batch prompting attacks we explored two approaches to defending the llms against the batch prompting attack. ## why does batch attack happen? our experiments in section 4.2 show that the batch prompting attack takes effect in the neural representations of the batch input. in this section, we seek to mechanistically understand how the interference between queries happens inside an llm. inspired by prior work @xcite @xcite , we hypothesize that there could similarly be attention heads that are responsible for the batch prompt attack. we study this hypothesis for the content attack using the llama-3.2-3b-instruct model. specifically, we followed through the analysis, we identified a subset of attention heads (e.g., l12h3, l15h19, and l13h17) that exhibit a strong causal effect on the success of the batch prompting attack. we dub these heads as "interference heads" in the context of batch prompting attack. these interference heads were found to consistently contribute to the batch prompting attacks across instructions and datasets. we examine their attention patterns and observe that these heads mostly attend to only the attack instructions. one example of l12h3 in ## related work batch prompting recent advances in prompting strategies have explored grouping multiple input samples into a single api call to reduce inference token usage and latency. @xcite introduces batch prompting as an efficient method that processes several samples simultaneously, leading to nearly inverse-linear cost reductions with increasing batch size. in a similar vein, @xcite not only employs batched inference but also augments it with batch permutation and ensembling to overcome performance degradation from naively increasing batch size. as a straightforward method, batch prompting is widely applied. @xcite leverages batch prompting within their reportgpt system to generate verifiable table-to-text outputs efficiently. @xcite demonstrates that batching multiple queries in many-shot in-context learning for multimodal foundation models not only cuts per-query latency and cost but can also yield performance gains in zero-shot settings. moreover, @xcite proposes a cost-effective framework that optimizes task decomposition and employs batch prompting for medical dialogue summary. although batch prompting has been widely adopted in domain-specific applications, the potential security issues have not been investigated. we first provide an in-depth discussion with an empirical evaluation and systematic analysis. prompt injection attacks prompt injection, which manipulates the prompt by appending malicious content to trigger unintended model behaviors, is a critical security vulnerability for llms @xcite . such attacks have been demonstrated through both human-designed prompts @xcite @xcite or and automated generation of adversarial inputs @xcite . these prompting injection works are based on an assumption that users of llms have malicious intent. however, several studies have indicated that in llm application scenarios of the real world, even users without malicious intent may still be exposed to potential security threats. in the retrieval-augmented llm, attackers may achieve attacks by contaminating the text to be retrieved @xcite . in the in-context learning scenario, @xcite propose that the malicious content can be in the demonstration examples to produce incorrect reasoning chains. besides, in llm-powered web agent scenarios, a malicious attack could be injected into the websites @xcite . in our paper, we focus on the prompting injection attack in the batch prompting scenario. our experiments demonstrate that current llms still exhibit significant limitations in defending the prompting injection in this scenario. ## conclusion in this work, we investigated the security risks associated with batch prompting. through the introduction of a comprehensive benchmark dataset batchsafebench, we systematically evaluated the llms and demonstrated that even state-ofthe-art models like gpt-4o and deepseek-r1 are not immune. to address these risks, we explored a prompting-based approach, which showed limited effectiveness, and a probing-based detection method, which achieves a high accuracy in identify-ing attacks. additionally, our mechanistic analysis uncovered a key role of "interference heads". our work underscores the importance of developing robust safeguards for batch prompting.