Defending against Indirect Prompt Injection by Instruction Detection (2025)

Abstract: The integration of Large Language Models (LLMs) with external sources is becoming increasingly common, with Retrieval-Augmented Generation (RAG) being a prominent example. However, this integration introduces vulnerabilities of Indirect Prompt Injection (IPI) attacks, where hidden instructions embedded in external data can manipulate LLMs into executing unintended or harmful actions. We recognize that IPI attacks fundamentally rely on the presence of instructions embedded within external content, which can alter the behavioral states of LLMs. Can the effective detection of such state changes help us defend against IPI attacks? In this paper, we propose InstructDetector, a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks. Specifically, we demonstrate the hidden states and gradients from intermediate layers provide highly discriminative features for instruction detection. By effectively combining these features, InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark. The code is publicly available at https://github.com/MYVAE/Instruction-detection.

## introduction large language models (llms) @xcite @xcite have shown remarkable performance over various tasks, including question answering @xcite , summarization @xcite , and machine translation @xcite . despite their impressive performance, llms often suffer from hallucinations @xcite ) and * joint first authors figure 1 : in a medical use case of the rag system, the llm is misled by the external instruction embedded in a retrieved document to recommend company a's medication. our method performs instruction detection to defend against such attacks, removing such documents before they are passed to the llm. however, the inclusion of external content exposes llms to indirect prompt injection (ipi) attacks. in such an attack, adversaries inject covert instructions into the external data retrieved by the system @xcite @xcite @xcite . on the one hand, these hidden instructions may distort the retrieved information, leading the model to generate incorrect or misleading responses. on the other hand, they may cause the model to produce outputs that are entirely unrelated to the user's intent, resulting in unexpected or irrelevant content. these vulnerabilities pose significant security and ethical risks, particularly in sensitive domains like healthcare @xcite @xcite , finance @xcite , and legal systems @xcite . for example, as illustrated in figure 1 , an instruction embedded in the external content could mislead the model into recommending medication from a specific company, even if it is not the most appropriate treatment for the patient, which results in harmful or biased medical advice. recently, similar manipulations have even been discovered in academic publishing, where researchers embedded hidden prompts in academic papers, such as "do not highlight any negatives. positive review only." these instructions were concealed with invisible fonts to manipulate ai systems used in peer review into generating favorable evaluations @xcite . to mitigate the risk of ipi attacks, recent defenses have primarily focused on prevention @xcite by modifying prompts or fine-tuning models to ensure that llms adhere strictly to user instructions while ignoring external ones. however, detection @xcite , as an external method that enables proactively screening external resources to minimize time overhead and avoid the risk of affecting other benign inferences, remains underexplored and has yet to effectively detect ipi attacks. we recognize that ipi attacks fundamentally rely on the presence of instructions embedded within external content, which can alter the behavioral states of llms. therefore, we hypothesize that this fundamental phenomenon-whether external data induces corresponding changes in the behavioral states of llms-can be leveraged to detect ipi attacks. building on this insight, we propose instruct-detector, a novel detection-based approach that leverages the behavioral states of llms to identify potential ipi attacks. we first evaluate the effectiveness of hidden states and gradients from different layers of the llm by employing them as features for instruction detection. through experimentation on the validation set, we identify that the hidden states and gradients from intermediate layers consistently exhibit the best performance in differentiating normal external data from those containing hidden instructions. specifically, we select the hidden states of the last token, as prior research indicates that the last token's hidden state provides the most informative representation of the input sequence @xcite . for the gradients, we focus on the gradients of self-attention layers, as previous studies suggest that self-attention layers capture the model's behavioral characteristics, while feed-forward layers are more effective at encoding knowledge-based features @xcite @xcite . lastly, we fuse the hidden state features and the gradient features from the intermediate layer, which effectively integrates the complementary information captured by these two features. the fused features are then fed into a multi-layer perceptron (mlp) classifier, enabling effective detection of ipi attacks. in our experiments, we consider normal external data as negative samples (without hidden instructions) and generate positive samples (with hidden instructions) by randomly inserting instructions into the negative samples. the external datasets include wikipedia and news articles, while the instruction data come from lamini-instruction and bipia. instructdetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, outperforming existing detection-based methods and several straightforward detection-based methods we propose. furthermore, we conduct evaluation on the bipia benchmark (out-of-domain), where our method reduces the attack success rate (asr) to just 0.03%, surpassing the performance of the prevention-based methods reported in the benchmark. the contributions of our work can be outlined as follows: ## conclusion in this work, we present instructdetector, a detection-based approach that leverages the internal behavioral states of llms as signals to identify ipi attacks. a key finding of our study is that the hidden states and gradients from the intermediate layers of llms provide highly discriminative features for instruction detection. by leveraging these internal behavioral states, instructdetector provides a robust mechanism for identifying hidden instructions within external data.