R ev PRAG : Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysis (2025)

Abstract: Retrieval-Augmented Generation (RAG) enriches the input to LLMs by retrieving information from the relevant knowledge database, enabling them to produce responses that are more accurate and contextually appropriate. It is worth noting that the knowledge database, being sourced from publicly available channels such as Wikipedia, inevitably introduces a new attack surface. RAG poisoning attack involves injecting malicious texts into the knowledge database, ultimately leading to the generation of the attacker’s target response (also called poisoned response). However, there are currently limited methods available for detecting such poisoning attacks. We aim to bridge the gap in this work by introducing RevPRAG, a flexible and automated detection pipeline that leverages the activations of LLMs for poisoned response detection. Our investigation uncovers distinct patterns in LLMs’ activations when generating poisoned responses versus correct responses. Our results on multiple benchmarks and RAG architectures show our approach can achieve a 98% true positive rate, while maintaining a false positive rate close to 1%.

## introduction retrieval-augmented generation (rag) @xcite has emerged as an effective solution that leverages retrievers to incorporate external databases, enriching the knowledge of llms and ultimately enabling the generation of up-to-date and accurate responses. rag comprises three components: knowledge database, retriever, and llm. fig. 1 visualizes an example of rag. the knowledge database consists of a large amount of texts collected from sources such as latest wikipedia entries @xcite , new articles @xcite and financial documents @xcite . the retriever is primarily responsible for retrieving the texts that are most related to the user's query from the knowledge database. these texts will later be fed to llm as a part of the prompt to generate responses (e.g., "everest") for users' queries (e.g., "what is the name of the highest mountain?"). due to rag's powerful knowledge integration capabilities, it has demonstrated impressive performance across a range of qa-like knowledge-intensive tasks @xcite . rag poisoning refers to the act of injecting malicious or misleading content into the knowledge database, contaminating the retrieved texts and ultimately leading the llm to produce the attacker's desired response (e.g., the target answer could be "fuji" when the target question is "what is the name of the highest mountain?"). this attack leverages the dependency between llms and the knowledge database, transforming the database into a new attack surface to facilitate poisoning. poisonedrag @xcite demonstrates the feasibility of rag poisoning by injecting a small amount of maliciously crafted texts into the knowledge database utilized by rag. the rise of such attacks has drawn significant attention to the necessity of designing robust and resilient rag systems. for example, in-structrag @xcite utilizes llms to analyze how to extract correct answers from noisy retrieved documents; robustrag @xcite introduces multiple llms to generate answers from the retrieved texts, and then aggregates the responses. however, the aforementioned defense methods necessitate the integration of additional large models, incurring considerable overheads. meanwhile, it is difficult to promptly assess whether the current response of rag is trustworthy or not. in our work, we shift our focus to leverage the intrinsic properties of llms for detecting rag poisoning, rather than relying on external models. our view is that if we can accurately determine whether a rag's response is correct or poisoned, we can effectively thwart rag poisoning attacks. we attempt to observe llm's answer generation process to determine whether the response is compromised or not. it is worth noting that our focus is not on detecting malicious inputs to llms, as we consider the consequences of malicious responses to be far more detrimental and indicative of an attack. the growing body of research on using activations to explain and control llm behavior @xcite provides us inspiration. specifically, we empirically analyze the activations of the final token in the input sequence across all layers of the llm. our findings demonstrate that the model exhibits distinguishable activation patterns when generating correct versus poisoned responses. based on this, we propose a systematic and automated detection pipeline, namely revprag, which consists of three key components: poisoned data collection, llm activation collection and preprocessing, and the detection model design. it is important to note that this detection method will not alter the rag workflow or weaken its performance, thereby offering superior adversarial robustness compared to methods that rely solely on filtering retrieved texts. to evaluate our approach, we systematically demonstrate the effectiveness of revprag across various llm architectures, including gpt2-xl-1.5b, llama2-7b, mistral-7b, llama3-8b, and llama2-13b. revprag performs consistently well, achieving over 98% true positive rate across different datasets. our contributions can be summarized as follows: ## conclusion in this work, we find that correct and poisoned responses in rag exhibit distinct differences in llm activations. building on this insight, we develop revprag, a detection pipeline that leverages these activations to identify poisoned responses in rag caused by the injection of malicious texts into the knowledge database. our approach demonstrates robust performance across rags utilizing five different llms and four distinct retrievers on three datasets. experimental results show that revprag achieves exceptional accuracy, with true positive rates approaching 98% and false positive rates near 1%. ablation studies further validate its effectiveness in detecting poisoned responses across different types and levels of poisoning attacks. overall, our approach can accurately distinguish between correct and poisoned responses.