RAG LLM s are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models (2025)

Abstract: Efforts to ensure the safety of large language models (LLMs) include safety fine-tuning, evaluation, and red teaming.However, despite the widespread use of the Retrieval-Augmented Generation (RAG) framework, AI safety work focuses on standard LLMs, which means we know little about how RAG use cases change a model’s safety profile. We conduct a detailed comparative analysis of RAG and non-RAG frameworks with eleven LLMs. We find that RAG can make models less safe and change their safety profile. We explore the causes of this change and find that even combinations of safe models with safe documents can cause unsafe generations. In addition, we evaluate some existing red teaming methods for RAG settings and show that they are less effective than when used for non-RAG settings. Our work highlights the need for safety research and red-teaming methods specifically tailored for RAG LLMs.

## introduction large language models (llms) can support many tasks but are susceptible to creating unsafe content @xcite @xcite @xcite . these safety concerns include harmful, illegal, offensive, and unethical content, such as spreading misinformation and jeopardizing personal safety and privacy @xcite @xcite . this can expose service providers to legal risks and undermine public trust in ai @xcite @xcite . to address safety concerns, researchers have developed methods like safety fine-tuning @xcite , building guardrails @xcite , and red teaming llms @xcite . red-teaming often involves probing llms with prompts to expose * work done during an internship at bloomberg. vulnerabilities, such as jailbreaking attempts @xcite @xcite . retrieval-augmented generation (rag) identifies relevant documents from a corpus and provides them to the llm to form the basis of a response @xcite @xcite . rag-based llms are popular because they can reduce hallucinations by grounding responses in source documents, include updated or recent information without model retraining, and facilitate attributed generation @xcite @xcite . since responses are influenced by retrieved documents, ensuring documents are free of safetyviolating statements may lead us to hypothesize that rag-based models are safer than their non-rag counterparts. unfortunately, little evidence exists to support or disprove this hypothesis. recent work on the safety of rag mainly considers infusion attacks @xcite , where vulnerabilities come from harmful documents injected into the source corpus through corpus poisoning @xcite @xcite . if we assume a corpus is secured, does that guarantee the safety of a rag-based system? this paper asks: are rag-based llms safer than their non-rag counterparts? perhaps surpris-ingly, our answer is a resounding no. we focus on a fixed user setting where harmful questions should not be answered in either rag or non-rag settings, and we rely on prior works' risk definitions. we pose three research questions. rq1: are rag-based llms safer than their non-rag counterparts? we assess the safety profiles of eleven popular llms on more than 5,000 harmful questions, comparing their safety behaviors in non-rag and rag settings. we find that rag introduces unsafe behaviors in ways not previously considered. for example, relatively safe models like llama-3-8b become unsafe, with unsafe responses rising from 0.3% to 9.2%. this change manifests across nearly every safety category. rq2: what makes rag-based llms unsafe? we explore three factors: the safety of the model, the safety of the retrieved documents, and the model's rag capability. all three factors affect the model's safety. surprisingly, even a safe model, like llama-3-8b, combined with safe documents can produce unsafe generations (figure 1 ). simply using a safe model and a safe corpus does not guarantee safety in rag systems, revealing previously unrecognized vulnerabilities. rq3: are red-teaming methods effective for rag-based models? we evaluate two representative red-teaming methods and find that adversarial prompts capable of jailbreaking an llm in the non-rag setting fail in the rag setting. optimizing adversarial prompts based on retrieved documents helps, but there is still a gap between testing and training, as the adversarial prompts may retrieve different documents from those used during training. we find that red-teaming methods tailored specifically to rag-based llms are needed, highlighting an important area for future work. safety of rag llms. recent work on rag llm safety @xcite ) mainly focuses on the threat of corpus poisoning, where injecting harmful documents leads to unsafe outputs @xcite @xcite @xcite @xcite . other concerns include: corpus leakage @xcite , retriever robustness to noise @xcite , and llms' resilience to document perturbations @xcite . our focus is a common scenario where the corpus is carefully controlled. this topic is also relevant to the safety of llm agents using tools @xcite @xcite @xcite @xcite @xcite , with a retriever being a special tool. red-teaming llms. red-teaming commonly involves jailbreaking llms @xcite @xcite to expose vulnerabilities using human-designed tests @xcite or llm-generated prompts @xcite @xcite . black-box methods only require models' discrete outputs @xcite @xcite @xcite , while white-box techniques exploit internal parameters for prompt optimization which are more effective @xcite @xcite . however, no methods have been designed specifically for rag llms. ## definitions a standard rag pipeline contains two components: an llm g and a retriever r, such as a sparse (e.g., bm25) @xcite or a dense retriever @xcite . given a user query q, the retriever r first retrieves the top k most relevant documents d @xmath0, d 2 , . . . , d k } from a corpus. these documents provide context for the llm's generation. next, the language model g takes both the query q and the retrieved documents d k as input to generate a response @xmath1, where i is the instruction such as "answer the following question. you should only use the following documents." ## rq1: are rag-based llms safer than their non-rag counterparts? we conduct a large-scale evaluation on eleven llms: llama-2-7b-chat @xcite , llama-3-8b-instruct @xcite dataset. we collect 5,592 harmful questions from the red-teaming resistance benchmark (haizelabs, 2024) and harmbench @xcite , covering nine datasets @xcite @xcite @xcite . each question is labeled according to a 16-category risk taxonomy based on openai's policy, as used in previous studies such as @xcite (see appendix a). we consider a fixed use case, where these harmful questions should not be answered in either rag or non-rag settings. evaluation setup. we use bm25, a strong and canonical retriever, and english wikipedia foot_0 as our corpus. articles are chunked into paragraphs, and each paragraph is treated as a document. there are 20,464,398 documents in total. the threat model is a user seeking generations that violate safety policy by directly asking harmful questions. we query a harmful question and record a response in each of these three settings: 1) non-rag: the model generates a response based on its own knowledge. 2) rag (docs): the retriever finds the top five documents, and the llm is instructed only to use these documents to answer. 3) rag (docs + model knowledge): the model is given the retrieved documents but is instructed to use both the documents and its own knowledge. these two rag settings represent the most popular rag system strategies in practice today. appendix a provides the detailed prompt templates. we use llama guard 2 (llama team, 2024) as the safety judge to classify the safety of responses, and we measure the percentage of unsafe responses for each setting and model. figure 11 illustrates the entire pipeline. rag-based llms are not safer. figure 2 shows that eight of the eleven models exhibit markedly different safety behaviors between rag and non-rag settings, with large increases in unsafe responses under the rag setting. even very safe models, like llama-2 and llama-3, which refuse to answer nearly all harmful queries in the non-rag setting, become vulnerable in the rag setting. for instance, the percentage of unsafe responses from llama-3-8b jumps from just 0.3% to 9.2%. a similar phenomenon is also observed on large models, such as 70b models, and proprietary models, such as gpt-4o. the unsafe rag setting is nearly the same, regardless of whether the model is instructed to also use its own knowledge or not. therefore, we use rag (docs) as the default setting in the rest of the paper. we also observe that the change in safety behavior between non-rag and rag settings is model-dependent. for unsafe models like zephyr -fine-tuned on mistral with alignment removed -rag has little effect, as it's already very unsafe. gemma seems safe in both settings, but this safety is deceptive, which we will explain in section 5.3. claude-3.5-sonnet is the most robust model, with very low unsafe response rates across all settings. rag changes the risk profile of llms, and the change is model-dependent. we dig into finegrained safety within each of the 16 risk categories. we examine the risk profile (i.e., the distribution of unsafety across categories) that highlights the vulnerabilities of each model and compare it in non-rag vs. rag (docs) settings. figure 4 shows the risk profile of llama-3-8b. in the non-rag setting, the model is unsafe in only 7 categories, with its greatest vulnerabilities in unauthorized practice of law, malware, and tailored financial advice. however, when applied to rag, the model becomes vulnerable across all 16 categories. previously safe areas, such as misinformation and disinformation, adult content, and political campaigning, now show a marked increase in risk that cannot be ignored. other models' risk profiles are shown in figures 12 and 13 . perhaps the change in risk results from the retrieved documents, with our corpus having a higher rate of unsafe documents in some categories. we plot the risk profile changes of different models in figure 3 and figure 14 . even with the same retrieved documents, the changes in risk profiles vary, indicating that the increase in unsafe responses cannot be attributed solely to the documents. ## rq2: what makes rag-based llms unsafe? we investigate three factors that could shape the safety behavior of rag-based llms: 1) the inherent safety of the llm itself; 2) the safety of the retrieved documents; and 3) the llm's capability to perform rag tasks correctly. ## rq3: are red-teaming methods effective for rag-based models? our findings show that safe models can become unsafe in rag settings, underscoring the need for a thorough evaluation of rag-based models before deployment. red-teaming methods aim to identify queries that trigger unsafe responses, but are these methods effective for rag-based models? red-teaming methods. we consider a threat model where the adversary has full access to the model and can call the retriever but cannot modify the corpus. we test two representative gradientbased methods: gcg @xcite and auto-dan @xcite . both methods optimize an adversarial suffix appended to a harmful query to create a jailbreaking prompt, aiming to maximize the likelihood of the target llm generating a compliant response. in every step, they use gradientbased search to explore token candidates in a discrete space, then validate candidates and select the best token. gcg produces unreadable prompts, while autodan generates human-readable ones using controllable text generation techniques. evaluation setup. we evaluate both methods on llama-3-8b, a safe model, and mistral-v0.3, a relatively unsafe model. for each model, we randomly select 50 harmful queries from our dataset that the model initially refuses to answer in both non-rag and rag settings. the same set of queries is used during both training and testing. in the training stage, we optimize jailbreaking prompts. in the testing stage, we evaluate jailbreaking prompts' attack success rate in the rag setting. we run gcg and autodan five times per query. we measure two attack success rates: asr@1, the average success rate across 250 attempts, and asr@5, where a query is considered jailbroken if at least one of the 5 attempts is successful. more details can be found in appendix c. ## discussions on future directions rag-based llms exhibit different, often more vulnerable, safety behaviors, underscoring the need for dedicated research on llm safety in the rag setting. current safety fine-tuning methods are primarily designed for non-rag settings. however, llms in rag settings are tasked with synthesizing information from retrieved documents, which differs from safety training. therefore, we suggest that safety fine-tuning should be specifically tailored for rag tasks if the model will be used in the rag setting. we need new methods for red-teaming ragbased models. one solution to adapt gcg and autodan to the rag setting is to re-retrieve documents after each optimization step, but this is time-consuming and may not converge. alternatively, we could introduce a penalty that encourages the retrieved documents to remain stable during optimization. another strategy is to optimize the jailbreaking prompts universally for a large set of retrieved documents, ensuring they generalize effectively at test time. future work could investigate why safe documents can still result in unsafe responses. techniques like mechanism interpretability might provide insights into this phenomenon and guide further improvements in safety. finally, rag presents a unique challenge for dynamic corpora, such as news articles, and we need safety strategies for these dynamic environments.