The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation ( RAG ) (2024)

Abstract: Retrieval-augmented generation (RAG) is a powerful technique to facilitate language model generation with proprietary and private data, where data privacy is a pivotal concern. Whereas extensive research has demonstrated the privacy risks of large language models (LLMs), the RAG technique could potentially reshape the inherent behaviors of LLM generation, posing new privacy issues that are currently under-explored. To this end, we conduct extensive empirical studies with novel attack methods, which demonstrate the vulnerability of RAG systems on leaking the private retrieval database. Despite the new risks brought by RAG on the retrieval data, we further discover that RAG can be used to mitigate the old risks, i.e., the leakage of the LLMs’ training data. In general, we reveal many new insights in this paper for privacy protection of retrieval-augmented LLMs, which could benefit both LLMs and RAG systems builders.

## introduction retrieval-augmented generation (rag) @xcite @xcite @xcite is an advanced natural language processing technique that enhances text generation by integrating information retrieved from a large corpus of documents. these techniques enable rag to produce accurate and contextually relevant outputs with augmented external knowledge and have been widely used in various scenarios such as domain-specific chatbots @xcite and email/code completion @xcite . rag systems typically work in two phases, as shown in fig 1 -retrieval and generation. when a user query is entered, relevant knowledge is first retrieved from an external database. the retrieved data is then combined with the original query to form the input to a large language model (llm). the llm then uses its pre-trained knowledge and the retrieved data to generate a response. in this paper, we focus on studying the risk of privacy leakage in the rag system, and we argue that the information from both retrieval dataset and the pre-training/fine-tuning dataset (of the llm) are potential to be released by rag usage. on one hand, the retrieval dataset can contain sensitive, valuable domain-specific information @xcite , such as patients prescriptions can be used for rag-based medical chatbots @xcite . on the other hand, the retrieval process in rag could also influence the behavior of the llms for text-generation, and this could possibly cause the llms to output private information from its training/fine-tuning dataset. notably, there are existing works @xcite @xcite @xcite observing that llms can remember and leak private information from their pre-training and fine-tuning data. however, how the integration of external retrieval data can affect the memorization behavior of llms in rag is still unclear and worth further exploration. therefore, these concerns motivate us to answer the research questions: in detail, take our study on rag for medical dialogue (section 3.2) as an example, the attacker can ask the model for general information or suggestions related to certain diseases. more importantly, we propose to append an extra "command prompt" (see section 3.2) during inquiry to improve the successful rate of extraction. after that, we examine the model's output to see whether it contains information about specific prescription records, which may hurt the privacy of patients. based our empirical study, we observe that our studied models (llama2-7b-chat and gpt-3.5-turbo) can output verbatim or highly similar records with very high rates (near 50%). this result reveals that rag systems are highly susceptible to such attacks, with a considerable amount of sensitive retrieval data being extracted. regarding rq2, while prior work has shown that llms exhibit a propensity to output memorized training data, verifying the influence of retrieval data integration remains unexplored. therefore, we conduct targeted and prefix attacks on llms' training corpus, comparing training data exposure with and without retrieval augmentation. we discover that incorporating retrieval data into rag systems can substantially reduce llms' tendency to output its memorized training data, achieving greater protection than noise injection or system prompts. from a training data security perspective, our findings indicate that rag may provide a safer architecture compared to using llms sorely. ## rq1: can we extract private data from the external retrieval database in rag? with the proposed targeted and untargeted attacks on the retrieval dataset in section 3.2 , we empirically investigated the privacy leakage of the retrieval dataset(rd). our evaluation revealed the rag system's high vulnerability to attacks on retrieval data. we also conducted ablation studies to examine various impact factors and explored possible mitigation strategies. ## rq2: can retrieval data affect the memorization of llms in rag? in this section, we aim to examine how incorporating retrieval data affects llms' tendency to reproduce memorized information from their training sets. to investigate this question, we conducted targeted and prefix attacks on llms and compared 6 more details can be found in appendix a.4. the leakage difference with and without retrieval data. next we first introduce the evaluation setup. ## conclusions in this paper, we extensively investigated the privacy risks associated with retrieval-augmented generation (rag) technique for llms. through our proposed attack methods, we first systematically evaluated and identified the significant risks of retrieval data extraction. meanwhile, we explored various defense techniques that can mitigate these risks. we also found that integrating retrieval data can substantially reduce llms' tendency to output its memorized training data, which suggests that rag could potentially mitigate the risks of training data leakage. overall, we revealed novel insights regarding privacy concerns of retrieval-augmented llms, which is beneficial for the proper usage of rag techniques in real-world applications.