Glue pizza and eat rocks - Exploiting Vulnerabilities in Retrieval-Augmented Generative Models (2024)

Abstract: Retrieval-Augmented Generative (RAG) models enhance Large Language Models (LLMs) by integrating external knowledge bases, improving their performance in applications like fact-checking and information searching. In this paper, we demonstrate a security threat where adversaries can exploit the openness of these knowledge bases by injecting deceptive content into the retrieval database, intentionally changing the model’s behavior. This threat is critical as it mirrors real-world usage scenarios where RAG systems interact with publicly accessible knowledge bases, such as web scrapings and user-contributed data pools. To be more realistic, we target a realistic setting where the adversary has no knowledge of users’ queries, knowledge base data, and the LLM parameters. We demonstrate that it is possible to exploit the model successfully through crafted content uploads with access to the retriever. Our findings emphasize an urgent need for security measures in the design and deployment of RAG systems to prevent potential manipulation and ensure the integrity of machine-generated content.

## introduction retrieval-augmented generative (rag) models @xcite @xcite @xcite represent a significant advancement in enhancing large language models (llms) by dynamically retrieving information from external knowledge databases. this integration improves performance in complex tasks such as fact checking @xcite and information retrieval @xcite . major search engines such as google search @xcite and bing @xcite are increasingly looking to integrate rag systems to elevate their perfor-figure 1 : example of a misleading search result. a query about "cheese not sticking to pizza" led google search to suggest using "non-toxic glue", influenced by a prank post on reddit, demonstrating rag system vulnerabilities to manipulated content. mance, leveraging databases that range from curated repositories to real-time web content. despite this remarkable progress, the openness to these databases poses potential risks. media reports highlight that ai-powered search engines can easily "go viral" foot_0 due to vulnerabilities in their knowledge sources. for example (in figure 1 ), when a user queried "cheese not sticking to pizza", google search suggested using "non-toxic glue". this misleading response resulted from the retriever behind google search retrieving a prank post from reddit foot_1 , and subsequently, the llm, gemini @xcite , was influenced to generate the deceptive reply. such vulnerabilities have forced google to scale back ai search answers 3 . based on this premise, our paper delves deeper into how such vulnerabilities can be exploited to influence rag systems' behaviors. we focus on a practical gray-box scenario: the adversary does not have access to the contents of user queries, existing knowledge in the database, or the internal parameters of the llm. the adversary only accesses the retriever and can influence the rag system outcomes by uploading or injecting adversarial contents. note that such exploitations are realistic threats given the public user interface of many knowledge bases used in rag systems. also, white-box retrievers such as contriever @xcite , contriever-ms (fine-tuned on ms marco), and ance @xcite remain popular and are freely accessible on platforms like huggingface foot_3 . these retrievers can be seamlessly integrated into online service like langchain for google search foot_4 , allowing for free local deployment. for instance, similar to the example in figure 1 , an adversary could upload, or inject, malicious content to its knowledge base, causing the search engine to return misleading or harmful information to other unsuspecting users. deriving such adversarial contents is not trivial. we conduct a warm-up study in section 4 and demonstrate that a vanilla approach that optimizes the injected content with a joint single-purpose objective will result in significant loss oscillation and prohibit the model from converging. accordingly, we propose to decouple the purpose of the injected content into a dual objective: ❶ it is devised to be preferentially retrieved by the rag's retriever, and ❷ it effectively influences the behaviors of the downstream llm once retrieved. then, we propose a new training framework, exploitative bi-level rag training (liar), which effectively generates adversarial contents to influence rag systems to generate misleading responses. our framework reveals these critical vulnerabilities and emphasizes the urgent need for developing robust security measures in the design and deployment of rag models. our major contributions are unfolded as follows: ⋆ threat identification. we are the first to identify a severe, practical security threat to prevalent rag systems. specifically, we demonstrate how malicious content, once injected into the knowledge base, is preferentially retrieved by the system and subsequently used to manipulate the output of the llm, effectively compromising the integrity of the response generation process. ⋆ framework design. we introduce the liar framework, a novel attack strategy that effectively generates adversarial contents serving the dual objective mentioned previously. ⋆ impact discussion & future directions: our experimental validation of the liar framework suggests strategies are needed for enhancing rag model security, or in broader terms, preserving the integrity and reliability of llms. query 𝑸 knowledge base 𝓚 ℎ ! ℎ " retriever 𝓡 top-𝑚 selection retrieved documents 𝓡(𝑸; 𝓚) primarily about their differing views on ai development and related practices… llm generator 𝒇 𝜽 why did lecun and musk argue recently why did lecun and musk argue recently 2 background retrival augmented generation (rag). as shown in figure 2 , rag systems @xcite @xcite @xcite are comprised of three fundamental components: knowledge base, retriever, and llm generator. the knowledge base in a rag system encompasses a vast array of documents from various sources. for simplicity, we denote the knowledge base as k, comprising n documents, i.e., @xmath0, d 2 , . . . , d n }, where d i denotes the ith document. this knowledge base can be significantly large, often containing millions of documents from sources like wikipedia @xcite . when a user submits a query, the retriever r identifies the top-m documents from the knowledge base that are most relevant to the query. this selection serves as the external knowledge to assist the llm generator g in providing an accurate response. for a given query q, a rag system follows two key steps to generate an answer. ## threat model in this section, we define the threat model for our investigation into the vulnerabilities of rag systems. this threat model focuses on adversaries who exploit the openness of these systems by injecting malicious content into their knowledge bases. we assume a gray-box setting, reflecting realistic scenarios where attackers have limited access to the system's internal components but can influence its behavior through external interactions. ## methods in this section, we begin by decoupling the structure and objective of attacking (i) the retriever r and (ii) the llm generator f θ (•) within the at framework. we empirically demonstrate that these individual components struggle to adapt to each other and fail to make a concerted effort within the at process in ## experiments we conduct a series of experiments to evaluate the effectiveness of liar. detailed experiment settings, including (1) dataset for attacks, (2) knowledge databases, (3) retriever models, (4) llm models, and (5) training details are included in appendix a. evaluation protocol: we set the attack success rate (asr) as the primary metric and evaluate the result by text matching and human judgment akin to @xcite . ## conclusion in this paper, we demonstrated the vulnerabilities of retrieval-augmented generative (rag) models to gray-box attacks. through a series of experiments, we showed that adversarial content could significantly impact the retrieval and generative components of these systems. our findings show the need for robust defense mechanisms to protect against such attacks, ensuring the integrity and reliability of rag models in various applications. in broader terms, we emphasize the urgent need to strengthen trustworthiness of llm applications.