Typos that Broke the RAG ’s Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations (2024)

Abstract: The robustness of recent Large Language Models (LLMs) has become increasingly crucial as their applicability expands across various domains and real-world applications. Retrieval-Augmented Generation (RAG) is a promising solution for addressing the limitations of LLMs, yet existing studies on the robustness of RAG often overlook the interconnected relationships between RAG components or the potential threats prevalent in real-world databases, such as minor textual errors. In this work, we investigate two underexplored aspects when assessing the robustness of RAG: 1) vulnerability to noisy documents through low-level perturbations and 2) a holistic evaluation of RAG robustness. Furthermore, we introduce a novel attack method, the Genetic Attack on RAG (GARAG), which targets these aspects. Specifically, GARAG is designed to reveal vulnerabilities within each component and test the overall system functionality against noisy documents. We validate RAG robustness by applying our GARAG to standard QA datasets, incorporating diverse retrievers and LLMs. The experimental results show that GARAG consistently achieves high attack success rates. Also, it significantly devastates the performance of each component and their synergy, highlighting the substantial risk that minor textual inaccuracies pose in disrupting RAG systems in the real world. Code is available at https://github.com/zomss/GARAG.

## introduction large language models (llms) @xcite have enabled remarkable advances in diverse natural language processing (nlp) tasks, especially in question-answering (qa) tasks @xcite . despite these advances, however, llms face challenges in having to adapt to ever-evolving * corresponding author or long-tailed knowledge due to their limited parametric memory @xcite , resulting in a hallucination where the models generate convincing yet factually incorrect text @xcite . retrieval-augmented generation (rag) @xcite has emerged as a promising solution by utilizing a retriever to fetch enriched knowledge from external databases, thus enabling accurate, relevant, and up-to-date response generation. specifically, rag has shown its superior performance across diverse knowledgeintensive tasks @xcite @xcite , leading to its integration as a core component in various real-world apis @xcite @xcite . given its extensive applications, ensuring robustness under diverse conditions of real-world scenarios becomes critical for safe deployment. thus, assessing potential vulnerabilities within the overall rag system is vital, particularly by assessing its components: the retriever and the reader. however, existing studies on assessing the robustness of rag often focus solely on either retrievers @xcite @xcite or readers @xcite @xcite . the robustness of a single component might only partially capture the complexities of rag systems, where the retriever and reader work together in a sequential flow, which is crucial for optimal performance. in other words, the reader's ability to accurately ground information significantly depends on the retriever's capability of sourcing query-relevant documents @xcite . thus, it is important to consider both components simultaneously when evaluating the robustness of an rag system. while concurrent work has shed light on the sequential interaction between two components, they have primarily evaluated the performance of the reader component given the high-level perturbed errors within retrieved documents, such as context relevance or counterfactual information @xcite @xcite . however, they have overlooked the impact of low-level errors, such as textual typos due to human mistakes or preprocessing inaccuracies in retrieval corpora, which often occur in real-world scenarios @xcite . additionally, llms, commonly used as readers, struggle to produce accurate predictions when confronted with textual errors @xcite . note that these are the practical issues that can affect the performance of any rag system in real-world scenarios, as illustrated in figure 1 . therefore, to deploy a more realistic rag system, we should consider: "can minor document typos comprehensively disrupt both the retriever and reader components in rag systems?" in this paper, we evaluate the rag system's robustness against textual typos in the database by generating a perturbed counterpart of the clean document retrieved for a given query. initially, we establish two attack objectives to qualitatively measure the negative impact of the adversarial document on the rag system's retrieval and grounding capabilities. to comprehensively assess system resilience under these objectives, we propose a novel black-box adversarial attack method, garag, which uses a genetic algorithm to search for the most adversarial document with low values for both loss objectives among the perturbed documents. the method begins by generating an initial population of adversarial documents by injecting minor textual errors into the original document while ensuring that answer tokens remain unaltered. through an iterative process of mutation, crossover, and selection to refine the population, the method searches for the most adversarial document for a given query by effectively exploring the vast search space of typos space and exploiting the most adversarial documents. to sum up, garag assesses the holistic robustness of an rag system against minor textual errors, offering insights into the system's resilience through iterative adversarial refinement. we validate our method on three standard qa datasets @xcite @xcite , with diverse retrievers @xcite and llms @xcite @xcite . the experimental results reveal that adversarial documents with low-level perturbation generated by garag significantly induce retrieval and grounding errors, achieving a high attack success rate of approximately 70%, along with a significant reduction in the performance of each component and the overall system. our analyses also highlight that lower perturbation rates pose a greater threat to the rag system, emphasizing the challenges of mitigating such inconspicuous yet critical vulnerabilities. our contributions in this paper are threefold: • we point out that the rag system is vulnerable to minor but frequent textual errors within the documents, prevalent in real-world scenarios. ## method here, we introduce our problem formulation and a novel attack method, garag. further details of the proposed method are described in appendix a. ## results in this section, we show our experimental results with an in-depth analysis of the adversarial attack. ## conclusion in this work, we highlighted the importance of assessing the overall robustness of the retriever and reader components within the rag system, particularly against noisy documents containing minor typos that are common in real-world databases. specifically, we proposed two objectives to evaluate the resilience of each component, focusing on their sequential dependencies. furthermore, to simulate real-world noises with low-level perturbations, we introduced a novel adversarial attack method, garag, incorporating a genetic algorithm. our findings indicate that noisy documents critically hurt the rag system, significantly degrading its performance. although the retriever serves as a protective barrier for the reader, it still remains susceptible to minor disruptions. our garag shows promise as an adversarial attack strategy when assessing the holistic robustness of rag systems against various low-level perturbations.