The RAG Paradox: A Black-Box Attack Exploiting Unintentional Vulnerabilities in Retrieval-Augmented Generation Systems (2025)

Abstract: With the growing adoption of retrieval-augmented generation (RAG) systems, various attack methods have been proposed to degrade their performance. However, most existing approaches rely on unrealistic assumptions in which external attackers have access to internal components such as the retriever. To address this issue, we introduce a realistic black-box attack based on the RAG paradox, a structural vulnerability that emerges from the system’s effort to enhance trust by revealing both the retrieved documents and their sources to users. This transparency enables attackers to observe which sources are used and how information is phrased, allowing them to craft poisoned documents that are more likely to be retrieved and upload them to the identified sources. Moreover, as RAG systems directly provide retrieved content to users, these documents must not only be retrievable but also appear natural and credible to prevent users from questioning the search results. Unlike prior work that focuses solely on improving document retrievability, our attack method explicitly considers both retrievability and user trust in the retrieved content. Through extensive offline and online experiments, we demonstrate that our method significantly degrades system performance without internal access, while generating natural-looking poisoned documents.

## introduction retrieval-augmented generation (rag) @xcite is a technique that retrieves documents relevant to a given query and utilizes them in the response generation process of large language models (llms). rag enables llms to access up-to-date information without requiring parameter updates and enhances the response quality based on this information @xcite . leveraging these advantages, numerous rag systems, such as chatgpt, gemini, and perplexity, have recently been introduced. ## realistic black-box rag attack in this section, we define a realistic black-box threat model for attacking rag systems ( §3.1), present an attack scenario ( §3.2), and describe our automated poisoning method ( §3.3). ## experiments to validate the effectiveness and feasibility of our realistic attack scenario, we conduct offline experiments using datasets and generators commonly used in rag research. we further perform a limited number of carefully controlled online experiments, conducted solely for research purposes to ensure safety and ethical compliance, targeting commercial rag systems. these experiments confirm that our attack method is effective in realworld deployment settings. the details of our experiments are provided in appendix §b. ## conclusion this study unveils the rag paradox, where retrieval-augmented generation (rag) systems face a fundamental dilemma between transparency and security. to enhance user trust, rag systems disclose retrieved documents along with their sources. however, this openness unintentionally exposes new attack surfaces and reveals to adversaries which sources can be targeted. conversely, withholding such information may reduce these vulnerabilities but would compromise transparency and erode user trust. to empirically expose this dilemma, we propose a realistic black-box attack scenario that does not require access to internal system components. our method leverages the disclosed documents to infer the retriever's preferences and generates poisoned documents that appear natural while effectively disrupting response generation. extensive offline and online experiments demonstrate that such attacks are both feasible and highly impactful under practical constraints. through this black-box attack, our work empirically reveals the inherent dilemma facing rag systems, offering a new perspective on their robust-ness. furthermore, it highlights the need for future research on defense strategies that can balance the trade-off between transparency and resilience. limitations while this study proposes a realistic black-box attack scenario and an effective poisoned document generation technique, several limitations remain. first, our experiments were conducted within a naive rag framework, and thus the effectiveness of the proposed attack method should be further validated in more diverse retrieval architectures and environments where additional filtering mechanisms are applied. such evaluations would provide a broader understanding of the generalizability and robustness of our attack across different rag settings. second, we adapted the naturalness evaluation score (nes) to suit our task by modifying its criteria for evaluating document naturalness. however, the use of llm-based evaluators inherently introduces subjectivity and consistency issues. moreover, the criteria for detecting artificial manipulation are uniformly applied across all domains, which may result in biased assessments, particularly in specialized domains such as law, healthcare, or technical fields where question-focused writing is naturally expected. future research should develop more domain-adaptive and fine-grained evaluation frameworks to address these limitations. despite these limitations, our study demonstrates that it is possible to infer the retriever's preferences solely from externally observable information and automatically generate poisoned documents that appear highly natural and trustworthy without any internal system access. in doing so, we highlight the rag paradox, where rag systems' efforts to enhance transparency by exposing external sources inadvertently create new attack surfaces.