The Silent Saboteur: Imperceptible Adversarial Attacks against Black-Box Retrieval-Augmented Generation Systems (2025)

Abstract: We explore adversarial attacks against retrieval-augmented generation (RAG) systems to identify their vulnerabilities. We focus on generating human-imperceptible adversarial examples and introduce a novel imperceptible retrieve-to-generate attack against RAG. This task aims to find imperceptible perturbations that retrieve a target document, originally excluded from the initial top-k candidate set, in order to influence the final answer generation. To address this task, we propose ReGENT, a reinforcement learning-based framework that tracks interactions between the attacker and the target RAG and continuously refines attack strategies based on relevance-generation-naturalness rewards. Experiments on newly constructed factual and non-factual question-answering benchmarks demonstrate that ReGENT significantly outperforms existing attack methods in misleading RAG systems with small imperceptible text perturbations.

## introduction rag has emerged as an important approach for mitigating hallucination in large language models (llms). by retrieving relevant documents from an external knowledge corpus to provide grounding, rag systems enhance the factual accuracy and reliability of model outputs @xcite @xcite . adversarial examples. deep neural networks can easily be deceived by adversarial examples, i.e., inputs modified with human-imperceptible perturbations, to induce incorrect predictions. in rag systems, both the retriever, which is typically neuralbased, and the llm are prone to inherit the adversarial vulnerabilities of neural networks @xcite @xcite 2022; @xcite @xcite , raising serious security concerns. initial studies have begun exploring adversarial attacks on rag systems @xcite . it is crucial to identify these vulnerabilities before real-world deployment, as this allows timely development of effective defenses. early studies into adversarial attacks against rag focus primarily on corpus poisoning, which typically involves injecting malicious prompts @xcite or introducing adversarial documents containing poisoned information into the corpus @xcite @xcite . despite exposing critical vulnerabilities, these studies overlook a key characteristic of effective adversarial examples: imperceptibility. injecting new poisoned knowledge into a corpus can easily attract the attention of administrators, while prompt injection attacks, as discussed in section 6, are susceptible to detection by rag systems' self-inspection mechanisms and may degrade the user experience. hence, developing imperceptible attacks against rag systems is of paramount importance. a new adversarial attack task against rag. we introduce the imperceptible retrieve-to-generate attack (irg-attack) task against rag systems. as illustrated in figure 1 , given an rag system and a query, our attack aims to identify and modify a target document outside the initial top-k candidate set from the knowledge corpus, to achieve three key objectives: (i) enter the top-k retrieved document list; (ii) influence the llm to generate targeted answer; and (iii) maintain imperceptibility to ensure the attack appears natural. since most real-world rag systems restrict access to their internal components, we consider a practical yet challenging decision-based black-box setting, where attackers have no access to model internals but can query the target rag system and obtain outputs. we consider two representative question-answering (qa) scenarios, i.e., factual qa and non-factual qa, and construct dedicated benchmarks for evaluation. an rl-based rag attack framework. the attack process in irg-attack can be viewed as a series of interactions between the attacker and the target rag system. during these interactions, the attacker should balance the trade-off between perturbation magnitude and attack effectiveness. to achieve this, we model the attack as a markov decision process (mdp) @xcite and propose a novel reinforced retrieve-to-generate attack framework (regent). regent iteratively refines small imperceptible perturbations and strategies to effectively manipulate both the retriever's search results and the llm's generated responses. we first train a surrogate retrieval model through coarse-grained training followed by fine-grained training to mimic the target rag system's retrieval preferences. combined with the llm, these models form the environment where the attack operates. the attack strategy, modeled as an agent, interacts with this environment to identify document vulnerabilities and generate effective perturbations. to guide perturbation generation, we propose a relevance-generation-naturalness reward that contrasts relevance and reference shifts between querydocument pairs across states while enforcing naturalness constraints. during the rl process, the attack strategy is continuously refined based on feedback from the environment, enabling more effective adversarial perturbations over time. experimental findings. our experiments across different qa scenarios demonstrate the vulnerability of rag systems to imperceptible adversarial attacks. by injecting only one imperceptibly perturbed document into the corpus containing over 8.8 million documents, regent achieves nearly 50% attack success rate. the perturbed documents maintain high semantic consistency with minimal perturbation rates, effectively evading rag's selfchecking mechanisms. human evaluation results further confirm the superiority of regent over existing baselines, with significantly higher naturalness scores for both document content and reasoning process. ## related work retrieval-augmented generation. rag has emerged as a powerful paradigm that combines llms with external knowledge, demonstrating superior capabilities in various tasks @xcite @xcite . recent studies primarily focus on effectiveness improvements, such as unified retrieval frameworks @xcite , fine-grained citations @xcite , and joint pipeline optimization @xcite , overlooking the adversarial robustness of rag systems, which is crucial given their increasing usage in deployment. adversarial attacks against retrieval models and llms. adversarial attacks against retrieval models primarily focus on manipulating document rankings with respect to queries through malicious modifications to documents @xcite @xcite . for llms, attacks focus on crafting inputs to make llms generate expected or abnormal responses, mainly through prompt injection @xcite and jailbreak attacks @xcite . llm attacks mainly focus on achieving targeted outputs with little regard for input naturalness. while current retrieval attacks do consider naturalness, they cannot be directly applied to rag systems due to the complex interactions between retrieval and generation components. adversarial attacks against rag systems. recent studies have revealed that rag systems are susceptible to various forms of manipulation and misguidance @xcite . a growing body of research has explored different attack approaches, particularly through the knowledge corpus @xcite . these attacks range from injecting poisoned documents @xcite to injecting malicious prompts @xcite . while current studies have revealed the vulnerability of rag systems, they overlook the critical aspect of attack naturalness. thus, developing attack methods that maintain naturalness while effectively manipulating both components of rag systems remains an open challenge. this work. we focus on maintaining naturalness from two critical perspectives: (i) from the rag system perspective, documents with obvious adversarial traits or significant deviations from the corpus patterns may trigger detection mechanisms; and (ii) from the user perspective, unnatural responses can undermine system credibility, as forced or unnatural responses will immediately reveal the attack when users question the reasoning. ## problem statement rag systems. a typical rag system has two main components: a retriever and a generator. given a query q, the retriever first identifies relevant documents from a knowledge corpus @xmath0, d 2 , ..., d n }. the retriever maps both query and documents into a shared embedding space r d using functions f q and f d through a dualencoder, and selects top-k documents based on similarity scores s (q, the relevant documents are denoted as r (q) = {d q 1 , . . . , d q k } ⊂ d. the generator then takes both the query q and the documents r(q) as input to produce the response @xmath1, r(q)). objective of the adversary. given a set of queries @xmath2, q 2 , . . . , q n }, the adversary aims to manipulate rag responses by promoting a target document d t into the top-k retrieved set r(q), where d t is initially excluded from the top-k set. based on these, the irg-attack task aims to fool the rag system by applying perturbations to the target document, seeking to influence the generator g to produce desired responses y * q . specially, we focus on maintaining the naturalness of the modified document to ensure the imperceptibility of the attack. formally, we define the attack objective as: ## method in this section, we introduce regent, our rlbased attack framework against rag systems. ## experimental setup benchmark construction. components of rag systems (knowledge corpus, retriever, and llm) as follows: (i) for knowledge corpus, we utilize the benchmark dataset; (ii) for retrievers, we consider co-condenser (fine-tuned on ms-marco) @xcite and contriever-ms (fine-tuned on ms-marco) @xcite . (iii) for llms, we employ lla-ma-3-8b @xcite , qwen-2.5-7b @xcite , and gpt-4o @xcite . unless otherwise specified, we use co-condenser as the default retriever and llama-3-8b as the default llm. our rag system retrieves the top-3 most similar documents from the corpus as context for each query. more experimental details can be found in app. a.2. hyperparameter selection. for the hyperparameters m i , m h , m n in training the surrogate model: (i) given our computational constraints, we followed the principle that the relevance differences between documents within the top-k should be small, with emphasis on the rank-1 document; for the hyperparameters in reward design: (i) λ r was used to keep generation and retrieval rewards at the same order of magnitude; (ii) the semantic preservation threshold τ required balance-too high would affect attack effectiveness, too low would affect naturalness. during our experiments, the semantic similarity between perturbed and original documents was above 99% in most cases; (iii) for the penalty p for excessive semantic loss, we set a relatively large value, indicating that word substitutions significantly deviating from the threshold are not permitted. comparison methods. we consider multiple attack baselines: (i) naive attack that directly injects target answers into the knowledge corpus. (ii) prompt injection attack @xcite with two variants: naive prompt attack and prompt hijacking attack @xcite . (iii) word substitution attack including prada -nrk @xcite and hotflip @xcite . more details on baselines can be found in the app. a.5. we also implement two variants of regent for ablation studies to validate the effectiveness of different components: (i) regent -nr ,which only iterates the target document using the surrogate retriever trained at coarse-grained level; for attack naturalness, we evaluate from both automatic and human perspectives: (i) automatic metrics including average perturbation rate (apr) and average document semantic preservation (adsp); (ii) human evaluation on answer reasoning naturalness (n r ) and document naturalness (n d ). for document naturalness n d , we followed previous works @xcite @xcite . specifically, we shuffled a mix of original and adversarial texts and asked human judges to rate their grammaticality and harmlessness on a likert scale of 1-5. for reasoning naturalness, we mainly examined whether the reasoning chain generating the final answer contained obvious logical inconsistencies. if the reasoning was logical, it received a score of 1; otherwise, score is 0. see app. a.6 for detailed metric definitions and evaluation protocols. ## experimental results performance of surrogate retriever. we trained surrogate models and evaluated their effectiveness in simulating rag retrieval preferences (see app. a.4). specifically, we assess how well the coarse-grained trained retriever c co-condenser and c contriever , and their corresponding fine-grained trained retriever f co-condenser and f contriever , serve as surrogate models for co-condenser and contriever respectively, in simulating the top-3 retrieval preferences of the original retriever in rag across both factual qa and stance-based qa scenarios. the experimental results demonstrate that injecting just one imperceptibly perturbed document into the corpus can significantly influence rag system responses: (i) for effectiveness, regent shows strong attack capability across both scenarios while maintaining high naturalness, as evidenced by the high document semantic preservation and low perturbation rates; (ii) the two qa scenarios show different vulnerabilities: stance-based qa achieves higher asr r while factual qa shows higher asr g . this disparity stems from their inherent nature: stance-based topics typically involve multiple viewpoints from various sources, which provides more candidate documents for retrieval attacks but also enables llms to cross-reference and maintain balanced outputs. in contrast, factual questions have limited information sources, which restricts retrieval attack options but makes llms more susceptible to manipulation due to reduced cross-verification opportunities. (iii) differences between models, llama-3-8b demonstrates the highest overall vulnerability to attacks, suggesting its stronger reliance on retrieved context. gpt-4o shows better resilience in factual qa, indicating its enhanced capability in fact-checking and verification. meanwhile, qwen-2.5-7b exhibits better robustness in stance-based qa, suggesting its superior ability to balance multiple points of view. overall, our regent achieves comparable attack effectiveness across both scenarios, while revealing distinct vulnerabilities. under our attacks, factual qa shows higher vulnerability in the generation phase, while stance-based qa exhibits more susceptibility in the retrieval phase. furthermore, we experimentally validated our method's effectiveness across different architectures(see a.7). attack and two variants of prompt injection attacks. according to the results in we further compare regent with two representative word substitution attacks: prada -nrk and hotflip. as shown in figure 3 , we find that: (i) all word substitution methods demonstrate good naturalness preservation, indicating their ability to make imperceptible modifications to target documents; (ii) regent significantly outperforms both furthermore, beyond the hard labels of answer changes, the prompt in app. a.1 provides a softlabel perspective on the results. our experiments reveal an interesting phenomenon: in factual qa, 13% of queries show increased reference to target documents when considering the generation reward during iteration, while this ratio reaches 24% in stance-based qa. it shows rag responses in stance-based qa are more susceptible to word variations, yet more resistant to actual answer changes (hard labels), while the factual qa is opposite. naturalness evaluation. as discussed in section 2, maintaining naturalness is crucial from both rag system and user perspectives. in this section, we evaluate the naturalness of our attack method from these two perspectives. for system perspective, we simulate a scenario where system administrators enhance the rag system's security through defensive prompts (refer to appendix a.1). we evaluate how this defense mechanism affects attack. as shown in figure 4 , we observe: (i) regent demonstrates strong robustness by maintaining relatively stable performance with only slight degradation; (ii) factual qa shows moderate performance drops while stance-based qa exhibits more dramatic reductions, suggesting that stance-based attacks are more sensitive to the naturalness constraints. ## conclusion in this paper, we introduced the irg-attack task against rag systems, which aims to manipulate rag outputs through generating imperceptible adversarial examples. we developed regent, an rl-based framework that guides attack strategies through surrogate retrieval models and relevancegeneration-naturalness rewards. our extensive experiments demonstrate that regent can effectively manipulate both retrieval and generation components of rag systems while maintaining high naturalness.