Investigating the Robustness of Retrieval-Augmented Generation at the Query Level (2025)

Abstract: Large language models (LLMs) are very costly and inefficient to update with new information. To address this limitation, retrieval-augmented generation (RAG) has been proposed as a solution that dynamically incorporates external knowledge during inference, improving factual consistency and reducing hallucinations. Despite its promise, RAG systems face practical challenges-most notably, a strong dependence on the quality of the input query for accurate retrieval. In this paper, we investigate the sensitivity of different components in the RAG pipeline to various types of query perturbations. Our analysis reveals that the performance of commonly used retrievers can degrade significantly even under minor query variations. We study each module in isolation as well as their combined effect in an end-to-end question answering setting, using both general-domain and domain-specific datasets. Additionally, we propose an evaluation framework to systematically assess the query-level robustness of RAG pipelines and offer actionable recommendations for practitioners based on the results of more than 1092 experiments we performed.

## introduction recent advancements in the capabilities of large language models (llms) have revolutionized the field of natural language processing (nlp) and have achieved impressive performance across a broad range of downstream applications. their success can largely be attributed to the massive text datasets on which they are trained and their increasing size in terms of model parameters. however, these factors that have enabled their success also limit their practical implementation in downstream applications. for example, a business seeking to implement an llm to answer questions about proprietary internal documents may lack the compute * work performed while at intel labs. even when an llm can be properly trained on domain-specific data, all existing models are prone to the well-known issue of hallucination @xcite . this phenomenon, where llms produce confident-sounding, factually inaccurate responses, is particularly problematic for applications in which downstream users may lack the necessary domain knowledge to identify and correct the inaccuracies. further compounding this issue is the inherent lack of transparency in how llms arrive at their generated responses @xcite . retrieval-augmented generation (rag) has been proposed as a solution for mitigating the aforementioned shortcomings of llms @xcite . specifically, a rag system utilizes a text retriever to identify the documents in a text corpus most relevant to a given query via a semantic similarity measure. the most similar retrieved documents are then provided as additional context to the llm, along with the query for context-augmented generation. by conditioning generation on retrieved documents, new information can be incorporated into llms' responses without additional training. furthermore, rag reduces the likelihood of hallucinations by grounding generation in documents which are a trusted source of truth and enables greater transparency by allowing end-users to inspect documents which were used to produce the response generated by an llm. while rag systems have achieved impressive performance, an essential question for their practical application in downstream systems is how variations in a user's query impact the relevancy of retrieved results. for instance, different users seeking the same information may phrase their queries differently or introduce typographical errors to the query. a desirable attribute of a rag system is that the elements in the system (e.g., retrievers) ## related work existing studies on rag robustness can be broadly grouped as focusing on the retriever, on the llm as the final generator, or on the entire rag pipeline. retriever-level robustness research in this category explores how retrievers maintain performance under various query perturbations @xcite @xcite @xcite . for example, zhuang and zuccon (2022) explores how bert-based retrievers cope with spelling errors and proposes encoding strategies and training procedures to improve robustness. in contrast, @xcite studies how generative retrievers handle query variants. meanwhile, @xcite evaluates retriever stability by perturbing queries' dense representations directly. ## data perturbations we investigate strategies to systematically evaluate the robustness of the rag pipeline under different input perturbations that commonly appear in realworld applications. for each type of perturbation, we also quantify how the performance of different modules in the rag pipeline changes. we focus on perturbations that do not significantly alter the semantic meaning of the query in practical rag use cases while having a high chance of occurrence. specifically, given an original query q, we apply a perturbation perturb(q) such that perturb(q) retains the same or very similar semantics as q. in this work, we generate the perturbed samples in two ways: either via prompting an llm or by inserting random typos. ## experiment details in this section, we describe the elements used in these experiments, such as the datasets and models, to assess the robustness of the rag systems. ## experiments in this section, we detail the steps in our analysis framework and describe our findings in the designed experiments. ## recommendations as a results of our broad experiments across different retrievers, generator models (i.e. llms) and data perturbation types, we provide evidence based practical recommendations to for the improvement of retrieval augmented generation pipelines. these insights are designed to help developers assess the robustness of their rag pipelines against different input transformations, essentially helping developers make robustness-aware decisions while increasing the stability of their system. first, we highlight how different perturbation types have different effects on the modules forming the rag system and its end-to-end performance. our experiments showed that certain perturbations and dataset combinations lead to more sensitivity on a specific rag component. therefore, we recommend that practitioners use our analysis framework on their we recommend that practitioners use our analysis framework on their own data for a better diagnosis of sensitivity in their pipeline. we acknowledge that the robustness of the generators is generally assessed in the closed-book setting without their use in a rag pipeline. however, as our results show, certain query perturbations affect the response generation differently when documents are presented in the context window of the generator. therefore, we recommend that practitioners assess the robustness of the response generation in their systems, especially in an oracle setting, as this setting estimates an upper bound for the system. furthermore, there is an active field of study investigating the training of retrieval augmented generation systems where the retriever and the llm are jointly trained @xcite . these systems benefit from training by becoming more robust against irrelevant contexts when generating the answer. however, the robustness against the query variations in the context of joint training is still underexplored. our findings can be integrated into these training regimes to develop robustnessaware end-to-end systems that are stable against query variations. lastly, as many methods of query disambiguation and expansion show, the query transformations help increase the performance of rag systems while introducing extra computational overhead. since these methods are greatly dependent on the initial query provided to the system and could be employed for different modules of the pipeline, we believe that our decoupling analysis could help practitioners identify the sensitive modules in their pipeline to employ these methods more efficiently for different perturbation types. therefore, we recommend using our provided analysis methodology to test the rag system before employing query transformations steps into the pipeline. ## conclusion in this work, we highlight a key issue of the retrieval augmented generation pipelines, namely their sensitivity to query variations. we perform extensive experiments on three question answering datasets and twelve rag pipelines which span four retrievers and three llm generators. our experiments probing these pipelines with five perturbation types show that slight variations in the input queries can result in significant performance discrepancies. our analysis framework enables the investigation of rag module sensitivity to query perturbations jointly and in isolation. using this framework, we provide practical insights and recommendations for the development of rag systems. we hope that our work brings greater attention to the importance of robustness research at the query level while contributing to the development of future robustness-aware retrieval augmented generation pipelines.