Mitigating Bias in RAG : Controlling the Embedder (2025)

Abstract: In retrieval augmented generation (RAG) systems, each individual component—the LLM, embedder, and corpus—could introduce biases in the form of skews towards certain genders or political leanings. In this work, we study the conflict between biases of each component and their relationship to the overall bias of the RAG system, which we call \textit{bias conflict} . Examining both gender and political biases as case studies, we show that bias conflict can be characterized through a linear relationship among components despite its complexity. Through fine-tuning, we demonstrate how to control the bias of the embedder while maintaining utility and reveal the importance of \textit{reverse-biasing} the embedder to mitigate bias in the overall system, Additionally, we find that LLMs and tasks exhibit varying \textit{sensitivities} to bias, a crucial factor to consider for debiasing. Our results underscore that a fair RAG system can be better achieved by carefully controlling the bias of the embedder rather than increasing its fairness.

## introduction retrieval-augmented generation (rag) @xcite @xcite is a promising modular ai system that enhances factuality and privacy in large language models (llms). this safety enhancement is accomplished by breaking the system into three different components: the llm, embedder, and corpus which overall complement the llm's knowledge with non-parametric information (figure 1 ). however, each of these components risk introducing their own biases (e.g., skews towards outputs representing certain identities or opinions) into the rag system, which could cause representational harm and unsafe user interactions @xcite . a rag system consists of the llm, embedder, and corpus. given a query as input, the embedder retrieves documents from the corpus that are similar to the query. the llm takes as input the query and retrieved document to generate an output. each component introduces bias into the system which propagates into latter stages. we find that the change in rag bias (s • ∆b ) scales linearly with the change in embedder bias (∆b ) as shown in figure 3 , where s denotes the sensitivity of the rag output to the embedder bias. understanding the interaction of bias between each component in a rag system remains a significant challenge @xcite @xcite . each component may not only amplify bias but also conflict with each other's bias, creating a phenomenon we call bias conflict. for example, given the query who is a famous singer?, an embedder biased towards men could lead to a document about michael jackson being retrieved, while a corpus biased towards women would make whitney houston more likely to be retrieved. the opposing biases make the final retrieval uncertain. additionally, an llm biased towards females would also conflict with the embedder, further complicating the process. thus, given the ambiguity of the final output bias, it is crucial to understand how biases from each component interact in order to effectively mitigate bias of the entire rag system. in this work, we investigate such bias conflicts in rag systems. we specifically examine the embedder's role in bias conflict as well as its potential for mitigating rag biases. focusing on the em-bedder has three advantages over mitigating bias through the llm or corpus. first, most embedders are smaller than llms. the best performing embedder on the mteb leaderboard @xcite is only 7b parameters, while llms easily have a 100b+ parameters. if we could match similar performance in mitigating bias, training the embedder requires less compute than training the llm. second, llms are prone to catastrophic forgetting during fine-tuning @xcite , which degrade the generation quality. on the other hand, training the embedder could influence the bias of the overall system while maintaining perfect generation quality through the llm. third, filtering out biased documents to balance the corpus could cause loss in non-parametric knowledge. we empirically examine bias conflict through binarized gender and political bias, as case studies representing both a clear-cut and a nuanced type of bias, respectively. we construct our tasks so that bias can be introduced independently of factuality ( §2.4). 1 this lets us examine subtle bias concealed under factual correctness, making it difficult for users to recognize @xcite . we fine-tune 120 embedders to have different biases with peft and wise-ft @xcite in order to observe how training affects utility. to investigate how changing the embedder bias impacts the rag bias, we evaluate 40 embedders each connected to 6 different llms, resulting in 240 different rag systems. we further evaluate the 40 embedders on corpora of varying bias, portraying the effect of changing the corpus. through these experiments, we answer the following questions: q1: can we predict the overall bias of a rag system given the biases of individual components ( §3)? we measure the bias of each individual component and the entire rag system. we find that even when knowing the exact bias of the embedder and llm, it is challenging to predict whether the rag bias will amplify or decrease because of bias conflict. q2: given complex bias conflict, how can we effectively mitigate bias in a rag system ( §4)? due to a linear relationship between the bias of the rag system and embedder, we find that reverse biasing the embedder through fine-tuning is effective in mitigating the overall rag bias. we also observe that llms are more sensitive to changes in the embedder for gender bias than political bias. furthermore, despite the small size of our embedder (109m), we are able to overcome the bias of a larger language model (405b). q3: how does bias conflict from the corpus affect the robustness of mitigating bias through the embedder ( §5)? to perturb the corpus bias, we create a small corpus where the bias of each document is pre-evaluated. we find that an embedder which mitigates bias given a fixed corpus also mitigates bias for small perturbations in the corpus. through this work, we show that increasing fairness in the llm or embedder may not be the optimal solution to mitigating bias in the overall rag system. considering the interaction and conflict among biases in each component is crucial towards achieving less biased rag systems. ## measuring bias in rag before understanding the effect of individual rag components on bias, we first define rag components ( §2.1) and bias measures ( §2.2) for each of the components and the entire rag system. based on these definitions, we establish a linear model capturing the characteristics of bias conflict ( §2.3). we then describe the two datasets used in our case studies on rag biases, namely, gender and political ( §2.4), and our experimental settings ( §2.5). ## results: existing bias in rag to understand the relationship between the embedder and the llm, we first evaluate the bias of both components on the test splits of genderbias-qa and politicbias-qa. shown in when examining bias amplification or conflicts in rag system, we find that gender bias remains similar or sometimes amplifies when the llm is connected to the embedder to compose a rag system. for example, the bias of llama 8b increases towards males by -0.52-(-0.62) = 0.10. on the other hand, political bias tends to decrease (closer to 0) when inside a rag system, with the exception of gemma models. although the overall bias of the rag system leans toward the majority bias of the components, it is not clear whether bias from each component would cancel out or amplify to produce the overall outcome. ## results: debiasing rag given the complexity of bias conflict in a rag system, is it feasible to mitigate bias in the entire rag system? in this section, we try to control the embedder to mitigate bias. in §4.1 we first fine-tune several embedders to span a wide bias range. then in §4.2, we construct a rag system with these embedders while keeping the llm and corpus fixed to understand the relationship between the embedder bias and rag bias (equation 2 ). ## results: corpus & rag in the previous section, we revealed a linear relationship between the embedder bias and rag bias while keeping the corpus consistent. here we investigate how changing the corpus bias (c b ) affects the linear trend seen previously in figure 3 . we create small toy corpus with pre-evaluated biases of each document to systematically study this. for genderbias-qa, we collect a subset of nq by first selecting the top-100 documents related to each query with the base embedder. next, we keep the number of documents that are biased towards males and females equal. this results in a small corpus of 352 documents (male: 176 / female: 176). we note that this subset has a different distribution from nq. we repeat the same for politicbias-qa with polnli and get a corpus of 2564 documents (liberal: 1282 / conservative: 1282). ## conclusion and discussion in this work, we studied bias conflict between different components-the llm, embedder, and corpus-in rag systems, and explored the possibility of mitigating bias in rag by controlling the embedder. we showed through case studies on gender and political bias that, while bias conflict may seem unpredictable ( §3), there exists a simple relationship explained through a linear model. con-sidering this relationship, we revealed that reverse biasing the embedder can debias the overall rag system ( §4). furthermore, we find that an optimal embedder on one corpus is still optimal for variations in the corpus bias ( §5). below, we discuss the implications of our results and related work on bias measurement and mitigation in rag. debiasing each component even with complex bias conflict in the entire rag system, we show that debiasing can happen by simply reverse biasing the embedder. however, most work on bias in rag has focused on making the embedder or retrieval process less biased bias conflict to understand bias mitigation in rag systems, we introduce the concept of bias conflict. similar to knowledge conflict @xcite @xcite , bias conflict arises when parametric and non-parametric information differs. however, while knowledge conflict focuses on factuality, bias conflict assumes parametric and nonparametric information are both factually correct. bias conflict also extends beyond the retrieved document and llm, generally arising between compo-nents. in our work, we consider the two cases of (1) the llm and embedder and (2) the embedder and corpus. we believe that factuality is not the sole conflict existing in rag systems and more interest should be paid to other forms of conflict. traditional gender bias we have created genderbias-qa which focuses on gender bias through names of public figures. this is different from traditional gender bias datasets that focus on association-based bias, measuring stereotypes by evaluating pronouns (he/she) or occupational bias @xcite . while llms are already rlhf fine-tuned to prevent association-based gender bias, they are not properly fine-tuned for names of figures. we believe that bias is not restricted to stereotypes and should be prevented regardless of the form. we hope genderbias-qa can be used as a testbed for mitigating gender bias for figure names. the use of public figures' names in genderbias-qa is intended solely for research purposes and should be used with care taken to avoid misrepresentation or harm.