E qualize IR : Mitigating Linguistic Biases in Retrieval Models (2025)

Abstract: This study finds that existing information retrieval (IR) models show significant biases based on the linguistic complexity of input queries, performing well on linguistically simpler (or more complex) queries while underperforming on linguistically more complex (or simpler) queries.To address this issue, we propose EqualizeIR, a framework to mitigate linguistic biases in IR models. EqualizeIR uses a linguistically biased weak learner to capture linguistic biases in IR datasets and then trains a robust model by regularizing and refining its predictions using the biased weak learner. This approach effectively prevents the robust model from overfitting to specific linguistic patterns in data. We propose four approaches for developing linguistically-biased models. Extensive experiments on several datasets show that our method reduces performance disparities across linguistically simple and complex queries, while improving overall retrieval performance.

## introduction neural ranking models have been extensively used in information retrieval and question answering tasks @xcite @xcite @xcite . we demonstrate that these models can show strong linguistic biases, where the retrieval performance is biased with respect to the "linguistic complexity" of queries, quantified by the variability and sophistication in productive vocabulary and grammatical structures in queries using existing tools @xcite @xcite . 1 figure 1 shows that the average linguistic complexity of the test queries in the nfcorpus @xcite and fiqa @xcite datasets 0.00 0.05 0.10 linguistic complexity 0.0 0.1 0.2 0.3 0.4 ndcg@10 0.02 0.04 0.06 0.08 linguistic complexity 0.15 0.20 0.25 0.30 0.35 figure 1: ndcg@10 of bm25 on the test set of nf-corpus @xcite (left) decreases and on the test set of fiqa @xcite (right) increases as the average linguistic complexity @xcite of queries increase. specifically, we observe a significant drop in ndcg@10, from 0.4 to 0, and a significant increase in ndcg@10, from 0.2 to 0.3. the result shows that bm25 is significantly biased toward linguistically easy and hard examples on different datasets. our contribution are (a): illustrating that the performance of current ir models vary based on the linguistic complexity of input queries, (b): a novel approach that trains a linguistically robust ir model with the help of a linguistically biased ir model to mitigate such biases, and (c): four approaches to obtain linguistically biased weak learners, all effective in mitigating biases in ir models. overview: equalizeir mitigates linguistic biases in an ir model using a linguistically-biased weak learner, f b . the process begins with training f b to learn linguistic biases present in a dataset. then, a linguistically robust model, f r , is trained based on the confidence of f b (which approximates the intensity of linguistic biases in input) and the prediction accuracy of f r . this approach has two purposes: firstly, f b guides f r to improve its robustness by learning from the identified biases of f b . secondly, f b can adjust the weights of training examples by prioritizing those that f r fails to predict, which effectively refines the training focus of f r toward more challenging examples. bi-encoder architecture: we consider a standard bi-encoder architecture with a query encoder f q and a document encoder f d @xcite @xcite . given the i-th batch b @xmath0, d + i , d - i,1 , . . . , d - i,n }, where q i denotes the query, d + i denotes a relevant document, and d - i,j , ∀j denote irrelevant documents, we encode them into embeddings h q i , h d + i , h d - i , and optimize the standard contrastive loss: ## conclusion we report that ir models are biased toward linguistic complexity of queries and introduce equalizeir, a framework that trains a robust ir model by regularizing it with four types of linguistically-biased weak learners (by amplifying linguistic constructs in queries, using a weaker model with limited capacity, training with fewer iterations to create an underdeveloped model, and training on less data to restrict generalization), to achieve equitable performance across queries of varying linguistic complexity.