An Empirical Study of Position Bias in Modern Information Retrieval (2025)

Abstract: This study investigates the position bias in information retrieval, where models tend to overemphasize content at the beginning of passages while neglecting semantically relevant information that appears later. To analyze the extent and impact of position bias, we introduce a new evaluation framework consisting of two position-aware retrieval benchmarks (SQuAD-PosQ, FineWeb-PosQ) and an intuitive diagnostic metric, the Position Sensitivity Index (PSI), for quantifying position bias from a worst-case perspective. We conduct a comprehensive evaluation across the full retrieval pipeline, including BM25, dense embedding models, ColBERT-style late-interaction models, and full-interaction reranker models. Our experiments show that when relevant information appears later in the passage, dense embedding models and ColBERT-style models suffer significant performance degradation (an average drop of 15.6%). In contrast, BM25 and reranker models demonstrate greater robustness to such positional variation. These findings provide practical insights into model sensitivity to the position of relevant information and offer guidance for building more position-robust retrieval systems. Code and data are publicly available at: https://github.com/NovaSearch-Team/position-bias-in-IR.

## introduction information retrieval (ir) underpins a broad range of applications, such as web search @xcite , question answering @xcite , and retrieval-augmented generation (rag) @xcite . a central challenge for ir systems is to accurately assess the semantic relevance between user queries and candidate passages. recent advances in neural ir models, particularly those leveraging pre-trained language models such as * project leader. ## related work prior work has extensively documented the existence of position bias in neural ir systems. @xcite observe that in ms marco @xcite , a widely used col-lection in the ir community, answer spans are disproportionately concentrated in the earlier portions of passages. bert-based neural ir models finetuned on ms marco tend to inherit and reinforce this position bias @xcite , potentially leading to overestimated performance due to shared distributional artifacts between training and evaluation sets @xcite . @xcite further dissect the training pipeline of a t5-based dense retriever on ms marco, showing that position bias primarily originates during contrastive pre-training and is amplified in contrastive fine-tuning. @xcite extend this analysis by uncovering multiple forms of bias-including position bias-in dense embedding models, and linking them to generation failures in rag pipelines. beyond ir, similar position-sensitive behaviors have been observed in llms, particularly in how attention is distributed across long contexts @xcite @xcite , suggesting that position bias may be a more general limitation of modern transformer architectures. ## conclusion we conduct a comprehensive study of position bias in the modern ir pipeline. to enable realistic evaluation, we introduce two position-aware retrieval benchmarks: squad-posq and fineweb-posq, repurposed from existing datasets while preserving semantic integrity. we further propose the position sensitivity index (psi), a simple and intuitive metric for quantifying position bias across retrieval models. our findings reveal that while position bias primarily arises in embedding-based retrievers, it can be substantially mitigated by downstream interaction-based reranker models.