Relevant for the Right Reasons? Investigating Lexical Biases in Zero-Shot and Instruction-Tuned Rerankers (2025)

Abstract: Large Language Models (LLMs) show strong potential for reranking documents in information retrieval (IR), but training with monolingual data often leads to monolingual overfitting and lexical bias, limiting generalization in cross-lingual IR (CLIR). To overcome these issues, we investigate instruction-tuning LLaMA-3.1-8B-Instruct on English and multilingual code-switched data, and evaluate on mMARCO and XQuAD-R. Results show that instruction-tuning on code-switched data substantially improves CLIR performance, while monolingual tuning remains more effective for monolingual reranking. We introduce a novel measure to analyze the relationship between lexical overlap and reranking performance, showing that the two factors are correlated. We finally conduct a causal analysis using counterfactual examples, where we evaluate whether rewriting passages that share overlapping keywords with the query causes models to change their relevance predictions. Overall, we find that code-switching serves as an effective and lightweight strategy to improve cross-lingual generalization in LLM-based re-ranking, while our analyses show that lexical overlap remains a major factor that can mislead reranking models.

## introduction large language models (llms) such as llama-3 @xcite , @xcite , gemini @xcite , and mistral @xcite have shown strong performance across a wide range of nlp tasks. in information retrieval (ir), which aims to return relevant documents from large text collections given a user query, recent advances have led to growing interest growing interest in leveraging llms as rerankers. in particular, llms have been explored as pointwise @xcite , pairwise @xcite , or list-wise rerankers (tang et al., * work done while at lmu munich. figure 1 : the first passage is semantically relevant to the query but shares no lexical overlap. in contrast, the second passage contains lexical overlap with the query terms "population" and "paris" but is topically unrelated. lexically biased llm rerankers may incorrectly favor the non-relevant passage. understanding whether llms determine relevance for the right reasons (i.e., semantic relevance), or whether they are biased towards lexical matches (i.e., shortcuts) is crucial for equitable information access and ensuring the trustworthiness of llm-based retrieval systems @xcite . biases in cross-lingual retrieval settings have been well-studied in the context of multilingual pre-trained language models (mplms). prior work includes studies on, e.g., language biases in mplm-based bi-encoders @xcite @xcite . our work is closest to @xcite , who study zero-shot cross-lingual transfer of mplm-based cross-encoders, where models trained on english data have been found to exhibit poor transfer performance to cross-lingual reranking tasks. the authors show that this monolingual overfitting can be mitigated by training on codeswitched data instead, which naturally reduces the lexical overlap between queries and documents. however, it remains unclear whether llms exhibit similar lexical biases when used as rerankers, and whether instruction-tuning those models on code-switched training data also leads to similar improvements. figure 1 illustrates this issue for a single pairwise cross-lingual reranking step: the model incorrectly prefers a lexically overlapping but semantically irrelevant passage, suggesting that relevance judgments may not always reflect genuine semantic understanding. this motivates our central question: are llm-based reranker outputs relevant for the right reasons? to address this, we investigate whether llmbased rerankers are affected by monolingual overfitting and lexical bias, and how instruction tuning strategies change this behavior. specifically, we compare direct zero-shot reranking (without further training) against instruction-tuning on monolingual english data, multilingual code-switched data and target language-pair data on both moir and clir. in addition to our reranking experiments, we also characterize the lexical bias through a correlation and causal analysis. our main contributions are: however, unlike mplm-based crossencoders, these gains come at the cost of a worse monolingual reranking performance. shortcut learning in language models. several recent studies have investigated shortcut learning behavior in llms, where models rely on superficial features in the input, such as lexical overlap or specific keywords, instead of performing genuine semantic reasoning. du et al. (2021) focus on bert-based models and show that these models tend to favor shortcut tokens early in training. tang et al. (2023) found that llms often rely on shallow cues from prompts during in-context learning, rather than understanding the task itself. sun et al. (2024) showed that instruction tuning and reinforcement learning with human feedback can increase shortcut learning in llms across tasks such as reasoning. yuan et al. (2024) provided a systematic evaluation of shortcut biases, including lexical overlap, in prompt-based inference. hagstrom et al. (2025) found that lm-based rerankers can be misled by lexical similarities, often favoring candidates with high surface overlap over semantically more relevant passages on english-only retrieval tasks. the study shows that these biases can lead to significant drops in model accuracy. taken together, these studies suggest that shortcut learning remains a major challenge for llms. however, these works do not explore how shortcut bias behavior changes when llms are finetuned for monolingual and cross-lingual pairwise reranking. we fill this gap and study shortcut learning behavior in prompt-based reranking tasks, and especially regarding the model's sensitivity to lexical overlap. ## methodology we conduct three different types of analyses: first we investigate how well llm rerankers instructiontuned on english data generalize to other monolingual reranking (moir) and cross-lingual reranking (clir) tasks, or whether they suffer from monolingual overfitting (section 3.1). we then propose a measure that captures the correlation between lexical overlap and reranking performance (sec-tion 3.2). finally, we introduce an evaluation protocol that facilitates a causal analysis of the impact of lexical overlap on reranking performance at the instance-level (section 3.3). ## results and discussion in the following, we first measure the performance gap of llms in monolingual reranking (moir) and cross-lingual reranking (clir). we specifically investigate how well different instruction-tuning strategies impact the generalization performance. we then validate our findings on xquad-r. ## further analysis in section 6.1, we first establish to what degree the reranking performance is correlated to the lexical overlap between queries and documents. we then investigate the reranking results at the instance level by inspecting individual pair-wise classification results (section 6.2). here, we evaluate whether removing lexical overlap causes models to recover from incorrect predictions. ## conclusion in this study, we investigate to what extent llmbased rerankers suffer from lexical biases as opposed to semantic relevance. our results on moir and clir show that instruction-tuning on english data is most effective for monolingual retrieval, whereas code switching provides the largest benefits in clir. we also show that the correlation between reranking performance and lexical overlap is stronger for models trained on monolingual data compared to those trained on code-switched data. our causal analysis reveals that spurious lexical cues can mislead the model, but their removal often restores correct semantic judgments. these findings highlight both the promise of code-switched data for improving cross-lingual generalization and the need to address lexical bias to ensure that llms are "relevant for the right reasons."