Evaluating Large Language Models for Cross-Lingual Retrieval (2025)

Abstract: Multi-stage information retrieval (IR) has become a widely-adopted paradigm in search. While Large Language Models (LLMs) have been extensively evaluated as second-stage reranking models for monolingual IR, a systematic large-scale comparison is still lacking for cross-lingual IR (CLIR). Moreover, while prior work shows that LLM-based rerankers improve CLIR performance, their evaluation setup relies on machine translation (MT) for the first stage. This is not only prohibitively expensive but also prone to error propagation across stages. Our evaluation on passage-level and document-level CLIR reveals that this setup, which we term noisy monolingual IR , is favorable for LLMs. However, LLMs still fail to improve the first-stage ranking if instead produced by multilingual bi-encoders. We further show that pairwise rerankers based on instruction-tuned LLMs perform competitively with listwise rerankers. To the best of our knowledge, we are the first to study the interaction between retrievers and rerankers in two-stage CLIR with LLMs. Our findings reveal that, without MT, current state-of-the-art rerankers fall severely short when directly applied in CLIR.

## introduction cross-lingual information retrieval (clir) aims to retrieve documents written in a different language than the query, which facilitates multilingual access to information. traditionally, clir systems have relied heavily on machine translation (mt) to convert either queries or documents into a shared language, effectively transforming the original crosslingual task into a monolingual task @xcite @xcite @xcite . while this mt-based setting has been the dominant approach, it introduces several practical and methodological limitations: translation increases the query latency; it remains unavailable or unreliable for many low-resource languages @xcite , adversely affecting cross-lingual retrieval when translations contain errors @xcite . recent work leveraging large language models (llms) in information retrieval has demonstrated promising gains over baseline systems @xcite , highlighting their capability in ranking tasks. while prior work focuses on monolingual retrieval and reranking, cross-lingual llmbased retrieval has been understudied. to the best of our knowledge, the only prior works on llmbased clir rely on mt to bridge the language gap in the retrieval stage @xcite and evaluates rerankers on a limited number of languages @xcite . in our study, we move beyond translation-based setups and conduct a large-scale evaluation of llms for cross-lingual retrieval and reranking. figure 1 illustrates our experimental setup. we compare the performance of state-of-the-art bi-encoders against mt-based lexical retrieval, and quantify the translation gap between rerankers applied on original language documents (og) versus translated documents (dt). our work is also the first to compare listwise and pairwise llm-based rerankers on clir. our research addresses the following key questions: rq1: how do recent dense multilingual bi-encoders and sparse bm25 differ in their performance as first-stage retrievers for cross-lingual retrieval? rq2: to what extent do llm rerankers improve retrieval performance when paired with different types of first-stage retrievers, and how does this interaction vary across high-resource and low-resource language settings? rq3: how do pairwise and listwise reranking approaches influence cross-lingual reranking performance? rq4: what is the impact of document length on listwise and pairwise reranking in clir? by addressing these questions, we provide a comprehensive view of llm rerankers' crosslingual capabilities and limitations, offering critical guidance for building more effective and adaptable multilingual retrieval solutions. in summary, our main contribution is a systematic evaluation of llms for cross-lingual reranking without fully relying on document translation. we release our code and resources to facilitate reproducibility and future research. foot_0 2 related work ## evaluation framework in the following, we describe our evaluation framework shown in figure 1 . to quantify to what extent gains in the retrieval stage translate to better reranking results, we experiment with translation-based lexical retrieval as well as multilingual bi-encoders. we also compare listwise and pairwise rerankers and quantify the impact of translation. ## conclusion in this paper, we conducted a systematic evaluation of llms for clir, evaluating both passageand document-level reranking without relying entirely on machine translation. our results reveal that further cross-lingual reranking gains can be achieved by substituting lexical mt-based retrievers with better-performing multilingual bi-encoder and hybrid retrieval approaches, and that the benefits resulting from document translation diminish with stronger listwise rerankers. we further demonstrate that instruction-tuned pairwise rerankers perform competitively with listwise rerankers like rankzephyr. finally, we highlight the sensitivity of reranking performance to document length and language resource disparities. our findings highlight the need for more robust approaches to harness llms for cross-lingual reranking without relying on machine translation.