XRAG : Cross-lingual Retrieval-Augmented Generation (2025)

Abstract: We propose XRAG, a novel benchmark designed to evaluate the generation abilities of LLMs in cross-lingual Retrieval-Augmented Generation (RAG) settings where the user language does not match the retrieval results. XRAG is constructed from recent news articles to ensure that its questions require external know-ledge to be answered. It covers the real-world scenarios of monolingual and multilingual retrieval, and provides relevancy annotations for each retrieved document. Our novel dataset construction pipeline results in questions that require complex reasoning, as evidenced by the significant gap between human and LLM performance. Consequently, XRAG serves as a valuable benchmark for studying LLM reasoning abilities, even before considering the additional cross-lingual complexity. Experimental results on five LLMs uncover two previously unreported challenges in cross-lingual RAG: 1) in the monolingual retrieval setting, all evaluated models struggle with response language correctness; 2) in the multilingual retrieval setting, the main challenge lies in reasoning over retrieved information across languages rather than generation of non-English text.

## introduction retrieval-augmented generation (rag) augments large language models (llms) by retrieval of relevant documents with the aim of improving response quality @xcite . the widespread adoption of rag has prompted many recent studies to evaluate specific capabilities of llms in rag settings, such as robustness to noise @xcite , information integration @xcite , time sensitivity @xcite , multi-hop reasoning @xcite and conversational qa @xcite . notably, these evaluations are in monolingual settings in which questions and retrieved documents are in the same language. real-world deployments of rag systems also need to handle cross-lingual use cases, where the user's language does not match that of the retrieved documents. the simplest scenario is cross-lingual rag with monolingual retrieval @xcite , where users in multiple locales are served by a single rag system that accesses an english-only knowledge base, as illustrated in figure 1a . this setup applies to, for example, a general-purpose rag system that relies solely on english web search or a corporate helpdesk with an internal database available only in english. a more complex scenario is cross-lingual rag with multilingual retrieval, where rag systems combine information from both english and the user's language to generate a response (see figure 1b ). this is a common situation in that native-language sources often contain culturally or geographically specific knowledge, with english resources providing additional, more general information. 2 due to the absence of relevant benchmarks, we lack an understanding of how well llms can handle such cross-lingual rag scenarios. a potential solution is to use existing cross-lingual open-domain question-answering datasets, such as xqa @xcite and xor qa @xcite , for evaluation @xcite . yet these datasets only cover limited cross-lingual scenarios; in particular, the documents used to answer questions are in english, which hinders the evaluation of llms in more complex multilingual scenarios (i.e., figure 1b ). moreover, the questions in these datasets tend to be relatively simple (e.g. span extraction questions) and often can be answered without retrieval. 3 due to these shortcomings, these datasets do not measure the true cross-lingual capabilities of llms in rag settings. to address this gap, we introduce xrag, a benchmark for evaluating the question answering capabilities of llms in cross-lingual rag scenarios, where some information must be extracted from retrieved documents that are not in the user's language. the benchmark features naturalsounding questions that require cross-document reasoning and are challenging for llms even in an english monolingual rag setting (gpt-4o achieves only 62.4% accuracy, see xrag comprises two sub-tasks, corresponding to the monolingual retrieval and the multilingual retrieval settings of cross-lingual rag. for each non-english language, we provide a directly com2 an initial study on a proprietary dataset of real-world llm traffic from non-english users in germany, japan, and spain found that using only english or native-language search results was inferior to combining both (see appendix a). 3 chirkova et al. (2024) shows that 47.5% of questions in xorqa can be answered by command-r without retrieval. we evaluate five llms, including both closedand open-source models, on xrag. in summary, our contributions are: @xcite we introduce xrag, a novel benchmark designed to evaluate the performance of llms in two cross-lingual rag scenarios. ## related work there are extensive recent investigations into characterizing the question answering capabilities of llms in rag settings. @xcite construct a a dynamic qa benchmark, freshqa, that tests the ability of llms to use up-to-date world knowledge to solve questions. @xcite introduce rgb to analyze fundamental abilities of llms in rag systems, such as noise robustness and negative rejection. tang and @xcite propose multihop-rag, which focuses on whether retrieval-enhanced llms can retrieve and reason over multiple pieces of supporting evidence. the comprehensive rag benchmark, created by @xcite , aims to assess whether llms are able to answer different types of questions, ranging from simple to complex. thakur et al. (2024b) present mirage-bench, a multilingual rag benchmark constructed from wikipedia, to evaluate rag systems performance in different languages. zhu et al. rag setting field language content question german wie viel haben walmart und aldi zusammen für die opfer des hurrikans helene 2024 gespendet? answer german die gesamten spenden überstiegen 11 millionen dollar. english walmart, sam's club and the walmart foundation are increasing their commitment to @xmath01,000,000. by making a donation to hurricane helene relief... english walmart canada reaches new giving milestone of $750 million raised and donated to charities and non-profits across canada... monolingual retrieval distracting articles english aldi has donated £2,000 to charities in gloucestershire to help support those in need during the school holidays. the donations... question german welches land gewann seine erste goldmedaille bei den olympischen spielen 2024 früher, die vereinigten staaten oder deutschland? answer german deutschland. beide länder gewannen am 27. juli bei den schwimmwettbewerben ihre ersten goldmedaillen, aber die usa gewannen ihre medaille erst im letzten wettkampf des tages -später als deutschland. german lukas märtens hat am 27. juli in paris den olympischen titel über 400 m freistil gewonnen. der magdeburger siegte in 3:41,78... supporting articles english in the last swimming race of july 27, the u.s. took its first gold medal of the 2024 olympics, winning the 4×100-meter freestyle... german im rahmen von noch bevorstehenden qualifikationsevents können sich weitere sportler noch für die spiele in paris qualifizieren... multilingual retrieval distracting articles english the united states olympic & paralympic committee have announced the 592-member 2024 u.s. olympic team ready to compete... evaluation of cross-lingual nlp systems is a long-standing research problem. relatively recent work has focused on performance in specific nlp tasks such as nli @xcite , summarization @xcite , retrieval question answering @xcite , and open-domain question answering @xcite . with the advent of large language models, cross-lingual evaluation has expanded to include few-shot or even zero-shot settings. @xcite investigate gpt-4 performance for crosslingual summarization in a zero-shot setting and find that it performs competitively with finetuned mbart-50. @xcite evaluate the performance of generative models on 15 tasks, covering classification, sequence labeling, and generation. we note that our focus is on retrieval augmented generation from multilingual document retrieval, and not on the document retrieval task itself. however in sections 4.1 and 4.4 we discuss how we use monolingual and multilingual dense document retrieval techniques in constructing xrag. our work aligns with recent efforts in evaluating crosslingual performance of llms, with a focus on retrieval augmented generation and cross-lingual answer generation in particular. ## xrag -cross-lingual rag benchmark we define the task of cross-lingual rag as follows: given a question q, the llm is prompted to generate an answer ã in the same language as the question by referring to a collection of m articles @xmath1, d 2 , ..., d m } that contains articles in a language different than the question: figure 1 shows two cases, in which llms need to use information from english articles to generate german responses to german questions. the goal of our benchmark is to enable an understanding of how well llms perform generation in such cross-lingual rag scenarios. monolingual retrieval setting multilingual retrieval setting news 1 identifying pairs of related articles 2 crossdocument q&a generation 3 quality control 4 human translation 5 english retrieval+ filtering 5 language x retrieval+ filtering news doc 1 doc 2 english + english doc 1 doc 2 x + english discard no yes doc 8 doc 7 doc 6 doc 5 doc 4 doc 3 english doc 8 doc 7 doc 6 doc 5 doc 4 doc 3 english x q&a q&a x + english d + d + d - d - 𝑞, 𝑎 figure 2: each instance (q, a, d + , d -) in xrag -where q is the question, a the gold answer, d + the supporting articles, and d -the distractors -is constructed as follows: (1) find two related articles; (2) generate an english cross-document q&a pair using the two articles; (3) evaluate the quality of the q&a pair; ( 4 ) translate the q&a pair into language @xmath2, spanish, chinese, arabic}; and (5) collect distracting articles for the question. each instance (q, a, d + , d -) in xrag consists of a question q, a golden answer a, two supporting articles d + , and several distracting articles d -. the supporting articles each contribute partial information needed to answer the question, and only together do they provide a complete answer; in contrast, the distracting articles are topically related but cannot answer the question. taken together, this simulates a realistic rag scenario with imperfect retrieval, where we can control the quality of the grounding by the inclusion of distractors. questions in xrag are cross-document questions, requiring reasoning across the two supporting articles to answer, while ignoring the distracting articles. our benchmark considers two real-world crosslingual rag scenarios: the monolingual retrieval scenario and the multilingual retrieval scenario. in the monolingual retrieval setting, llms rely on english articles to generate an answer. this occurs when users in multiple locales are served by a single cross-lingual rag system that has access only to an english knowledge base. in this paper, we consider questions in four languages: german (de), spanish (es), chinese (zh), and arabic (ar): language(q) ∈ {de, es, zh, ar} these four are widely used in the research community @xcite and represent a range of cross-lingual challenges, ranging from easy (es-en) to challenging (zh-en) @xcite . in a multilingual retrieval setting, llms use articles in both the question language and other languages to answer a question. this corresponds to a cross-lingual rag scenario where documents in a resource-rich language provide additional information for llms to answer questions in a second language. similarly, we consider four languages: language(q) ∈ {de, es, zh, ar} ## xrag construction figure 2 shows the overall xrag construction process. we begin with english, german, spanish, chinese, and arabic news articles from news crawl between @xcite . this timeframe ensures that the articles are dated after the knowledge cutoff of llms such as gpt-4 and claude 3.5. questions created from these articles are more likely to require external knowledge to answer. ## conclusions we introduce xrag, a benchmark for evaluating the generation abilities of llms in cross-lingual rag settings. we introduce a novel llm-based workflow for creating questions that require complex reasoning and external documents to answer. experiments reveal that llms significantly underperform humans on xrag, even without crosslingual elements, highlighting its utility for assessing reasoning ability. further analysis shows that llms struggle with response language correctness in the xrag monolingual retrieval setting and with reasoning over retrieved content across languages in the xrag multilingual retrieval setting.