On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generation (2025)

Abstract: Retrieval-augmented generation (RAG) with large language models (LLMs) has demonstrated strong performance in multilingual question-answering (QA) tasks by leveraging relevant passages retrieved from corpora. In multilingual RAG (mRAG), the retrieved passages can be written in languages other than that of the query entered by the user, making it challenging for LLMs to effectively utilize the provided information. Recent research suggests that retrieving passages from multilingual corpora can improve RAG performance, particularly for low-resource languages. However, the extent to which LLMs can leverage different kinds of multilingual contexts to generate accurate answers, independently from retrieval quality, remains understudied. In this paper, we conduct an extensive assessment of LLMs’ ability to (i) make consistent use of a relevant passage regardless of its language, (ii) respond in the expected language, and (iii) focus on the relevant passage even when multiple ‘distracting passages’ in different languages are provided in the context. Our experiments with four LLMs across three QA datasets covering 48 languages reveal a surprising ability of LLMs to extract relevant information from passages in a different language than the query, but a much weaker ability to produce a full answer in the correct language. Our analysis, based on both accuracy and feature attribution techniques, further shows that distracting passages negatively impact answer quality regardless of their language. However, distractors in the query language exert a slightly stronger influence. Taken together, our findings deepen the understanding of how LLMs utilize context in mRAG systems, providing directions for future improvements. All codes and data are released at https://github.com/Betswish/mRAG-Context-Consistency.

## introduction retrieval-augmented generation has shown strong results in multilingual question-answering (qa) tasks @xcite . given a query in the user language, informative passages are retrieved from a reference corpus and provided jointly with the query, promoting the large language model (llm) to generate more precise responses @xcite . in multilingual rag (mrag), retrieval can be performed either monolingually or crosslingually. in the former, retrieval is performed only over passages in the same language as the query @xcite @xcite , which can be successful for high-resource languages. however, this approach is marginally useful, or even harmful, when the question is posed in a low-resource language, since relevant information is likely to be available only in different languages @xcite . in addition, for questions regarding a specific geographical region or culture, essential information may be present only in corpora of the languages spoken in that region. to address this issue, cross-lingual retrieval attempts to extract useful information simultaneously from multiple languages @xcite , leading to visible gains in lowresource languages @xcite evaluating rag pipelines is notoriously difficult due to the open-endedness of the retrieval task, and to the complex interactions of retrieval quality with model understanding and generation abilities. on top of this, multilinguality adds another layer of complexity. ideally, retrieved passages should be equally useful when the same question is posed in different languages. besides, llm-generated answers should be consistently correct across languages so that users with different language backgrounds enjoy a similar experience. however, despite the reported accuracy improvements, the abilities of llms to exploit cross-lingually retrieved contexts in mrag remain poorly understood. in this paper, we conduct an in-depth assessment of these abilities, using standard accuracy evaluation as well as feature attribution analysis. unlike recent mrag evaluations @xcite , which test the llm performance for each language in the entire rag pipeline (i.e., retrieval + generation), we disentangle these two factors and focus on the llm's ability to exploit context independently from retrieval quality. as shown in figure 1 , our key contributions include: (a) we evaluate how llms leverage retrieved passages in different languages in various multilingual qa tasks, revealing remarkably robust input understanding but much more brittle generation abilities. (b) besides the standard accuracy evaluation, we apply a recently proposed rag answer attribution method based on model internals @xcite to confirm that llms consistently incorporate retrieved content from various languages, providing insights from an interpretability perspective. (c) we consider both single-passage and multi-passage mrag setups and examine how distracting passages in different languages affect model performance, shedding light on the complex interplay between relevance and content of the retrieved passages. taken together, our results deepen our understanding of how llms utilize context in mrag systems and reveal important areas for future improvements. ## experimental setup consider a multilingual qa setup where q ℓ is a query in language ℓ and a ℓ is the gold answer in the same language. for each query, a set of relevant passages p @xmath0, . . . , p n } in multiple languages is retrieved from a reference multilingual corpus. a relevant passage (@xmath1, or non-gold ('distracting') p otherwise. to perform mrag, a subset of relevant passages c q ⊂ p q is selected and provided as extra context to the llm along with query q ℓ . in an ideal mrag setting, the model should answer more accurately when provided with c but it should also be agnostic to the languages in which the passages @xmath2, in terms of both answer accuracy and feature attribution results. following muller et al., 2023, we use the term 'in-language' for the same language as the user query language, and 'out-language' for different languages than the user language. given this setup, we study llms' ability to handle multilingual context in different retrieval scenarios, which we simulate by varying (i) the number of gold and non-gold ('distracting') passages provided in c, and (ii) the languages of those passages. ## single-passage mrag we start from a simple scenario where, for each query q ℓ , only one gold passage is provided to the model either in the query language (in-language; @xmath3; @xmath4, ℓ ′ ̸ = ℓ). as a baseline, we calculate answer accuracy when no context is provided to the model (@xmath5. in xquad, where gold passages are translated into 12 languages, we iterate over the 11 outlanguage passage versions for each query and report the average accuracy. we also report accuracy for the passage language that yielded the best (or variants can be decisive, particularly for phonologically similar languages, we do not adopt the variant of the softer lexical metric @xcite (3-gram recall), which tolerates minor orthographic differences and could blur the distinctions. figure 2 : performance on xquad, mkqa, gmmlu-open, and gmmlu-choice, where the llms are provided with no retrieved passage or one gold passage in either in-language or out-language. the shading on the bars represents the ratio of questions that can be correctly answered but in the wrong passage language, which does not apply to gmmlu-choice since the evaluation on it is not affected by the generation language. ## multi-passage mrag real-world rag settings are further complicated by the presence of multiple passages c @xmath6, . . . , p n }, some of which may be related to the query but not functional to answering it correctly (i.e. 'distracting' passages p). we investigate how the language of different passages in the context affects llms' ability to locate the right information, assuming this is included in at least one passage of the context. in particular, we aim to assess model robustness in a challenging scenario where the important information is only provided in a different language than the query, along with several in-language distractors. for simplicity, we set the maximum number of passages to 4 and simulate two practical scenarios: (i) a weak retriever finds one out-language gold passage while the other three are distractors; (ii) a strong retriever finds three out-language gold passages while the remaining one is a distractor. in both cases, we compare accuracies when the distractors are in-language vs. out-language. we conduct experiments on mkqa and gmmlu-choice. xquad is excluded because it is an extractive qa dataset, unsuitable for multi-passage mrag. ## conclusion in this work, we explored the challenge of consistent context utilization in mrag systems. specifically, we assessed the ability of various state-ofthe-art llms to handle various kinds of multilingual context while strictly controlling for retrieval quality. our experiments across three diverse qa datasets, using standard accuracy evaluation as well as feature attribution analysis, reveal a remarkable ability of llms to understand multilingual contexts and to locate the important information in relevant passages regardless of their language. in fact, models provided with multiple gold passages in languages different from that of the query are more likely to answer correctly than when provided with a single gold passage in the query language, reflecting the potential of retrieving cross-lingually rather than monolingually for mrag. at the same time, we also detected some important directions for future improvement. firstly, poor generation abilities in many languages push the models to respond in a different language than that of the query, resulting in answers that would be deemed useless by most end-users. importantly, we showed that this also happens when the retrieval works optimally. this suggests that, rather than just trying to optimize the retriever, it may be more effective to invest on the model generation abilities in a specific (set of) user language(s) -for instance by continued pre-training @xcite on generic corpora of those languages-or to apply techniques that push the model to decode in a given language, such as contrastive decoding @xcite . sec-ondly, the presence of distracting passages (i.e., relevant to the query topic, but not directly functional to answer it) in the context can have a very negative effect on answer accuracy in open-domain qa. while this effect is rather similar regardless of the distractors' language, it does highlight the importance of carefully ranking the retrieved passages and to aim for precision when selecting which passages are provided to the model. to conclude, our work underscores the potential of cross-lingual retrieval in enhancing multilingual qa performance, and stresses the importance of focusing not only on retrieval optimization but also on improving language-specific generation. we believe this dual focus will be key to unlocking more robust and user-friendly mrag systems that can operate effectively across diverse language settings.