Investigating Language Preference of Multilingual RAG Systems (2025)

Abstract: Multilingual Retrieval-Augmented Generation (mRAG) systems enhance language models by integrating external multilingual information to produce context-aware responses. However, mRAG systems struggle with retrieving relevant information due to linguistic variations between queries and documents, generating inconsistent responses when multilingual sources conflict. In this work, we systematically investigate language preferences in both retrieval and generation of mRAG through a series of experiments. Our analysis indicates that retrievers tend to prefer high-resource and query languages, yet this preference does not consistently improve generation performance. Moreover, we observe that generators prefer the query language or Latin scripts, leading to inconsistent outputs. To overcome these issues, we propose Dual Knowledge Multilingual RAG (DKM-RAG), a simple yet effective framework that fuses translated multilingual passages with complementary model knowledge. Empirical results demonstrate that DKM-RAG mitigates language preference in generation and enhances performance across diverse linguistic settings. Code is available at https://github.com/jeonghyunpark2002/LanguagePreference.git

## introduction multilingual retrieval-augmented generation (mrag) extends traditional retrieval-augmented generation (rag) @xcite ) by leveraging multilingual external sources to generate accurate, contextually and linguistically aware responses. however, mrag systems face challenges in retrieving relevant information due to linguistic discrepancies between queries and documents @xcite . moreover, conflicts among multilingual sources can lead to inconsistencies in the generated responses @xcite . beyond retrieval challenges and source conflicts, language preference is another critical issue in mrag systems, often leading to inaccurate outputs. as illustrated in case 1 of figure 1 , the retriever may prioritize particular languages-especially high-resource or query-language documents-at the expense of truly relevant information in lowresource languages. consequently, the large language model (llm) either produces an incorrect answer or deems the query unanswerable due to irrelevant content in the documents. likewise, in case 2, even if relevant documents are retrieved, the generator might favor passages in the query language or latin scripts, ignoring essential evidence in lower-resource languages and resulting in inaccurate outputs. these preferences ultimately limit the effectiveness of mrag, yielding biased rankings, reduced answer quality, and inconsistencies across languages @xcite . prior studies @xcite @xcite have investigated this issue by introducing language fairness metrics to assess whether documents from different languages are ranked equitably via statistical equivalence testing, by proposing language-preference-based re-ranking for multilingual information retrieval, and investigating llm's linguistic preference in across-language rag-based information search setting. however, these approaches primarily focus on a limited set of languages and fail to reflect the true ranking dynamics of documents across languages. in this work, we aim to understand language preference phenomena in mrag systems comprehensively. we focus on the following three key research questions: to address these questions, we present a comprehensive evaluation of language preferences throughout the entire mrag pipeline across multiple languages. to systematically investigate the language preference problem of multilingual retrievers, we propose multilingualrankshift (mlrs), a novel metric that quantifies language preference at the retriever level by measuring the shift in document rankings when non-query-language passages are translated into the query language. our extensive experiments with diverse language combinations demonstrate that the retriever strongly prefers documents that are in high-resource languages and also share the same language as the query, confirming the presence of significant preference ( §4). at the generator level, we evaluate language preference by generating responses in multiple languages for the same query and the same retrieved document set, measuring their semantic similarity. our results show that the generator favors both query languages and latin script languages, with a relatively modest preference for query languages. this ultimately results in a decline in answer quality. moreover, we uncover a nuanced relationship between language preference and overall mrag performance. we observe that a strong preference for high-resource languages does not always lead to improved mrag performance ( §5). this occurs because the retriever may retrieve high-resource but irrelevant documents so that the generator cannot generate accurate answers from them. therefore, language preference can degrade performance by overlooking lower-resource but relevant documents, thereby causing inconsistencies. finally, we propose dual knowledge multilingual rag (dkm-rag), a simple yet effective framework that mitigates the language preference of mrag. dkm-rag enhances mrag by combining externally retrieved, translated passages with internally rewritten passages enriched by the model's knowledge. empirical results demonstrate that dkm-rag significantly reduces language preference issues in the generation process, leading to improved performance across a range of linguistic settings ( §6). ## multilingualrankshift to evaluate the language preference of a multilingual retriever in the mrag system, we introduce multilingualrankshift (mlrs), a novel metric that quantifies how much the ranking of retrieved documents improves when non-query language documents are translated into the query language. as shown in figure 2 , mlrs is computed in three stages: (i) retrieving documents across multiple languages, (ii) translating documents that are not in the query language into the query language, and (iii) re-ranking the translated documents to measure rank improvements. ## language preference of retrievers in this section, we examine two factors that may affect the retriever's language preference: (i) the relationship between the query language (l q ) and the document language (l d ), and (ii) the resource availability of the languages involved. ## language preference of generators in this section, we explore llm generators' language preferences in mrag and their impact on overall performance. ## dual knowledge multilingual rag translating retrieved documents into the query language benefits mrag, but it may also reflect retrieval outputs from high-resource languages including irrelevant content. therefore, leveraging the llm's internal knowledge can help filter inaccuracies and enrich the retrieved information with more reliable content. so we rewrite translated passages to refine the relevancy of documents by leveraging llm's internal information. based that leverages both external translated passages and internal knowledge as shown in figure 4 . first (#1), we retrieve documents for a given query from the all strategy and re-rank them. next (#2), we obtain external translated passages, p translated by translating into the query language. and (#3), the rewriter llm refines each translated passage in the context of the given query to produce refined passages, p refined . this refining process utilizes a prompt to guide the model in integrating its internal knowledge, removing redundancy, and highlighting relevant information in a coherent and consistent style. for detailed prompts, please refer to appendix b. finally (#4), we concatenate the two sets to form the final passage set as input to the generator llm, ensuring that responses are both contextually enriched and linguistically aligned with the query. results. as shown in ## conclusion in this work, we investigate language preferences in mrag systems. we propose a metric that measures the language preference of retrievers by checking the rank difference between the translated passage and the original one. our experiments reveal that retrievers prefer high-resource and query language but do not always yield better generation performance. we also find that generators often favor the query language or latin scripts, resulting in inconsistent outputs. to address this, we propose dkm-rag which integrates translated passages with internal knowledge. empirical results show that dkm-rag consistently enhances mrag performance across diverse languages.