Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness (2025)

Abstract: The paradigm of retrieval-augmented generated (RAG) helps mitigate hallucinations of large language models (LLMs). However, RAG also introduces biases contained within the retrieved documents. These biases can be amplified in scenarios which are multilingual and culturally-sensitive, such as territorial disputes. We thus introduce BordIRLines, a dataset of territorial disputes paired with retrieved Wikipedia documents, across 49 languages. We evaluate the cross-lingual robustness of this RAG setting by formalizing several modes for multilingual retrieval. Our experiments on several LLMs show that incorporating perspectives from diverse languages can in fact improve robustness; retrieving multilingual documents best improves response consistency and decreases geopolitical bias over RAG with purely in-language documents. We also consider how RAG responses utilize presented documents, finding a much wider variance in the linguistic distribution of response citations, when querying in low-resource languages. Our further analyses investigate the various aspects of a cross-lingual RAG pipeline, from retrieval to document contents. We release our benchmark to support continued research towards equitable information access across languages, at https://huggingface.co/datasets/borderlines/bordirlines.

## introduction large language models (llms) have gained widespread adoption due to their remarkable text generation capabilities @xcite @xcite . despite this, their tendencies to hallucinate and their static knowledge from training-time impacts their applicability for knowledge-intensive tasks. the paradigm of retrieval-augmented generation (rag) addresses these pitfalls by using an information retrieval (ir) system to access documents from an external database, enabling an llm to generate grounded and more up-to-date responses @xcite . rag can be particularly useful in multilingual settings by helping an llm generate more accurate responses @xcite , and by allowing it access to different perspectives on more subjective, nuanced issues @xcite . while several recent studies have investigated rag in small-scale multilingual settings @xcite , they only consider high-resource languages and do so in artificially construed scenarios with synthetic documents. a wider approach has been taken in work on openretrieval multilingual qa @xcite , but these primarily focus on simple fact-seeking questions where right answers are easily memorized by llms, in which cases, rag can even be harmful @xcite . in this work, we investigate the impact of rag in a massively multilingual setting. we focus on queries about territorial disputes, which exist at the intersection of cultural and linguistic fault lines. although these queries are fact-seeking in nature, their answers are highly dependent on one's perspective. @xcite found that llms' responses are inconsistent depending on the language of interaction, and that such inconsistency increases the societal risks of llms by promoting divisive viewpoints across cultures. meanwhile, a rag system's answers are highly dependent on the selection of documents, particularly over different perspectives in the multilingual setting. many questions arise: how does the linguistic composition of the set of documents impact responses? does sourcing information from different languages increase or decrease consistency? and is presenting conflicting information to llm's base preferences better expressed in certain languages? our study makes the following contributions: ## experimental results we report results on our two main experiments: on cross-lingual robustness for the direct response format, and on llm's use of documents in the citation response format. we consider 4 llms here, and all 7 in appendix ## prior work multilingual rag there are several recent and studies on multilingual rag. @xcite study the linguistic inequities of rag (8 languages), finding a selection bias towards english documents. @xcite study how rag addresses information disparities across languages, finding a selection bias towards in-language documents. our study considers 49 languages on a real-world, not hypothetical, task. ## conclusion the rag paradigm enables llms to retrieve over external information. while significant efforts have focused on how rag can reduce hallucinations, and how better ir systems further improve performance, less attention has been paid to ensuring balance of the information sources included, thus and the perspectives contained within them. in this paper, we presented bordirlines a benchmark of territorial disputes, and used it to evaluate the cross-lingual robustness of rag. we formalized several ir modes to compare how different linguistic information access settings affect llms' responses. we first found that retrieving over a diversity of languages improves cross-lingual consistency over in-language retrieval. furthermore, despite the broad multilingual support of modern llms, they exhibit much higher variability in incorporating citations for low-resource languages. additional experiments and case studies showed how these behaviors are impacted by various aspects of the rag pipeline. we release our benchmark and code to enable further research towards llm systems which more equitably consider and address information across languages.