Retrieval-augmented generation in multilingual settings (2024)

Abstract: Retrieval-augmented generation (RAG) has recently emerged as a promising solution for incorporating up-to-date or domain-specific knowledge into large language models (LLMs) and improving LLM factuality, but is predominantly studied in English-only settings. In this work, we consider RAG in the multilingual setting (mRAG), i.e. with user queries and the datastore in 13 languages, and investigate which components and with which adjustments are needed to build a well-performing mRAG pipeline, that can be used as a strong baseline in future works. Our findings highlight that despite the availability of high-quality off-the-shelf multilingual retrievers and generators, task-specific prompt engineering is needed to enable generation in user languages. Moreover, current evaluation metrics need adjustments for multilingual setting, to account for variations in spelling named entities. The main limitations to be addressed in future works include frequent code-switching in non-Latin alphabet languages, occasional fluency errors, wrong reading of the provided documents, or irrelevant retrieval. We release the code for the resulting mRAG baseline pipeline at https://github.com/naver/bergen, Documentation: https://github.com/naver/bergen/blob/main/documentations/multilingual.md.

## introduction retrieval-augmented generation (rag) @xcite has recently emerged as a promising solution for incorporating up-to-date or domainspecific knowledge into large language models (llms) and improving llm factuality, especially in knowledge-intensive tasks such as open-domain question answering or fact-checking. rag augments user queries with relevant context retrieved from the internet or a given collection and figure 1 : multilingual retrieval-augmented generation pipeline. we study which components are required to build a well performing mrag pipeline, that can be used as a strong baseline in future works. ## related work despite mrag being not well studied in the literature, some of the individual components of the rag pipeline were rather well developed for multilingual settings, e.g. multilingual retrievers and generator llms; we discuss them in section 3. the closest line of work to ours is multilingual open question answering @xcite @xcite defined as a the task of answering non-english questions from a large collection of multilingual documents, as introduced in @xcite . those aforementioned works train task-specific models combining cross-lingual retrievers and multilingual generation models, e.g. with iterative extension of annotated data used in the cora approach @xcite . the key difference of our work is that we compose the mrag system in a zero-shot manner, using off-the-shelf components without dedicated training. this approach, dominating nowadays in the literature, is enabled by recent advances in llms and retrieval and makes the system more robust and easy-to-extend. it's important to note that our goal is not to outperform the mentioned models such as cora, but to evaluate the state of the described zero-shot mrag setting, understand its open problems, and provide an experimental ground for future development of mrag. another related and orthogonal effort is @xcite which release a nomiracl dataset for evaluating llm robustness in mrag across 18 typologically diverse languages. ## multilingual rag pipeline the high-level illustration of the mrag pipeline is presented in figure 1 . the input is represented by a user query q in language l q . this could be an arbitrary user request to an llm. following the common practice of testing rag systems on open-domain question answering, we assume q is an information-seeking question. the model is expected to output response r which correctly answers the given question. an important (and reasonable) expectation is that the model replies in the user language, i.e. r is written in l q . step 1: retrieval. the first step in mrag is retrieving context c relevant to the query q from the internet or a particular collection c, using the retriever system r: @xmath0, c), @xmath1. here q denotes an optional query generation model which infers a search query q from a user query c, e.g. it can be an llm prompted to reformulate the query, or simply copying the user query q. following a standard practice in testing rag systems, we use wikipedia as our collection c. in most of the experiments we assume monolingual c in language l c (english or user language), but we also experiment with retrieving from the multilingual c. the retriever system r usually consists of two stages. the first stage ranker r 1 encodes queries q and documents @xmath2: h @xmath3, h @xmath4, allowing to precompute document representations offline and enabling fast search over large collections, e.g. @xmath5, k denotes the number of retrieved documents. the second-stage reranker r 2 processes a (small) subset c of documents from c retrieved by r 1 and encodes documents together with queries: h q,@xmath6, d) ∈ r, enabling semantically richer representations and selecting k most relevant documents: @xmath7,d . both r 1 and r 2 are often based on bert-like models and trained on retrieval datasets such as ms-marco @xcite . in our work we rely on retrievers and rerankers developed specifically for the multilingual setting. step 2: generation. the second stage of mrag pipeline consists of generating a response r based on the user query q and retrieved relevant context c with a generator llm: @xmath8, c). state-ofthe-art llms follow the wide-spread paradigm of pretraining a decoder-only transformer model on a large set of unsupervised data and then tuning it for instruction following and alignment with user preferences. this second step of instruction tuning and alignment often introduces a template, representing formatting rules for passing data into the llm. template usually contains placeholders for user queries q, model responses r and also for a system prompt, which is put in the beginning of the template and describes the task / role for the llm. a simplest example of the system prompt is "you are a helpful assistant.". in our work we study several generator llms and experiment extensively with various prompting strategies for mrag. below we describe how we instantiate different components of our mrag pipeline. ## experimental details retrieval. we follow @xcite and @xcite and construct passages by splitting wikipedia article into chunks of 100 words (or 100 unicode characters for non whitespace separated languages, namely chinese, japanese, and thai) and prepending the article title to each chunk. in most of the experiments we retrieve either from english wikipedia (kilt version foot_9 ) or wikipedia in the user language foot_10 , but we also experiment with retrieving from concatenation of two mentioned wikipedias and from wikipedia in all considered languages. for each question in the evaluation data, we retrieve 50 relevant passages and pass them to the reranker to select top-5 relevant ones which will be inserted in the llm context during generation. generation. we use greedy decoding, limit generation to maximum 128 new tokens and run all experiments with model quantized into int4. evaluation. we rely on the commonly-used squad evaluation script foot_11 , but use it on the character 3-gram level, as discussed in section 3 and illustrated in prompting strategy: importance of translating the system prompt into target languages and specifying the desired language of the response. the left part reporting correct language rate (clr) allows us to assess how often the model replies in the user language. due to multilingual pretraining and instruction tuning, command-r-35b, equipped with the default system prompt ("reply short (en)"), replies in the user language in most, but not all, cases. importantly, it gets "distracted" by the english context when retrieving from english wikipedia and replies in english for around 50% of non-english user queries. english-centric solar-10.7b, provided with the default system prompt, also often replies in english. these results demonstrate the need for using more advanced language-related prompting strategies for both models. explicitly specifying an instruction to reply in the given user language, while keeping the system prompt itself in english ("+ reply in ul (en)"), substantially alleviates the problem of generation in english and correspondingly increases recall, but still does not enable correct language rate (crl) close to 100%. in appendix the high clr is necessary but not sufficient for high overall performance, as llms may use code-switching and tend to insert english named entities in their responses in user languages. in appendix generator model: importance of using a strong multilingual base model. despite command-r-35b being a leader model for non-english, its recall in english is much lower than of english-centric solar-10.7b which is possibly due to the "curse of multilinguality" effect. this highlights the need for future models which would be fluent and accurate in both english and non-english. ## conclusion in this work we study rag in multilingual settings and build a strong pipeline to be used as a baseline in future works. better understanding of mrag would enable reliable information access across different languages and cultures. we analyze an impact of each mrag component impact on overall performance and provide guidelines and future research direction to further improve it. possible research directions include: