Leveraging LLM s for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval (2024)

Abstract: There has been limited success for dense retrieval models in multilingual retrieval, due to uneven and scarce training data available across multiple languages. Synthetic training data generation is promising (e.g., InPars or Promptagator), but has been investigated only for English. Therefore, to study model capabilities across both cross-lingual and monolingual retrieval tasks, we develop **SWIM-IR**, a synthetic retrieval training dataset containing 33 (high to very-low resource) languages for fine-tuning multilingual dense retrievers without requiring any human supervision. To construct SWIM-IR, we propose SAP (summarize-then-ask prompting), where the large language model (LLM) generates a textual summary prior to the query generation step. SAP assists the LLM in generating informative queries in the target language. Using SWIM-IR, we explore synthetic fine-tuning of multilingual dense retrieval models and evaluate them robustly on three retrieval benchmarks: XOR-Retrieve (cross-lingual), MIRACL (monolingual) and XTREME-UP (cross-lingual). Our models, called SWIM-X, are competitive with human-supervised dense retrieval models, e.g., mContriever-X, finding that SWIM-IR can cheaply substitute for expensive human-labeled retrieval training data. SWIM-IR dataset and SWIM-X models are available at: https://github.com/google-research-datasets/SWIM-IR.

## introduction dense retrieval models have demonstrated impressive performance in ad-hoc information retrieval (ir) tasks, e.g., web search, outperforming traditional retrieval systems such as bm25 @xcite et al., 2020; @xcite @xcite . a major reason for its success lies in the availability of largescale supervised training datasets in english, such as ms marco @xcite or nq @xcite , and coupled with effective training strategies, such as custom hardnegative mining @xcite , or teacher distillation @xcite . however, there is a limited exploration of dense retrieval models in multilingual retrieval, 1 due to with about 850,000 residents, the comoros is one of the least-populous countries in the world, but its population density is high, with an average of 275 inhabitants per square kilometre (710/sq mi). in 2001, 34% of the population was considered urban, but the urban population has since grown; in recent years rural population growth has been negative, while overall population growth is still relatively high. ## swim-ir dataset overview in our dataset overview, we first describe the sap design formulation for multilingual query generation ( §2.1), data construction details ( §2.2), and fi- ## ablation studies optimal value of k-shot exemplars. we investigate the optimal value of few-shot exemplars required by palm 2 and the variation in the retrieval performance on xor-retrieve. 12 from figure 4 (right), we observe a linear improvement in recall@5kt with increase in k. best recall@5kt is observed with @xmath0. sap is unable to perform well zero-shot (i.e., @xmath1. byt5 tokenizer. we evaluate whether the poor performance of swim-x on low-resource languages in xtreme-up can be attributed to low-quality language tokenization. we replicate swim-x using a byt5-base model as backbone, which contains a language independent tokenizer extension @xcite . from our results in training split query replacement. next, we evaluate the impact of human-generated versus llm-generated queries on retrieval performance on xtreme-up. we replace all human-generated queries in the xtreme-up training split with only synthetic queries generated using palm 2 (s). from ## cost comparison heatmap showing mrr@10 denoting language-based transfer ability of swim-x (120k) across indo-european languages available in xtreme-up @xcite . (all) denotes swim-x fine-tuned on all xtreme-up languages. model plm query gen. brx mni mrr@10 1. models with byte-level (utf-8) tokenizer mcontriever-x ♡ byt5 human 1.8 1.0 2.1 swim-x (120k) m t byt5 palm 2 2.1 4.9 13.3 swim-x (120k) byt5 palm 2 5.1 5.8 15.4 2. human-generated query replacement in xtreme-up mcontriever-x ♡ mt5 human --13.5 swim-x (≈10k) mt5 palm 2 --11.5 thakur et al., 2021; wang et al., 2022; thakur et al., 2022). recently, using llms for query generation has gained interest. bonifacio et al. (2022) proposed inpars, where they few-shot prompt gpt-3 (brown et al., 2020) to generate synthetic queries. similarly, complementary works (sachan et al., 2022; jeronymo et al., 2023; boytsov et al., 2024; saad-falcon et al., 2023; dua et al., 2023) all follow a similar setup as in bonifacio et al. (2022). dai et al. (2023) proposed promptagator, which studied task-dependent few-shot llm prompting and used the synthetic data for both retrieval and ranking models. similarly, hyde (gao et al., 2023) and genread (yu et al., 2023) generate synthetic documents instead of queries. however, prior work has focused on english, with the exception of hyde. in our work, we robustly investigate how llms can be used for improving multilingual retrieval systems. multilingual datasets. prior work investigates techniques to build multilingual datasets for better fine-tuning or evaluation of dense retrieval models. datasets such as neuclir @xcite , mkqa @xcite have been constructed using human annotators. similarly, mmarco (bonifacio et al., 2021) has been generated using machine translation of ms marco @xcite . however, as translated documents are not written by native speakers, mmarco and similar datasets suffer from artifacts such as "translationese" @xcite . a concurrent work, jh-polo @xcite , prompts gpt-3 to generate english queries from language specific passages in neuclir. ## conclusion in this work, we present swim-ir, a synthetic multilingual retrieval training dataset with 28 million training pairs across 33 diverse languages. swim-ir allows synthetic fine-tuning of multilingual dense retrieval models cheaply without human supervision. swim-ir is constructed using sap, which stands for summarize-then-ask prompting, assisting the llm to identify the relevant sections of the input passage, improving the quality of the generated multilingual query. our rigorous evaluation across three multilingual retrieval benchmarks assesses our dataset quality. we find that swim-x, fine-tuned on swim-ir (keeping model training parameters unchanged) outperforms the best supervised crosslingual baseline by 7.1 points recall@5kt on xor-retrieve and 11.7 points mrr@10 on xtreme-up, while remaining competitive in monolingual retrieval on miracl.