Benchmarking and Building Zero-Shot H indi Retrieval Model with H indi- BEIR and NLLB -E5 (2025)

Abstract: Given the large number of Hindi speakers worldwide, there is a pressing need for robust and efficient information retrieval systems for Hindi. Despite ongoing research, comprehensive benchmarks for evaluating retrieval models in Hindi are lacking. To address this gap, we introduce the Hindi-BEIR benchmark, comprising 15 datasets across seven distinct tasks. We evaluate state-of-the-art multilingual retrieval models on the Hindi-BEIR benchmark, identifying task and domain-specific challenges that impact Hindi retrieval performance. Building on the insights from these results, we introduce NLLB-E5, a multilingual retrieval model that leverages a zero-shot approach to support Hindi without the need for Hindi training data. We believe our contributions, including the release of the Hindi-BEIR benchmark and the NLLB-E5 model, will be a valuable resource for researchers and promote advancements in multilingual retrieval models.

## introduction information retrieval (ir) models are indispensable in our digital age, enabling swift and accurate extraction of relevant data from vast amounts of information. these models enable quick and accurate retrieval of relevant data, thereby facilitating informed decision-making across various downstream applications and domains. in this modern era of llms, retrievers have become even more relevant and almost necessary in curbing hallucinations from large language model (llm) generations by means of much popular retrieval augmented generation (rag) @xcite methods. however, the focus of information retrieval (ir) research has predominantly centered * work done as an intern at ibm research on the english language, which can be attributed to the availability of comprehensive evaluation benchmarks, such as the beir @xcite . although, more recently, there have been efforts to build multi-lingual retrievers and associated evaluation benchmarks @xcite on non-english languages, multi-lingual expansion of robust benchmarks is still a work in progress, with many languages like hindi lacking a beir like a comprehensive benchmark. hindi is one of the official languages of india foot_0 , a language of immense importance and global reach, with over half a billion speakers worldwide and ranking as the third most widely spoken language foot_1 . however, from nlp research point of view, hindi can be categorized as a low-resource language, lacking comprehensive resource and benchmarks to advance scientific research and this includes lack of a good retrieval benchmark too. though there has been some efforts in recent time for the development of indic language dataset for various nlp task like question answering @xcite , @xcite ) and summarization @xcite , @xcite , @xcite ), the evolution of hindi or, in general, indic language retrieval datasets has been rather ad-hoc, restricted to a small subset of domains, languages and tasks. although a recent effort attempts to incorporate multiple indic languages @xcite for a retrieval benchmark, the absence of diversity in domains and tasks remains a major limitation, affecting the robust evaluation of the current state-of-the-art retriever models in practical applications, which often involve varying tasks and domains. in this work, we posit that creating a beir-like diversified and robust benchmark is the necessary first step to track and advance any retrieval research effectively in a new language. such a benchmark will be more useful than ad-hoc datasets with multiple languages. thus, we choose the most widely spoken language among indic languages, hindi, as our pivot language to pioneer a large-scale diversified retrieval benchmark hindi-beir, spanning 15 diverse datasets across 7 tasks. hindi-beir aims for two key objectives:(1) establish a standardized retrieval benchmark to assess, compare, and advance the state-of-the-art retriever models and (2) provide workable insights into the potential research directions on building retrieval models for hindi. with multiple tasks and domains in hindi-beir, we posit that hindi-beir will provide a more realistic and accurate assessment of retriever models for zero-shot usage in unseen domains and applications. on the choice of hindi language it is important to note here, besides being a language of high usage, a retrieval benchmark in hindi offers some unique challenges too such as: the introduction of hindi-beir benchmark lets us assess and compare the existing multi-lingual retriever models more accurately for their zeroshot performance in hindi across diverse domains and tasks covered in the benchmark. based on this performance comparison, we notice that a key limitation of existing models is the need for a substantial amount of language-specific training data, which is often not available for low-resource lan-guages such as hindi. to address this problem, we propose nllb-e5, a multilingual retrieval model, inspired by recent works like langbridge @xcite and @xcite , which distills information from a monolingual retrieval model into a multilingual retrieval model in a zero shot setup via a multi-lingual encoder. the key strength of nllb-e5 lies in its ability to perform distillation without reliance on multilingual training data, thereby eliminating the necessity for supervised multilingual datasets. we summarize our contributions as follows: ## related works one of the known sources of testing retriever performance on indic languages was to test on the indic language subset contained in multi-lingual ir datasets such as miracl @xcite , mr.tydi @xcite having instances from indic languages such as hindi, bengali, and telugu. however, two major drawbacks here were: more recently, a relatively large-scale retrieval benchmark ms marco @xcite got translated into multiple languages to produce mmarco @xcite , including hindi from the set of indic languages. mmarco was further extended to more indic languages by in-dicirsuite @xcite , which is the most recent retrieval benchmark in indic languages. however, the common problem for both mmarco and indicmarco is that these benchmarks, too, are tied to a single type of retrieval task and web domain as was originally in ms marco and, therefore, do not provide a robust benchmark. in terms of language specific benchmarks, snegirev et al. (2024) introduce the rumteb benchmark, an embedding model evaluation benchmark in russian with retrieval being a subtask, @xcite introduce the spanish ir dataset while @xcite release the ir benchmark in the polish language. in indian languages such as hindi, the forum for information retrieval evaluation (fire) 3 has released numerous sub-tasks and datasets over the years to facilitate the benchmarking and development of information retrieval models. however, these datasets have predominantly been sourced from various news outlets and are not freely or openly accessible to the public. to the best of our knowledge, hindi-beir presents the first comprehensive ir benchmark, which spans diverse domains and tasks and therefore, provides the first beir @xcite equivalent comprehensive retrieval benchmark in hindi. for multilingual retrieval model development, the most promising works are from @xcite and @xcite , but both these models are heavily dependent on multilingual data and do not scale for languages and domains where data is not readily available in the target language. @xcite proposed an approach to equip existing llms with multilingual capability without any multilingual supervised data. they vertically stack a multilingual encoder and an llm, adding a projection/alignment layer between them. the input first passes through the multilingual encoder, the representation from the multilingual encoder passes through the alignment layer and then the output from the alignment layer is passed as input to the llm. they use english-labeled data to fine-tune the overall model and update only the embedding layers and the alignment layer. the model showed better multilingual capabilities. to ensure that the hindi-beir benchmark is comprehensive, challenging, and accessible to the public for future research and evaluation, we adhered to the following objectives:1) diverse domains and tasks: we include datasets from diverse domains like wikipedia and news articles to niche domains like scientific publications, finance to test the robustness and generalization ability of the retrieval models. additionally, a good retrieval model should handle documents and queries of varying lengths equally well. we have ensured that the datasets included in the hindi-beir benchmark exhibit a wide range of query and document lengths among the datasets. 2) difficulty level: we ensure that the datasets are challenging and systems relying solely on lexical overlap have a hard time retrieving the correct document. 3) public availability: all datasets curated in hindi-beir benchmark have user-friendly licenses and are publicly available at hindi-beir we discuss the hindi-beir benchmark creation in the following subsections. please refer to appendix a.2 for a more detailed analysis of each dataset. ## nllb-e5: multilingual retriever in this section, we build a strong multilingual retrieval model based on the approach proposed by @xcite and @xcite . figure 1 shows the architecture of nllb-e5 with the different components and data flow between them. we build nllb-e5 on the back of two components (1) a multilingual encoder nllb @xcite and (2) a monolingual retriever e5 @xcite . we bank on the multilingual encoder to project semantically similar sentences across languages to a shared representation space. we hypothesize that embeddings coming from such a language-agnostic shared representation space can act as a strong prior that can be easily aligned for producing multilingual task-specific embeddings. moreover, given the multilingual nature of the embedding coming directly from the multilingual encoder, the aforementioned task-specific alignment is language agnostic and, therefore, can be done with task-specific data in a high-resource language, too. figure 1 demonstrates how we follow the above intuition in designing nllb-e5. we use nllb @xcite as the strong multilingual encoder that produces robust multilingual embeddings owing to its translation capabilities for over 200 languages. to align these embeddings for the retrieval task, we learn a projection layer from the multilingual nllb encoder to a monolingual retriever e5. the alignment happens by learning the projection layer parameters between nllb and e5 through distillation from e5 embeddings, which are specifically tuned for retrieval tasks. since this alignment to the retrieval task is agnostic to a specific language, we use english data and monolingual retriever e5 to do the distillation, eliminating the need for hindi training data. we chose e5-large as the teacher and student model coupled with nllb owing to its satisfactory retrieval performance on the beir benchmark. ## training methodology the nllb-e5 model is constructed by integrating the monolingual e5 model @xcite atop the multilingual encoder from nllb @xcite , connected via a learnable projection layer. the parameters of both the monolingual and multilingual base models remain frozen throughout training. to enable adaptation, we introduce @xmath0, a linear projection layer mapping the n-dimensional output space of the multilingual encoder to the d-dimensional input space expected by the e5 model. we employ knowledge distillation @xcite to transfer the capabilities of the monolingual e5 model to the nllb-e5 model, facilitating multilingual sentence representation learning. specifically, the original e5 model serves as the teacher, producing high-quality english embeddings, while the nllb-e5 model, which includes the learnable linear projection w , acts as the student. by minimizing the discrepancy between the teacher and student embeddings, the student model learns to approximate the e5 model's performance across hindi and other languages supported by nllb. the frozen multilingual encoder ensures effective cross-lingual transfer, enabling generalization across all nllb supported languages. foot_4 ## experiments this section provides a detailed overview of our experiments on hindi-bier datasets, including our experimental setup and the baseline models used in our experimentation. ## conclusion and future work in this work, we introduced hindi-beir, a comprehensive benchmark for hindi language information retrieval. the benchmark consists of 15 datasets, with a corpus size exceeding 27 million and a query size of over 200k, covering a diverse range of 7 tasks. this provides the first-of-its-kind robust benchmark for assessing the performance of retriever models in hindi. our experimental results offer insights into the strengths and limitations of current multilingual retrievers on hindi-beir, in-dicating the pressing need for further research in this domain. additionally, we also proposed the nllb-e5 model, a novel architecture for multilingual retriever tuning without needing multilingual training data, capable of handling data from over 200 languages, including hindi, which outperformed existing multilingual models on hindi-beir. future work will focus on expanding the hindi-beir benchmark to include more diversity by curating additional domains such as law and medicine. we also plan to extend this benchmark to cover languages beyond hindi and explore alternative multilingual encoders to optimize representation. we believe our work will have a lasting impact in developing inclusive and scalable information retrieval systems across diverse languages.