I ndic IRS uite: Multilingual Dataset and Neural Information Models for I ndian Languages (2024)

Abstract: In this paper, we introduce Neural Information Retrieval resources for 11 widely spoken Indian Languages (Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu) from two major Indian language families (Indo-Aryan and Dravidian). These resources include (a) INDIC-MARCO, a multilingual version of the MS MARCO dataset in 11 Indian Languages created using Machine Translation, and (b) Indic-ColBERT, a collection of 11 distinct Monolingual Neural Information Retrieval models, each trained on one of the 11 languages in the INDIC-MARCO dataset. To the best of our knowledge, IndicIRSuite is the first attempt at building large-scale Neural Information Retrieval resources for a large number of Indian languages, and we hope that it will help accelerate research in Neural IR for Indian Languages. Experiments demonstrate that Indic-ColBERT achieves 47.47% improvement in the MRR@10 score averaged over the INDIC-MARCO baselines for all 11 Indian languages except Oriya, 12.26% improvement in the NDCG@10 score averaged over the MIRACL Bengali and Hindi Language baselines, and 20% improvement in the MRR@100 Score over the Mr. Tydi Bengali Language baseline.

## introduction information retrieval (ir) models process user queries and search the document corpus to retrieve a ranked list of relevant documents ordered by a relevance score. classical ir models, like bm25 @xcite , retrieve documents that have lexical overlap with the query tokens. recently, there has been a notable upsurge in adopting neural ir models utilizing language models such as bert @xcite , which enable semantic matching of queries and documents. this shift has proven highly effective in retrieving and reranking documents. colbertv2 @xcite , one of the state-of-art neural ir models, has shown 18.5 points improvement in ndcg@10 score over the bm25 model baseline on the ms marco dataset @xcite . the importance of dataset size outweighs domain-matching in training neural ir models @xcite . due to the scarcity of largescale domain-specific datasets, neural ir models are first trained on the ms marco passage ranking dataset @xcite , and they are subsequently evaluated on domain-specific datasets in a zero-shot manner. ms marco dataset contains 39 million training triplets (q, +d, -d) where q is an actual query from the bing search engine, +d is a human-labeled passage answering the query, and -d is sampled from unlabelled passages retrieved by the bm25 model. the ms marco dataset is in english, implying that neural ir models trained on it are effective only with english queries and passages. monolingual ir for non-english languages @xcite , multilingual ir @xcite , and cross-lingual ir @xcite extend the english ir paradigm to support diverse languages. in monolingual ir for non-english languages, the query and passages are in the same language, which is not english. in cross-lingual ir, the query is used to create a ranked list of documents such that each document is in the same language, which is different from the query language. in multilingual ir, the query is used to create a ranked list of documents such that each document is in one of the several languages, which can be the same or different from the query language. in this work, we focus on monolingual ir for non-english languages. monolingual ir for non-english languages involves training an encoder like mbert @xcite , on a large-scale general-domain monolingual dataset for non-english languages to minimize the pairwise softmax cross-entropy loss. the trained models are subsequently finetuned or used in a zero-shot manner on small-scale domainspecific datasets. however, there is a notable lack of large-scale datasets like mmarco @xcite for training monolingual neural ir models on many low-resource indian languages. we introduce neural ir resources to address this scarcity and facilitate monolingual neural ir across 11 indian languages. our contributions are: to the best of our knowledge, this is the first effort for a neural ir dataset and models on 11 major indian languages, thereby providing a benchmark for indian language ir. ## related work the size of datasets holds greater importance than ensuring domain matching in the training of neural ir models @xcite . in terms of size and domain, mmarco @xcite is the most similar to our work as it introduces a largescale machine-translated version of ms marco in many languages, hindi being the only indian language. miracl @xcite and mr. tydi @xcite also introduce datasets and models for monolingual neural ir in hindi, bengali, and telugu. fire foot_0 was the most active initiative from 2008 to 2012 for multilingual ir in indian languages. fire developed datasets for multilingual ir in six indian languages (bengali, gujarati, hindi, marathi, oriya, and tamil). however, the size of these datasets is not large enough to train neural ir systems based on transformer models like mbert @xcite and xlm @xcite . in addition, the text in the fire dataset comes from newspaper articles @xcite , which is domain-specific; hence, the models trained on such datasets cannot generalize well to other domains. due to the lack of large-scale datasets, cross-lingual knowledge transfer via distillation has become popular for neural ir in low-resource languages @xcite . the key distinction in our work from the earlier approaches is that we introduce monolingual datasets and neural ir models in 11 major indian languages (assamese, bengali, gujarati, hindi, kannada, malayalam, marathi, oriya, punjabi, tamil and telugu), that can also benefit crosslingual and multilingual ir models from the crosslingual transfer effects when trained on a large number of indian languages @xcite . ## experiment setup we train 11 distinct indic-colbert (icol) models separately for 50k iterations with a batch size of 128 on the first 6.4 million training triplets from the indic-marco dataset to optimize the pairwise softmax cross entropy loss function, where each -t ) and mdpr model scores @xcite ; mcol (mcolbert trained on ms marco), and icol (indic-colbert trained on indic-marco) are tested in a zero-shot manner. for the miracl dev-set, we use official bm25, mdpr, and mcol(mcolbert) model scores @xcite ; icol (indic-colbert trained on indic-marco) is tested in a zero-shot manner. ## results indic-colbert (icol) outperforms baseline models (bm25, bm25-t, mdpr, mcol) by 20%, in mrr@100 score and on mr. tydi test-set (refer ## ablation study in this section, we perform ablation study with three different machine translation models and two different document splitting schemes. we compare the ndcg@10 scores of indic-colbert models trained on machine translated ms-marco data using nllb-600m, nllb-1.3b and indictrans2. as shown in for the miracl dev-set, we use official bm25, mdpr, and mcol(mcolbert) model scores @xcite ; icol (indic-colbert trained on indic-marco) is tested in a zero-shot manner. on retrieval effectiveness follows a different trend for each language. in terms of chrf++ score, in-dictrans2 performs better than nllb-1.3b which performs better than nllb-600m on flores-200 devtest @xcite . for telugu, we observe a negative correlation between translation quality and retrieval effectiveness, where the indic-colbert trained on data translated using nllb-600m model, which has the lowest chrf++ score among the three machine translation models, gives the best retrieval effectiveness. for hindi, we observe a positive correlation between the translation quality and retrieval effectiveness. for bengali, we don't observe any correlation between translation quality and retrieval effectiveness. each document in ms-marco dataset is first split into sentences, each sentence is translated by the machine translation model and finally the translated sentences are merged back into the document. we experimented with two different document splitting schemes. we compare the ndcg@10 scores for indic-colbert models trained on machine translated ms-marco dataset using nllb-1.3b model on sentences obtained from moses splitting and full-stop splitting schemes. as shown in ## summary, conclusion, and future work we present indicirsuite, featuring indic-marco, a multilingual neural ir dataset in 11 indian languages, and indic-colbert, comprising 11 monolingual neural ir models based on col-bertv2. our results demonstrate performance enhancements over baselines in mr. tydi, mir-acl, and indic-marco, particularly benefiting low-resource languages like assamese. indic-marco proves valuable for such languages, not supported by models like mbert but linguistically akin to bengali. we also perform an ablation to find the impact of translation quality and sentence splitting on retrieval effectiveness. future work includes expanding indicirsuite to multilingual and crosslingual ir.