Towards Leaving No I ndic Language Behind: Building Monolingual Corpora, Benchmark and Models for I ndic Languages (2023)

Abstract: Building Natural Language Understanding (NLU) capabilities for Indic languages, which have a collective speaker base of more than one billion speakers is absolutely crucial. In this work, we aim to improve the NLU capabilities of Indic languages by making contributions along 3 important axes (i) monolingual corpora (ii) NLU testsets (iii) multilingual LLMs focusing on Indic languages. Specifically, we curate the largest monolingual corpora, IndicCorp, with 20.9B tokens covering 24 languages from 4 language families - a 2.3x increase over prior work, while supporting 12 additional languages. Next, we create a human-supervised benchmark, IndicXTREME, consisting of nine diverse NLU tasks covering 20 languages. Across languages and tasks, IndicXTREME contains a total of 105 evaluation sets, of which 52 are new contributions to the literature. To the best of our knowledge, this is the first effort towards creating a standard benchmark for Indic languages that aims to test the multilingual zero-shot capabilities of pretrained language models. Finally, we train IndicBERT v2, a state-of-the-art model supporting all the languages. Averaged across languages and tasks, the model achieves an absolute improvement of 2 points over a strong baseline. The data and models are available at https://github.com/AI4Bharat/IndicBERT .

## introduction recent advances in natural language understanding are largely driven by pretrained multilingual models @xcite @xcite . one of the advantages of such models is that they can potentially reduce the performance gap between high and lowresource languages through zero-shot knowledge transfer @xcite . however, in practice, the benefits of such models are fertility (↓) 2.8 2.2 1.7 1. 7 first, current multilingual models often have a poor representation of low-resource languages. for example, out of the 22 languages listed in the 8 th schedule of the indian constitution, only 15 languages are supported by the popular xlm-r model @xcite . this is mainly due to the non-availability of pretraining data for languages like bodo, dogri, kashmiri, etc. in large multilingual corpora such as cc-100 @xcite , or mc4 @xcite . hence, dedicated efforts towards collecting pretraining data for these languages by discovering and crawling language-specific sources are needed. second, even for low-resource languages supported by existing multilingual models, the size of pretraining data is much smaller than that of english and other resource-rich languages @xcite . due to this disparity, low-resource languages get a very poor share of the model's capacity and vocabulary, and thus the performance on these languages is poor @xcite . indeed, a few recent efforts @xcite @xcite show that multilingual models trained using pretraining data from a smaller set of related languages leads to better performance on downstream tasks than large scale models which support many languages. hence, there is a need for training language models only on indic languages thereby ensuring that the model capacity is not dominated by unrelated high-resource languages. the third reason is the poor representation of these languages in existing evaluation benchmarks. for example, in the xtreme-r @xcite benchmark, out of the 10 tasks only three contain evaluation data for more than two indic languages. further, the maximum number of indic languages for any task is just seven. in effect, 15 of the 22 constitutionally recognized indic languages have no representation in xtreme-r for any task. thus, a human supervised evaluation benchmark tailored for indic, and other low-resource language families is essential for furthering inclusivity and equity in nlp research @xcite . in this work, we make contributions toward addressing all the three challenges. we focus on the 22 languages listed in the 8 th schedule of the indian constitution spanning 4 language families and spoken by over a billion speakers (8 of these languages being amongst the top-20 most spoken languages globally). some of these languages are also widely spoken and/or are official languages in neighbouring countries viz., bangladesh, nepal and pakistan. our first contribution towards serving these languages is to release indiccorp v2, the largest collection of corpora for languages spanning 4 indic language families with 20.9 billion tokens and 1.1 billion sentences. our third, and perhaps, the most important contribution is indicxtreme, a human supervised benchmark containing evaluation sets for nine diverse tasks with each task covering 7-18 indic languages per task. these include five classification tasks, two structure prediction tasks, one qa task, and one text retrieval task. of the total 105 evaluation sets, summed across languages and tasks, 52 have been newly created as a part of this benchmark. all the newly added evaluation sets have been created manually with the help of in-house language experts with several years of experience in language annotation and translation. the datasets for three tasks, viz., ner, qa, and paraphrase detection were created from scratch without any translation from english sources. we consciously make an effort to include languages spanning all the classes from the inclusion taxonomy introduced in @xcite . according to their classification ( using indicxtreme, we evaluate indicbert and show that it outperforms strong baselines on 7/9 evaluation tasks. we also do a series of ablation tests to show that (i) the translation language modeling (tlm) objective slightly improves zero-shot performance when high-quality parallel data is used, (ii) using noisy parallel data during pretraining leads to sub-optimal zeroshot performance, (iii) using in-language-family development sets allows better model selection, and (iv) zero-shot transfer via hindi, as opposed to english, leads to better performance. all the datasets, code, and models developed as a part of this work will be open-sourced. all the datasets and models developed as a part of this work are available at https://ai4bharat.iitm. ac.in/language-understanding . broadly classified into the following three broad areas. ## indicxtreme the indicxtreme benchmark includes 9 tasks that can be broadly grouped into sentence classification (5), structure prediction (2), question answering (1), and sentence retrieval (1). since the benchmark is designed to evaluate models in a zeroshot setting, we only create test sets. ## indiccorp v2 in this section, we describe the process followed to build indiccorp v2, the largest collection of texts for indic languages consisting of 20.9 billion tokens of which 14.4b tokens correspond to 23 indic languages and 6.5b tokens of indian english content curated from indian websites. ## indicbert v2 this section describes the various aspects of training indicbert, a language model trained on indic-corp and evaluated on indicxtreme. in our experiments, we train with bert architecture and ablate on objective functions and training data. compared to indicbert v1 @xcite , trained on the smaller albert @xcite architecture, this version has ∼7.5x more param-eters and is able to transfer across languages in zero-shot settings. the model has 278m parameters and supports all 24 languages in indiccorp. training objectives we experiment with two objective functions: masked language modeling @xcite and translation language modeling @xcite . we use the document-level data created as part of in-diccorp for mlm objective training. pretraining hyperparameters are listed in appendix c. ## experiments we compare indicbert v2 with the following lms -indicbert v1 @xcite , mbert @xcite , xlmr @xcite and muril @xcite . we describe our choice of baseline models, and their similarities and differences in appendix d. we then briefly introduce our fine-tuning details and the various ablation studies conducted. ## results the results for each task in indicxtreme averaged across languages are shown in massively multilingual vs indic models it is clear that there is no single best model on the benchmark. however, indicbert v2 family of models beat the baselines in 7/9 tasks. the languagespecific results for all experiments can be found in appendix o. when averaged across tasks (see the results show that models trained only on indic languages perform better since languages do not have to compete for model capacity. we see that indicbert v2 trained only on mlm, by itself performs much better than the standard baselines. the only exception to this is that muril outperforms indicbert v2 in the paraphrase detection and ner tasks. we also see that adding the 7 samanantar data is sentence-level parallel and is not ideal. but document-level parallel data for indic languages are scarce. tlm objective with (i) high-quality parallel data increases the model performance across the board, and (ii) machine-translated data hurts performance. effect of monolingual corpora ## conclusion through this work, we distinctively contribute towards all the fundamental requirements of developing indic language technologies; these include indiccorp v2, the largest pretraining corpus for 24 indic languages, indicbert v2 a language model pretrained on indiccorp v2 and a holistic crosslingual nlu benchmark, indicxtreme, for 20 indic languages. we provide empirical evidence for our design decisions and show that pretraining models only on indic languages result in much better performance on indicxtreme. ganni for their help in coordinating the annotation work. most importantly we would like to thank all the annotators who spent their time helping create the indicxtreme benchmark. we also thank raghavan ak for helpful discussions on corpus cleaning and harshita diddee for insightful discussions on model pretraining.