I ndic NLPS uite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for I ndian Languages (2020)

Abstract: In this paper, we introduce NLP resources for 11 major Indian languages from two major language families. These resources include: (a) large-scale sentence-level monolingual corpora, (b) pre-trained word embeddings, (c) pre-trained language models, and (d) multiple NLU evaluation datasets ( IndicGLUE benchmark). The monolingual corpora contains a total of 8.8 billion tokens across all 11 languages and Indian English, primarily sourced from news crawls. The word embeddings are based on FastText , hence suitable for handling morphological complexity of Indian languages. The pre-trained language models are based on the compact ALBERT model. Lastly, we compile the ( IndicGLUE benchmark for Indian language NLU. To this end, we create datasets for the following tasks: Article Genre Classification, Headline Prediction, Wikipedia Section-Title Prediction, Cloze-style Multiple choice QA, Winograd NLI and COPA. We also include publicly available datasets for some Indic languages for tasks like Named Entity Recognition, Cross-lingual Sentence Retrieval, Paraphrase detection, etc. Our embeddings are competitive or better than existing pre-trained embeddings on multiple tasks. We hope that the availability of the dataset will accelerate Indic NLP research which has the potential to impact more than a billion people. It can also help the community in evaluating advances in NLP over a more diverse pool of languages. The data and models are available at https://indicnlp.ai4bharat.org .

## introduction distributional representations are the corner stone of modern nlp, which have led to significant advances in many nlp tasks like text classification, ner, sentiment analysis, mt, qa, nli, etc. particularly, word embeddings @xcite volunteer effort for the ai4bharat project et al., 2013b), contextualized word embeddings @xcite , and language models @xcite can model syntactic/semantic relations between words and reduce feature engineering. these pre-trained models are useful for initialization and/or transfer learning for nlp tasks. they are also useful for learning multilingual embeddings which enable cross-lingual transfer. pretrained models are typically learned from large, diverse monolingual corpora. the quality of embeddings is impacted by the size of the monolingual corpora @xcite , a resource not widely available for many major languages. in particular, indic languages, widely spoken by more than a billion speakers, lack large, publicly available monolingual corpora. they include 8 out of top 20 most spoken languages and ∼30 languages with more than a million speakers. there is also a growing population of users consuming indian language content (print, digital, government and businesses). further, indic languages are very diverse, spanning 4 major language families. the indo-aryan and dravidian languages are spoken by 96% of the population in india. the other families are diverse, but the speaker population is relatively small. almost all indian languages have sov word order and are morphologically rich. the language families have also interacted over a long period of time leading to significant convergence in linguistic features; hence, the indian subcontinent is referred to as a linguistic area @xcite . indic languages are thus of great interest and importance for nlp research. unfortunately, the progress on indic nlp has been constrained by the unavailability of large scale monolingual corpora and evaluation benchmarks. the former allows the development of pretrained language models and deep contextualised word embeddings which have become drivers of modern nlp. the latter allows systematic evaluation across a wide variety of tasks to check the efficacy of new models. with the hope of accelerating indic nlp research, we address the creation of (i) large, general-domain monolingual corpora for multiple indian languages, (ii) word embeddings and multilingual language models trained on this corpora, and (iii) an evaluation benchmark comprising of various nlu tasks. our monolingual corpora, collectively referred to as indiccorp, contains a total of 8.8 billion tokens across 11 major indian languages and english. the articles in indiccorp are primarily sourced from news crawls. using indiccorp, we first train and evaluate word embeddings for each of the 11 languages. given the morphological richness of indian languages we train fasttext word embeddings which are known to be more effective for such languages. to evaluate these embeddings we curate a benchmark comprising of word similarity and analogy tasks @xcite ), text classification tasks, sentence classification tasks @xcite , and bilingual lexicon induction tasks. on most tasks, the word embeddings trained on our indiccorp outperform similar embeddings trained on existing corpora for indian languages. next, we train multilingual language models for these 11 languages using the albert model @xcite . we chose albert as the base model as it is very compact and hence easier to use in downstream tasks. to evaluate these pretrained language models, we create an nlu benchmark comprising of the following tasks: article genre classification, headline prediction, named entity recognition, wikipedia section-title prediction, cloze-style multiple choice qa, natural language inference, paraphrase detection, sentiment analysis, discourse mode classification, and crosslingual sentence retrieval. we collectively refer to this benchmark as indicglue and it is a collection of (i) existing indian language datasets for some tasks, (ii) manual translations of some english datasets into indian languages done as a part of this work, and (iii) new datasets that were created semi-automatically for all major indian languages as a part of this work. these new datasets were created using external metadata (such as website/wikipedia structure) resulting in more complex nlu tasks. across all these tasks, we show that our embeddings are competitive or better than existing pre-trained multilingual embeddings such as mbert @xcite and xlm-r @xcite . we hope that these embeddings and evaluations benchmarks will not only be useful in driving nlp research on indic languages, but will also help in evaluating advances in nlp over a more diverse set of languages. in summary, this paper introduces indicnlp-suite containing the following resources for indic nlp which will be made publicly available: • indiccorp: large sentence-level monolingual corpora for 11 languages from two language families (indo-aryan branch and dravidian) and indian english with an average 9-fold increase in size over oscar. ## related work text corpora. few organized sources of monolingual corpora exist for most indian languages. the emille/ciil corpus @xcite was an early effort to build corpora for south asian languages, spanning 14 languages with a total of 92 million words. wikipedia for indian languages is small (the largest one, hindi, has just 40 million words). the leipzig corpus @xcite contains small collections of upto 1 million sentences for news and web crawls (average 300k sentences). in addition, there are some language specific corpora for hindi and urdu @xcite . in particular, the hind-monocorp @xcite is one of the few larger indian language collections (787m tokens). the commoncrawl foot_0 project crawls webpages in many languages by sampling various websites. our analysis of a processed crawl for the years 2013-2016 @xcite for indian languages revealed that most indian languages, with the exception of hindi, tamil and malayalam, have few good sentences (≥10 words) -in the order of around 50 million words. the oscar project @xcite , a recent processing of commoncrawl, also contains much less data for most indian languages than our crawls. the cc-net @xcite and c4 @xcite projects also provide tools to process common crawl, but the extracted corpora are not provided and require a large amount of processing power. our monolingual corpora is about 4 times larger than the corresponding oscar corpus and two times larger than the corresponding cc-100 corpus @xcite . word embeddings. word embeddings have been trained for many indian languages using limited corpora. the polyglot @xcite and fasttext @xcite projects provide embeddings trained on wikipedia. fasttext also provides embeddings trained on wikipedia + commoncrawl corpora. we show that on most evaluation tasks indicft outperforms existing fasttext based embeddings. pretrained transformers. pre-trained transformers serve as general language understanding models that can be used in a wide variety of downstream nlp tasks @xcite . several transformer-based language models such as gpt @xcite , bert @xcite , roberta @xcite , albert @xcite , etc. have been proposed. all these models require large amounts of monolingual corpora for training. for indic languages, two such multilingual models are available: xlm-r @xcite and multilingual bert @xcite . however, they are trained across ~100 languages and smaller indic language corpora. nlu benchmarks. benchmarks such as glue @xcite , superglue @xcite , clue (chinese) @xcite , and flue (french) @xcite are important for tracking the efficacy of nlp models across languages. such a benchmark is missing for indic languages and the goal of this work is to fill this void. datasets are available for some tasks for a few languages. the following are some of the prominent publicly available datasets 2 : word similarity @xcite , word analogy @xcite ), text classification, sentiment analysis @xcite , paraphrase detection (anand @xcite , qa @xcite , discourse mode classification @xcite , etc.. we also create datasets for some tasks, most of which span all major indian languages. we bun- in this section, we describe the creation of our monolingual corpora. data sources. our goal was the collection of corpora that reflect contemporary use of indic languages and cover a wide range of topics. hence, we focus primarily on crawling news articles, magazines and blogposts. we source our data from popular indian language news websites. we discover most of our sources through online newspaper directories (e.g., w3newspaper) and automated web searches using hand-picked terms in various languages. we analyzed whether we could augment our crawls with data from other smaller sources like leipzig corpus @xcite , wmt newscrawl, wmt commoncrawl @xcite , hindencorp (hindi) @xcite , etc. amongst these we chose to augment our dataset with only the commoncrawl data from the oscar corpus @xcite . article extraction. for many news websites, we used boilerpipe foot_1 , a tool to automatically extract the main article content for structured pages without any site-specific customizations @xcite . this approach works well for most of the indian language news websites. in some cases, we wrote custom extractors for each website using beautifulsoup foot_2 , a python library for parsing html/xml documents. after content extraction, we applied filters on content length, script, etc., to select good quality articles. text processing. first, we canonicalize the representation of indic language text in order to handle multiple unicode representations of certain characters. next, we split the article into sentences and tokenize the sentences. these steps take into account indic punctuations and sentence delimiters. heuristics avoid creating sentences for initials (p. g. wodehouse) and common indian titles (shri., equivalent to mr. in english) which are followed by a period. we use the indic nlp library foot_3 @xcite for processing. the final corpus for a language is created after combining our crawls with oscar corpus foot_4 and de-duplicating and shuffling sentences. we used the murmurhash algorithm (mmh3 python library with a 128-bit unsigned hash) for de-duplication. due to copyright reasons, we only release the final shuffled corpus described below. ## indicglue: multilingual nlu benchmark we now introduce indicglue, the indic general language understanding evaluation benchmark, which is a collection of various nlp tasks as described below. the goal is to provide an evaluation benchmark for natural language understanding capabilities of nlp models on diverse tasks and multiple indian languages. as discussed earlier, very few public nlp datasets are available for all indian languages. hence, we adopted a two-pronged approach to construct this benchmark. one, we use existing datasets that address some tasks. however, such datasets are available for just 4-5 indian languages. we also manually translated some english datasets into a few indian languages. we summarize statistics of these datasets in appendix a. two, we create new datasets that span all major indian languages. these datasets are curated semi-automatically using external metadata like website/wikipedia structure and are designed to present reasonably complex nlu tasks. further details (such as the min, max, average number of words per training instance) can be found in appendix c. standard train and test splits for all datasets are publicly available on the website for reproducibility. for publicly available datasets, we used the original split if provided. ## indicft: word embeddings we train fasttext word embeddings for each language using indiccorp, and evaluate their quality on: (a) word similarity, (b) word analogy, (c) text classification, (d) bilingual lexicon induction tasks. we compare our embeddings (referred to as indicft) with two pre-trained embeddings released by the fasttext project trained on wikipedia (ft-w) @xcite and wiki+commoncrawl (ft-wc) @xcite ) respectively. ## indicbert: multilingual nlu model in this section, we introduce indicbert which is trained on indiccorp and evaluated on indicglue. we specifically chose albert as the base model as it has fewer parameters making it easier to distribute and use in downstream applications. further, similar to mbert, we chose to train a single model for all indian languages with a hope of utilizing the relatedness amongst indian languages. in particular, such joint training may be beneficial for some of the under-represented languages (e.g., odia and assamese). ## conclusion and future work we present the indicnlpsuite, a collection of large-scale, general-domain, sentence-level corpora of 8.9 billion words across 11 indian languages, along with pre-trained models (indicft, indicbert) and nlu benchmarks (indicglue). we show that resources derived from this dataset outperform other pre-trained embeddings on many nlp tasks. the sentence-level corpora, embeddings and evaluation datasets are publicly available under a creative commons attribution-noncommercial-sharealike 4.0 international license. we hope the availability of these resources will accelerate nlp research for indian languages by enabling the community to build further resources and solutions for various nlp tasks and opening up interesting nlp questions.