Naamapadam: A Large-Scale Named Entity Annotated Data for I ndic Languages (2023)

Abstract: We present, Naamapadam , the largest publicly available Named Entity Recognition (NER) dataset for the 11 major Indian languages from two language families. The dataset contains more than 400k sentences annotated with a total of at least 100k entities from three standard entity categories (Person, Location, and, Organization) for 9 out of the 11 languages. The training dataset has been automatically created from the Samanantar parallel corpus by projecting automatically tagged entities from an English sentence to the corresponding Indian language translation. We also create manually annotated testsets for 9 languages. We demonstrate the utility of the obtained dataset on the Naamapadam-test dataset. We also release IndicNER , a multilingual IndicBERT model fine-tuned on Naamapadam training set. IndicNER achieves an F1 score of more than 80 for 7 out of 9 test languages. The dataset and models are available under open-source licences at https://ai4bharat.iitm.ac.in/naamapadam .

## introduction named entity recognition (ner) is a fundamental task in natural language processing (nlp) and is an important component for many downstream tasks like information extraction, machine translation, entity linking, co-reference resolution, etc. the most common entities of interest are person, location, and, organization names, which are the focus of this work and most work in nlp. given high-quality ner data, it is possible to train goodquality ner systems with existing technologies @xcite . for many high-resource languages, publicly available annotated ner datasets @xcite @xcite as well as high-quality taggers @xcite are available. figure 1 : illustration of named entity projection. we perform (i) ner with fine-tuned english bert model, followed by (ii) word alignment between parallel sentence pair and (iii) projection of the english entities onto indic sentence. however, most indic languages do not have sufficient labeled ner data to build good-quality ner models. all existing ner corpora for indic languages have been manually curated @xcite @xcite @xcite . given the number of languages, the expenses and the logistical challenges, these datasets are limited on various dimensions viz. corpus size, language coverage, and broad domain representation. in recent years, zero-shot crosslingual transfer from pre-trained models, fine-tuned on task-specific training data in english has been proposed as a way to support various language understanding tasks for low-resource languages @xcite . however, this approach is more suitable for semantic tasks and the cross-lingual transfer does not work as well for syntactic tasks like ner when transferring across distant languages like english and indian languages @xcite @xcite . hence, there is a need for in-language ner training in recent years, the paradigm of mining datasets from publicly available data sources has been successfully applied to various nlp tasks for indic languages like machine translation @xcite , machine transliteration @xcite and many natural language understanding and generation tasks @xcite . these approaches have led to the creation of large-scale datasets and models with broad coverage of indic languages in a short amount of time. taking inspiration from these successes, we also explore the automatic creation of ner datasets by utilizing publicly available parallel corpora for indian languages and high-quality english-named entity taggers. in this work, we undertake the task of building large-scale ner datasets and models for all major indic languages. the following are the contributions of our work: • we build naamapadam 1 , the largest publicly available ner dataset for indic languages for 11 languages from 2 language families. naamapadam contains 5.7m sentences and 9.4m entities across these languages from three categories: person, name, and location. this is significantly larger than other publicly available ner corpora for indian languages in terms of the number of named entities and language coverage. ## related work we discuss the state of ner datasets for indian languages and common methods used to improve ner for low-resource languages. ## mining ner corpora following @xcite , our method for building ner corpora is based on projecting ner tags from the english side of an english-indic language parallel corpora to the corresponding indic language words. for our work, we use the samanantar parallel corpus @xcite which is the largest publicly available parallel corpora between english and 11 indic languages. figure 1 illustrates our workflow for extracting named entity annotated indic sentences from an english-indic parallel sentence pair. it involves the following stages: (a) tagging the english sentence with a high-accuracy english ner model (sec 3.1), (b) aligning english and indic language words in the parallel sentence pair (sec 3.2), (c) projecting ner tags from the english sentence to indic words using the word alignments (sec 3.3). these stages are further described in this section. ## testset creation we have created naamapadam-test: manually annotated test set for indian language ner evaluation. the naamapadam-test comprises 500-1000 annotated sentences per language for 9 languages namely bengali, gujarati, hindi, kannada, malayalam, marathi, punjabi, tamil, and, telugu. the annotators were provided sentences with named entity annotations obtained using the methodology described in section 3. the annotators had to verify if the projected ner annotations were correct and rectify the annotations if incorrect. they were asked to follow the conll 2003 annotation guidelines @xcite . the human annotations were contributed by volunteers who are native language speakers. ## experimental setup we analyze the performance of models trained on the naamapadam-train dataset with alternative approaches for low-resource ner and to models trained on publicly available datasets. to this end, we investigate the following research questions: rq1: are models trained on data obtained from projection approach (naamapadam-train) better than zero-shot and teacher-student models? rq2: how does the model trained on publiclyavailable dataset fare against the model trained on naamapadam-train data? we evaluate it on the naamapadam-test set. ## conclusion we take a major step towards creating publicly available, open datasets and open-source models for named entity recognition in indic languages. we introduce naamapadam, the largest entity recognition corpora for 11 indic languages containing more than 100k training sentences per language, and covering 11 of the 22 languages listed in the indian constitution. naamapadam also includes manually labelled test set for 9 indic languages. we also build indicner, an mbert based multilingual named entity recognition model for 11 indic languages. we also provide baseline results on our test set along with a qualitative analysis of the model performance. the datasets and models will be available publicly under open-source licenses. we hope the dataset will spur innovations in entity recognition and its downstream applications in the indian nlp space.