Fine-tuning Pre-trained Named Entity Recognition Models For I ndian Languages (2024)

Abstract: Named Entity Recognition (NER) is a use-ful component in Natural Language Process-ing (NLP) applications. It is used in varioustasks such as Machine Translation, Summa-rization, Information Retrieval, and Question-Answering systems. The research on NER iscentered around English and some other ma-jor languages, whereas limited attention hasbeen given to Indian languages. We analyze thechallenges and propose techniques that can betailored for Multilingual Named Entity Recog-nition for Indian Languages. We present a hu-man annotated named entity corpora of ∼40Ksentences for 4 Indian languages from two ofthe major Indian language families. Addition-ally, we show the transfer learning capabilitiesof pre-trained transformer models from a highresource language to multiple low resource lan-guages through a series of experiments. Wealso present a multilingual model fine-tunedon our dataset, which achieves an F1 score of∼0.80 on our dataset on average. We achievecomparable performance on completely unseenbenchmark datasets for Indian languages whichaffirms the usability of our model.

## introduction named entities are usually real world objects that are denoted by proper names such as "location", "person", "organization", etc. named entity recognition (ner) is defined as a process of classifying each named entity into a category within a given piece of text. ner is very useful in the understanding of the structure and content of the textual information, and it also plays a pivotal role in various nlp applications. india has a wide range of languages, where each language has a unique structure, script, grammar, and other linguistic characteristics. considering india's linguistic diversity, designing accurate and robust ners for indian languages bears even greater significance. we also encounter different challenges while working with ner in an indian language setup, mainly hindi, urdu, telugu and odia. these challenges mainly arise due to the following reasons: the emergence of models such as bidirectional encoder representations from transformers (bert) @xcite and many of its variants has added a new dimension to ner with the possibility of developing multilingual ner solutions. this was made possible due to the training data of these models, that consisted of multiple languages. these models, unlike traditional machine learning models, demonstrated the ability of knowledge transfer across languages. this made ner more adaptable and accessible to low resource languages, like many of the indian languages, which are still largely unexplored and low resourced. many indian languages suffer from lack of labelled data, linguistic resources, and nlp toolkits which is required for designing specific language related features for most of the machine learning models. this issue can easily be resolved by the multilingual neural models by offering a viable solution of knowledge transfer from high to low resource languages. fine-tuning a single multilingual model can leverage the linguistic knowledge encoded with the model. we experiment with different multilingual pre-trained models and show their efficacies with a strong focus on the availability of resources. ## related work the previous works in this field of ner have mainly explored the challenges and opportunities of ner techniques in multilingual settings. researchers have developed and fine tuned some multilingual ner models, that help perform ner across multiple languages @xcite . these models rely on pre-trained transformer based architectures, for example: bert, roberta @xcite , xlm-roberta @xcite . it has been observed that cross lingual transfer learning is extremely useful and effective for low resource languages, where ner models pre-trained on high resource languages are adapted for low resource languages. the research has also focused on creating and curating multilingual corpora encompassing a large range of languages, that prove to be valuable resources for training and evaluating multilingual ner models. there has been significant amount of work regarding datasets and other resources using pre-trained transformer models. naamapadam @xcite and hiner @xcite are two widely used publicly available datasets for indian language ner. 1. naamapadam dataset: naamapadam consists of data from 11 major indian languages from two language families. the dataset contains more than 400k sentences annotated with a total of at least 100k entities from three standard entity categories (person, location, and, organization) for 9 out of the 11 languages. it is a significant resource for ner in indian languages. ## named entity annotation for the task of ner, we annotated data from two domains, general and governance. at least 2 annotators with post graduation education were involved in the task for each language. named entities are annotated for following 4 languages where 3 are from the indo aryan family and 1 from dravidian family (shown in sequence): hindi, odia, urdu, and telugu. for hindi, 7 annotators were included. the average inter-annotator agreement for all four languages was 0.95, which shows good agreement among the annotators. the agreement scores are evaluated on 200 sentences for each language. we compute cohen's kappa measure for this. for hindi, we compute the average of cohen's scores among all possible combinations of the raters. language-wise inter-annotator agreement scores are reported in ## methodology we first explored various datasets and models available for hindi named entity recognition. as our named entity annotated corpus is annotated with a different tagset, we could not make use of the existing models directly. in this pursuit, we explored different fine-tuning techniques to develop a model tailor-made for our tagset. we experiment with two approaches for the creation of monolingual models. first approach is to fine-tune a baseline bert model for our task, and the second approach fine-tunes a bert based ner model for our task, on our annotated dataset. as our basic model, we select xlm-roberta-base (conneau et al., 2020) model, which is a transformer based architecture designed for multilingual natural language understanding tasks. this model is pretrained on a vast multilingual corpus and hence is capable of efficiently handling multiple languages, which makes it well suited for the multilingual ner task. the selection of this model for multilingual ner in indian languages can be further justified by its strong performance in various nlp tasks and its ability to generalize well across languages. its multilingual pre-training enables it to capture linguistic nuances in different languages, including those present in indian languages. as our main focus had been creating a multilingual model for low resource languages, we found multiple ways of improving the results for ner for low resource languages, some of them are as follows: out of all these available methods, we find the approach that uses cross lingual and progressive transfer learning, to train language models for low resource languages with language model for high resource languages by appending the vocabulary. this method worked well for languages belonging to the same language family. we also try taking a different approach of converting the scripts from native to roman script and carrying out the experiments on the multilingual model, but it was observed that the model trained on native scripts was performing better than the model trained on the roman scripts. a reason for this behaviour can be the absence of roman scripts for the corresponding native scripts of the language in the training data of the pretrained xlm roberta @xcite base model. hence, no further exploration was done in this direction. we also evaluated the dataset on the crf @xcite ) model, which as expected did not give a good result due to the fact that it was not a pre-trained model. the major limitation of a crf model lies in it inability to transfer knowledge for reusability. hence, we did not continue any exploration in that direction. ## experiments our experiments include reviewing of the earlier methods including conditional random fields and neural based named entity taggers. in this, we analyze the pre-trained models and datasets released as indic ner model and naamapadam dataset @xcite and hiner @xcite . our experiments include testing indic ner and hiner on our annotated dataset, where we record an f1 score between 0.55 to 0.65 for the dev and test sentences of the gold dataset. we refer to our dataset as gold dataset and this convention is used in the future tables and figures. these experiments are conducted to visualize the performance of different models and adapting them towards developing a customized model for our gold dataset. as an initial experiment, we test the publicly available models on each other to assess their performance which are reported in we then proceed towards creating a monolingual model for hindi. our hypothesis is that a model that is already trained on ner task is expected to outperform the base model with no knowledge about the ner task. we validate our hypothesis by fine-tuning a baseline bert model (not trained for an ner task) on our annotated dataset and finetuning a bert based ner model (hiner) on our annotated dataset. this experiment is carried out on all the tags of our dataset. we report accuracies between bert @xcite based ner model and baseline bert based model. as expected, the model which is a result of fine-tuning on hiner model performs better than fine-tuning on baseline bert model. we then combine all the data from different languages and train a multilingual model. we experiment with changing of scripts i.e converting all the data to the same script before finetuning, to check whether the new model performs better or worse than the original model. we convert all our data to roman script for this purpose. we then fine-tune the roberta base model on naamapadam dataset and gold dataset as the part of the comparative study between native script and roman script. in the fine-tuning approach used, we combine all the training data for all languages and fine-tune the monolingual model on this combined data. we then analyze the performance of each language on the multilingual model. ## conclusion and future work we introduce a specialized ner dataset tailored for four indian languages. our experiments with established ner models on this dataset provide valuable insights for fine-tuning. our proposed fine-tuning technique paves a way for ner in low resource languages. techniques such as transfer learning and architectural modifications can further be explored to improve the model. we propose augmenting our dataset with additional annotated sentences. adding data from other indian languages can potentially lead to substantial performance improvements.