T elugu NER : Leveraging Multi-Domain Named Entity Recognition with Deep Transformers (2022)

Abstract: Named Entity Recognition (NER) is a successful and well-researched problem in English due to the availability of resources. The transformer models, specifically the masked-language models (MLM), have shown remarkable performance in NER during recent times. With growing data in different online platforms, there is a need for NER in other languages too. NER remains to be underexplored in Indian languages due to the lack of resources and tools. Our contributions in this paper include (i) Two annotated NER datasets for the Telugu language in multiple domains: Newswire Dataset (ND) and Medical Dataset (MD), and we combined ND and MD to form Combined Dataset (CD) (ii) Comparison of the finetuned Telugu pretrained transformer models (BERT-Te, RoBERTa-Te, and ELECTRA-Te) with other baseline models (CRF, LSTM-CRF, and BiLSTM-CRF) (iii) Further investigation of the performance of Telugu pretrained transformer models against the multilingual models mBERT, XLM-R, and IndicBERT. We find that pretrained Telugu language models (BERT-Te and RoBERTa) outperform the existing pretrained multilingual and baseline models in NER. On a large dataset (CD) of 38,363 sentences, the BERT-Te achieves a high F1-score of 0.80 (entity-level) and 0.75 (token-level). Further, these pretrained Telugu models have shown state-of-the-art performance on various existing Telugu NER datasets. We open-source our dataset, pretrained models, and code.

## introduction named entity recognition (ner) aims to identify various named entities from the raw text. typically these named entities are broadly categorized into person names, locations, organizations, and other categories depending on the domain. identifying these named entities is necessary and is proven to be very helpful in natural language processing (nlp), information retrieval (ir), and information extraction (ie). moreover, when so much data is generated daily today, ner becomes very important in processing and extracting meaningful information from the text. however, most ner work is limited to the resource-rich english language due to the availability of annotated datasets, efficient feature representations, and tools to process the data. english has many huge annotated datasets like @xcite , ontonotes @xcite and wnut @xcite . traditional models like conditional random fields (crf) @xcite have been used for ner modeling by training them on these datasets. with the development in deep learning, solutions like @xcite and @xcite used long short-term memory (lstms) for sequencelabelling tasks like ner. further, the combination of the lstm-crf model proposed by @xcite has achieved even better performance. recently, transformer models @xcite have proven to be achieving similar results to the state-of-the-art models @xcite . hence, we can infer that there has been extensive and rapid research in ner for english with significant advancements. however, ner developed in english cannot be generalized and extended due to the rich morphological nature of indian languages. unlike english, most of the resources created for indian languages are for machine translation. however, in the ner task, the meaning of context, the roles of named entities, differentiations amongst categories, and syntactic and semantic structures will be lost if we translate english sentences to telugu. examples of telugu language ner sentences, their wx notation (a standard notation used for indian languages) 2 , and their english translations are reported in figure 1 . from the examples, we can notice that telugu's context and the actual ner tags are not captured by english-translated sentences when given to the stanford corenlp ner tool 3 . therefore, we understand the need for ner to address these challenges even in morphologically rich languages like telugu. hence, we created an annotated dataset for ner in telugu, which will be a good resource for those working in telugu nlp areas such as dialog systems, text summarization, machine translation, and question answering. furthermore, we used pretrained telugu transformer models @xcite and finetuned on the telugu ner dataset to achieve ner in multiple domains. in this paper, we aim at creating resources for ner in telugu. overall, we make the following contributions to this paper: (1) we publicly release two diverse annotated ner datasets, which will be pioneering resources for building automated ner systems in telugu, (2) we build ner models using telugu pretrained transformer models to analyze the entity-level and token-level class performance across the multi-domain datasets and (3) we achieve the state-of-the-art results on existing ner datasets. our extensive experiments also lead us to these crucial insights: (i) telugu pretrained transformer 2 https://en.wikipedia.org/wiki/wx_ notation 3 https://corenlp.run/ models fine-tuned for the ner task outperform the existing baseline methods. (ii) it is widely known that language-specific models (bert-te and roberta-te) outperform the existing pretrained multilingual models (mbert, xlm-r, and in-dicbert), this holds to be true for telugu as well. traditional methods: the early ner experiments were studied to identify specific categories of named entities like proper names @xcite , organizations, and locations @xcite . they were based on rules, heuristics, and gazetteers. however, they could not handle outof-gazetteer and ambiguous cases. unlike earlier work, @xcite and @xcite proposed crf and hmm models to handle numerous sequence to sequence tasks such as ner and pos tagging. nevertheless, the main limitation of these models is the computational complexity and that they cannot handle unknown words. later, it was found that deep learning (dl) based models like lstm-crf @xcite and bilstm-crf @xcite focused on long-term dependencies and handled the feedback mechanism on sequence labeling tasks with high accuracy. however, these models compute token representation one by one (sequentially), which hinders the full exploitation of parallel computation and bidirectional context. transformers based ner: in recent years, transformers @xcite have successfully performed various nlp tasks like machine translation, language modelling, and semantic role labeling. recently introduced bidirectional encoder representations from transformers (bert), developed by @xcite , is a powerful language modeling technique to handle masked-language modelling (mlm) and nextsentence prediction tasks. furthermore, by finetuning the bert model on the conll dataset, a high f1 score of 92.8% was reported in @xcite for ner. the success of bert led to other variations like roberta @xcite and electra @xcite . ner for telugu: though ner is a wellresearched problem in english, very few works describe ner for telugu. existing ner systems mainly use small datasets and limited categories like person, location, and organisation. in addition, these systems are developed based on heuristics @xcite , traditional ml @xcite or dl @xcite methods. to the best of our knowledge, we are the first to create such a large and diverse annotated dataset of 38,363 sentences for the ner task in telugu. further, we create a multi-domain dataset that incorporates both newswire and medical domains. finally, we take inspiration from the transformer models and use bert-te to model ner in telugu. ## annotated dataset for ner task existing ner datasets are small and mainly focus on limited categories like person (per), location (loc), and organisation (org). there are two significant existing datasets for ner in telugu: (i) wikiann @xcite (ii) lrec-ner @xcite . the wikiann dataset has per, loc, and org entity types, with a total of 6, 495 annotated sentences. on the other hand, even though the lrec-ner dataset has 32, 610 sentences, it consists only of per, org, loc, and miscellaneous named entity category (misc). hence, we came up with three datasets consisting of diverse named entity categories for ner in telugu: (i) newswire dataset (nd), (ii) medical dataset (md), and (iii) combined dataset [newswire+medical] (cd). the nd focuses on the general named entity categories in the news domain, while the md focuses on data related to the biomedical domain. ultimately, by combining nd and md, we form the cd. detailed statistics of the three datasets are shown in figures 2a , 2b , and 2c. further, details regarding the dataset have been discussed below. data collection and preprocessing: for the nd, we crawled around 50,000 sentences from telugu360 foot_1 , greatandhra foot_2 , and eenadu foot_3 websites that generally publish articles related to current affairs, sports, movies, gossips, and the latest news. however, while doing so, we noticed that in the prevailing covid-19 situation, much information on the telugu websites focuses on health and diseases. so then, we created a separate dataset by crawling 20,000 sentences for md. we collected this data from boldsky foot_4 and telugu-wikipedia foot_5 websites. after crawling, we cleaned and preprocessed the data by removing the unwanted urls, hashtags, hyperlinks, english text, and duplicate sentences. entity types in datasets: after analyzing the preprocessed data, we identified the following named entity categories that would best suit to describe the data: ## results this section presents the entity and token-level macro-averaged classification metrics for models trained on nd, md, and cd in tables 2 and 3 . to further examine each class's performance, we show the performance of eight models on each dataset in section 5.1 and answer several research questions. entity-level results: we make the following observations from wrt to precision, recall & f1-score, finetuned telugu pretrained transformer models such as bert-te, roberta-te, and electra-te show an improved performance than crf, lstm-crf, and bilstm-crf models. (iii) specifically, the bert-te, roberta-te, and electra-te models yield the highest, second-highest, and third-highest recall and f1 scores for all the classes except for oth and cardinal categories, as shown in figures 3(a ) and 3(b). (iv) we observe that the bert-te model is better than all the models for nd (0.83) and cd (0.80) in terms of f1-score, whereas roberta-te model performs the best on md (0.73). this demonstrates that the pre-training models capture the word context better. (v) the performance of all models on md is comparatively low compared to nd and cd. this can be explained by analyzing entity class differences across the eight training models as discussed in 5.1. token-level results: class distribution performance: to understand the performance of models on each class, we show the individual class performance wrt entity-level macro-average classification metrics, including precision, recall, and f1-score. entity-level class distribution: figures 3(a), 3(b), and 3(c) display each class performance at entity-level wrt f1-score on three datasets. we also report the f1-score of three best performing models such as bert-te, roberta-te, and electra-te for each class at entity-level on three datasets in figures 4 , 5 , and 6 . further, we showcase the recall of each class at entity-level on three datasets (refer to figures 10, 11, and 12 in appendix). overall, the results indicate that the transformer-based models outperform crf and lstm-crf based models in terms of recall and f1 score across the three datasets. bert-te achieves the highest recall and f1-score in 7 out of the 9 classes. however, the crf and lstm-crf based models have similar performance but display relatively lower class performance in terms of recall and f1-score when compared to the finetuned telugu pretrained models. specifically, lstm-crf and bilstm-crf models with ft as cardinal loc org oth per time 0.5 0.7 0.9 bert-te roberta-te electra-te newswire dataset tags f1-score figure 5: distribution of f1 scores across three bestperforming systems on newswire dataset. input have shown a trend of lower performance in most classes. token-level class distribution: figure 7 shows the token-level class performance wrt f1score across eight models on cd. similar to entity-level, the transformer-based models bert-te, roberta-te, and electra-te outperform the other models wrt f1-score in figure 7. bert-te and roberta-te show an increasing f1-score performance for every class, while lstm-crf-ft and bilstm-crf-ft report an overall lower f1score across all the classes. model #sentences #parameters mbert 2.5tb 110m xlm-r 2.5tb 125m indicbert 452.8m 11m bert-te 8.2m 108m roberta-te 8.2m 125m electra-te 8.2m 14m here, we compare the performance of three finetuned telugu pretrained models (bert-te, roberta-te, and electra-te) with existing multilingual transformer models (mbert @xcite , xlm-r @xcite , and indicbert @xcite ) for the ner task. figure 8 showcases the entity-level class performance across the three datasets. from figure 8 , we observe that bert-te, and roberta-te outperform mbert, xlm-r, and indicbert across the three datasets. on the other hand, the electra-te model has a similar performance as mbert and xlm-r. further, we report the pretrained model parameters of each model, as depicted in here, we noticed that electra-te and indicbert models have comparatively fewer parameters than other models. ## conclusion this paper presented annotated datasets and an empirical study of the performance of various finetuned telugu pretrained transformer models for the ner task. we compare these results with the commonly used architectures like crf, lstm-crf, and bilstm-crf models in all three datasets. we even compare these pretrained telugu models to existing multilingual models like mbert, xlm-r, and indicbert. we conclude that finetuned telugu pretrained transformer models outperform all the other models across multiple domains and they give state-of-the-art performance on existing datasets. we also notice that electra-te yields significantly equal performance when compared with multilingual models even after being trained on a much smaller corpus. in the future, we would like to perform fine-grained ner and also expand ner to more domains for the telugu language.