ML lab4 CS at S em E val-2023 Task 2: Named Entity Recognition in Low-resource Language B angla Using Multilingual Language Models (2023)

Abstract: Extracting of NERs from low-resource languages and recognizing their types is one of the important tasks in the entity extraction domain. Recently many studies have been conducted in this area of research. In our study, we introduce a system for identifying complex entities and recognizing their types from low-resource language Bangla, which was published in SemEval Task 2 MulitCoNER II 2023. For this sequence labeling task, we use a pre-trained language model built on a natural language processing framework. Our team name in this competition is MLlab4CS. Our model Muril produces a macro average F-score of 76.27\% , which is a comparable result for this competition.

## introduction an interesting and well-studied area of natural language processing (nlp) is the extraction of entities and detecting relationships between entities from unstructured text. named entity recognition (ner) is a sub-task of information extraction, where entities are extracted from a given text and classified into some pre-defined classes. ner has many practical use cases in various nlp tasks, including text summarization @xcite , information retrieval, machine translation @xcite , and question answering @xcite . a wide range of methodologies have been investigated by researchers to perform ner tasks including rulebased techniques @xcite , supervised feature-based approaches @xcite , unsupervised methods @xcite , deep learning-based methods @xcite and transformer-based approaches @xcite . recent applications of pre-trained language models (plms) have shown significant improvement * these authors contributed equally to this work in entity extraction and relation extraction tasks. plms can learn highly accurate linguistic, semantic, and factual information from a sizable amount of unlabeled data through self-supervised pre-training @xcite . the second edition of the shared task was conducted in semeval 2023 and was named multilingual complex named entity recognition (multiconer ii). the goal of this task was to build a system, which is capable of extracting fine-grained complex entities of different categories, as provided in the data. the complete list of categories is -facility, otherloc, hu-mansettlement, station, visualwork, musical-work, writtenwork, artwork, software, musical-grp, publiccorp, privatecorp, aerospacemanufacturer, sportsgrp, carmanufacturer, org, scientist, artist, athlete, politician, cleric, sports-manager, otherper, clothing, vehicle, food, drink, otherprod, medication/vaccine, medical-procedure, anatomicalstructure, symptom, and disease @xcite . the organizers @xcite invited the participants to submit ner models for each of the 13 languages listed on the website. they provided a different dataset @xcite for each language. each dataset was made up of three parts: training, development, and testing. during the evaluation period of this competition, only the tokens were provided in the testing dataset and the named entity tags were provided for the training and development datasets. after the end of the evaluation phase, the organizers released the gold-standard test data and we used it to analyze our system. we participated in the task to build a ner model in the bangla language track. we participated in semeval multiconer ii 2023 and developed a neural framework to extract the entities from a bangla language dataset (track 11). note that bangla is a low-resource asian language used primarily in bangladesh and in some parts of eastern india. our proposed system performs entity extraction from a low-resource bengali dataset using muril language model. it uses a transfer learning strategy by fine-tuning muril @xcite . furthermore, we apply a conditional random field (crf) decoder to implement the entity extraction task. experimental results show that the proposed system achieves comparable results on this task. ## related work named entity recognition (ner), also known as entity extraction, is a fundamental task in natural language processing. early works in ner systems used handcrafted features @xcite , but even state-of-the-art systems developed for one domain did not generally perform well for other domains. these models also struggle with multilingual datasets, and while they may obtain better precision, it comes at the cost of a lower recall and higher cost of work. since the early 2000s, ner has attracted much attention and has seen various applications in bioinformatics, molecular biology, natural language processing, and information extraction. in recent years, deep-learning methods have been employed in ner tasks @xcite and have shown considerable improvement over conventional methods. deep learning models can perform better than feature-based approaches as ner benefits from non-linear transformations, which help the model to learn complex features of the dataset. pre-trained contextual embeddings like bert, luke, and xlm-r have achieved near-human performance in certain ner tasks. these embeddings are generally trained on large datasets like wikipedia in order to improve the contextual representation of entities. however, these models significantly underperform on unseen entities and noisy datasets (which are more "real-world"-like) @xcite . some models implement knowledge bases or gazetteers, into neural architectures in order to incorporate external information. but, such systems have only reported limited improvements @xcite . models that use gazetteer features have been shown to suffer feature undertraining, where the model relies heavily on either context or gazetteer. hence, such models are not reliable as utility of gazetteer is variable. recent studies @xcite , @xcite , @xcite have developed multi-lingual, multi-task models that achieve state-ofthe-art performance in a cross-lingual setting, with much lower resource requirements than earlier models. in some cases, these models are able to perform at the level of systems that use data augmentation, by using transfer learning. they also provide comparable results in scenarios with significantly fewer cross-lingual resources. the semeval 2023 task 2 is a continuation of multiconer 2022 @xcite which also featured the aforementioned tracks along with a code-mixed track. the 2022 datasets also featured complex entities with limited context @xcite . methods that augmented external knowledge into transformer models achieved the best results, but they were unable to perform at the same level for indian languages. ner in multi-lingual domain has benefited from advances in pre-trained multi-lingual transformers, but these systems are bottlenecked by limited entity knowledge across all languages @xcite ). another key challenge for pre-trained models is the presence of emerging entities in real-world scenarios like web-queries. fine-tuning pre-trained embeddings is a significantly effective approach for many downstream nlp tasks including ner. recent approaches propose to train already finetuned embeddings on specific training data. shi et al. @xcite proposed to first train on a general multilingual model and then fine-tune it for a specific language. multi-stage fine-tuning has also shown to accelerate the training process. 3 . ## data there are 33 different fine-grained ner entity types in the bangla dataset @xcite from ## methodology in this section, we introduce the problem definition followed by the proposed methodology of our neural system called (muril + crf + bangla) muril crf bangla foot_0 . ## results and analysis we explore the effectiveness of several available multilingual plms for our downstream entity extraction task. we conduct the required training experiments on these six proposed models using bangla training data. throughout our experiments, we utilize the flair @xcite framework and train all the neural models for 30 epochs with a batch size of 64, using adamw @xcite optimizer with a very small initial learning rate of 5e -5 and a stopping criterion as mentioned in @xcite . to carry out our experiments, we used nvidia a100 and nvidia rtx a600 gpus on the paperspace cloud. ## conclusion this paper discusses a transformer-based deep neural network for extracting complex named entities from a bangla (low-resource language) dataset released by semeval multiconer ii 2023 shared task team. our system, muril crf bangla , has achieved a macro f-score of 76.27% on the test data. our future work will include few-shot learning strategies to extract complex ners from lowresource languages. we also plan to look into datadriven factored modeling techniques for dealing with class imbalance issues.