Sartipi-Sedighin at S em E val-2023 Task 2: Fine-grained Named Entity Recognition with Pre-trained Contextual Language Models and Data Augmentation from W ikipedia (2023)

Abstract: This paper presents the system developed by the Sartipi-Sedighin team for SemEval 2023 Task 2, which is a shared task focused on multilingual complex named entity recognition (NER), or MultiCoNER II. The goal of this task is to identify and classify complex named entities (NEs) in text across multiple languages. To tackle the MultiCoNER II task, we leveraged pre-trained language models (PLMs) fine-tuned for each language included in the dataset. In addition, we also applied a data augmentation technique to increase the amount of training data available to our models. Specifically, we searched for relevant NEs that already existed in the training data within Wikipedia, and we added new instances of these entities to our training corpus. Our team achieved an overall F1 score of 61.25% in the English track and 71.79% in the multilingual track across all 13 tracks of the shared task that we submitted to.

## introduction the multiconer 2023 task 2 was initiated with the purpose of developing ner systems that can accurately detect fine-grained nes across multiple languages. the shared task was organized into 13 tracks, with 12 monolingual tracks and one multilingual track, to facilitate a thorough evaluation of the participating systems @xcite . despite the inherent complexity and ambiguity of the dataset instances, the task presented two main features that are worth mentioning. the first feature was the identification of fine-grained nes, which required the systems to detect and classify a wide range of entities with varying levels of specificity. the second feature involved the augmentation of test data for some languages with simulated errors to increase the difficulty and realism of the task @xcite . these features posed significant challenges for the participating systems and necessitated the use of advanced nlp techniques. the work presented in this paper makes two main contributions to the field of ner. the overall architecture of the model used for finetuning can be seen in figure 1 . ## related work ner is a natural language processing (nlp) task that involves identifying and classifying nes in text, such as person names, organization names, location names, and others, into predefined categories @xcite . ner is widely used in many nlp applications, such as information extraction @xcite ), text summarization (khademi and fakhredanesh, 2020), and question answering @xcite ; mollá figure 3: number of nes for training, development, test, and augmentation set per fine-grained labels et al. @xcite . fine-grained ner is a more specific variant of ner that aims to recognize more detailed categories of nes @xcite . moreover, various ner datasets have been released in both coarse-grained @xcite @xcite and fine-grained @xcite @xcite domains. additionally, there exists an automatic translation of popular ner benchmarks, for cross-lingual ner evaluation @xcite . multiconer was initially introduced as part of semeval 2022 task 11 with the objective of developing multilingual (ner) systems capable of identifying coarse-grained entities. the competition featured a total of 13 tracks, comprising 11 monolingual tracks, one code-mixed track, and one multilingual track @xcite . the multiconer dataset is an extensive multilingual dataset for (ner) that includes three domains: wiki sentences, questions, and search queries. the dataset is designed to address modern ner challenges, including low-context scenarios, such as short and uncased text, complex entities like movie titles, and long-tail entity distributions @xcite . in its second iteration, multiconer 2023 aimed to build ner systems capable of identifying nes across 12 languages, including english (en), spanish (es), hindi (hi), bangla (bn), chinese (zh), swedish (sv), farsi (fa), french (fr), italian (it), portuguese (pt), ukrainian (uk), and german (de). the shared task was subdivided into 13 tracks, comprising 12 monolingual tracks and one multilingual track. two main features of this task are worthy of mention: firstly, the identification of fine-grained nes, such as symptom, politician, and writtenwork. secondly, for some languages, namely english, chinese, italian, spanish, german, french, portuguese, and swedish, the test data was augmented with simulated errors to increase the difficulty and realism of the task @xcite . @xcite presents several challenges that current datasets and models do not adequately address. these challenges include short-text inputs, long-tail entity distributions, emerging entity types, and complex entities that are linguistically difficult to parse. these challenges pose problems for current ner systems, which are primarily trained on news texts with long sentences that discuss multiple entities. to overcome these challenges, the authors build gazetteers that incorporate external knowledge and contextual information, which is represented using transformers such as bert. contextual features from bert and gazetteers are combined through a fusion process, and the resulting features are then fed into a conditional random field (crf) layer. this enables the model to incorporate both external knowledge from gazetteers and contextual information from bert to better handle the challenges. to extend these challenges to multilingual and code-mixed settings, @xcite have introduced two datasets: mlowner, a multilingual ner training dataset for short texts in six languages, and ember, a code-mixed ner dataset covering the same languages as mlowner. these datasets can assist in training models to recognize complex nes and provide a basis for evaluating the models' performance which is included in multiconer. augmentation in order to augment the dataset and fine-tune our ner models, we utilized the wikipedia python library 1 to generate additional instances for some of the shorter instances in the dataset. to accomplish this, we constructed sets of entities from the existing entities in each language, excluding instances labeled as "o". we then used the wikipedia library to search for these entities, which provided a corresponding paragraph for each entity. in order to segment these paragraphs into 1 https://github.com/goldsmith/wikipedia sentences, we leveraged stanza @xcite . for each paragraph, we selected one sentence containing the entity and positioned in the middle or end of the sentence, rather than the beginning. subsequently, we assigned the "o" tag to the other tokens in the sentence and labeled the corresponding fine-grained category for the entity. we followed this process for all languages, with the exception of bn where no sentence segmenter was available in stanza. our aim was to maintain consistency in approach across all languages. it is important to note that certain entities in wikipedia had multiple descriptions available, but we opted to utilize only one for the sake of simplicity. given the time-consuming nature of searching for each entity in wikipedia, we employed dask @xcite to expedite the search process. with the aid of dask, it took approximately two hours for each language to search all entities. the data presented in figure 5 include one primary instance for each language, along with an augmented version of that instance. these datasets are publicly available in this foot_0 git repository. the primary motivation behind this work is to increase the diversity of instances that are used for training. additionally, when a ne appears in a short sentence or with limited context, generating more instances of that ne with different contexts can help to provide a more comprehensive understanding of its meaning and usage. this can improve the model's ability to correctly identify and classify nes in a variety of different contexts. data statistics out of all the test sets, corrupted data was found in six languages, namely en, es, fr, it, pt, sv, model name lang bert-base-spanish-wwm-uncased @xcite ) es bert-base-german-uncased foot_1 de roberta-hindi foot_2 hi chinese-roberta-wwm-ext @xcite zh bert-base-swedish-cased @xcite sv bert-base-italian-xxl-uncased @xcite it bert-large-portuguese-cased (souza et al., 2020) pt bert-base-french-europeana-cased (schweter, 2020a) fr banglabert @xcite bn roberta-large-wechsel-ukrainian foot_3 uk deberta-v3-large @xcite en bert-base-parsbert-uncased @xcite fa xlm-roberta-large @xcite multi ## methodology in recent years, transformer-based models such as bert @xcite have revolutionized the field of nlp, resulting in significant improvements in ner performance. these models are pre-trained on massive amounts of text data, enabling them to capture complex patterns and relationships between words in the text. they generate highly contextualized embeddings for each token in a sentence, allowing them to understand the meaning of words in context. to leverage the power of these models, we fine-tuned a plm for each language on the training data. hyper-parameters: we used the same hyperparameters for all of our experiments. to train our model we used the hugging face @xcite trainer and all models were trained for 15 epochs and saved the best model according to lower validation loss. we set 32, 2e-5, and 0.01 for batch size, learning rate, and weight decay, respectively. fine-tuning during this phase, we utilized transformer-based encoders. the models used for fine-tuning in the evaluation phase are listed in tafigure 7: heat map of the base system which is trained on main training data coarse-grained classes are creative works (cw), location (loc), person (per), group (grp), product (prod), medical (med) ble 1. additionally, we also fine-tuned the roberta-large (liu et al., 2019) and bert-large-uncased (devlin et al., 2018) models during the practice phase, achieving f1 scores of 63.03% and 65.13%, respectively. however, the deberta-v3-large model yielded a higher f1 score of 65, leading us to choose this model for further analysis. ## results and analysis in this section, we present the official results as reported by the organizers. base model figure 7 presents an analysis of the performance of base systems, which were trained on the training data without any augmentation. the results show that recognizing artwork and musi-calwork proved to be more challenging within the creative work class. similarly, otherloc emerged as the most difficult entity to detect within the location class. in the person class, the names of scientists and otherpersons were found to be the most challenging entities, while humansettlement was more easily recognizable. moreover, subclasses such as privatecorp and aerospacemanufacture within the group class were particularly demanding, whereas sportgrp had the best f1 values. these findings highlight the categories and languages in which ner systems struggle to accurately detect named entities. figure 8 illustrates the discrepancies between base systems and augmentation. the data in this table indicates that increasing the quantity of data in each category can have either a positive effect or a negative one in terms of f1 score, depending on the language and sub-class. the negative impact of augmentation is depicted by the black color. overall, data augmentation had the most positive impact on sub-classes such as pri-vatecorp, symptoms, and scientists. moreover, in terms of languages, hindi and french exhibited the highest improvements due to data augmentation. ## conclusion in this work, we utilized (plms) to build a system for recognizing complex nes. to increase the number of training examples and improve the performance of the system, we applied a simple data augmentation technique. however, we observed that this approach led to mixed results, with improvements in some subclasses but a reverse effect in others. one possible reason for this outcome is that the augmentation technique involves assigning "o" tags to the rest of the tokens in a sentence, which may lead to some loss of information. furthermore, the augmented data may be more unbalanced than the original data, with some instances being increased more than others. to address this issue, it may be necessary to use more sophisticated augmentation techniques or balance the data more effectively to ensure that the model can learn from a representative set of examples.