Minanto at S em E val-2023 Task 2: Fine-tuning XLM - R o BERT a for Named Entity Recognition on E nglish Data (2023)

Abstract: Within the scope of the shared task MultiCoNER II our aim was to improve the recognition of named entities in English. We as team Minanto fine-tuned a cross-lingual model for Named Entity Recognition on English data and achieved an average F1 score of 51.47\% in the final submission. We found that a monolingual model works better on English data than a cross-lingual and that the input of external data from earlier Named Entity Recognition tasks provides only minor improvements. In this paper we present our system, discuss our results and analyze the impact of external data.

## introduction nowadays, language models are capable of solving various problems, amongst others grammar checking, speech (pattern) recognition and translation. but there are still some challenges. the recognition of complex named entities is one of them. named entities (nes) occur in various domains and can be, for instance, the name of a person, a city or the title of a book or a movie. as these nes can be ambiguous like "on the beach", which can either be a preposition or a movie title, their recognition is more challenging than that of a simple noun. furthermore, the amount of nes is not as stable as the vocabulary of a language, but increases fast. the focus of the multiconer shared task @xcite is to address this challenge in monolingual cases as well as in the multilingual case. the main challenges this year are the fine-grained entity taxonomy with over 30 classes and the induced noise in the test dataset, whereby the task becomes considerably harder. ## related work the same task was already organized last year at semeval-2022. @xcite presented semeval-2022 task 11: multilingual complex named entity recognition (multiconer), which is a shared task focusing on the identification of complex named entities like media titles, products, and groups in text documents across multiple languages and domains. they also discussed the evaluation metrics used in the task, including the f1 score, and presented baseline results for each language and domain. for each language in the dataset the participants developed models that can identify the boundaries of complex named entities, classify them into predefined entity types, and recognize nested entities and entity modifiers. in general, the top-performing systems for each language and domain used neural network models, such as transformers like bert @xcite and xlm-roberta @xcite , and ensemble models that combine multiple neural network models and made use of multilingual pre-training or language-specific pre-training to improve the performance of the model. the baseline result for english in multiconer 2022 was a f1 score of 61.36% and the best performing team outperformed the baseline with a score of 91.22%. they used a knowledge retrieval module to retrieve k most relevant paragraphs from a knowledge base (i.e., wikipedia). these paragraphs were concatenated together with the input, and token representations were passed through a conditional random field to predict the labels. they employed multiple such xlm-roberta models with random seeds and then used a voting strategy to make the final prediction @xcite . the key challenge of this year's multiconer dataset @xcite is to deal with complex named entities with limited context. as explained by @xcite named entity recognition (ner) for web queries can be very challenging because of the limited context, especially for code-mixed queries. they found that the infusion of multi-lingual gazetteers helps to identify nes in code-mixed queries more accurately. but, according to @xcite , the use of gazetteer features can also lead to poor generalization. therefore, the appropriate use of gazetteer knowledge integration is still challenging. ## data the multiconer dataset @xcite used in the semeval-2023 task 2 includes 12 languages (bangla, german, english, spanish, farsi, french, hindi, italian, portuguese, swedish, ukrainian, chinese). its fine-grained taxonomy contains 33 named entity classes from six categories (location, creative work, group, person, product, medical). the data was collected from three domains: wikipedia sentences, questions and search queries. the multiconer dataset was designed to address the ner challenges for complex entities as described before. this task can be completed using either a monolingual track, a multilingual track, or a code-mixed track where the entity is from one language and the rest of the text is written in another language. in addition to that, we also trained our model with external data from the conll 2003 ++ and wnut 2016 datasets 1 . as the labels in the external data were different than those of the training data, we needed to translate them and lost some named entities, because there was no comparable label to translate it to. the test data additionally included noisy instances with typing errors in the context tokens and entity tokens. ## methodology in this section, the methodology of our proposed model is presented. it consists of two main parts: the pre-processing of the data and the fine-tuning of the pre-trained xlm-roberta-base model for the ner task. the third part is the training on external data. ## conclusion participating at semeval-2023 task 2 as team minanto we fine-tuned xml-roberta-base for named entity recognition on english data. in the final submission we obtained as rank 28 out of 34 teams an overall fine-grained macro-averaged f1 score of 51.47%. after the submission, we realized that our model performed better based on roberta-base than on xlm-roberta-base, because roberta-base is trained on english data only. furthermore, we found that the additional training on external data from conll 2003 ++ and wnut 2016 did not increase the performance much, because these datasets are not as fine-grained as the multiconer dataset. as every external dataset we found had different class labels, we needed to translate them, which led to loss and potential errors in translation. standardized fine-grained labels would make future work with external data much more easier. besides, the use of noisy external data could be much more helpful with regard to the noisy test data.