DS 4 DH at S em E val-2022 Task 11: Multilingual Named Entity Recognition Using an Ensemble of Transformer-based Language Models (2022)

Abstract: In this paper, we describe our proposed method for the SemEval 2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER). The goal of this task is to locate and classify named entities in unstructured short complex texts in 11 different languages. After training a variety of contextual language models on the NER dataset, we used an ensemble strategy based on a majority vote to finalize our model. We evaluated our proposed approach on the multilingual NER dataset at SemEval-2022. The ensemble model provided consistent improvements against the individual models on the multilingual track, achieving a macro F1 performance of 65.2%. However, our results were significantly outperformed by the top ranking systems, achieving thus a baseline performance.

## introduction named entity recognition (ner) is the process of identifying pre-defined categories of named entities, such as people, places, organizations, from unstructured text. ner usually serves as an important first component in various natural language processing (nlp) tasks, such as question answering @xcite , information retrieval @xcite and machine translation @xcite . thus, the performance of the ner system can influence the quality of many downstream nlp applications. despite the high performance achieved by the current ner systems, they still face some critical challenges @xcite . ner models are typically trained on a wellformed news text containing a variety of entities within a relatively long context. in addition, most of the existing ner datasets usually include a large number of common entities between train set and test set. as a result, the performance of the models drops dramatically in the real world applications as they must deal with unseen entities and noisy texts. furthermore, previous studies on ner have mostly focused on english and as a result, many other languages specially low-resource ones, such as turkish, korean, and persian, have not been as well studied @xcite . in this context, semeval-2022 proposes the task of multilingual complex named entity recognition (multiconer) @xcite , which is concerned with detecting semantically ambiguous and complex entities in short and low-contextual settings for 11 languages (i.e. english, spanish, dutch, russian, turkish, korean, farsi, german, chinese, hindi, and bangla). in this paper, we present a multilingual ner method based on ensemble of deep neural language models. we first trained multiple ner models on the official training dataset and then utilized an ensemble strategy based on a majority of votes from the top-3 bestperforming models. based on the macro-average f1-score of 65.2, achieved by our model, we placed 20th in the multilingual track of the competition. the rest of the paper is organized as follows. section 2 reviews published work related to the ner task. section 3 and section 4 explain our proposed ner system and the experimental setup respectively. the results and detailed analysis of the model performance are discussed in section 5 and the conclusion and future work are reported in section 6. ## related work over the last decade, deep learning approaches have significantly improved the results of different ner tasks @xcite . the most recent works on ner utilize pre-trained language models like bert in a supervised setting @xcite @xcite . these models use pre-trained language models that have been trained on a large monolingual or multilingual corpus to fine-tune ner models. @xcite intro-duced a number of current challenges of developed ner datasets and systems. the challenges include the presence of long-tail entities, i.e., entities with large distribution and millions of values, emerging entities, i.e., domains with growing entities, or complex entities, i.e., linguistically complex entities such as gerunds and full clauses, in the context of the systems' inputs. in addition, as discussed in @xcite the context of search queries and questions usually include a short amount of words which could be problematic for ner systems. to overcome the above issues, @xcite created three new ner datasets, including short sentences, questions, and search queries, and a novel ner system which uses a contextual gazetteer representation (cgr) encoder and a mixture of experts (moe) gating network to feed a crf layer for final predictions. @xcite also tackled the challenge of the code-mixed queries in which entities and non-entity query terms coexist simultaneously. they developed a large-scale ner dataset in six languages with four different scripts as well as a novel multi-lingual ner method for code-mixed queries which integrates external knowledge into the multilingual setting. ## method our multilingual ner system takes sentences in 11 different languages and automatically identifies and classifies named entities within each sentence. for each sentence, the system utilizes three different bert-like models (fined-tuned on the multilingual ner dataset) to perform entity prediction independently. next, for each entity, the label with the majority of votes will be chosen as the final prediction. in the following, we provide details on different ner models we used in our pipeline and our ensemble strategy for label prediction in section 3.1 and section 3.2, respectively. ## experimental setup this section discusses the dataset we used to conduct our experiments, followed by the parameters we used to train the models. ## conclusion in this paper, we presented our multilingual ner method that uses an ensemble of different finetuned models to identify the named entities in the unstructured texts. using a variety of multilingual pre-trained language models, we first fine-tuned several ner models and then applied a vote-based ensemble strategy to make the final prediction. our submission achieved an overall f1 score of 65.2, ranking 20th in the multilingual track of task 11 of semeval-2022. our next step would be to examine other possible types of ensemble strategies as it has shown to be effective in the performance of the ner models.