S em E val-2022 Task 11: Multilingual Complex Named Entity Recognition ( M ulti C o NER ) (2022)

Abstract: We present the findings of SemEval-2022 Task 11 on Multilingual Complex Named Entity Recognition MULTICONER. Divided into 13 tracks, the task focused on methods to identify complex named entities (like names of movies, products and groups) in 11 languages in both monolingual and multi-lingual scenarios. Eleven tracks required building monolingual NER models for individual languages, one track focused on multilingual models able to work on all languages, and the last track featured code-mixed texts within any of these languages. The task is based on the MULTICONER dataset comprising of 2.3 millions instances in Bangla, Chinese, Dutch, English, Farsi, German, Hindi, Korean, Russian, Spanish, and Turkish. Results showed that methods fusing external knowledge into transformer models achieved the best results. However, identifying entities like creative works is still challenging even with external knowledge. MULTICONER was one of the most popular tasks in SemEval-2022 and it attracted 377 participants during the practice phase. 236 participants signed up for the final test phase and 55 teams submitted their systems.

## introduction processing complex and ambiguous named entities (nes) is a challenging nlp task in practical and open-domain settings but has not received sufficient attention from the research community. complex nes, like the titles of creative works (movie/book/song/software names) are not simple nouns and are harder to recognize @xcite . they can take the form of any linguistic constituent, like an imperative clause ("dial m for murder"), and do not look like traditional nes @xcite . this ambiguity makes it challenging to recognize them based on their context. such titles can also be semantically ambiguous, e.g. "on the beach" can be a preposition or refer to a movie. 2 finally, such entities usually grow at a faster rate than traditional categories, and emerging entities pose yet another challenge. neural models (e.g. transformers) have produced high scores on benchmark datasets like conll03/ontonotes @xcite . however, as noted by @xcite , these scores are driven by the use of well-formed news text, the presence of "easy" entities (e.g. person names), and memorization due to entity overlap between train/test sets; these models perform significantly worse on complex/unseen entities @xcite . researchers using ner on downstream tasks have also noted that a significant proportion of their errors are due to ner systems failing to recognize complex entities @xcite . examples of such challenges are highlighted in for this task, we created the multiconer dataset @xcite to address the aforementioned challenges. multiconer provides data from three domains (wikipedia sentences, questions, and search queries) across 11 different languages, which are used to define 11 monolingual subsets of the shared task. additionally, the dataset has multilingual and code-mixed subsets. we received 1,884 submissions from 55 teams during the test phase and 34 system description papers were submitted. results showed that usage of external data and ensemble strategies played a crucial role in the strong performance on in-domain data and also contributed to domain adaptation. external knowledge brought large improvements on classes containing names of creative works and groups, allowing these systems to achieve the best overall performance. ## multiconer dataset the multiconer dataset was designed to address the ner challenges described in §1. it represents three domains (wiki sentences, questions, and search queries) and includes 11 languages, plus multilingual and code-mixed subsets. for a detailed description of the multiconer dataset, we refer the reader to the dataset paper @xcite . the dataset is publicly available. 3 2.1 ner taxonomy multiconer leverages the wnut 2017 @xcite taxonomy entity types, which defines the following ner tag-set with six classes: this taxonomy allows us to capture a wide array of entities, including those with more complex entity structures, such as creative works. ## task description and evaluation the shared task is composed of 11 monolingual and 2 multilingual tracks. the monolingual tracks invited participants to build monolingual models for 11 languages addressed by the shared task. the multilingual track invited multilingual models capable of identifying entities from monolingual texts from any of the 11 languages. the code-mixed track called for models to identify entities in mixedlanguage texts (any language pair from the 11 languages). that means the multilingual models for multilingual and code mixed tracks should be able to process texts from any language and show competitive performance for all the languages. we used the macro-averaged f1 scores to evaluate and rank systems. additionally, we report precision, recall, and per-domain performance. ## baseline system we train and evaluate a baseline ner system using on xlm-roberta (xlm-r) @xcite , a multilingual transformer model. the xlm-r model computes a representation for each token, which is then used to predict the token tag using a crf classification layer @xcite . the xlm-r baseline is highly suited for multilingual application scenarios, such as our. it supports up to 100 languages and provides a solid baseline upon which the participants can build. the baseline was trained with a learning rate of 2e -5 and a maximum number of 50 epochs, with an early stopping criterion of a non-decreasing validation loss for 5 epochs. the code and scripts for the baseline system were provided to the participants to use its functionalities and further extend it with their approaches. 5 ## participating systems and results we have received submissions from 55 different teams. among the monolingual tracks, we have observed the highest participation of 30 teams in the english track. ordered by the number of participating teams, the other monolingual tracks are chinese @xcite , bangla (18), spanish (18), hindi (17), korean (17), german (16), dutch (15), farsi (15), turkish (15), and russian (14). the number of participating teams for the multilingual and code-mixed tracks are 25 and 21, respectively. most of the top-performing teams aimed at building their system targeting the multilingual track, and then retrained it for the other tracks separately and made submissions to all the 13 tracks. therefore, in the rest of this section, we will first discuss the approaches by focusing on the multilingual track. then, we will discuss teams that built their systems for one or more monolingual tracks. finally, we will summarize the methods (e.g. language models, toolkits) and resources used. ## conclusion in this paper, we have presented an overview of the semeval shared task on identifying complex entities in multiple languages. in this shared task, we have received system submissions from 55 competing teams, and 34 system description papers. on average, the wining systems for all the tracks outperformed the baseline system by a large margin of 35% f1. most of the top-performing teams in multiconer utilized external knowledge bases like wikipedia and gazetteer. they also tend to use xlm-roberta as the pre-trained language model. in terms of modeling approaches, ensemble strategies helped the systems to achieve strong performance. results from the top teams indicate that identifying complex entities like creative works is still difficult among all the classes even with the usage of external data.