S em E val-2023 Task 2: Fine-grained Multilingual Named Entity Recognition ( M ulti C o NER 2) (2023)

Abstract: We present the findings of SemEval-2023 Task 2 on Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2). Divided into 13 tracks, the task focused on methods to identify complex fine-grained named entities (like WRITTENWORK, VEHICLE, MUSICALGRP) across 12 languages, in both monolingual and multilingual scenarios, as well as noisy settings. The task used the MultiCoNER V2 dataset, composed of 2.2 million instances in Bangla, Chinese, English, Farsi, French, German, Hindi, Italian., Portuguese, Spanish, Swedish, and Ukrainian. MultiCoNER 2 was one of the most popular tasks of SemEval-2023. It attracted 842 submissions from 47 teams, and 34 teams submitted system papers. Results showed that complex entity types such as media titles and product names were the most challenging. Methods fusing external knowledge into transformer models achieved the best performance, and the largest gains were on the Creative Work and Group classes, which are still challenging even with external knowledge. Some fine-grained classes proved to be more challenging than others, such as SCIENTIST, ARTWORK, and PRIVATECORP. We also observed that noisy data has a significant impact on model performance, with an average drop of 10% on the noisy subset. The task highlights the need for future research on improving NER robustness on noisy data containing complex entities.

## introduction complex named entities (ne), like the titles of creative works, are not simple nouns and pose challenges for ner systems @xcite . they can take the form of any linguistic constituent, like an imperative clause ("dial m for murder"), and do not look like traditional nes @xcite . this syntactic ambiguity makes it challenging to recognize them based on context. we organized the multilingual complex ner (multiconer) task @xcite at semeval-2022 to address these challenges in 11 languages, receiving a positive community response with 34 system papers. results confirmed the challenges of processing complex and longtail nes: even the largest pretrained transformers did not achieve top performance without external knowledge. the top systems infused transformers with knowledge bases and gazetteers. however, such solutions are brittle against out of knowledgebase entities and noisy scenarios (e.g. spelling mistakes and typos). for entities with fine-grained classes, apart from the entity surface form, the context is critical in determining the correct class. multiconer 2 expanded on these challenges by adding fine-grained ner classes, and the inclusion of noisy input. fine-grained ner requires models to distinguish between sub-types of entities that differ only at the fine-grained level, e.g. scientist vs. athlete. in these cases, it is crucial for models to capture the entity's context. in terms of noise, we assessed how small perturbations in the entity surface form and its context can impact performance. noisy scenarios are quite common in many applications such as web search and social media. these challenges are described in ## multiconer vdataset the multiconer v2 dataset was designed to address the ner challenges described in §1. the data comes from the wiki domain and includes 12 languages, plus a multilingual subset. some examples from our data can be seen in figure 1 . for a detailed description of the multiconer v2 data, we refer the reader to the dataset paper @xcite . the dataset is publicly available. ## task description and evaluation the shared task is composed of 12 monolingual and 1 multilingual track. the multilingual track invited multilingual models capable of identifying entities from monolingual texts from any of the 12 languages. as described in section 2.4, 30% of the test sets of the en, zh, it, es, fr, pt, and sv monolingual tracks are corrupted with simulated noise. we refer the subsets with corruption as noisy subsets and the rest as clean subsets. for evaluation, we used the macro-averaged f1 scores to evaluate and rank systems. the f1 scores are computed over the fine-grained types using exact matching (i.e. the entity boundary and type must exact match the ground truth), and averaged across all types. we also report the performance on noisy subsets and clean subsets in appendix a to study the impact on noise in §6. ## baseline system similar to the 2022 edition @xcite , we train and evaluate a baseline ner system using xlm-roberta (xlm-r) @xcite , a multilingual transformer model. the xlm-r model computes a representation for each token, which is then used to predict the token tag using a crf classification layer @xcite . xlm-r is suited for multilingual scenarios, supporting up to 100 languages. it provides a solid baseline upon which the participants can build. it was trained with a learning rate of 2e -5 and for 50 epochs, with an early stopping criterion of a non-decreasing validation loss for 5 epochs. the ## participating systems and results we have received submissions from 47 different teams. netease.ai @xcite ranked 1 st in zh. their proposed system consists of multiple modules. first, a bert model is used to correct any potential errors in the original input sentences. the ner module takes the corrected text as input and consists of a basic ner module and a gazetteer enhanced ner module. this approach boosted the performance on the entity level noise and gave the system a strong advantage over the other teams ( ## insights from the systems integrating external knowledge: to overcome the challenges of complex entities, unseen entities, and low context, the integration of external data was a common theme among the submitted systems, similar to the prior edition. however, this time we observed many new and diverse knowledge sources and novel ways to inject the data into the models for ner prediction. for example, apart from using paragraphs retrieved from wikipedia using search engine, participating teams used wikidata, wikipedia infoboxes, and conceptnet. some of these approaches used knowledge sources to compute better representation of the entity labels. multilingual models: most participants in the multilingual track opted to use the task's baseline model, xlm-roberta. additionally, some par-ticipants used mluke, mdeberta, and mbert. in terms of external multilingual resources, participants made mostly use of wikipedia. complex entities: our task includes several classes with complex entities such as media titles. the most challenging entities at the coarse level were from prod class, where the average macro-f1 score across all participants was 0.68. this classes contains challenging entities, with highly complex and ambiguous surface forms, such as clothing, where the average across all participants was macro-@xmath0.58. there is a high variation among on the challenging coarse types, such as prod. for instance, for en the top ranked system, damo-nlp, achieves an f1 of 0.88, while the lowest ranking system ixa achieves a f1 of 0.21. this is highly related to whether the systems used external knowledge. figure 2 shows a confusion matrix of coarsegrained performance. we note that prod, med and cw have low recall with more than 25% of the entities not being identified correctly. grp is misclassified in 4.2% of the cases with other types such as loc or cw, highlighting the surface form ambiguity of this type. on the other hand, per obtains the highest score with 93.7%, yet at finegrained level often there is confusion among the different per fine-grained types. impact of fine-grained classes: for coarse types such as per, participants obtain very high scores, e.g. damo-nlp obtains an f1 of 0.97 on the noise-free test set. however, if we inspect the performance at the fine-grained level we notice high variance. for instance, scientist and otherper obtain significantly lower scores with f1 scores of 0.70. this gap provides two main insights. first, while the per class is often very easy to spot, distinguishing the more fine-grained types is much more challenging given their high ambiguity. second, for fine-grained ner, captur-ing context is important. in this case we see that for a class like scientist, where its entities are often in scientific reporting context (e.g. research breakthroughs), pre-trained transformer models often confuse such entities as either artist or politician, for which such models have much more pretrained knowledge. appendix b provides an in-depth error analysis at the fine-grained entity type level for all coarse grained types. impact of noise: evaluation on the noisy subsets shows that most of the participants were impacted significantly. comparing the difference in terms of macro-f1 on the noisy and the clean subsets, we notice that across all participants and languages, there is an average performance drop of 10%. the most impact is observed for zh, where the gap can be as high as macro-@xmath1. finally, we note that noise is mostly harmful when it affects named entity tokens, while noise on other has a minor impact in terms of ner performance. across all participants and languages, the average performance dropped 11.1% when corruption was applied to entity tokens and 4.3% when it was applied to context tokens. chatgpt and llms: our evaluation concluded in jan 2023, and participants did not use chatgpt for the submissions. damo-nlp @xcite reported that the performance of chatgpt on multi track is poor and it only achieved 14.78% f1 score. this matches the results of @xcite where they evaluated chatgpt on multiconer task from last year @xcite . ## conclusion we presented an overview of the semeval shared task on identifying complex entities in multiple languages. we received system submissions from 47 teams, and 34 system papers. on average, the wining systems for all tracks outperformed the baseline system by a large margin of 35% f1. all top-performing teams in multiconer 2 utilized external knowledge bases like wikipedia and gazetteers to provide additional context. we have also observed systems that provided information about the entity classes to help models know the definition of the entity. in terms of modeling, ensemble strategies helped the systems achieve strong performance. finally, the impact of noise was significant for all submitted systems, with the macro-f1 dropping significantly when compared between the noisy and clean subsets of test data.