M ulti C o NER v2: a Large Multilingual dataset for Fine-grained and Noisy Named Entity Recognition (2023)

Abstract: We present MULTICONER V2, a dataset for fine-grained Named Entity Recognition covering 33 entity classes across 12 languages, in both monolingual and multilingual settings. This dataset aims to tackle the following practical challenges in NER: (i) effective handling of fine-grained classes that include complex entities like movie titles, and (ii) performance degradation due to noise generated from typing mistakes or OCR errors. The dataset is compiled from open resources like Wikipedia and Wikidata, and is publicly available. Evaluation based on the XLM-RoBERTa baseline highlights the unique challenges posed by MULTICONER V2: (i) the fine-grained taxonomy is challenging, where the scores are low with macro-F1=0.63 (across all languages), and (ii) the corruption strategy significantly impairs performance, with entity corruption resulting in 9% lower performance relative to non-entity corruptions across all languages. This highlights the greater impact of entity noise in contrast to context noise.

## introduction named entity recognition (ner) is a core task in natural language processing that involves identifying entities and recognizing their type (e.g., person or location). recently, transformer-based ner approaches have achieved new state-of-theart (sota) results on well-known benchmark datasets like conll03 and ontonotes @xcite . despite these strong results, as shown by @xcite , there remain a number of practical challenges that are not well represented by existing datasets, including low context, diverse domains, missing casing information, and text with grammatical and typographical errors. furthermore, wide use of coarse-grained taxonomies often contribute to a lower utility of ner systems, as additional fine-grained disambiguation steps are needed in many practical systems. furthermore, models trained on datasets such as conll03 perform significantly worse on unseen entities or noisy texts @xcite . ## ner model performance we assess if multiconer v2 is challenging (cf. §1.1) by training and testing with xlm-roberta @xcite . metrics we measure the model performance using precision (p), recall (r), and f1, where for f1 we distinguish between micro/macro averages. ## conclusions and future work we presented multiconer v2, a new largescale dataset that represents a number of current challenges in ner. the results from xlmr indicate that our dataset is difficult, with a macro-f1 score of only 0.63 on the multi test set. the results presented illustrate that the multiconer v2, with its detailed ner taxonomy and noisy test sets, poses substantial challenges to large pre-trained language models' performance. it is our hope that this resource will help further research for building better ner systems. this dataset can serve as a benchmark for evaluating ner methods that infuse external entity knowledge, as well as assessing ner model robustness on noisy scenarios. we envision several directions for future work. the extension of multiconer v2 to additional languages is the most straightforward direction for future work. expansion to other more challenging domains, such as search queries @xcite , can help explore fine-grained ner within even shorter inputs. finally, evaluating the performance of this data using zero-shot and few-shot cross-lingual models @xcite or even unsupervised ner models (iovine et al., 2022) can help us better understand the role of data quantity in building effective ner systems. besnik fetahu, sudipta kar, zhiyu chen, oleg rokhlenko, and shervin malmasi. 2023. semeval-2023 task 2: fine-grained multilingual named entity recognition (multiconer 2). in proceedings of the 17th international workshop on semantic evaluation (semeval-2023). association for computational linguistics. andrea iovine, anjie fang, besnik fetahu, oleg rokhlenko, and shervin malmasi. 2022. cyclener: an unsupervised training approach for named entity recognition. in proceedings of the acm web conference 2022, pages 2916-2924. shervin malmasi, anjie fang, besnik fetahu, sudipta kar, and oleg rokhlenko. 2022a. multiconer: a large-scale multilingual dataset for complex named entity recognition. in proceedings of the 29th international conference on computational linguistics, pages 3798-3809, gyeongju, republic of korea. international committee on computational linguistics. shervin malmasi, anjie fang, besnik fetahu, sudipta kar, and oleg rokhlenko. 2022b. semeval-2022 task 11: multilingual complex named entity recognition (multiconer). in proceedings of the 16th international workshop on semantic evaluation (semeval-2022), pages 1412-1437, seattle, united states. association for computational linguistics. stephen mayhew, tatiana tsygankova, and dan roth. 2019. ner and pos when nothing is capitalized. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6256-6261, hong kong, china. association for computational linguistics. tao meng, anjie fang, oleg rokhlenko, and shervin malmasi. 2021. gemnet: effective gated gazetteer representations for recognizing complex entities in low-context input. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, naacl-hlt 2021, online, june 6-11, 2021, pages 1499-1512. association for computational linguistics. erik tjong kim sang and fien de meulder. 2003. introduction to the conll-2003 shared task: languageindependent named entity recognition. in proceedings of the seventh conference on natural language learning at hlt-naacl 2003, pages 142-147. dingmin wang, yan song, jing li, jialong han, and haisong zhang. 2018. a hybrid approach to automatic corpus generation for chinese spelling check. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 2517-2527. appendix a ner challenges & motivation