M ulti C o NER : A Large-scale Multilingual Dataset for Complex Named Entity Recognition (2022)

Abstract: We present AnonData, a large multilingual dataset for Named Entity Recognition that covers 3 domains (Wiki sentences, questions, and search queries) across 11 languages, as well as multilingual and code-mixing subsets. This dataset is designed to represent contemporary challenges in NER, including low-context scenarios (short and uncased text), syntactically complex entities like movie titles, and long-tail entity distributions. The 26M token dataset is compiled from public resources using techniques such as heuristic-based sentence sampling, template extraction and slotting, and machine translation. We tested the performance of two NER models on our dataset: a baseline XLM-RoBERTa model, and a state-of-the-art NER GEMNET model that leverages gazetteers. The baseline achieves moderate performance (macro-F1=54%). GEMNET, which uses gazetteers, improvement significantly (average improvement of macro-F1=+30%) and demonstrates the difficulty of our dataset. AnonData poses challenges even for large pre-trained language models, and we believe that it can help further research in building robust NER systems.

## introduction named entity recognition (ner) is a core task in natural language processing which involves identifying entities in text, and recognizing their types (e.g., classifying entities as a person or location). recently, the development of transformerbased ner approach have results in new stateof-the-art (sota) results on well-known benchmark datasets like conll03 and ontonotes @xcite . despite these strong results, there remain a number of practical challenges that may not be represented by these existing datasets. as noted by @xcite , increasingly higher scores on these datasets are driven by several factors: accordingly, models trained on existing benchmark datasets such as conll03 tend to perform significantly worse on unseen entities or noisy text @xcite . ## multiconer dataset overview the multiconer dataset was designed in order to address the ner challenges described in §1.1. it represents 3 domains (wiki sentences, questions, and search queries) and includes 11 languages, including multilingual and code-mixed subsets. multiconer was collected and released as part of the semeval 2022 task#11, serving more than 236 participants across all the different languages @xcite . this taxonomy allows us to capture a wide array of entities, including those with more complex entity structure, such as creative works. ## dataset construction in this section, we provide a detailed description of the methods used to generate our dataset. ## ner model performance to evaluate whether our new dataset poses realworld challenges (cf. §1.1), we train and test two existing ner systems: (1) xlm-roberta (conneau et al., 2020), a large multilingual transformer model; and (2) gemnet @xcite @xcite , a state of the art model that integrates gazetteers into xlm-roberta. ## conclusions and future work we presented multiconer, a new large-scale dataset that represents a number of current challenges in ner. results obtained on our data showed that our dataset is challenging. a xlmr based model achieves only approx. 50% f1 in average while gemnet improves f1 performance by more than 30% using gazetteers. these results demonstrate that multiconer represents challenging scenarios where even large pre-trained language models fail to achieve good performance without external resources. it is our hope that this resource will help further research for building better ner systems. this dataset can