M ulti NERD : A Multilingual, Multi-Genre and Fine-Grained Dataset for Named Entity Recognition (and Disambiguation) (2022)

Abstract: Named Entity Recognition (NER) is the task of identifying named entities in texts and classifying them through specific semantic categories, a process which is crucial for a wide range of NLP applications. Current datasets for NER focus mainly on coarse-grained entity types, tend to consider a single textual genre and to cover a narrow set of languages, thus limiting the general applicability of NER systems. In this work, we design a new methodology for automatically producing NER annotations, and address the aforementioned limitations by introducing a novel dataset that covers 10 languages, 15 NER categories and 2 textual genres. We also introduce a manually-annotated test set, and extensively evaluate the quality of our novel dataset on both this new test set and standard benchmarks for NER.In addition, in our dataset, we include: i) disambiguation information to enable the development of multilingual entity linking systems, and ii) image URLs to encourage the creation of multimodal systems. We release our dataset at https://github.com/Babelscape/multinerd .

## introduction named entity recognition (ner) represents a milestone in information extraction, and its aim is to identify and classify key information in unstructured texts, i.e. named entities @xcite . it is widely used in a broad spectrum of downstream applications, like machine translation @xcite , question answering @xcite , automatic text summarization @xcite , and entity linking @xcite , inter alia. with the advent of pretrained language models like bert @xcite or luke @xcite ) -this latter with a particular focus on named entities -the ner field observed astonishing results on conventional benchmarks @xcite . however, such benchmarks are limited in size, cover a single textual genre, and are available only for a narrow set of languages. moreover, they focus on coarse-grained entity types, and often overlook more complex entities like titles of books, songs and movies. these latter entities are not simple nouns and can be both syntactically and semantically ambiguous. specifically, they can assume the form of any linguistic constituent (e.g. singin' in the rain) which makes them difficult to extract. indeed, in the last decade, the ontonotes 5.0 dataset @xcite has become very popular thanks to its high quality, size and finegrained categories. nevertheless, it covers only 3 languages, namely, english, arabic and chinese. since the manual creation of training data for ner is expensive and time-consuming -especially when many languages have to be covered -several studies have tried to address data scarcity by producing training data automatically @xcite @xcite , recently showing that automaticallygenerated annotations can boast a quality comparable to that of manually-created ones @xcite . unfortunately, although these studies have considered a wider range of languages, they have still focused on coarse-grained entities and on a single textual genre, i.e. encyclopedic texts from wikipedia foot_0 @xcite . in this paper, inspired by the success of the ontonotes 5.0 dataset and by recent achievements in automatic data creation, we fill the aforementioned gaps and propose the following novel contributions: additionally, although in this work we focus on ner, we also contribute to the entity disambiguation (also known as entity linking) task, i.e. the task of linking entities mentioned in texts with their corresponding entry in a knowledge base. specifically, for a given entity, we provide disambiguation information together with its ner tag in order to enable training, validation and testing of multilingual entity linking models. finally, we also include image urls to encourage the creation of multimodal systems. to enable comparability on our benchmark, we release our data and software at https: //github.com/babelscape/multinerd . ## ner classes our 15 ner classes are a subset of the newly introduced 18 classes of @xcite designed to reduce the intrinsic sparsity of the entity linking task. specifically, they are: person (per), location (loc), organization (org), animal (anim), biological entity (bio), celestial body (cel), disease (dis), event (eve), food (food), instrument (inst), media (media), plant (plant), mythological entity (myth), time (time) and vehicle (vehi). we prefer these classes to the ontonotes ones, because they cover a wider range of macro categories. for instance, the ontonotes' product class -which groups very heterogeneous entities -is split into food, vehi and inst. over and beyond these, our new set contains animals, plants, biological entities, celestial bodies, diseases and mythological entities that are not present in ontonotes. ## multinerd in this section we describe our language-agnostic strategy for automatically generating a fine-grained and multilingual resource to train robust ner and ed systems. specifically, our methodology widely extends previous state-of-the-art strategies, and it is characterized by the following five steps: i) preprocessing of wikipedia and wikinews articles (section 4.1), ii) identification of entities (section 4.2), iii) tagging the identified entities with the ner labels (section 4.3), iv) propagation of the annotations (section 4.3), v) enhancement of the annotations (section 4.4). ## experiments in this section, we describe our experimental setup (section 5.1), the datasets foot_9 we use to train (section 5.2) and evaluate (section 5.3) our methodology, and finally the results obtained (section 5.4). ## conclusions in this work we introduced multinerd, a novel resource for training robust multilingual and finegrained named entity recognition (and disambiguation) systems. to create it, we presented a new language-agnostic strategy for generating high-quality silver-standard ner and ed annotations. this strategy uses a knowledge-based semantic classifier to automatically annotate wikipedia and wikinews articles, and then iteratively enhances the annotations produced by means of a selfimprovement algorithm which builds upon neural models. our experiments showed that multin-erd outperformed previous state-of-the-art dataproduction methods across all tested languages and domains, while covering a much wider set of ner categories. additionally, we also included image urls in our dataset to encourage the development of multimodal ner and ed systems. this visual information could also be exploited to further improve the quality of the annotations by ensembling the predictions of nlp and computer vision models. we release our multin-erd dataset and software at https://github . com/babelscape/multinerd.