NEREL : A R ussian Dataset with Nested Named Entities, Relations and Events (2021)

Abstract: In this paper, we present NEREL, a Russian dataset for named entity recognition and relation extraction. NEREL is significantly larger than existing Russian datasets: to date it contains 56K annotated named entities and 39K annotated relations. Its important difference from previous datasets is annotation of nested named entities, as well as relations within nested entities and at the discourse level. NEREL can facilitate development of novel models that can extract relations between nested named entities, as well as relations on both sentence and document levels. NEREL also contains the annotation of events involving named entities and their roles in the events. The NEREL collection is available via https://github.com/nerel-ds/NEREL .

## introduction knowledge bases (kbs) encompass a large amount of structured information about real-world entities and their relationships, which is useful in many tasks: information retrieval, automatic text summarization, question answering, conversational and recommender systems @xcite @xcite . even the largest knowledge bases are inherently incomplete, but their manual development is time-consuming and expensive. automatic population of knowledge bases from large text collections is usually broken down into named entity (ne) recognition, relation extraction (re), and linking entities to a knowledge base. in turn, training and evaluating models addressing these problems require large and high-quality annotated resources. currently, most of the available resources of this kind are in english. in this paper, we present nerel (named entities and relations), a new russian dataset with annotated named entities and relations. in developing the annotation schema, we aimed to accommodate recent advances in information extraction methods and datasets. in particular, nested named entities and relations within named entities are annotated in nerel. both of these provide a richer and more complete annotation compared with a flat annotation scheme. current datasets with nested named entities @xcite are not annotated with relations. therefore, most state-of-the-art relation extraction models @xcite do not work with relations between nested and overlapping entities. nerel aims to address these deficiencies with the addition of nested named entities and relations within nested entities. secondly, nerel relations are annotated across sentence boundaries at the discourse level allowing for more realistic information extraction experiments. figure 1 illustrates annotation of nested entities, relations between overlapping entities, as well as cross-sentence relations on a sample nerel sentence. finally, nerel provides annotation for factual events (such as meetings, negotiations, incidents, etc.) involving named entities and their roles in the events. future versions of the dataset can easily expand the current inventory of entities and relations. nerel is the largest dataset for russian annotated with named entities and relations. nerel features 29 entity and 49 relation types. at the time of writing the dataset contains 56k entities and ## related work the nosta-d collection of german wikipedia articles and online newspapers is annotated with nested named entities of four main classes. each class can appear in a nominal (proper noun) form, as a part of a token, or as a derivative (adjective) such as "österreichischen" (austrian). the digitoday corpus for finnish is annotated with six types of named entities (organization, location, person, product, event, and date). it permits nested entities with the restriction that an internal entity cannot be of the same class as its top-level entity. for example, microsoft research is annotated as a flat entity, without additional annotation of the microsoft entity. both nosta-d and digitoday datasets allow at most two levels of nesting within entities. amongst ner datasets in russian, rured @xcite provides the largest number of distinct entities with 28 entity types in the rured dataset of economic news texts. rured annotation scheme of named entities mainly follows the ontonotes guidelines with addition of extra named entities (currency, group, family, country, city, etc). currently, factrueval @xcite is the only russian dataset annotated with nested named entities with at most 2 levels of nesting. in factrueval, person mentions (per) can be subdivided into first/last names, patronymics, and nicknames; while organizations and locations -into their description/type and names (e.g. ## experiments we exploit multiple deep learning models, which deliver state-of-the-art results for the english data for two tasks, available at nerel: (i) nested named entity recognition (ner), (ii) relation extraction. to this end, we subdivided nerel into train, dev, and test sets -746/94/93 documents, respectively. ## conclusion we presented a new russian dataset nerel annotated with both nested named entity and relations, which is significantly larger than existing russian datasets. nerel dataset has several significant distinctive features, including nested named entities, relations over nested named entities, relations on both sentence and discourse level, and events our experiments with baseline models for extraction of entities and relations show that there is room for improvement in both. in the nearest future we plan to enrich the dataset by linking the annotated named entities to wikidata items.