Razmecheno: Named Entity Recognition from Digital Archive of Diaries “Prozhito” (2022)

Abstract: The vast majority of existing datasets for Named Entity Recognition (NER) are built primarily on news, research papers and Wikipedia with a few exceptions, created from historical and literary texts. What is more, English is the main source for data for further labelling. This paper aims to fill in multiple gaps by creating a novel dataset “Razmecheno”, gathered from the diary texts of the project “Prozhito” in Russian. Our dataset is of interest for multiple research lines: literary studies of diary texts, transfer learning from other domains, low-resource or cross-lingual named entity recognition. Razmecheno comprises 1331 sentences and 14119 tokens, sampled from diaries, written during the Perestroika. The annotation schema consists of five commonly used entity tags: person, characteristics, location, organisation, and facility. The labelling is carried out on the crowdsourcing platfrom Yandex.Toloka in two stages. First, workers selected sentences, which contain an entity of particular type. Second, they marked up entity spans. As a result 1113 entities were obtained. Empirical evaluation of Razmecheno is carried out with off-the-shelf NER tools and by fine-tuning pre-trained contextualized encoders. We release the annotated dataset for open access.

## introduction modern named entity recognition (ner) systems are typically evaluated on datasets such as ace, ontonotes and conll 2003, collected from news or wikipedia. other common setups to test ner systems include cross-lingual evaluation @xcite and evaluation in domains, other than general, such as biomedical domain @xcite . additionally, the vast majority of ner datasets are in english. a few large-scale datasets for other languages are nosta-d @xcite (german), norne @xcite (norwegian), aqmar @xcite (arabic), ontonotes @xcite (arabic, chinese), factrueval @xcite . we present in this work a new annotated dataset for named entity recognition from diaries, written in russian, -"razmecheno" 1 . the texts are provided by the project "prozhito" foot_1 which digitizes and publishes personal diaries. diaries exhibit different surface and style features, such as complex narrative structure, and author-centricity, mostly expressed in simple sentences with predominance of verbs and noun phrases. ner annotation is the first step for summarisation and coreference resolution tasks. design choices, made for the corpus construction, are the following. we follow the standard guidelines of named entity annotation and adopt four commonly-used types person (per), location (loc), organization (org), facility (fac). we add one more type, char, which is used for personal characteristic (e.g., nationality, social group, occupation). texts, used in the corpus, are sampled from the diaries, written in the late 1980s, the time period addressed as perestroika. we utilized crowdsourcing to label texts. our dataset enables assessing performance of the ner models in a new domain or in a cross-domain transferring. we make the following contributions: ## related work most of the standard datasets for named entity recognition, as ace @xcite and conll @xcite , consist of general domain news texts in english. for our study, there are two related research lines: ner for the russian language and ner in digital humanities domain. ## evaluation we've benchmarked two groups of models on the presented dataset. off-the-shelf tools were evaluated without any modifications, while transformerbased models were evaluated after a fine-tuning. ## conclusion this paper introduces razmecheno, a novel dataset for named entity recognition. the texts in the dataset are sampled from the project "prozhito", which comprises personal diaries, written in russian, from the 17th century up to the end of the 20th century. in particular, texts, marked up in razmecheno belong to the mid-1980 years, the period in russia, commonly known as perestroika. razmecheno is a middle-scale dataset so that it contains enough data to carry out literal and historical studies. the annotation schema, used in razmecheno, is simplistic. it consists of five named entity types, of which four are commonly used in ner datasets, namely, persons, locations, organization, and facilities. an only named entity type, introduced in this project, is characteristics of the different groups of people. the annotations are flat; overlapped, or nested entities are not allowed at the moment. as our annotation schema matches a commonly used inventory of named entity types, it is possible to leverage upon pre-trained models and transfer learning techniques. the experimental evaluation of razmecheno is two-fold. first, we carry out an extensive analysis of how available off-the-shelf ner tools cope with the task. the results reveal, that natasha outperforms other tools under consideration by a small margin. however, of five named entity types, the off-the-shelf tools used to support only three. next, we experiment with four state-ofthe-art pre-trained transformers. a monolingual model, rubert significantly outperforms other transformers, followed by a multilingual model xlm-roberta. there are a few directions for razhmecheno development. we plan to annotate the collected sentences for other information extraction tasks, including co-reference resolution, relation extraction, and entity linking. providing ner is the first step to present the diary's plot in a concise form. this can be beneficial for studying the narratives and events present in diaries. this way, razhmecheno could serve as a test-bed for end-to-end information extraction models. experiments in domain adaptation and cross-lingual transfer from other languages are another research line. finally, we have set up the whole environment to annotate texts from "prozhito", so that diaries from other periods can be marked up with a little effort. appendix f transformers confusion matrix