S erbian NER &Beyond: The Archaic and the Modern Intertwinned (2021)

Abstract: In this work, we present a Serbian literary corpus that is being developed under the umbrella of the “Distant Reading for European Literary History” COST Action CA16204. Using this corpus of novels written more than a century ago, we have developed and made publicly available a Named Entity Recognizer (NER) trained to recognize 7 different named entity types, with a Convolutional Neural Network (CNN) architecture, having F1 score of ≈91% on the test dataset. This model has been further assessed on a separate evaluation dataset. We wrap up with comparison of the developed model with the existing one, followed by a discussion of pros and cons of the both models.

## introduction the "distant reading for european literary history" foot_0 (cost action ca16204) has started in 2017 with the purpose of using computational methods to analyse large collections of literary texts @xcite . the main goal of this ongoing action is to compile a multilingual open-source collection, named european literary text collection (eltec), containing linguistically annotated sub-collections of 100 novels per language written more than 100 years ago. in this paper, we present a collection of serbian texts in this corpus, named srpeltec. alongside, we describe our efforts in developing its named entity (ne) layer, defined previously as one of the main action's deliverables. for this purpose, we adjusted and used the existing rule-based ne recognizer for serbian, dubbed srpner, that we will describe in section 2 together with some approaches to ne recognition in literary texts. this srpner model was applied to the raw version of the selected texts from srpeltec collection, presented in section 3. based on the specifically tailored guidelines, different evaluators performed careful checks and corrections, yielding a gold standard @xcite . this enabled us to train a cnn-based ne recognizer, named srpcnner, presented in section 4. having the gold dataset, prepared as described in subsection 4.1, we trained (subsection 4.2) and evaluated the model in two different settings: first, we discussed our model's performance on the srpeltec-gold test subset, as shown in subsection 4.3. afterwards we carried out a detailed evaluation on a collection of novels that were not present in the gold standard, named srpeltec-eval, with the findings and a thorough discussion given in section 5. finally, conclusions and plans for the future work were stated in section 6. ## related work the existence of large-scale lexical resources for serbian, e-dictionaries in particular @xcite , coupled with local grammars in the form of finite-state transducers @xcite , enabled the development of a comprehensive rule-based system for ner srp-ner. this system presented by @xcite targeted 11 classes of nes: dates and time (moments and periods), money and measurement expressions, geopolitical names (countries, settlements, oronyms and hydronyms), and personal names (one or more last names with or without first names and nicknames). the system was developed to recognize nes in newspapers and similar texts. it was manually evaluated on a sample of unseen newspaper texts. the overall f 1 score of the model was ≈ 96%. to the best of our knowledge, so far there were no attempts to produce a ner system for serbian literary texts. the enhanced version of srpner was later utilized by @xcite for the preparation of a gold standard annotated with personal names, which was used for building training sets for 4 different levels of annotation, on which two ml-based ne recognizers were trained and evaluated (spacy and stanford). as a support for the developed ner models, @xcite joined several existing tools and developed various new tools, combined into a web platform ner&beyond. 2 although ner systems in general were developed mostly for newspaper and similar texts, there were some endeavours to produce functional systems for literary texts as well. enrichment of french renaissance texts with proper names @xcite faced two challenges: text diversity due to various spellings of words, and need to deal with numerous xml-tei tags used to preserve the format of original editions. authors' solution was based on the cascades of finite-state automata and both general dictionaries and those built specifically for the project. the evaluation showed that the slot error rate of name tagging was 6.1%. a dataset of literary entities comprising 210,532 tokens evenly drawn from 100 different english literary texts annotated with ace entity categories (person, location, geo-political entity, facility, organization, and vehicle) foot_2 was published in @xcite . the authors' main motivation was to asses ner models' performance on different types of texts. their conclusion was that recognition improved for almost all entity types when literary texts were used for the both training and evaluation (on average @xmath0.1%, @xmath1.6% and f 1 = 68.3%), whilst for training on general texts, such as news data, and testing on literary texts the results were much poorer (on average @xmath2.8%, @xmath3.7% and f 1 = 45.7%). shinra2020-ml shared-task @xcite targeted the categorization of wikipedia entities using the extended named entity (ene) hierarchy in 30 languages (serbian was not one of them). ene included about 220 fine-grained categories of nes in a hierarchy of up to four layers. some traditional ne types such as location were specified as either geopolitical location ("city", "province", "country", etc.) or geological region ("mountain", "river", "lake", etc.). ene also included some new ne types like "products", "event", "position", etc. dekker et al. (2019) experimented with different off-the-shelf ner tools for the extraction of social network graphs from classic and modern english fiction novels. the authors wanted to find out to what extent are these tools suitable for identifying fictional characters in novels, and what are differences and similarities that can be discovered between social networks extracted for different novels. distant reading training school for named entity recognition and geo-tagging for literary analysis organized within the cost action 16204 foot_3 covered ner approaches in general, annotation campaigns, practical work with ner tools, annotating ner in tei, analyzing ner annotation for literary characters and place names and ner data analysis. different types of ner systems were tested for several languages, some based on symbolic methods, relying on rules developed by experts and dictionaries (gazetteers), others using statistical and data-driven approach. the ne layer of eltec corpus has presently been produced for three languages: hungarian, portuguese, slovene. @xcite reported on the ner annotation of the portuguese sub-collection of the eltec corpus. authors used the palavras-ner parser, a constraint grammar (cg) system, in which ner is an integrated task of grammatical tagging, implemented with the basic tagset of 6 ne categories (person, organization, place, event, semantic products and objects) with about 20 subcategories at three levels, disambiguated by cg-rules: known lexical entries and gazetteer lists, pattern-based name type prediction and context-based name type inference for unkno-wn words. this system was applied to eight novels that were fully human revised. evaluation results varied for precision from 64.6% to 80.8%, and recall from 64.3% to 82.0%. at the mentioned distant reading training school it was concluded that spacy module foot_4 for python was used for training ner models for many involved languages, already having tagsets that could be mapped to the eltec annotation scheme, elaborated later in section 3. partalidou et al. (2019 ) developed a pos-tagger and a ner for greek using spacy, based on newspaper articles and wikipedia dataset, able to recognize the following entity types: location, organization, person and facility. jabbari et al. (2020) created a corpus consisting of news articles in french, which served as a dataset for training and evaluation of a ner and a relation extraction algorithms using spacy. modrzejewski et al. (2020) incorporated ner trained in spacy into an english/german machine translation system, with the aim to improve ne translation. morever, jiang et al. (2016) conducted a comparative evaluation of different publicly available ner tools. based on different criteria, authors concluded that spacy was among best performing across all tested datasets. having all this in mind, we decided for spacy as a framework for developing a serbian ner model on a collection old literary texts. ## serbian collection in the eltec as described earlier in section 1, the focus of the cost action ca16204 is to compile the eltec corpus containing collections of old european novels published between 1840 and 1920 in various languages. in order to make these sub-collections decent representatives of their corresponding languages, the novels were selected to evenly represent a) novels of various sizes: short, medium, long; b) four twenty-year time periods within the examined time span, c) canonical novels as well as those not known to wider audience or completely forgotten, as judged by the number of reprints, and d) female and male authors @xcite . the last version of the eltec (v. 1.1.0) was released in april 2021. 6 it contained 14 language sub-collections each with at least 50 novels, while 8 collections contained targeted 100 novels per language. the srpeltec corpus 7 in the latest el-tec release has 90 novels. the work on this collection is still in progress with the aim to obtain the complete collection by the end of the project. contrary to a number of other european languages involved in this action, the serbian corpus is being produced from scratch, because the vast majority of novels from the selected time period were not digitized before, they were not digitized in the proper manner or were not available @xcite . this preparation procedure involved several steps: selection of novels, retrieval of hard copies, scanning, ocr, automatic correction of ocr errors (for which a specialized tool based on the serbian morphological dictionaries was produced @xcite ), correction of remaining errors by a number of volunteer readers, and production of metadata. one of the important aspects of this eltec collection is to feature annotations of certain named entities. at this moment, annotation of named entities is carried out for nine languages, including serbian. according to the guidelines, the common ner tagset includes the following 7 categories: demonyms (demo), professions and titles (role), works of art (work), person names (pers), places (loc), events (event) and organizations (org). 8 4 srpcnner model for serbian in this section we first explain how we have turned the srpeltec corpus into a dataset for ner. afterwards, we describe the training of the ner model srpcnner, followed by a detailed evaluation. web users can navigate to http://ner.jerteh.rs/ in order to apply the srpcnner model directly on input text. the model can also be applied to a customsize collection of text files using the previously mentioned ner&beyond web platform. ## separate evaluation set despite the encouraging results obtained on the srpeltec-gold, shown in subsection 4.3, we 14 hashembedcnn, https://spacy.io/api/architectures# hashembedcnn wanted to further asses our model's performance. for this purpose, we prepared an independent evaluation set, dubbed srpeltec-eval, containing corrected annotated chapters from three novels that were not included in the training procedure. we applied the same evaluation procedure for the both recognizers. after running a them on srpeltec-eval, we took the strictest approach and differentiated between the following three situations: in the subsections that follow, we analyze the performances of our newly trained model srpcnner and the adjusted srpner on the srpeltec-eval corpus. finally, we discuss their strengths and weaknesses and make certain statements about their applicability in different contexts and situations. ## conclusions and future work we presented the corpus of old serbian novels, which served as a basis for training a cnn-based ner model srpcnner using the spacy module's framework for python. after comparing this newly developed model for serbian with the existing rule-based srpner, we came to the conclusion that the previously developed one performs better on this type of texts, due to its adaptability. however, it is not easy to set it up and use it, while the model trained in spacy can be easily and efficiently applied to the large text collections, and there is still a lot of room for improvement. first of all we need to remove observed flaws from srpeltec-gold. moreover, in the future we intend to use the pre-trained word embedding vectors instead of the default tok2vec layer. the integration of pos-tagging and lemmatization with ner into tei eltec level 2 schema foot_10 is an ongoing activity, where a pipeline starts with srpner annotation, followed by pos-tagging and lemmatization by a tree-tagger @xcite . as a result, first 16 novels from srpeltec collection were annotated with pos, lemmas, and ne in a format agreed by the cost action.