Training and Evaluation of Named Entity Recognition Models for Classical L atin (2023)

Abstract: We evaluate the performance of various models on the task of named entity recognition (NER) for classical Latin. Using an existing dataset, we train two transformer-based LatinBERT models and one shallow conditional random field (CRF) model. The performance is assessed using both standard metrics and a detailed manual error analysis, and compared to the results obtained by different already released Latin NER tools. Both analyses demonstrate that the BERT models achieve a better f1-score than the other models. Furthermore, we annotate new, unseen data for further evaluation of the models, and we discuss the impact of annotation choices on the results.

## introduction commonly an important precursor to information extraction, text summarisation and the creation of knowledge bases, named entity recognition (ner) has become a ubiquitous task in natural language processing (nlp). for modern highresource languages, generic ner off-the-shelf solutions, focusing mainly on identifying locations, organizations and people, can produce highly accurate annotations. for historical languages, even prolific ones like latin, the task remains a challenge, in part due to a lack of annotated corpora and tools @xcite . we pursue three main objectives with this paper: the paper contributes to the application of nlp to latin on a methodological level, since we propose a thorough analysis of the results of ner on latin and identify the most critical points. in addition, the paper is associated with the publication of ner models and datasets, and documents the choices that have been implemented. the paper is structured as follows: after introducing existing work and datasets related to ner for classical languages (section 2), we describe the data used, and the training of the models and their performance on in-domain and out-of-domain test sets (section 3). section 4 provides a qualitative error analysis of the best performing model based on f1 metrics. in section 5, we introduce the annotation of new data from the lasla corpus, and analyse the results of the automatic annotation by the best-performing model. the data and code related to this paper are made available on a github repository. 1 2 related work previous work has highlighted the challenges linked to ner for latin. @xcite identified among others the following relevant challenges concerning ner on historical documents: variable and sparse feature space (generalizing over different genres and domains, cf. @xcite ), dynamics of language such as spelling variations and change in naming conventions, general lack of resources (e.g. typologies from modern languages not fitting for historical documents). in addition, burns (2023) underlined another difficulty of the already scarce resources: differences in orthographic conventions and annotation schemes. lastly, both @xcite and torres aguilar (2022) consider the frequency of overlapped and nested entities in latin as a challenge. when it comes to existing models, @xcite trained a crf-based model on latin medieval charters from burgundy. later torres aguilar (2022) tested two approaches for creating a multilingual pipeline for medieval charters (french, spanish and latin): the first uses contextual and static embeddings coupled to a bilstm-crf (bidirectional long-short term memory) classifier, and the second employs a fine-tuning method using the pre-trained multilingual bert and roberta models. for both of these efforts, custom charter corpora were annotated. in the context of the herodotos project -which aims to catalogue ancient ethno-political groups and their interactions @xcite created a neural, bilstm-crf based entity recognizer @xcite trained on classical latin texts. in addition, ner is included in text analysis pipelines for latin, such as the classical language toolkit (cltk; @xcite and latincy (burns, 2023). in recent years, transformer-based models (with the bert architecture as one of the prime instantiations) have become the norm for various nlp applications @xcite @xcite . these models have been leveraged, inter alia, for latin morphosyntactic tagging @xcite @xcite and translation alignment for ancient languages @xcite , which could also be leveraged for named entity projection from modern languages given a parallel corpus @xcite . for greek ner, a bert-based approach equally proved to be effective @xcite . there already exists a transformer-based model for latin (latinbert; bamman and burns, 2020) but to the best of our knowledge, it has not yet been finetuned on the task of named entity recognition. regarding datasets, the herodotos dataset (at the time of training) is the only available ner dataset for classical latin @xcite . additionally, the authors of the latincy pipeline are planning to make their custom dataset publicly available @xcite . lastly, the multilingual medieval charter dataset, which includes non-classical latin @xcite , is avail- on the whole dataset, minimal preprocessing was performed to iron out formatting mistakes. afterwards, the five works were divided into two parts: in-domain, used for training and in-domain testing, and out-domain, used exclusively for out-domain testing. the latter should assess the model's generalizing capabilities to texts that are significantly different from the data it was trained on. in this experiment, the in-domain part consisted of the prose texts, (bgall., bciv., ep. and nh.) the out-domain part consisted of the one poetry text, ars. am.. the in-domain texts were then split into three sets: a training set (75%), a validation set (12.5%) and an in-domain test set (12.5%). as the bertmodel processes input on the sentence level, the sentence order was randomized. the sentences containing rare multi-word locations and groups were identified and split separately. each of those splits was later appended to one of the three sets to ensure that each contained entities of every type. the frequencies of the entity types can be found in to ensure representative testing, the data was augmented with manually annotated test sets from the lasla corpus in the second part of this paper (see section 5), both for in-domain prose and outdomain poetry. ## annotation of the lasla corpus in what follows, we discuss the performance of the same ner models on the lasla latin corpus. 3 as the lasla corpus includes a diverse range of classical latin texts, it represents an interesting test set to investigate the generalisability of the models. with this procedure, we also establish criteria for the annotation of the most problematic classes. in addition, we augment the test set by including both prose and poetry works (resp. in-domain and outof-domain) which do not appear in the training data and that belong to different genres with respect to the training data. overall, this process allows us to reach conclusions on the urgency of guidelines, of data generation, and the generalisability of existing models across different projects. the portion of the lasla corpus used for this experiment is composed of 1,738,435 tokens, belonging to 130 latin literary texts by 21 authors ranging from the 2nd century bce to the 2nd century ce. it is linked to the lila knowledge base, an open-ended knowledge base of linguistic linked data @xcite . the uris for lemmas and tokens provided by the linking are published to ensure interoperability and reusability of the data. foot_3 ## conclusions and future work the process of training two new models on existing data, comparing their results on previously and newly annotated data, and comparing their performance to existing models allows us to draw several conclusions. first, the good performance of latinbert1 and 2 demonstrates the interest of applying transformer-based models for the ner task on latin. especially for the category 'pers' the two models yield satisfactory results. however, the analysis of the annotations and the errors has shown that the development of guidelines is crucial to ensure the consistent annotation of datasets that can be reused as training-and test-sets across different projects and for different models. in addition, the significantly worse performance of the models on poetry indicates the need for training data for this specific type of texts. future work should also consider improving the preprocessing and normalization of training data (e.g. harmonizing the use of the 'v/u' 'i/j' pairs), and testing the use of multilingual bert models that include latin (mbert, xlm-roberta) @xcite . likewise, additional linguistic information available in the lasla corpus (e.g. lemmatization and pos tagging) might improve the results of the ner. finally, after we establish a system for named entity disambiguation employing information from existing extensive resources, we will explore the potential of mutual reinforcement, i.e. we will consider whether results from one system can improve the other and vice-versa as argued by @xcite . gold label 1 & 2 wrong 1 correct 2 correct o 0 11 25 b-pers 13 47 24 i-pers 2 1 1 b-loc 14 12 14 i-loc 0 0 2 b-grp 18 16 12 i-grp 1 0 0 total 58 87 78