People and Places of Historical E urope: Bootstrapping Annotation Pipeline and a New Corpus of Named Entities in Late Medieval Texts (2023)

Abstract: Although pre-trained named entity recognition (NER) models are highly accurate on modern corpora, they underperform on historical texts due to differences in language OCR errors. In this work, we develop a new NER corpus of 3.6M sentences from late medieval charters written mainly in Czech, Latin, and German.We show that we can start with a list of known historical figures and locations and an unannotated corpus of historical texts, and use information retrieval techniques to automatically bootstrap a NER-annotated corpus. Using our corpus, we train a NER model that achieves entity-level Precision of 72.81–93.98% with 58.14–81.77% Recall on a manually-annotated test dataset. Furthermore, we show that using a weighted loss function helps to combat class imbalance in token classification tasks. To make it easy for others to reproduce and build upon our work, we publicly release our corpus, models, and experimental code.

## introduction named entity recognition (ner) techniques enable the extraction of valuable insights from unstructured information in various domains. with the advancements in optical character recognition @xcite , ner can now be applied to scanned historical texts spanning over a millennium. however, despite the significant interest in ner, resources for training models to recognize entities in medieval texts remain scarce @xcite . in this work, we present a new multilingual ner corpus of 3.6m sentences from the ahisto project, which aims to build a searchable web database of late medieval charters from scanned images. in section 2, we review recent related work in historical ner. in section 3 and 4, we describe the database and the automatic pipeline used to bootstrap our corpus in. in section 5, we use our corpus to train models for historical ner and evaluate them on a manually-annotated test dataset. we show that our models are highly-accurate, reaching precision of up to 94% with up to 82% recall. additionally, we show that the use of a weighted loss function is crucial in token classification tasks with high class imbalance. we conclude in section 7. to facilitate reproducibility and future research, we publicly release our corpus under open cc0 license as well as our models and code. foot_0 2 related work @xcite created a corpus of british parliamentary proceedings from the late 17th and early 19th centuries using ocr techniques and expert annotation. they developed and evaluated a rule-based ner classifier, achieving an f 1 -score of 71%, noting the detrimental effect of ocr errors. @xcite created a corpus of 32 historical czech newspapers from 1872 using ocr and expert annotation. @xcite achieved a state-of-the-art f 1 -score of 82% on the corpus by fine-tuning a pre-trained slavicbert model @xcite . @xcite fine-tuned pre-trained monolingual transformer models on early modern ner corpora in english, french, and german, achieving a near-state-of-the-art f 1 -score of 62%. they also found that the character-based charbert model @xcite was more robust against ocr errors than bert @xcite . torres aguilar (2022) developed a corpus of 7,576 medieval charters ranging from the 10th century to the 15th century using expert annotation. they showed that finetuning pre-trained multilingual transformer models such as mbert @xcite and xlm-roberta @xcite gave comparable results to state-of-the-art bi-lstm-crf ner models @xcite . ## data description the ahisto project database includes various document types, as shown in figure 1 . the main focus is on charters from europe during the hussite era (1419-1436), each with an accompanying abstract written by project historians in contemporary czech. historians also manually annotate all named entities (people and places) in the abstracts. the original text of charters, if available, can be found in books. ocr texts of books are available @xcite , but named entities are not annotated. the database includes 4,182 abstracts with 5,621 sentences, and 15,100 unique named entities: people (62.53%) and places (37.47%). it also includes 872 books with 268,669 pages and 3.6m sentences, mostly in medieval czech (43.23%), latin (36.32%), and german (16.89%). of these pages, 3,553 (1.32%) with 50k sentences were selected as relevant to medieval charters by project historians. ## corpus annotation we developed five ner corpora from the database of the ahisto project. we created abstracts-tiny from abstracts and books-small, medium, large, and huge from books. the statistics of all our corpora are listed in ## experiments in this section, we describe the ner models that we used to produce the books-medium, large, and huge corpora and how we evaluated them. ## conclusion despite the large interest in named entity recognition (ner) in the last few decades, studies targeting late medieval historical texts are still scarce. in our work, we have developed a new silverstandard ner corpus of 3.6m sentences from late medieval charters. we described our automatic pipeline for bootstrapping a corpus using a list of known named entities. we also showed that our corpus can be used to train highly accurate models for historical ner. lastly, we have demonstrated the usefulness of using weighted loss functions in token classification tasks with high class imbalance.