Named Entity Recognition for F rench medieval charters (2021)

Abstract: This paper presents the process of annotating and modelling a corpus to automatically detect named entities in medieval charters in French. It introduces a new annotated corpus and a new system which outperforms state-of-the art libraries. Charters are legal documents and among the most important historical sources for medieval studies as they reflect economic and social dynamics as well as the evolution of literacy and writing practices. Automatic detection of named entities greatly improves the access to these unstructured texts and facilitates historical research. The experiments described here are based on a corpus encompassing about 500k words (1200 charters) coming from three charter collections of the 13th and 14th centuries. We annotated the corpus and then trained two state-of-the art NLP libraries for Named Entity Recognition (Spacy and Flair) and a custom neural model (Bi-LSTM-CRF). The evaluation shows that all three models achieve a high performance rate on the test set and a high generalization capacity against two external corpora unseen during training. This paper describes the corpus and the annotation model, and discusses the issues related to the linguistic processing of medieval French and formulaic discourse, so as to interpret the results within a larger historical perspective.

## introduction named entity recognition (ner) is a fundamental task aiming at detecting and classifying words used as rigid designators in a text. typically the operation consists in a lexical segmentation of texts separating entities from common words and a subsequent classification of them according to a set of predefined categories. ner has quickly become part of the nlp toolbox used by digital humanities to structure and mine textual collections. however, its application to historical texts still involves some challenges. not only medieval charters use low-resource languages such as medieval versions of latin until the 15th century and vernacular languages (e.g. old and middle french) from the 13th c. onwards, but they are also written in diverse linguistic versions due to language change over space and time. moreover their strong topic-dependency further complicates the use of general classifiers and popular nlp tools used for modern languages, since charters contain mainly property deeds whose wording was framed by well-defined documentary models using stereotyped structures and a restricted and formulaic vocabulary. in the last years, digitizing and scholarly editing original manuscript sources have gained great momentum, coinciding with their reappraisal in medieval studies. new tools are needed to explore and mine these specialized sources in order to foster insights from millions of documents and support new hypotheses. named entity recognition, in particular, is a key element in order to provide an indexed structure to historical texts. it would allow the implementation of information retrieval techniques and adapt diplomatics and historical research methods to large scale corpora. our contribution can be summarized as follows: (1) an annotated corpus built upon three different collections of medieval acts in french, (2) an adequate training and validation framework, to create supervised models able to automatically distinguish places and person names in unstructured texts; (3) we suggest a test protocol to evaluate the models' ability to generalize on a wide range of acts regardless of regional and chronological differences. ## related work sequence tagging, including ner, is a classic nlp task. ner processing libraries such as spacy, flair, and stanford corenlp have become popular in digital humanities, because one can easily train custom models and apply diverse neural approaches based on rnn architectures using control gates as in the case of lstm approaches or adding a double-scope lecture (bi-lstm) @xcite . indeed, lstm architectures using statistical classifiers such as crf and lda have become the most successful training methods, esp. when feature engineering cannot be deployed as it happens with low-resources languages. while supervised corpus-based methods have generally become the paradigm in ner research, ner approaches on ancient western languages as classical latin and greek are still deploying ruled-based analyzers coupled with gazetteers and patronymic lists @xcite due to the lack of relevant annotated corpora. moreover, research on ner for heritage resources focused on 19th century ocred newspaper collections and it has not been confronted with the intense spelling variations of pre-modern, so-called "pre-orthographic" sources @xcite . currently there are only few available nlp resources for pre-modern french: a dependency treebank, the syntactic reference corpus of medieval french @xcite , and two lemmatizers for old french, one trained on the base de franc ¸ais médiéval @xcite using treetagger and the other, deucalion, based on a encoder-decoder architecture @xcite . but there is a lack of large language models for tasks such as topic clustering and named entity recognition, given that pos tools only detect, but do not classify proper names or deal with their length and composition. @xcite show that robust ner models can be trained even in the absence of wordlevel features (e.g. lemma, pos), but that one then has to add character-based word representations to encode all lexical phenomena, concatenated with word embeddings vectors. leveraging on pre-trained language models increase significantly the performance compared with traditional approaches @xcite . yet, static embeddings and contextualized word representations such as bert @xcite and @xcite , require large-scale corpora for training and fine-tuning, and they are not available for ancient language versions ("état de langue") or domain-specific texts. ## corpus description to remedy the lack of relevant training corpora, we created a relatively large dataset for the present task, composed of ca. 500,000 words, from three different sources: diplomata belgica, home-alcar, and the arbois (cbma). ## evaluation all three models obtain high performance results, both in pers (0.944 to 0.959) and loc (0.945 to 0.956) categories. the first metric shows that performance between b-and i-tags are harmonic which implies that our three models are able to correctly detect the boundaries of the entities regardless of their length. the second metric confirms that false negatives and false positives are marginal both in loc and pers thus achieving a very good result in multi-class tasks. the custom bi-lstm shows a better performance in most categories. specifically it generalizes better as indicated by the lower number of false negatives and false positives compared to the flair and spacy models. it can be explained by the use of a denser set of features including pos and french medieval embeddings more adapted to the medieval charters. however, the flair and spacy performance is only 1 to 2 points lower, confirming in general, the three models perform slightly less well on i-loc suggesting some issues on loc compound entities. average results are not affected severely since only 5-10% of the entities belongs to this type (see ## discussion this work clearly proves that robust tools to classify named entities on french medieval charters can be modeled using character-based neural network approaches (bi-lstm-crf) on our annotated corpus. the test sets being of different dates (end of 13th c. and end of 14th c.), the high performance proves that the models are fairly robust against language change. this may be explained as follows. ## conclusion we present an annotated corpus to french medieval charters and three neural ner models. the evaluation returns a strong performance reaching 0.96 in both pers and loc categories on the homogeneous test set and 0.95 in pers and 0.92 in loc on unseen data which confirms that the model can be used on charters from other chronologies and origins. besides, we can confirm that our models are able to produce a double hypothesis which implies a high confidence on the recognition of nested entities extensively used in medieval charters. while this work concerns the development of a neural ner model, it can benefit several research areas including indexation systems, data visualization and distant reading methods. named entities are the base of several digital and classical humanities research methods on networks, timelines, event-lines and gis-maps. these models and the annotated data on which it is built, which are themselves new contributions, can be easily integrated into other pipelines, thus contributing to enhance the toolbox for old and middle french regarding other supervised methods.