Transferring Modern Named Entity Recognition to the Historical Domain: How to Take the Step? (2021)

Abstract: Named entity recognition is of high interest to digital humanities, in particular when mining historical documents. Although the task is mature in the field of NLP, results of contemporary models are not satisfactory on challenging documents corresponding to out-of-domain genres, noisy OCR output, or old-variants of the target language. In this paper we study how model transfer methods, in the context of the aforementioned challenges, can improve historical named entity recognition according to how much effort is allocated to describing the target data, manually annotating small amounts of texts, or matching pre-training resources. In particular, we explore the situation where the class labels, as well as the quality of the documents to be processed, are different in the source and target domains. We perform extensive experiments with the transformer architecture on the LitBank and HIPE historical datasets, with different annotation schemes and character-level noise. They show that annotating 250 sentences can recover 93% of the full-data performance when models are pre-trained, that the choice of self-supervised and target-task pre-training data is crucial in the zero-shot setting, and that OCR errors can be handled by simulating noise on pre-training data and resorting to recent character-aware transformers.

## introduction due to the massive effort to digitize and transcribe historical documents, the field of digital humanities is facing the challenges of digesting and analyzing large quantities of texts. with the continuous advancements of natural language processing (nlp), there is a growing interest in applying tools such as named entity recognition (ner) on historical documents. indeed, identifying people, places, and other historical entities is crucial to understand the historical context, and having the ability to do so automatically is a major step forward. it facilitates the exploration of massive corpora by identifying, counting and extracting textual clues, among others to enrich a database, which can help to systematically explore the information reported in these documents. but the variety present in historical texts, compared to modern ones, makes the evaluation and application of nlp techniques quite difficult. in particular, apart from the fact that these documents relate to different domains, the evolution of language, as well as the noise due to optical character recognition (ocr) errors, preclude using off-the-shelf systems. ner success, like for other nlp tasks, is highly dependent on the corpus on which the system has been trained, and most of the available named entity (ne) corpora use contemporary texts with contemporary concerns. for example, off-the-shelf ner systems such as spacy @xcite or @xcite do not yield acceptable results on historical texts as evidenced in the goal of this study is to evaluate the effort required to obtain relevant ner results on historical documents. this effort can relate to annotation in the target domain, transfer of contemporary models, pre-training on matching resources, or adaptation to ocr errors. compared to other domain transfer approaches @xcite @xcite , where the evaluation is carried out on specific contemporary domains, this work considers ner from a historian's point of view: we wish to process historical texts and understand why some approaches do not yield usable results. in that context, would off-the-shelf systems be appropriate? if not, can we reduce the amount of annotation needed to obtain reasonable results by using existing resources? and finally, if so, is this approach robust to the characteristic difficulties of historical documents? our contribution is to tentatively answer the following questions: (1) what annotation effort in the target historical domain? (2) is it worth adapting initial pre-trained word representations? (3) what is the impact of ocr errors on transfer performance? we perform experiments on the lit-bank @xcite and hipe @xcite annotated historical ner datasets using prototypical ner systems built on the previous and current generation of models, trained on contemporary annotations from ace 2005 @xcite and various amounts of target data. given the progress on this task and its need for applications, many off-the-shelf models pre-trained on modern data are made available. spacy @xcite , flair @xcite , stanza @xcite , allennlp @xcite offer models trained on ontonotes 5.0, stanford corenlp @xcite provides a model trained on a mixture of conll, muc-6, muc-7 and ace. in this paper we mainly focus on ner in english but this task is also a subject of research in other languages. for instance, @xcite proposed an adversarial transfer learning strategy to make full use of the boundary information shared by tasks and prevent the task-specific functions of chinese word segmentation. @xcite process 41 languages using truth inference to model the transfer annotation bias from diverse source-languages models. @xcite created a 0-shot ner systems by aligning monolingual embeddings from english to spanish, german, and dutch, and then translated the english conll dataset into these languages, and built a self-attentive bi-lstm-crf model using the translated languages. cross domain ner cross-domain approaches have been developed to enhance the generalization of ner models to a given target domain. most existing approaches are in a supervised setting where both source and target domains have labeled data, the goal being to improve performance over only using instances from the target data. baselines jointly train models on source and target data with shared parameters, add adaptation layers on top of source models for capturing target domain specifics, or design label-aware feature representations for ner adaptation @xcite . more specific methods use multi-task approaches which have been shown to be effective for this cross-domain task, to reduce the gap between the two domains. for example, @xcite propose to use extrinsic data in both the source and target domains to train language models for domain adaptation. @xcite propose a parameter transfer learning between feature representations from bi-lstm and two conditional random fields. @xcite propose a multitask learning objective that learns domain labels as an auxiliary task and @xcite propose a dual adversarial transfer network which aims at addressing representation difference and resource data imbalance problems. methods such as transfer learning also show that knowledge sharing is effective for cross-ner. for instance, @xcite utilize transfer learning by initializing a target model with parameters learned from source-domain ner, and rely on labeled target domain data to finetune the model. @xcite propose an adversarial transfer learning framework for chinese ner task, which can exploit task-shared word boundaries features and ensure proper information usage from the word segmentation task. lin and lu (2018) perform adaptation across two domains using adaptation layers augmented on top of the existing neural model. @xcite propose a fine-grained knowledge fusion model to balance the learning from the target data and learning from the source model. ## datasets in this study, we deal with target domain annotated datasets (historical texts), and source domain annotated datasets (contemporary texts, typically news). two target datasets are used, each with two different subdomains, specific difficulties towards this task and a non-comparable annotation guideline. litbank @xcite is an annotated dataset of 100 english-language literary public-domain texts from project gutenberg, annotated with ace entity categories except for the weapon category (person, location, geopolitical entity (gpe), facility, organization (org), and vehicle). in contrast to existing datasets built primarily on news (focused on gpes and orgs), literary texts offer strikingly different distributions of entity categories, with much stronger emphasis on people and description of settings. all texts were one source dataset is used. ace 2005 @xcite multilingual training corpus was developed by the linguistic data consortium (ldc) and contains approximately 1,800 documents of mixed genre texts in english, arabic, and chinese annotated for entities, relations, and events. the genres include newswire, broadcast news, broadcast conversation, blog, discussion forums, and conversational telephone speech. the dataset is annotated with 7 entity types: person, location, gpe, facility, organization, vehicle, and weapon. we followed the same pre-processing of the data as those presented by @xcite . in general, all corpora share annotations in persons, locations and organizations, these three types also represent the majority of annotations. ace and litbank share 100% of their entity types and ace and hipe share 3 out of 6 entity types, which represents 92.5% of entity instances of shared type in the test set. a shared entity type is an entity type ## experimental settings we used three different systems which represent the typical kind of systems that could be implemented in an industry-provided solution for historical ner given current technology. bert @xcite , with an extra layer to predict ner categories. the pre-trained contextual embeddings are finetuned on the source/target dataset using their proposed approach. it's a de facto baseline for nlp systems, and is implemented with the transformers library @xcite . we also used bert models (bert-hist) @xcite finetuned on 47,685 books (5.1b tokens) in english from the year 1760 to 1900 from the microsoft british library corpus. this model, compared to bert-base learned from wikipedia and bookcorpus, allows us to question the impact of diachronic language changes on these embeddings applied to the historical domain. charbert @xcite which accounts for characters in addition to bpe tokens. the model is pre-trained with a noisy lm objective for obtaining robust character-level representations. we expect such model to perform well on noisy ocr, and rely on the available implementation foot_1 . lstm-crf @xcite initialized with fasttext @xcite noncontextual embeddings, an architecture that has shown its robustness for ner and is standard in many available systems. this allows to compare its performance to contextual ones. the flair library @xcite is used for this model. all results presented in the experiments section are averages over 10 random initializations. since the objective is not to maximize performance on a particular dataset or on a particular architecture, the same hyperparameters for all experiments of a given system are used. the learning rate is initialized to 0.1 for the lstm-crf and to 3e-5 for the transformers. for the three architectures, the size of the hidden layers are set to 512 and the training is performed on 3 epochs on the source data (10 epochs for the lstm-crf) and finetuned on 10 epochs on the target data. the rest of the parameters are the defaults proposed by the libraries for the different models. ner performance is computed with f1score (f1). because not all corpora contain nested entities, the embedded entity mention is removed and the smaller mention is kept in the case where the datasets proposed this type of entity. moreover, as previously mentioned, the hipe dataset do not share all entity types. the evaluation of a system is only performed on the entity types of the test set, and all the predicted entities that are not part of the test set tagset are removed prior to evaluation. ## discussion we identify replicable steps for the application of ner on historical documents. first, select contemporary resources, pre-trained models or annotated datasets, using a guideline close to the target needs. then annotate a few sentences on the target data to adapt the learned models on the source data. 250 sentences can recover 93% of the full-data performance and 1000 sentences 98%. in a second step, the choice of the model will depend on the quality of the target documents to process. if the target is noisy, charbert, even compared to a bert learned on historical noisy data, is more robust to process this type of documents. moreover, if the quality information of the target documents is available, applying noise following the same distribution to the source documents will allow the system to be more robust on the target data. finally, dapt can be applied if a large amount of unannotated target data is available (> 2m sentences) and if some target data is annotated. this approach does not seem to work in the case where no annotated target data is available. in the case of 0-shot, using word embeddings adapted to the source data will bring better performance on the target data. ## conclusion in this work, we investigate the potential transfer of contemporary named entity recognition models to the historical domain. experiments show that finetuning contemporary pre-trained transformers allows reducing considerably the annotation effort and can be further reduced by making an informed choice of the data sources for transfer. adapting pre-trained word representations prior to learning the task (dapt) allows a low-cost adaptation to the target domain and improves performance in the full settings depending on the dataset. processing noisy data is still challenging but the choice of an architecture relying on pre-trained character representations, and the simulation of target noise on the source domain allows recovering acceptable performance compared to a bert baseline. to further this study, we would like to systematically look into recent transformer architectures and pre-train them on large corpora of historical texts instead of crawled web data. it would be an opportunity to infuse the models with knowledge of the target temporal span. in addition, we would like to study how levels of ner performance impact historian's findings, and whether current technology is acceptable for reliably mining large quantities of historical documents.