Alleviating Digitization Errors in Named Entity Recognition for Historical Documents (2020)

Abstract: This paper tackles the task of named entity recognition (NER) applied to digitized historical texts obtained from processing digital images of newspapers using optical character recognition (OCR) techniques. We argue that the main challenge for this task is that the OCR process leads to misspellings and linguistic errors in the output text. Moreover, historical variations can be present in aged documents, which can impact the performance of the NER process. We conduct a comparative evaluation on two historical datasets in German and French against previous state-of-the-art models, and we propose a model based on a hierarchical stack of Transformers to approach the NER task for historical data. Our findings show that the proposed model clearly improves the results on both historical datasets, and does not degrade the results for modern datasets.

## introduction with the emergence of large scale archives of digitized contents, the need for efficient preservation and accessibility of historical documents through appropriate technologies increased exponentially. at the same time, there is a growing interest in extracting relevant information from historical sources. in this paper, we address the named entity recognition (ner) task which aims at identifying real-world entities, such as names of people, organizations, and locations within historical documents. since most of the state-of-the-art research focuses on ner for modern available datasets, the performance of the ner systems grew at a fast pace, enabled by the representational capacity of neural networks and off-the-shelf pre-trained word embeddings @xcite @xcite . more recently, ner models based on contextual word and subword representations provided by elmo @xcite , flair @xcite , or bert @xcite , achieved impressive improvements. the transformer-based @xcite architectures for ner became popular since the release of the bert (bidirectional encoder representations from transformers) model. however, while most ner systems have been developed to generally address contemporary data, ner systems for processing historical documents are less common. to extract entities from historical documents, ner tools face additional challenges. as the majority of these documents are hardcover, they are scanned and processed by an ocr to transcribe the text. however, an ocr tool can occasionally misrecognize letters and improperly identify its textual content. this can be due to the level of degradation of the actual document being scanned, to the digitization artifacts and also to the quality of the ocr tool. this leads to digitization errors in the transcribed text, such as misspelled locations or person names. languages evolve through time and certain words can have a different meaning depending on the period of time analyzed @xcite . the spelling of words can also change due to new orthographic conventions or cultural tendencies @xcite . this high level of spelling differences can be incompatible with modern orthography and the produced noise can severely affect modern nlp systems @xcite . to address these challenges of ner on historical documents, we propose a robust ner model based on a stack of transformers that includes fine-tuned bert encoders. we study the impact of such a model, and we conclude that this type of model is suited for the extraction of entities from historical documents. the remainder of this paper is organized as follows. in section 2, we present and discuss a selection of works concerning ner in modern and historical documents. then, in section 3, the datasets explored in this work are presented. the proposed model is detailed in section 4. the experiments are described in section 5. we present and discuss the obtained results in section 6. finally, section 7 concludes this paper and hints at future work. ner for historical documents historical documents pose multiple challenges that either depend on the quality of digitization or the historical variations of a language. studies on how the ner models can be impacted by the digitization process @xcite @xcite have clearly shown that the performance scores of a ner model can significantly decrease when applied on historical documents. the increased interest in contributing to historical language resources is driven forward by the creation of new gold standards for historical document processing. for example, @xcite created and annotated a corpus using scanned czech historical newspapers, and @xcite proposed a german gold standard for ner in historical biodiversity literature. a recent competition organized by the identifying historical people, places, and other entities (hipe) lab at clef 2020 foot_0 , not only that it created a gold standard for german and french historical texts, but also encouraged researchers to participate in two sub-tasks, named entity recognition and classification and entity linking. considering the high level of spelling differences between modern and historical documents, variance (inconsistency), and uncertainty (digitization errors) found in historical documents, the recent methods assess these shortcomings differently. @xcite presented a crf-based model with handcrafted features for latin historical texts and motivated the choice of part-of-speech (pos) tagger by the fact that this nlp tool leverages the highly informative morphological complexity of latin. the bilstm-based model proposed by @xcite applied a characterbased cnn to encode the different spellings of words. similar to the latter approach, we also consider that the ner model itself can help in alleviating the historical documents issues, without the use of language-specific engineered features. differently, we introduce the ner for historical documents to the language model methods based on the transformer architecture @xcite and bert @xcite methods, that, to our knowledge, have not been approached in previous research, with regard to processing historical documents. with new needs and resources in the context of historical ner processing, we evaluate our proposed model on the dataset proposed by the hipe competition, and we also propose a new gold standard for german and french, to assess our assumptions. ## datasets we conduct experiments on two datasets that comprise digitized historical newspapers, hipe and newseye datasets in french and german. additionally, we study how the proposed methods behave in the case of contemporary data, by experimenting on the english conll 2003 dataset @xcite . the hipe dataset was created by the clef 2020 evaluation lab hipe challenge @xcite . it is composed of articles from several swiss, luxembourgish, and american historical newspapers from 1790 to 2010 @xcite . more concisely, the german articles were collected from 1790 to 1940, and the french articles, from 1790 to 2010. the corpus was manually annotated by natives following the annotation guidelines derived from the quaero annotation guide foot_1 . we also present the newseye dataset, composed of historical newspapers in @xcite and german . the documents were collected through the national libraries of france foot_2 (bnf) and austria foot_3 (onb), respectively. this dataset was annotated following guidelines derived from the quaero annotation guide foot_4 . the annotation process was made by native speakers for each language using the transkribus tool foot_5 . in order to compute the inter-annotator agreement (iaa), we used the kappa coefficient introduced by @xcite . several pages from each corpus (german and french) have been annotated twice by two groups of annotators. satisfactory iaa scores were reached for the two corpora (0.90 for french and 0.91 for german). the newseye corpus is split into 80% for training and 20% for both validation and testing. the conll 2003 dataset consists of newswire from the reuters rcv1 corpus and it includes standard train, development, and test sets. ## model we based our ner model on the pre-trained model bert proposed by @xcite . although original recommendations suggest that unsupervised pre-training of bert encoders are expected to be sufficiently powerful on modern datasets, we consider that adding extra transformer layers could contribute to the alleviation of word errors or misspellings. first, we use a pre-trained bert model, and second, we stack n transformer blocks on top, finalized with a crf prediction layer. we refer to this model as bert+n×transf where n is a hyper- more specifically, the built-in tokenizer of bert first performs simple white-space tokenization, then applies a byte pair encoding (bpe) based tokenization, wordpiece @xcite . for example, word can be split into character n-grams (e.g. @xmath0, '##pa', '##ti', '##bility'), where ## is a special symbol for representing the presence of a sub-word that was recognized. between the types of ocr errors that can be encountered in historical documents, the character insertion modification has the minimum influence @xcite , because the tokenization at the sub-word level of bert would not change much in some cases, such as 'practically' → 'practicaally'. meanwhile, the substitution and deletion errors can hurt the performance of the tokenizer the most due to the generation of uncommon samples, such as 'professionalism' → 'pr9fessi9nalism' that is tokenized as 'pr', '##9', '##fes', '##si', '##9', '##nal', '##sm'. bert has been demonstrated to have a sensitivity to its sub-word segmentation when it comes to such words, as the meaning of the sub-words can diminish the initial meaning of the correctly spelled word @xcite . thus, these new noisy tokens could influence the performance of bert-based models foot_7 . on top of bert, we add a stack of transformer blocks (encoders). a transformer block (encoder), as proposed in @xcite , is a deep learning architecture based on multi-head attention mechanisms with sinusoidal position embeddings. it is composed of a stack of identical layers. each layer has two sub-layers. the first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. a residual connection is around each of the two sub-layers, followed by layer normalization. all sub-layers in the model, as well as the embedding layers, produce outputs of dimension 512. in our implementation, we used learned absolute positional embeddings @xcite instead, as it is a common practice foot_8 . @xcite found that the two versions produced nearly identical results. we assume that the additional transformer layers can alleviate the sensitivity of the built-in tokenizer of bert towards oov, ocr errors, or misspellings, and contribute to the learning or finding the proper informative words around entities. ## results in this section, we provide experimental results of the baseline model and the proposed method. in order to assess the ability of both models with regard to the presence of errors provided by an ocr, we present several experiments: from the results in the moreover, they generally manage to maintain a balance between recall and precision, while the baseline models vary, depending on the language. we also notice that, while in general, both models obtain a more or less precision-recall balance, there are two cases where there is a large imbalance, more specifically in the newseye german dataset. comparing with the baseline models, the bert+n×transf only achieves a 20 percentage points difference between precision and recall, while the baseline suffers from 40 points difference. in the context of transfer learning applied for the baseline models, two performance results, for newseye in german, and for hipe in french are higher due to the fine-tuning on these datasets, while the others are not degraded by the pretraining on larger contemporary datasets. this observation confirms the previous studies done on this type of model regarding their robustness to misspellings @xcite . we also notice that for german both datasets, the results for transfer learning from contemporary ger-man datasets are statistically significant (< 0.01%), while contemporary datasets the performance difference for both french datasets was minimal (@xmath1.5 for french newseye @xmath2.9 for french hipe). in the context of modern data, in the ## conclusions and future work we presented a deep learning architecture for ner based on stacked transformer layers that includes a fine-tuned bert encoder and several transformer blocks. results on two historical datasets in french and german showed the fitness of the proposed model to process noisy digitized text corpora in distinct languages. at the same time, the approach did not degrade the performance over modern data. thus, this type of model appears to be adapted for the ner of historical document collections. while the improvements brought by the proposed ner model are clear, our analysis of the results highlighted several factors that could influence the results. further analysis remains to be done. thus, hereafter, we will investigate detailed variations of our architecture. in addition, we intend to explore data augmentation techniques, simulating digitized data by adding noise to digitallyborn documents. this could be a solution to increase the size and expand the diversity of training datasets for performing nlp tasks over historical documents.