Transfer Learning for C zech Historical Named Entity Recognition (2021)

Abstract: Nowadays, named entity recognition (NER) achieved excellent results on the standard corpora. However, big issues are emerging with a need for an application in a specific domain, because it requires a suitable annotated corpus with adapted NE tag-set. This is particularly evident in the historical document processing field. The main goal of this paper consists of proposing and evaluation of several transfer learning methods to increase the score of the Czech historical NER. We study several information sources, and we use two neural nets for NE modeling and recognition. We employ two corpora for evaluation of our transfer learning methods, namely Czech named entity corpus and Czech historical named entity corpus. We show that BERT representation with fine-tuning and only the simple classifier trained on the union of corpora achieves excellent results.

## introduction recently, named entity recognition (ner) achieved outstanding results on standard ner corpora. particularly on english conll-2003 corpus 90% fmeasure has been overcome, which is sufficient for several real applications. however, big issues are emerging with a need for an application in the specific domain which requires an appropriately annotated corpus including adapted ne tag-set. this issue is particularly evident in the case of historical documents in czech language on which we focus on the project "modern access to historical sources" 1 . manual annotation of this corpus is very expensive and time consuming. moreover, the presence of a linguist is necessary. we will use the information about 1 http://www.portafontium.eu/ named entities as additional metadata for information retrieval and knowledge extraction. transfer learning targets at reusing information obtained from one corpus to improve the results of a model learned on an analogous task with few resources. to overcome this issue, we propose and evaluate several transfer learning approaches to improve the results of the czech historical ner when the annotated resources are limited. the following information sources are considered and studied for this task: we employ two neural based models, namely recurrent bidirectional long short-term memory (bilstm) @xcite and bidirectional encoder representations from transformers (bert) model @xcite with a simple perceptron for ner modelling and recognition. we use two corpora for evaluation of our experiments. note that, to the best of our knowledge, this is the first attempt to employ transfer learning in the field of czech historical ner. @xcite presented reproduction paper focused on transfer learning for entity recognition. they compared seven new corpus pairs results and other researches that were published previously. they showed that if there is a small labelled target dataset, the simpler approaches work better in compare to neural transfer approaches that work better for larger labelled target data set. they reached an f1 score of 71.81% for the period 1814-1817 and 70.35% for the period 1685-1691. ## corpora we experimented with two corpora: czech named entity corpus (cnec) and czech historical named entity corpus (chnec). cnec corpus contains almost 9,000 sentences and more than 35,000 occurrences of the czech named entities. the corpus uses two-level nes annotation scheme and the firstlevel contains 10 main ne types and the secondlevel is composed of 62 ne subtypes. to be able to map nes from cnec to chnec, we use only five ne types from the first annotation level which are same for both corpora. chnec 7 contains 73,647 tokens and 4,017 named entity occurrences. the corpus was created from czech historical newspaper posel od čerchova from second half of 19 th century and distinguishes five ne types: personal names, institutions, geographical names, time expressions and artifact names/objects. the corpus is encoded in iob format @xcite , where b represents one-token entity or the beginning of multi-token named-entity, i inside tokens of multitoken named-entity and o stands for all tokens that are not a named-entities. ## evaluation and results qualitative analysis in section 5.1 is based on the observed linguistics phenomena in both development and test data sets. the first part of the second part of the table shows the experiments using bilstm model with different approaches for training. these results show that it is possible to obtain f-measure 45% using only the different corpus (any additional annotation is not required). moreover, it has been demonstrated that the transfer from cnec into chnec does not bring any positive impact on the final ner. the last part of another observation is that the values of the precision and recall are balanced, however in the previous series of experiments, these values differ significantly. the previous experiment does not consider the individual ne types. however, this information could be very interesting for further improvement of the training strategy and the model itself. we assume, that the size of the training data and the ## conclusions in this paper, we proposed and evaluated several transfer learning approaches in order to improve the results of the czech historical ner. we considered and studied the following information sources: pre-trained fasttext and word2vec word representations, pre-trained slavic bert contextual text representation and another ne corpus from different domain. we used two popular models, namely recurrent bilstm and bert with a simple perceptron for ner modelling and recognition. we have shown that fasttext representation gives significantly better results than word2vec model. it has been also demonstrated that the transfer from cnec into chnec with bilstm model does not improve the final ner score. we have further presented that bert representation with fine-tuning and only the simple classifier trained on the union of corpora brings the best results (f-measure 82%). based on the analysis of the errors, we can also conclude that the combination of the different models / approaches would not bring any further improvement.