Lemmatization of Historical Old Literary F innish Texts in Modern Orthography (2021)

Abstract: Texts written in Old Literary Finnish represent the first literary work ever written in Finnish starting from the 16th century. There have been several projects in Finland that have digitized old publications and made them available for research use. However, using modern NLP methods in such data poses great challenges. In this paper we propose an approach for simultaneously normalizing and lemmatizing Old Literary Finnish into modern spelling. Our best model reaches to 96.3% accuracy in texts written by Agricola and 87.7% accuracy in other contemporary out-of-domain text. Our method has been made freely available on Zenodo and Github.

## introduction finnish language has a long literary history starting from the 16th history. a large portion of the books printed in finnish is currenlty openly available for research, and especially the bibliographic record they form has already been studied. @xcite investigated the document size and language in the bibliographic metadata records, and @xcite took into account a number of other metadata fields including the titles, and recognize the lack of full-text documents as a downside of investigation that is possible. although the whole body of books printed in finland is not available as full-text, there are numerous smaller corpora that can already be used in various ways. both the national library of finland and institute for the languages of finland have produced a large number of digitized material from the era of the old literary finnish. former has made a large amount of scanned and text recognized publications available (national library of @xcite , and latter has created large plain text corpora from selected works (institute for the languages of @xcite . the agricola corpus used in this study has been morpho-syntactically annotated, and is to our knowledge the only annotated resource in the old literary finnish (institute for the languages of finland & university of @xcite . another very central resources is the dictionary of old literary finnish. 1 thus far the old literary finnish has largely eluded any attempts of nlp research as the historical written form cannot be processed easily with currently available nlp tools for finnish, as they are designed for modern finnish. we present our approach for normalizing and lemmatizing old literary finnish automatically to modern finnish orthography. as the resources for processing historical finnish text are scarce, we have released the models presented in this paper on zenodo 2 and through an easy-to-use python library 3 . the use of existing nlp tools targeted for modern finnish on historical materials can be made possible through normalization. previous work conducted on english data indicates that normalization is a viable way of improving the accuracy of nlp methods such as pos tagging @xcite . another direction of digital humanities study has benefited from normalization of historical data in studying the use of neologisms in old letters @xcite . in their approach, without normalization, they would have been able to cover only a small subset of the corpus. the same corpus has also been studied without nlp tools @xcite . the history of printed written finnish starts from the 16th century with the works of mikael agricola. his primer and religious works were followed by a continuous increase in the amount of the written finnish materials. the language form used by agricola is known as old literary finnish (vanha kirjasuomi). the majority of the early finnish publications included religious materials, although the text types started to diversify already in the 18th century. the majority of the oldest texts are translations. the period of old literary finnish is often estimated to have lasted until 1810, after which the written finnish started to transform into early modern finnish. eventual changes in printing laws and printing technology, which expanded the amount and variation of the printed materials, and the creation of regular finnish newspapers, contributed to the stabilization of the written standard. linguistically one of the exceptional features of old literary finnish is the variation it displays. the orthography was not yet entirely established, and there was extensive spelling variation. the age of these materials adds also a historical dimension, as there are linguistic features that are not present in the modern finnish, or exist currently only in the dialects. our main contributions in this work are : ## related work historical text normalization has been studied in the past for other languages than finnish. a recent literature review @xcite finds that there are five categories in which modern normalization approaches can be divided : substitution lists like vard @xcite and norma @xcite , rule-based methods @xcite , edit distance based approaches @xcite , statistical methods and most recently neural methods @xcite . statistical machine translation (smt) based methods have been the most successful ones in the past in terms of statistical methods. the key idea behind these methods is to approach the task as a character-level machine translation problem, where a word is translated character by character to its normalized form. these methods have been applied to historical text @xcite and dialect normalization @xcite . in the recent years, normalization has been approached as a character-level neural machine translation (nmt) problem similarly to the previous smt approaches. the additional advantage is that a neural model does not need a separate language model like smt does. @xcite have shown that a bi-directional long short-term memory (bi-lstm) can be used to normalize historical german texts. the paper presents a so-called multi-task learning setting where auxiliary data is added to improve the performance of the model. multi-task learning setting generally improved the results. their system outperformed the existing conditional random fields and norma based approaches in terms of accuracy. text written in uyghur language has been normalized with an lstm and a noisy channel model (ncm) @xcite . they use a relatively small set of gold annotated data for training (around 200 hand normalized social media sentences). they augment this data by synthetically generating non-normalized text by introducing random changes in normalized text. in the same fashion, another research has used an lstm model to normalize code-mixed data @xcite . recently @xcite have shown that bi-directional recurrent neural networks (brnn) outperform regular unidirectional recurrent neural networks (rnn) when normalizing historical english data. interestingly, additional layers and different attention models do not improve the results. additional data such as time period, social metadata or pronunciation information in ipa characters makes the results worse. according to them, post-processing can boost the accuracy of a character level nmt model more than changing the network structure. a simple dictionary filtering method improved the results. omorfi @xcite is a popular rule-based tool used to do morphological analysis and lemmatization of modern finnish. while omorfi itself is not relevant for our work, there is a github fork of the project known as historical omorfi 4 . the fork introduces several improvements to better cater for historical finnish text. currently, this tool is the only tool available for lemmatizing historical finnish. thereby we compare the results of our model to this tool. ## dataset of historical finnish in order to use machine learning methods, data is needed to train a model. the data needs to have text written in old literary finnish and its normalized lemmas. the lemmas should be aligned on a word level with the historical data in order to train the normalization more accurately. fortunately, such a dataset exists. the corpus we use is the morpho-syntactic database of mikael agricola's works (institute for the languages of finland & university of turku, 2020) that contains 522,237 tokens and 38,222 sentences. the corpus includes all nine finnish books translated by mikael agricola. the corpus is openly available in the language bank of finland. 5 in our testing we also use the dictionary of old literary finnish 6 (institute for the languages of @xcite , from which a small number of sentences has been sampled and manually normalized and lemmatized. both resources are available under creative commons licenses. since the the morpho-syntactic database of mikael agricola's works is licensed under cc by-nd 4.0 (clarin pub), we do not redistribute the training material ourselves, but it can be accessed in the language bank of finland's concordance service 7 . naturally, since this data is so old, it is already in public domain, and many of the original works are entirely openly accessible. for example, the agricola's prayer book is available as high quality scans by the national library of finland 8 . although there has not been prior use of this dataset in the computational linguistics, the corpus has been used in the linguistic studies. to illustrate this with few recent examples, toropainen (2018) investigated the compounds in this variety, and @xcite studied nouns that contain an initial adjective. salmi (2020) discussed recently the german influence in the agricola's language. also annotating the corpus into it's current stage has been a long undertaking, and inaba (2015) investigated the use of two finnish cases with a prototype of the current database. it is beyond doubt that the materials of old literary finnish still have much to contribute to the linguistic research. we believe that by creating new tools for natural language processing of these and similar materials we can further expand toward these goals. the research concerning agricola's language and old literary finnish in general is naturally much wider and has a long research history, especially in finland, and we primarily wanted to illustrate in this section some of the previous studies where the same database was used. ## neural normalization in this section, we describe our artificial neural network model for normalizing and lemmatizing historical finnish into standard finnish. we begin by explaining how the dataset is used and how it is preprocessed. thereafter, we elucidate the architecture of the model along with the technical details of its hyperparameters. the corpus contains nine distinct works. in order to test the model's accuracy realistically, we selected seven books into the training data, and left the remaining two into the test set. thereby the training data was 393,779 tokens and the test set 128,294 tokens. the books in the test set were specifically 'messu eli herran echtolinen' from 1549 and 'rucouskiria' from 1544. additionally 15% of the training data was used in the validation set. we considered it important not to select the test set from the entire corpus, as this would not give a clear picture about how much the model generalizes into new use cases, which would be the other books written in the same language variety. with the current sparsity of manually annotated data, the agricola works in the currently used corpus, but which we kept unseen for the model, were the best option we had. we also extended the testing into other examples of old literary finnish, but in a smaller scale. the data has the original written form of each sentence and a token level normalized lemma for each word. we use this parallel data to train our models. we model the problem as a character level nmt problem. in practice, we split words into characters separated by white space and mark actual spaces between words with an under score (_). this allows to pass word boundary information to the model while the characters themselves are separated by spaces. we train the model to predict from the old literary finnish word forms to the lemmas. as previous research @xcite has found that using chunks of words instead of full sentences at a time improves the results, we train different models with different chunk sizes. this allows direct comparison of different chunk sizes. this way we train the models to predict one word at a time, two words at a time all the way to five words at a time. an example of the data can be seen in ## results and evaluation our initial evaluation results were very good as seen in because there is no other dataset freely available that would both be written in old literary finnish and lemmatized to modern orthography, we take randomly 50 sentences from the example sentences of the dictionary of old literary finnish 9 that is available online. altogether these sentences have 562 words (excluding punctuation). we lemmatize these sentences with the model that has been trained with chunks of 3 as it worked the best out of the models and verify the lemmatization by hand. the results of this experiment are seen in chunk 3 omorfi accuracy 87.7% 47.9% the results drop in this evaluation, but it is only to be expected given that out-of-domain performance is typically lower for neural models. nevertheless, we see clearly that the model does well in out-ofdomain data and beats the current state of the art. the fact that historical omorfi gets better results in this dataset is a good indication that the text is very different from what is in the agricola dataset. however, it clearly is not entirely distinct when we consider how well the model still performed. if we look at the results more closely, analyzing the errors, we can see that the model usually does not predict non-words but rather words that are a part of the finnish vocabulary. this indicates that the model has learned a good target representation. there are several errors in which the model has normalized the historical word correctly, but it has not lemmatized it, for example runsast was normalized to runsaasti although the lemma would be runsas 'plenty'. another example is ulosteon that was already written as in the modern orthography was normalized unchanged to ulosteon, while the lemma is ulosteko. this example also illustrates how there is variation and historical change in spelling, as we find in the agricola corpus comparable words spelled with the initial v, whereas in the later materials we find the variant above that is spelled closely the current standard. analysing how this relates to the changes in characters used in different centuries is beyond the scope of our study, but it illustrates well the kind of variation we can find in the historical texts and their digitized versions. ## conclusions and future work our results have a clear indication, both with in-domain and out-of-domain test data, of working successfully in lemmatizing old literary finnish in the modern orthography. the models have been released on zenodo and in a python library 10 . by sharing the models we are making nlp research on historical finnish data more widely accessible for the research community, as the currently available historical omorfi does not work well for texts that are this old. our study also creates a benchmark into which the further work can easily be compared. having lower accuracy in another dataset is a reminder of the importance of evaluating normalization models on data that comes from a different distribution. this is something seldom seen in the previous work on historical spelling normalization. despite this, the accuracy has remained relatively high and we have identified several possibly problematic phenomena in the test data that are more prone to errors. this error analysis helps in understanding the biases that using our model might introduce in historical data when it is used to lemmatize a corpus completely new to the model. further research is needed to evaluate how the error rate varies when the distance grows to the materials of agricola. it is beyond doubt that more diverse training material is needed to successfully process the entire corpus of old literary finnish, but our study certainly has improved the position to initiate and continue such work. the work we presented here also makes it possible for the current research on analyzing historical finnish newspapers, such as @xcite , to standardize post-ocr historical finnish, which in turn permits employing state-of-the-art finnish nlp methods and tools on such data (e.g. sentiment and semantic analysis @xcite ). when we work with the currently available resources, we must also remain aware that the digital versions have been edited in various ways, and do not contain necessarily all features of the original printed text @xcite . now when we increasingly have access also to the original prints as high quality scans, it is important to think how these different resources can be connected to one another. this will need a combination of both text recognition and nlp tools. in the future, we are interested in conducting work on semantic change on historical data. this should be greatly facilitated by the fact that we can now considerably reliably lemmatize historical text. this means that training word embeddings models will become more accurate as the model is trained on lemmas instead of inflectional forms.