Linguistic change and historical periodization of Old Literary F innish (2021)

Abstract: In this study, we have normalized and lemmatized an Old Literary Finnish corpus using a lemmatization model trained on texts from Agricola. We analyse the error types that occur and appear in different decades, and use word error rate (WER) and different error types as a proxy for measuring linguistic innovation and change. We show that the proposed approach works, and the errors are connected to accumulating changes and innovations, which also results in a continuous decrease in the accuracy of the model. The described error types also guide further work in improving these models, and document the currently observed issues. We also have trained word embeddings for four centuries of lemmatized Old Literary Finnish, which are available on Zenodo.

## intoduction in this study, we investigate linguistic drift and historical periodization of old literary finnish. we use a historical finnish lemmatizer model trained on the works of mikael agricola, and apply the model to the remaining currently available corpus of old literary finnish (institute for the languages of @xcite . this allows us to examine both the differences in the model's performance and how the lexicon of old literary finnish has changed and evolved over time. we hypothesize that the contexts where the model's quality changes significantly correlate, in fact, with changes in the actual form of the literary language. these can be innovations in the orthography, or other kinds of linguistic changes that are known to have happened during the period finnish has been a written language. careful error detection should also reveal something about the nature of these changes. as long as the model's quality remains above a specific threshold, we should also be able to monitor the use of specific lexemes over time. we trained the word embeddings for this purpose. the corpus size being limited, and divided to time period of 1543-1809, we concluded that more data is needed to follow the actual semantic changes. ## related work natural language processing for old literary finnish is still in a very early stage, while extensive work already exists for historical variants of other languages @xcite @xcite . most work has been done with historical newspapers, which represent only later periods of this language variety, starting from 1771. many studies are connected to improving ocr accuracy, which remains as an important task for old printed materials. recognizing named entities is another line of research that has been developed relatively far, especially by @xcite , @xcite and @xcite . this connects to other work in ner of other finnish varieties @xcite . also evaluation and post processing approaches are closely connected to our study. @xcite and @xcite used a morphological analyser adapted for historical finnish to evaluate ocr accuracy in these newspapers. later on, ocr accuracy has been improved through unsupervised post-correction in finnish newspapers @xcite . koskenniemi and kuutti (2017) studied alignment and analysis of old literary finnish, using a helsinki finite-state transducer @xcite . lexical change through neologisms has been studied in historical data by comparing word occurrences in a historical corpus to earliest attestations recorded in dictionaries @xcite . ## data we use the old literary finnish corpus (institute for the languages of @xcite . this is the only proofread corpus of old literary finnish currently available, it aims to be representative, and is created especially for the purposes of lexicography. the current corpus is 4.13 million tokens in size. the distribution of tokens by year is shown in figure 1 . to contextualize the distribution, the bible translation from year 1642 contains over a million tokens. the corpus has 1.5 million tokens where the year is not defined in the metadata, and thereby were not included in our study. in order to better understand the relationship this data has to the entire old literary finnish corpus, we can compare it to various adjacent sources. the first logical point of comparison is the national metadata catalogs, which should contain relatively complete information about all books that have ever been printed. this data was already analysed by @xcite , and their figures are certainly worth comparing in this context, too. as text sources, however, these materials are only useful for us if they have been digitized and can be accessed. to understand this context, we examined the number of digitized pages from the same time period in the collections of the national library of finland foot_0 . the distribution of digitized finnish pages is shown in figure 2 . this shows that our current sample is still relatively small, and many different sample constellations could be imagined. comparing to @xcite , for example, it seems that the dip in digitized pages we see in figure 2 in the first half of the 18th century does not seem to correlate with a reduced printing activity in this time period. similarly the old literary finnish corpus has four larger peaks, representing, presumably, the goal to include all four centuries of this language variety to a comparable degree. besides the proofread portion of the corpus, the materials of agricola have been published as a morphosyntactically annotated version (institute for the languages of finland and university of turku, 2020). each resource type we have discussed above is narrower than the one before, as specialized annotation, proofreading and digitization are all resource demanding activities. our work explores what we can do with the current data, existing annotations, and how we can build nlp solutions around these materials to extend and enrich the available resources. publishing our word embeddings also contributes to this goal. 2 for evaluation purposes, we have also created our own manually lemmatized dataset. 3 this ground truth material was created where possible with the dictionary of old literary finnish (kotimaisten kielten keskus, 2021). since the dictionary only currently extends to the word perstauta 'to rot; to decay', however, there are instances where we could not consult this resource, and had to decide the evaluated lemma with our own linguistic intuition. for example, one description of metallurgy practices from 1797 contains the segment jotka makawat palkein ylitze ja wääteillä 'which lie over the bellows and [unknown word]' [rin1797-49]. the wordform wääteillä is not in the dictionary, and it occurs only in this decade in the currently available corpus. we have lemmatized this lexeme as vääde, with full knowledge that this may be erroneous. as our dataset is openly available, the errors are easily corrected later. this illustrates how extremely complicated tasks normalization and lemmatization of historical texts are, and we approach this question with the goal to evaluate the currently available methods, and to improve our understanding on how to improve our models. ## experiment design we used an old literary finnish lemmatizer @xcite trained with manually lemmatized corpus form agricola (institute for the languages of finland and university of turku, 2020). the lemmatization model reached 96.3% accuracy in texts written by agricola, and 87.7% accuracy in out-of-domain data @xcite . the model follows the same lstm architecture that has been found useful both for modern finnish normalization @xcite and dialectalization @xcite . our hypothesis is that if we evaluate the errors the model makes with the texts originating from different periods, we can use the errors as a proxy for progressing changes. these results can be later verified and dated more accurately with larger corpora as such resources become available. for our error analysis, we have selected 25 sentences from different decades, and manually lemmatized them. if a decade had a smaller number of sentences, then we took all the sentences available. the manual annotations are used as the gold standard against which the model's predictions are compared. this results in a manually corrected dataset of 476 sentences. ## result we find that the word error rate (wer) of the lemmatization model fluctuates between 1-23% in our test dataset. the wer, however, increases gradually when measured by the decade, and our hypothesis is that this change represents the linguistic distance that increases when new vocabulary and conventions are added to the written standard. this can be tested through a detailed error classification and analysis, which we conduct in the next section. whenever possible, we aim to provide estimations of when different features emerge, which hopefully allows to detect various periods that can be distinguished from one another. 6 error analysis 6.1 agricola texts although the model was trained with the agricola data, there are still individual errors even in this material. these relate often to personal names such as ziphi and zipheis, which in the original corpus were normalized as sifi. in the vicinity of these words the normalization is very good, and generally we do not find that lemma level errors would impact a more extensive sentence. we presume these are words which have not occurred in the original training data, where part of the agricola corpus was used as the test data, or then forms are simply too rare or exceptional. needless to say, as the figure 3 shows the accuracy is almost flawless in the earliest portion of the corpus. we can illustrate the accuracy with an example from agricola's prayer book. the original sentence is mine rucolen sinua sinun poias cautta 'i pray for you through your son' @xcite . the correct lemmatization is minä rukoilla sinä sinä poika kautta, which is exactly what the model outputs. ## conclusion we show that analysing the errors produced by a neural network that is trained for one task in one specific material serves as a good indicator for salient and emerging differences between the texts. the methodological contribution of our study is that we can use neural networks effectively to track these changes. we could not successfully split the material into distinct periods, but we propose this can be done. still, we were able to trace the changes in some phenomena, especially the punctuation conventions. we see more of a gradual process than clear phases, which also indicates that our initial goal of periodization may not be ideal. the most important finding of our study is that the proposed method works. as the error rate of the neural network increases linearly with newer material, we are convinced that this signals the increasing differentiation of the data in these periods when compared to the texts written by agricola. although in reality there is no need to process old literary finnish materials with the data from agricola alone, besides the fact that only this material is available for training, we think the experiment design also has relevance for nlp research more generally. the language changes also in our day, and the models we train should be able to handle innovations that are only currently emerging. therefore the test setting, although artificial, asks a question that is worth presenting. very importantly, our study provides a clear roadmap for the further development of normalization and lemmatization of old literary finnish. as we published our models and materials openly in zenodo, our analysis is easy to reproduce, and our initial benchmark can be improved. mikko tolonen, leo lahti, hege roivainen, and jani marjanen. 2019. a quantitative approach to bookprinting in sweden and finland, 1640-1828. historical methods: a journal of quantitative and interdisciplinary history, 52(1):57-78. asahi ushio and jose camacho-collados. 2021. t-ner: an all-round python library for transformerbased named entity recognition. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: system demonstrations, pages 53-62, online. association for computational linguistics.