An Unsupervised method for OCR Post-Correction and Spelling Normalisation for F innish (2021)

Abstract: Historical corpora are known to contain errors introduced by OCR (optical character recognition) methods used in the digitization process, often said to be degrading the performance of NLP systems. Correcting these errors manually is a time-consuming process and a great part of the automatic approaches have been relying on rules or supervised machine learning. We build on previous work on fully automatic unsupervised extraction of parallel data to train a character-based sequence-to-sequence NMT (neural machine translation) model to conduct OCR error correction designed for English, and adapt it to Finnish by proposing solutions that take the rich morphology of the language into account. Our new method shows increased performance while remaining fully unsupervised, with the added benefit of spelling normalisation. The source code and models are available on GitHub and Zenodo.

## introduction nature language processing (nlp) is arguably tremendously difficult to tackle in finnish, due to an extremely rich morphology. this difficulty is reinforced by the limited availability of nlp tools for finnish in general, and perhaps even more so for historical data by the fact that morphology has evolved through time -some older inflections either do not exist anymore, or are hardly used in modern finnish. as historical data comes with its own challenges, the presence of ocr errors makes the data even more burdensome to modern nlp methods. obviously, this problematic situation is not unique to finnish. there are several other languages in the world with rich morphologies and relatively poor support for both historical and modern nlp. such is the case with most of the languages that are related to finnish like erzya, sami and komi, these uralic languages are severely endangered but have valuable historical resources in books that are not yet available in a digital format. ocr remains a problem especially for endangered languages @xcite , although ocr quality for such languages can be improved by limiting the domain in which the ocr models are trained and used @xcite . automated ocr post-correction is usually modelled as a supervised machine learning problem where a model is trained with parallel data consisting of ocr erroneous text and manually corrected text. however, we want to develop a method that can be used even in contexts where no manually annotated data is available. the most viable recent method for such a task is the one presented by @xcite . however, their model works only on correcting individual words without considering the context in sentences, and as it focuses on english, it completely ignores the issues rising from a rich morphology. extending their approach, we introduce a self-supervised model to automatically generate parallel data which is learned from the real ocred text. later, we train sequence-to-sequence (seq2seq) nmt models on character level with context information to correct ocr errors. the nmt models are based on the transformer algorithm @xcite , whose detailed comparison is demonstrated in this article. as more and more digital humanities (dh) work start to use the large-scale, digitised and ocred collections made available by national libraries and other digitisation projects, the quality of ocr is a central point for text-based humanities research. can one trust the output of complex nlp systems, if these are fed with bad ocr? beyond the common pitfalls inherent to historical data (see @xcite for a very thorough overview), some works have tried to answer the question stated above: hill and hengchen (2019) use a subset of 18th-century corpus, ecco foot_2 as well as its keyed-in counterpart ecco-tcp to compare the output of common nlp tasks used in dh and conclude that ocr noise does not seem to be a large factor in quantitative analyses. a conclusion similar to previous work by @xcite in the case of ner and to @xcite for authorship attribution, but in opposition to @xcite who focus on topic modelling for historical newspapers and conclude that ocr does play a role. more recently and still on historical newspapers, @xcite conclude that while ocr noise does have an impact, its effect widely differs between downstream tasks. it has become apparent that ocr quality for historical texts has become central for funding bodies and collection-holding institutions alike. reports such as the one put forward by @xcite rise ocr initiatives, while the library-of-congress-commissioned report by @xcite underlines the importance of ocr for culturage heritage collections. these reports echo earlier work by, among others, @xcite who tackle the digitisation of british newspapers, the eu-wide impact project foot_3 that gathers 26 national libraries, or @xcite who set out to analyse the quality of ocr made available by the swedish language bank. ocr post-correction has been tackled in previous work. specifically for finnish, @xcite correct the ocr of newspapers using weighted finite-state methods, accordance with, @xcite do the same for finnish (and erzya). most recent approaches rely on the machine translation (mt) of "dirty" text into "clean" texts. these mt approaches are quickly moving from statistical mt (smt) -as previously used for historical text normalisation, e.g. the work by @xcite -to nmt: dong and smith (2018 ) use a word-level seq2seq nmt approach for ocr post-correction, while hämäläinen and hengchen (2019), on which we base our work, mobilised character-level nmt. very recently, nguyen et al. (2020) use bert embeddings to improve an nmt-based ocr postcorrection system on english. ## experiment in this section, we describe our methods for automatically generating parallel data that can be used in a character-level nmt model to conduct ocr post-correction. in short, our method requires only a corpus with ocred text that we want to automatically correct, a word list, a morphological analyzer and any corpus of error free text. since we focus on finnish only, it is important to note that such resources exist for many endangered uralic languages as well as they have extensive xml dictionaries and fsts available (see @xcite ) together with a growing number of universal dependencies @xcite treebanks such as komi-zyrian @xcite , erzya @xcite , komi-permyak @xcite and north sami @xcite . ## evaluation we evaluate all proposed models and the natas baseline on the ground truth finnish fraktur dataset foot_6 made available by the national library of finland, a collection of 479 journal and newspaper pages from the time period @xcite @xcite . the data format is constructed as a csv table with 471,903 lines of words or characters and there are four columns of ground truth (gt) aligned with the output coming from 3 different ocr methods tesseract, @xcite . despite the existence of character-level benchmarks for ocr post-correction (e.g. @xcite ), we elect to evaluate models on the more realistic setting of whole words. we would like to note that finnish has very long words, and as a result this metric is actually tougher. in the previous section, our models are trained without non-alphabet characters, so all the tokens which have non-alphabet will be removed. we also removed the blank lines which have no result from ocr. after having the ground truth and ocr text cleaned, the number of tokens for each ocr method (tesseract, old, fr11) are 458,799, 464,543 and 470,905 with accuracies of 88.29%, 75.34% and 79.79% respectively. the ocr words will be used as input data for the evaluation of our post-correction systems. the translation processes apply for each ocr method separately with the input tokens formatted based on the model's requirement. in natas, we used opennmt to translate with the default settings. in transformer models with context, we created a sliding window over the rows of the ocred text. for the non-context model, we only need a single token for source input. these models do the translation with beam search @xmath0. the result is shown in ## conclusion and future work in this paper, we have shown that creating and using an artificial error dataset clearly outperforms the natas baseline @xcite , with a clear advantage for the t rain over the rand configuration. another clear conclusion is that a larger context window results in increasing the accuracy of the models. comparing the new results for all three ocr methods, we see the models are most effective with fr11 when the ratio of fixing wrong words (45.17%) is high enough to overcome the issue of breaking the right words (6.7%). our methods also work very well on old with ability to fix 36.03% of wrong words and handle more than 93% of right words correctly. however, our models are not compelling enough to beat the accuracy achieved by tesseract, a conclusion we see as further work. in spite of the effectiveness of the postcorrection strategy, it does not guarantee that all the words with w/v replaced are correct, nor that uralicnlp manages to recognize all the existing finnish words. for example: the wrong ocr word mcntoistamuotiscn was fixed to metoistavuotisen which is the correct one according to the gold standard, but uralicnlp has filtered it out due to not considering that is the valid finnish word. this is true, as the first syllable kol was dropped out due to a line break in the data, and without the line break, the word would be kolmetoistavuotisen ("13 years old"). this means that in the future, we need to develop better strategies more suitable to ocr contexts for telling correct and incorrect words apart. this implies that in reality the corrected cases can be higher if we don't revert the already normalized w/v words. in addition, if there is a better method to ensure a word is valid in finnish, the result could be improved. thus, our evaluation provides an overall view of how the transformer and trained error generator models with context words could improve the post ocr correction notably. our methods also show that using artificial dataset from a modern corpus is very beneficial to normalize the historical text. importantly, we would like to underline that our method does not rely on huge amounts of hand annotated gold data, but can rather be applied for as long as one has access to an ocred text, a vocabulary list, a morphological fst and error-free data. there are several endangered languages related to finnish that already have these aforementioned resources in place. in the future, we are interested in trying our method out in those contexts as well.