Generating Errors: OCR Post-Processing for I celandic (2023)

Abstract: We describe work on enhancing the performance of transformer-based encoder-decoder models for OCR post-correction on modern and historical Icelandic texts, where OCRed data are scarce. We trained six models, four from scratch and two fine-tuned versions of Google’s ByT5, on a combination of real data and texts populated with artificially generated errors. Our results show that the models trained from scratch, as opposed to the fine-tuned versions, benefited the most from the addition of artificially generated errors.

## introduction optical character recognition (ocr) is used to digitize texts by converting scanned documents into machine-readable text. unfortunately, ocr errors are prevalent, particularly when it comes to old texts, where data tends to be scarce, and post-correction is often required to improve the extracted texts' accuracy (e.g. @xcite . transformer-based encoder-decoder models have been shown to be effective in various natural language processing tasks, including machine translation @xcite ) and text summarization @xcite . in this study, we investigate the use of such models for ocr post-correction under scarce data condition, as a sequence-to-sequence problem, similar to how neural machine translation (nmt) systems approach the problem of translation. to address the lack of resources available for training models when dealing with ocred texts, we propose the use of artificially generated errors to improve the performance of the models, which has been shown to be an effective way of generating data for text correction @xcite . the main contribution of this study is an examination of the effectiveness of using artificially generated errors to improve the performance of transformer-based encoder-decoder models for ocr post-correction when data scarcity is a limiting factor. furthermore, we publish our best performing models under the apache 2.0 license. 1 the paper is structured as follows. section 2 discusses related work while section 3 describes the dataset and error generation methods used in this study. section 4 presents the proposed models and training methods. section 5 presents the experimental results and analysis, section 6 discusses limitations and future work, and finally section 7 concludes. ## related work previously, @xcite developed a tool for post-processing icelandic 19 th century texts based on an error model containing statistical information on word and character errors and an ngram language model. their tool correctly identifies and corrects 52.9% of errors in their evaluation set. @xcite report a 63% error correction rate with their ocr post-processing tool on an english text from the 18 th century. they used a scoring system based on string-similarity to find possible substitutions for perceived errors and a language model to evaluate the edited sentences. ## data we used a combination of real ocred texts, processed by abbyy finereader, foot_1 and digital texts not scanned with ocr, the latter of which were populated with artificially generated errors, for training and evaluating our ocr post-correction models. the evaluation data solely comprised real ocred texts, alongside their manually corrected counterparts, which were used to ensure that the models' performance is reflective of real-world ocr output. the ground truth (gt), foot_2 i.e. the data from which the errors were extracted, consists of around 375k tokens from 80 texts published between 1874 and 1913, which were manually corrected, while the data used in training and validation, which include the ocred texts as well as the texts populated with the artificial errors, amount to roughly 9.2m tokens. the data into which the artificial errors were inserted were taken from the icelandic gigaword corpus (igc; @xcite ) and the icelandic text archive (ita). 4 their publication dates range from the late 18 th century to the early 21 st century, with roughly 40% of them having been published between 1830 and 1920. overall, the training data consist of 7.8m tokens, whereas the validation set, which is approximately 15% of the whole dataset, consists of around 1.4m tokens. the evaluation set, totalling 44k tokens, is composed of manually corrected texts. all datasets contain texts from different eras, including texts from the 19 th century and the early 20 th century, as well as texts from the last two decades of the 20 th century. it should be noted that none of the data are based on texts printed in gothic font, which has been reported to be harder to recognize than other fonts @xcite . the evaluation set is divided into two parts, with 26k tokens being from modern texts and 18k tokens from texts from the 19 th century and the early 20 th century. this allows for an evaluation of the model's performance on different types of texts and ocr errors, which is crucial to ensure that the model is robust and generalizable. it is important to note that the dataset used in this study is relatively small in size. one of the reasons is the scarcity of available corrected ocred texts. additionally, we observed that too large a proportion of modern texts in the training set resulted in the models over-generalizing and changing historical spellings to modern spellings. however, we aim for diplomatic transcription, preserving the original spelling. therefore, we ensured that the dataset included texts from different eras while also avoiding over-generalization and alteration of historical spellings by limiting the amount of modern texts into which we inserted artificial errors. ## models in total, six models were trained. two of them follow the architecture of model 1, laid out in for every pair of the models, one was trained on the 375k tokens in the gt, and the other one on the whole dataset, around 7.8m tokens. this was done to study the artificially generated errors' impact on the models' output. we experimented with various hyperparameter configurations, evaluating the models we trained from scratch on the validation sets, and these specific configurations resulted in the highest performance. it is well established that transformer models require large amounts of data to be trained effectively. in this study, our gt had a limited number of examples, which likely contributed to the poorer performance of the models trained from scratch on the smaller dataset, in some instances even performing worse than the base ocr process. ## results the six models we trained for post-processing of ocred texts were applied to modern and historical texts to measure the impact and viability of using artificial errors to improve such models when available data are scarce. the results of these models were compared to the base output from the ocr process using four metrics: chrf @xcite and bleu @xcite , character error rate (cer) and word error rate (wer). bleu score is calculated by comparing texts on a word-level, while chrf score is calculated on a character-level and can be more accurate for inflected languages @xcite . note that while the models generally perform better on the historical texts, the addition of artificial errors improve their performance proportionally more on the modern ones. this could stem from the fact that the artificially-erroneous dataset includes modern texts, while the gt does not. when evaluating the models on modern texts, we found that they were less capable in reducing errors in modern texts than in historical texts. this could be due to the fact that the gt only comprised historical data, suggesting that using solely historical ocred texts is not a viable approach when designing an ocr post-processing tool for modern texts. the lower error rate reduction (err) on the modern texts presumably also stems from the higher base score on the modern texts, as opposed to the base score of the historical ones, leaving less room for improvement. ## limitations and future work the cost of manually correcting ocr output is high, making it difficult to obtain a larger dataset for training. this has a direct impact on the ability of the models to perform well on a wider range of texts and ocr errors. the models have the unwanted tendency to adapt to modern spellings when using a large amount of modern texts populated with artificial errors. this could lead to the alteration of historical spellings, which is not in line with our objectives, to produce diplomatic transcriptions. to mitigate this risk, more corrected texts are needed for the period of texts being ocred. moreover, the use of artificially generated errors to enhance the performance of the models may not fully capture the complexity and diversity of real-world ocr errors. future studies may benefit from incorporating a more diverse range of error types and more realistic error generation methods. we are interested in investigating optimal methods for generating realistic errors to use in training the models. as previously mentioned, the artificial errors used in this study were generated by randomly inserting errors that were extracted from the gt into other texts. however, there may be more effective methods for generating errors that better ## conclusion our findings demonstrate that while fine-tuning pre-trained models on smaller datasets is an effective approach to improving the performance of ocr error correction models, it is possible to achieve comparable results by training an encoderdecoder transformer model from scratch. model 2, which was trained from scratch, emerged as the best performer in our study, achieving a 52.60% word error rate reduction (werr) and a 47.55% character error rate reduction (cerr) on the historical texts, and a word error rate reduction of 34.67% and a character error rate reduction of 15.20% on the modern texts, see these results indicate that with proper architectural design, it is possible to train effective ocr error correction models without relying on pretrained models or large datasets. however, the use of artificially generated errors in the training process was found to be effective in countering the challenges posed by data scarcity. the fact that the models' performance improved proportionally more on the modern texts after the introduction of the artificial errors, which were by and large inserted into modern texts, indicates that in order to train a designated ocr post-processing tool for modern texts, a dataset consisting of modern texts is needed.