Post- OCR Correction of Digitized S wedish Newspapers with B y T 5 (2024)

Abstract: Many collections of digitized newspapers suffer from poor OCR quality, which impacts readability, information retrieval, and analysis of the material. Errors in OCR output can be reduced by applying machine translation models to “translate” it into a corrected version. Although transformer models show promising results in post-OCR correction and related tasks in other languages, they have not yet been explored for correcting OCR errors in Swedish texts. This paper presents a post-OCR correction model for Swedish 19th to 21th century newspapers based on the pre-trained transformer model ByT5. Three versions of the model were trained on different mixes of training data. The best model, which achieved a 36% reduction in CER, is made freely available and will be integrated into the automatic processing pipeline of Sprakbanken Text, a Swedish language technology infrastructure containing modern and historical written data.

## introduction the ocr (optical character recognition) quality of printed documents in general and historical documents in particular is often low. historical documents often suffer from stains, faded print, and ink bleed-through from other pages, which leads to poor ocr accuracy. this, in turn, causes a range of challenges for natural language processing (nlp) systems such as information retrieval and analysis. one way to achieve higher accuracy is to apply a post-ocr correction method to the ocr output. in this context, post-ocr can be compared to machine translation where the input is a set of character strings in one form that should be mapped onto the corrected form @xcite . since the transformer architecture was introduced in 2017 @xcite , it has pushed the state-of-the-art in many nlp tasks, including machine translation. it has also led to the emergence of large pre-trained models. one of these is byt5, which is pre-trained on the large web-scraped multilingual dataset mc4 @xcite . byt5 operates on the character level, as illustrated in figure 1 . this approach preserves words that are not covered by the model's vocabulary due to, for example, ocr errors (han-delstidnidgens) or age (hvalfisken), at the cost of increased sequence length. since the sizes of language models' vocabularies are fixed, a word-level model would map all out-of-vocabulary words (e.g., misspelled or obsolete words) to the same out-of-vocabulary @xmath0, losing all information about these words. this characterlevel approach has reached state-of-the-art results in transliteration and grapheme-to-phoneme tasks @xcite , diacritics restoration in 13 languages @xcite , and post-ocr correction in sanskrit @xcite . however, it has not yet been explored for swedish post-ocr correction. the main contributions of this work are: (i) we demonstrate how effective a fine-tuned byt5 is in the task of correcting ocr errors in 19th and 20th century swedish newspapers. (ii) we show what effect does further training on data from books and other domains have on the model's performance on newspapers. (iii) we release a freely available post-ocr correction model with state-of-the-art performance on historical swedish text, and a demo for testing the model. the model is available at https://huggingface. co/viklofg/swedish-ocr-correction , with a demo at https://huggingface.co/spaces/ viklofg/swedish-ocr-correction-demo . ## related work previous work in post-ocr correction of swedish historical text have explored both statistical and neural network based approaches. @xcite used an svm classifier in combination with a word list to detect and correct ocr errors in 17th to 19th century texts. @xcite proposed a method for increasing ocr accuracy of the swedish newspapers by merging outputs from two ocr engines. @xcite explored using the reference material prepared by @xcite to train an lstm-based model from scratch, but found that it was not large enough to yield satisfactory results. another successful method for correcting ocr errors in swedish historical text involves deep cnn-lstm hybrid models @xcite . drobac and lindén (2020) utilized deep cnn-lstm hybrid networks for the post-ocr task. they employed both finnish and swedish historical newspapers from 17th to 18th century, and reached stateof-the-art results for both. brandt skelbye and dannélls (2021) experimented with mixed deep cnn-lstm hybrid models directly on the character model within the ocr engine. while they have achieved state-of-the-art results for swedish ocr, their method is not directly comparable to ours as it was not applied as a post-processing step. in many other languages, post-ocr correction approaches based on neural machine translation have shown promising performance. examples include the winner of the 2019 icdar competition @xcite , which was trained on ten european languages (but not swedish). more recent examples include models for finnish @xcite , icelandic @xcite , and english @xcite . @xcite note that while these models tend to outperform other techniques, they need a lot of training data to be successful. nevertheless, the emergence of pre-trained transformer models, which require less training data and perform better than traditional methods such as lstm networks, gives us hope that these models will overcome some of the previously reported limitations for swedish ocr. all datasets come with a ground truth. an overview of all datasets is given in ## conclusion in this paper, we present state-of-the-art results in post-ocr correction of swedish 19th to 21th century newspapers. we fine-tuned three models from byt5-small, the smallest available version of google's pre-trained character-level transformer model byt5, using mixes of training data from different domains. the most successful model in terms of cer was fine-tuned on a mix of swedish newspapers and blackletter texts. it achieved a 36% reduction in cer over the entire evaluation set, but despite this, it was found to increase the cer in 7.7% of the evaluation samples. we made this model freely available and will integrate it into the automatic processing pipeline of språkbanken text, 6 a swedish language technology infrastructure containing modern and historical written data. further work aims to minimize the amount of introduced errors taking context information into account. we also intend to conduct several evaluations to learn more about the type of errors the model makes, in particular, on new digitized resources.