OCR Error Post-Correction with LLM s in Historical Documents: No Free Lunches (2025)

Abstract: Optical Character Recognition (OCR) systems often introduce errors when transcribing historical documents, leaving room for post-correction to improve text quality. This study evaluates the use of open-weight LLMs for OCR error correction in historical English and Finnish datasets. We explore various strategies, including parameter optimization, quantization, segment length effects, and text continuation methods. Our results demonstrate that while modern LLMs show promise in reducing character error rates (CER) in English, a practically useful performance for Finnish was not reached. Our findings highlight the potential and limitations of LLMs in scaling OCR post-correction for large historical corpora.

## introduction digitizing and transcribing historical documents and literature is vital for preserving our cultural heritage and making it accessible for modern digital research methods. the transcription process relies on ocr, which naturally introduces noise into the output. the noise varies in severity depending on the quality of the source material and the ocr technology used, impacting the research usage of the data @xcite . the ocr output at two noise levels is illustrated in figure 1 . although modern ocr systems are becoming increasingly accurate @xcite , reprocessing large collections of historical literature remains a significant challenge, as the resources available to the institutions maintaining these collections are often insufficient for such an undertaking. consequently, ocr error post-correction has been suggested as means of improving the historical collections without the need to repeat the whole transcription process @xcite . recent studies @xcite have proposed the application of llms to the task, with varying degrees of success. currently, there is no clear consensus as to how llms can be applied to the task and how to deal with the various methodological challenges it poses. our objective is to address some of these challenges as well as to assess several open llms to correct ocr-generated text when prompted to. we study hyperparameter optimization, quantization levels, input lengths, output post-processing and several novel correction methods so as to benchmark and improve the llm performance on this task. as our long-term goal is to post-correct two large historical datasets, one in english and the other in finnish, we focus on these two languages as well as open-weight llms, since post-correction of large datasets with commercial models is infeasible costwise. ## related work despite decades of active research, post-correction of historical documents remains a challenge. the icdar 2017 and 2019 shared tasks @xcite addressed the lack of adequate benchmarks for evaluating ocr performance across several languages, introducing two tracks: detecting ocr errors, and correcting previously detected errors. this setting has naturally guided the development towards two-stage systems, and the best performing models in the icdar 2019 edition were based on the bert model finetuned separately for each task @xcite . such two-step approaches are still actively pursued, with e.g. @xcite applying a bert classifier for error detection, and an lstm-based seq2seq model for error correction in bulgarian. recently, llms have been effectively applied to text correction problems, for example, penteado and perez ( 2023) and östling et al. (2024) in the zero-shot, prompt-based line of work, @xcite evaluated a variety of models and prompts on several multilingual historical datasets. interestingly, the results of the study were mostly negative, concluding that llms (including the commercial gpt-4 model) are not effective at correcting transcriptions of historical documents, in many cases the llm actually decreasing the quality instead of improving it. bourne (2024) conducted a similar study on three historical english datasets, arriving at the opposite conclusion. they achieved over 60% reduction of character error rate at best, with most of the evaluated models improving the data quality. finally, several studies also pursue approaches that include the original image as an input, together with the ocr output to be post-corrected. here, e.g. @xcite combine a state-of-the-art transformer-based ocr system with the character-based charbert model for handwritten text recognition, and @xcite propose a model iterating between ocr and post-correction steps for farsi. such image-text approaches are, nevertheless, beyond the scope of the present study. ## data in our study, we utilize manually corrected samples of two large historical datasets, one for english and the other for finnish. ## experiments first, we set out to evaluate the basic performance of different llms on the ocr error correction task and establish how the generation and model hyperparameters (e.g. sampling parameters and quantization) affect the results. the page lengths in our data vary, with the ecco-tcp pages on average at 200 words, and the finnish newspaper pages at about 1,000 words. to improve comparability of the results, we split the pages to segments of 200 ocr words for english and 100 ocr words for finnish, both corresponding to roughly 300 sub-words in openai's gpt-4 tokenization for the language in question. the length of roughly 300 sub-words was established as suitable in our initial experiments, however, we will carry out a more detailed evaluation of segment lengths as a separate experiment in section 6. since the finnish data is originally word-aligned, obtaining these shorter-than-page segments is trivial. for the english data, which is only pagealigned, we utilize the character-level ocr-gt alignments produced during data filtering (described in section 3.1), allowing us to find corresponding points. in cases where the segment boundary falls within a region of poor alignment, we shift the boundary to the next reliable word (the word starting the next aligned region). therefore, the exact segment length may vary depending on how well the ocr and gt strings could be aligned. given the substantial volume of our data, and the number of llm runs necessary in our experiments, we randomly sample for each language a development and a test set, each containing 200 examples (i.e. segments of about 300 sub-words in length). these constitute 244k+243k gt characters for english, and 162k+165k gt characters for finnish. the development set is used to set the generation parameters and the test set is used to report all results, unless otherwise stated. ## results the main results are shown in however, the llama-3.1-70b still shows a notable improvement of 38.7%. in figure 2 we illustrate the cer values for english test examples before and after the llama-3.1-70b correction. most examples demonstrate improved cer scores, regardless of whether they initially had mild or severe noise levels. in terms of wer%, all models show positive improvement on english, the two best models achieving an improvement of 59.1% (gpt-4o) and 46.3% (llama-3.1-70b). the relative order of the models seems to mostly follow the number of model parameters, bigger models generally performing better, expect for mixtral which is clearly worse than the others, and the two gemma models performing almost equally in terms of wer%, although the gemma-2-27b version clearly outperforms the 8b model in terms of cer%. for finnish, on the other hand, gpt-4o is the only model achieving a positive improvement in either metric, albeit considerably smaller in absolute terms than for english with 11.9 cer% and 33.5 wer%. seeing these entirely negative results for finnish, we are forced to conclude that prompt-based ocr post-correction is presently infeasible for this language using any of the tested open-weight models. this is disappointing, but not surprising since none of the models officially support finnish. the striking effect of a common but meaningpreserving difference between historical and modern language becomes apparent when measuring the effect of modern spelling produced by the llms such as the ſ vs. s and w vs. v variation discussed in section 4.1. a typical example for both languages is illustrated in figure 3 . without the applied normalization, the results of gpt-4o would have been 34.9 cer% and 35.5 wer% (compared to 58.1% and 59.1% with normalization) for english, and -10.1 cer% and -4.8 wer% (compared to 11.9% and 33.5%) for finnish. this demonstrates a substantial impact on the reported scores, and while the relative model ranking is unlikely to change, we can see that the conclusion w.r.t. this model's performance on finnish would have been the opposite, and the improvements seen in english would have been lot smaller. given the entirely negative results for finnish with the open-weight models, we carry out all further analyses on english only. furthermore, we also remove the mixtral-8x7b from follow-up experiments as it performs notably worse than the other models. ## segment length and continuation the results in the previous sections were reported for text segments about 300 sub-words in length. the actual texts in the historical collections are naturally substantially longer, necessitating splitting the input into segments of appropriate length. this raises two related questions: (1) how long the input segments should optimally be for best postcorrection accuracy, and (2) how should the outputs be combined to minimize degradation on segment boundaries. our english data is on the level of pages, which we cannot simply naively concatenate, we need other means to obtain sufficiently long documents. for this experiment, we sample long pages of at least 600 whitespace delimited ocr words in length from the development data, taking at maximum two pages from any one book. this resulted in a sample of 53 development set pages. these sampled pages are then divided into nonoverlapping segments of 50, 100, 200, and 300 words, using the same alignment-based splitting strategy as described in section 4. the segments are corrected individually and the cer% improvement over the segments is calculated. the results are shown in figure 4 . shorter segments (50-100 words) get notably worse cer% score for all models, with the gains diminishing past about 200-300 words, but our page-level data does not have longenough examples to allow us to reach the point where the performance would start consistently decreasing as the segments become too long. in the future, we plan to develop a book-level version of the data, and study the correction performance on even longer segments. ## conclusion we set out to establish the ability of recent openweight models to post-correct ocr errors in a zeroshot, prompt-based setting. in the first set of experiments, we established that for historical english these models achieved notable improvements (llama-3.1-70b-instruct reaching a cer improvement of 38.7%), even though still far behind the method model bas. lcc luc llama-3-8b -0.2 -2.9 -2.8 llama-3.1-8b 13.7 8.4 10.6 llama-3.1-70b 33.2 36.0 34.6 gemma-2-9b 29.2 27.9 28.3 gemma-2-27b 38.1 39.9 39.7 in a separate set of experiments we examine how segment-wise correction of long documents should be approached. we devise and evaluate a number of methods to incorporate additional context for the correction of individual segments. we find that some of these methods have a strong positive effect in the immediate proximity of segment boundaries, however, for smaller models the more complicated prompt may cause unexpected degradation in performance when the whole text is considered. further work will be necessary to resolve these issues. as future work, we will pursue a large correction run of the ecco collection as well as a fine-tuned model for finnish post-correction. all datasets and evaluation scripts used in this study are available at https://github.com/turkunlp/ ocr-postcorrection-lm to support result replication and comparability.