Effective Synthetic Data and Test-Time Adaptation for OCR Correction (2024)

Abstract: Post-OCR technology is used to correct errors in the text produced by OCR systems. This study introduces a method for constructing post-OCR synthetic data with different noise levels using weak supervision. We define Character Error Rate (CER) thresholds for “effective” and “ineffective” synthetic data, allowing us to create more useful multi-noise level synthetic datasets. Furthermore, we propose Self-Correct-Noise Test-Time Adaptation (SCN-TTA), which combines self-correction and noise generation mechanisms. SCN-TTA allows a model to dynamically adjust to test data without relying on labels, effectively handling proper nouns in long texts and further reducing CER. In our experiments we evaluate a range of models, including multiple PLMs and LLMs. Results indicate that our method yields models that are effective across diverse text types. Notably, the ByT5 model achieves a CER reduction of 68.67% without relying on manually annotated data

## introduction the task of preserving historical data has seen a transformative impact due to advancements in optical character recognition (ocr) @xcite . while these systems have facilitated the digitization of historical books, they have also introduced a new set of challenges, particularly when dealing with texts involving complex layouts, unusual typefaces, or degraded quality @xcite . post-ocr methods have been proposed to correct these errors @xcite , improving the usability of digitized texts -an important factor for cultural analytics research and digital humanities in general. recent research has framed the post-ocr task as a seq2seq neural machine translation (nmt) task @xcite @xcite . the reliability and generalization performance of nmt models is closely tied to the quality and the volume of the training data @xcite . therefore, synthetic data is commonly used in this area. however, most previous work considers synthetic data with only a single noise ratio for model training @xcite @xcite , which is not consistent with real digitized collections where quality will often vary. even in cases where researchers have considered different noise ratios, this has involved conducting separate sub-experiments for each noise level @xcite . to the best our knowledge, there has been no attempt to merge synthetic data with multiple different noise levels for model training, nor has anyone explored which noise levels to merge to construct the most useful synthetic data. in addition to the original errors that occur during digitization, subsequent post-ocr procedures can introduce further inaccuracies. this issue becomes particularly severe with proper nouns (pns), like personal and place names, which are more susceptible to these post-ocr errors and are challenging to correct. this often arises either because the pns in the test data were not present in the training data, or because they are hard to distinguish. for instance, in ocr text, it is unclear whether "marl" should be corrected to "mark" or "marl". errors like these can negatively impact downstream tasks, such as retrieval and recommendation systems @xcite . in response to the challenges above, in section 3.1 we propose a process that employs a weakly-supervised learning approach to generate synthetic data with varying noise levels. furthermore, we explore how to merge synthetic data with different noise levels to create the most useful dataset for model training. in section 3.2 we evaluate a range of different models using this synthetic data, achieving competitive results for post-ocr correction on multiple benchmarks. in section 4.1 we also propose self-correct-noise test-time adaptation (scn-tta) to enable the model to better handle pns, further improving the quality of the output text. the experiments in section 4 demonstrate that this method is effective across diverse text types and can reduce the cer by up to 68.67%. ## related work post-ocr has evolved over many years, with several methods leveraging multiple ocr engines and combining their outputs to yield improved results @xcite . recent advances involve using pre-trained models and seq2seq architectures for error detection and correction @xcite . @xcite employed both nmt and statistical machine translation (smt) models for post-ocr tasks. following this, schaefer and neudecker (2020) introduced a two-step method that involves detection followed by correction. an unsupervised approach combining multiple ocr views is proposed by @xcite . @xcite split texts into character ngrams and combine their individual corrections into the final output. other methods have involved using lstms for post-processing @xcite , establishing benchmarks using pre-trained models on niche datasets @xcite and using an adapted hill-climbing algorithm @xcite . a number of recent studies have used encoder-decoder pre-trained language models @xcite @xcite and large language models @xcite . synthetic data can enhance model performance across various deep learning domains @xcite . in post-ocr and grammatical error correction (gec) tasks, synthetic data is mainly generated using noise injection @xcite . work by @xcite developed confusion sets for each word in a corpus vocabulary, replacing words in correct sentences at a fixed ratio to generate synthetic data. ingólfsdóttir et al. (2023) claimed that highly-noisy text provided optimal training examples for error correction. both krishna et al. (2018) and davydkin et al. (2023) at-tempted to generate images of text and then create synthetic data with a single error level. jasonarson et al. (2023), rijhwani et al. (2020) and xie and anastasopoulos (2023) injected single level ocr errors into clean text according to their occurrence frequency in the corpus. d'hondt et al. (2017) generated datasets with different noise levels and trained models using a single noise ratio corresponding to the noise level found in the test set. moreover, guan and greene (2024) mixed data with different error ratios in the training data and proposed a novel approach for constructing synthetic data based on glyph similarity measures. weak supervision involves training models with noisy, partial, or imprecise labels. examples include whisper's robust speech recognition using weakly supervised transformer models @xcite , and multir's relation extraction from noisy data @xcite . @xcite demonstrated the effectiveness of weak supervision for text classification. test-time adaptation refers to methods that allow a model to dynamically adapt to test data during inference, without the need for labeled data @xcite . tta has been applied in various tasks such as question answering @xcite @xcite and image segmentation @xcite . this stategy is especially useful in scenarios where the test data distribution may differ from the training data. ## effective synthetic data we now introduce a new approach for generating effective synthetic data for the post-ocr task. ## self-correct-noise test-time adaptation in tasks such as gec and translation, models often struggle with rare words, like proper nouns, that appear in the training data, which impacts their performance on these words @xcite . common solutions include data augmentation, generating new synthetic sentences with rare words @xcite , or aligning rare words to their sources and then translating them with a word-level model @xcite . although these methods can help the model to learn rare words in the training data, they often fail to adapt to the distribution of the test data. in typical translation tasks, each data point contains limited information, whereas in post-ocr tasks, the data often consists of longer texts, like novels. this potentially allows us to use text from the test data to improve model performance. ## conclusion in this paper, we explored how to generate the most effective synthetic data for post-ocr correction tasks. our experiments show that using weak supervision and effective synthetic data with multiple noise levels can reduce the character error rate of ocr texts by 62.95% when used in conjunction with the byt5 model. additionally, we have demonstrated that, by incorporating a novel testtime adaptation approach scn-tta, we can improve the model's ability to handle to handle proper nouns, leading to a 68.67% reduction in cer.