BART for Post-Correction of OCR Newspaper Text (2021)

Abstract: Optical character recognition (OCR) from newspaper page images is susceptible to noise due to degradation of old documents and variation in typesetting. In this report, we present a novel approach to OCR post-correction. We cast error correction as a translation task, and fine-tune BART, a transformer-based sequence-to-sequence language model pretrained to denoise corrupted text. We are the first to use sentence-level transformer models for OCR post-correction, and our best model achieves a 29.4% improvement in character accuracy over the original noisy OCR text. Our results demonstrate the utility of pretrained language models for dealing with noisy text.

## introduction newspapers capture daily life from a moment in history. their contents may not ever be published in history books, but they are rife with stories about everyday people, and connecting people to these stories empowers journeys of personal discovery. connecting people to interesting and relevant articles within a newspaper is often challenging because newspaper pages are stored as scanned images. computer vision models have been developed to automatically separate pages into individual articles (via object recognition) and convert each article from image to text (via ocr). still, because of the degradation of old documents and variation in typesetting, the resulting text often contains errors, which can cause problems for downstream applications of the data. the field of natural language processing (nlp) has undergone a rapid shift in the last several years due to the popularization of transformers as a new, powerful tool for language modeling @xcite . many pretrained models like bert @xcite and gpt @xcite 1 work completed during an internship at ancestry.com have advanced the state-of-the-art on numerous nlp tasks, from text classification to question answering and translation. the key to the success of these models is their flexibility; pretraining on generic tasks like masked language modelling or next sentence prediction gives models generalized language knowledge, which then allows them to be easily adapted to more specific tasks @xcite . while pretrained language models are highly adaptable to new tasks, they have been shown to be very sensitive to noise. models like bert break down easily in the presence of irregular spellings or errors in text @xcite . this work shows that in fact this type of pretrained language model can actually be used to correct noise. in this work, we test the utility of transformer models to automatically correct noisy text generated by ocr. specifically, we fine-tune bart @xcite on the icdar 2017 post-ocr correction dataset @xcite . unlike most previous approaches, fine-tuning bart does not require character-level alignment of the training data. our results indicate that the generalized knowledge attained by bart during pretraining is enough to perform well at ocr text correction with just a small amount of task-specific fine-tuning. our best model improves the character accuracy of the evaluation set by 29.4%. correcting ocr errors is a challenging task, but significantly impacts performance on any downstream use of the data. the following report is organized as follows: in section 2 we describe previous work on ocr post-correction, in section 3 we outline the bart model architecture and present our methodology for fine-tuning the model, in section 4 we report the results of our experiments, and finally we conclude in section 5. ## previous work ocr post-correction has been an important problem since the inception of ocr technology. traditional approaches to ocr post-correction relied on n-gram or dictionary-based techniques @xcite . recently, neural methods have become more popular. the international conference on document analysis and recognition (icdar) has held two competitions for ocr post-correction. the first, in 2017, was dominated by statistical and neural machine translation approaches @xcite . the winner of the competition was amrhein and clematide (2018), who still hold the current state-of-the-art on the icdar 2017 benchmark. they use an ensemble of character-based statistical and neural machine translation models. the second icdar competition for ocr postcorrection was held in 2019, after the introduction of transformer-based language models. the winning team, from clova ai, used bert embeddings as input to train cnn classifier, then characterlevel sequence-to-sequence (bilstm) for correction @xcite . since then, there has been continued interest in applying pretrained language models to the task of text correction. nguyen et al. (2020) use bert embeddings to train an error detection network, and then apply character-level nmt for correction. bert has also been used in the correction of errors from other sources; zhang et al. (2020) correct errors generated by human typos. they use pretrained bert embeddings as input to a bigru to detect errors, and then fine-tune bert on a masked language modeling task to correct the sentence, where the errors detected in the first step are softmasked. @xcite use bert embeddings plus edit distance between the errors and candidate replacements to correct pre-identified errors. the above studies demonstrate the potential for pretrained language models in text correction. the present study builds on previous work by testing the potential of this type of model to handle text correction on their own in a single step, without additional infrastructure. we were interested in whether the generalized language knowledge acquired during pretraining could be directly transferred to ocr post-correction with a small amount of task-specific fine-tuning. to test this, we use bart, a sequence-to-sequence transformer model pretrained on text denoising, and fine-tune it on the icdar 2017 dataset @xcite . this approach is novel in that error detection and correction are handled together in one step, and the model uses full sentences rather than performing character-level translation. to our knowledge, this is the first time that bart has been applied to the task of ocr post-correction. our results suggest that bart can in fact be satisfactorily tuned as a stand-alone text correction model. ## methods next we outline the original bart model, followed by the methods used to fine-tune it for ocr post-correction. ## results next we discuss the results of fine-tuning bart for ocr post-correction. our model can handle a wide variety of error types, and improves the overall text accuracy by 29.4% on the evaluation set. ## conclusion in this work, we explored the use of bart for automatic correction of noisy text data generated by ocr. our best model achieves 29.4% improvement in text accuracy. the resulting improvement in ocr text quality impacts all subsequent applications of the data. the approach described above is not specific to errors introduced by ocr; the same methods could in principle be applied to correct any type of noisy data, from the output of handwriting recognition models, to human-generated typos. future work might test the utility of fine-tuning bart on a wider range of noisy text types, including finetuning mbart for correction of non-english data, as well as comparing bart with other sequenceto-sequence language models on text correction. error correction is a critical first step for any nlp task when working with noisy data sources, and thus remains an important problem for the field.