Unsupervised Multi-View Post- OCR Error Correction With Language Models (2021)

Abstract: We investigate post-OCR correction in a setting where we have access to different OCR views of the same document. The goal of this study is to understand if a pretrained language model (LM) can be used in an unsupervised way to reconcile the different OCR views such that their combination contains fewer errors than each individual view. This approach is motivated by scenarios in which unconstrained text generation for error correction is too risky. We evaluated different pretrained LMs on two datasets and found significant gains in realistic scenarios with up to 15% WER improvement over the best OCR view. We also show the importance of domain adaptation for post-OCR correction on out-of-domain documents.

## introduction digital scans of printed paper are still one of the main sources of digitized text across industries, libraries and governmental organizations. scanned documents need to be processed by optical character recognition (ocr) systems in order to be consumed by natural language processing (nlp) pipelines. unfortunately, ocr errors are pervasive and input noise can severely hinder downstream nlp applications, e.g., search @xcite , neural machine translation @xcite or nlu in general @xcite . prior work used text generation techniques or redundancy in similar passages for ocr error correction, which is not appropriate in cases of low corpus redundancy or weak document contextual information. for example, this may pose a risk in sensitive documents in the legal or financial domain, where documents tend to be templatized and specific information, such as legal entities or numbers (e.g., interest rates or amounts), are specific to single documents and cannot be safely inferred. other prior work uses multiple ocr views of a * work done during an internship at goldman sachs ocr1: total 0 ft suppl ies : 23 .64 ocr2: total 0 % suppl tea .3 .64 reconciled: total 0 % suppl ies : 23 .64 document and reconciles them in a supervised way, requiring expensive and difficult to acquire training data. therefore, we investigated post-ocr correction with multiple ocr views on the same document in an unsupervised way. a key assumption of this work is that different ocr views make mistakes in different parts of the input document. to create a better ocr view from multiple ocr inputs we use a language model (lm) to pick the most probable reconciliation. see figure 1 for an example. the key question of this study is if it is possible to use a lm to reconcile the ocr views such that their combination contains fewer errors than the individual views. the advantage of the proposed approach is that (i) it does not require supervision, and that (ii) it is well suited for risky scenarios. we explore two different settings: (i) using an off-the-shelf pre-trained lm (i.e., gpt/2 family and a n-gram model), and (ii) domain adaptation of the lm. we evaluated our approach on two datasets that we adapted for our experiments foot_0 . the re-tas dataset (yalniz and manmatha, 2011), consisting of 20 english books with a total of 100 ocr views, and the more challenging icdar scanned receipts dataset @xcite , with 625 scanned receipts. our results indicate that the proposed unsupervised approach is able to improve over the individual ocr systems. on retas we measured a 8% gain over the best ocr view and on icdar a gain of 15% over the best ocr view. we also found that domain adaptation is crucial for documents from domains distinct from the lm's training data, as we can show that domain adaptation improved the error rates on critical numerical data. ## proposed approach our approach focuses on a setting with multiple ocr views where the challenge is to pick the best segments of each view where the aligned ocr inputs differ. given two ocr views of the same document our approach consists of the following steps: (i) align the ocr outputs and spot differences, and then (ii) score the different choices of those differences to pick the best solution. step 1: spot differences. suppose we have two sequences s 1 = c 1,1 , . . . , c 1,n and s 2 = c 2,1 , . . . , c 2,n , where c i,j is the j-th character of sequence i. further we denote d i,j as the j-th chunk in sequence i which is not present in the other sequence, while e i is a chunk shared by both inputs. finding the longest common subsequence between s 1 and s 2 is equivalent to finding the shortest edit script @xcite . denote the shortest edit script by diff(s 1 , s 2 ). for example, suppose that it takes the form diff(s 1 , s each difference yields a binary choice to either pick d 1,j or d 2,j . all possible outputs that can be produced from the pair of inputs (s 1 and s 2 ) is the set of all root-leaf paths in a binary tree. ## experiments the goal of the experiments is to understand if the proposed approach is able to generate a combined ocr view which contains fewer errors than the individual inputs. ## conclusions and future work we presented an approach for post-ocr corrections in an unsupervised way, relying on multiple ocr inputs and lms for reconciliation. our results show that the approach consistently improves over the single best input. we also show that in a dataset with a different domain with respect to the pretraining data, a domain adaptation step is able to significantly improve the performance. questions that are not addressed in this study and which are open for future work are: (i) how can one deliberately generate different ocr views in our setting; (ii) if there is also an unsupervised way to pick good ocr views, because as the random setting on retas shows that just randomly merging views does not automatically yield the best possible result; (iii) whether it is possible to use a bidirectional lm; (iv) whether it is possible to assess in advance the quality of the underlying ocrs.