Lexically Aware Semi-Supervised Learning for OCR Post-Correction (2021)

Abstract: Much of the existing linguistic data in many languages of the world is locked away in non- digitized books and documents. Optical character recognition (OCR) can be used to produce digitized text, and previous work has demonstrated the utility of neural post-correction methods that improve the results of general- purpose OCR systems on recognition of less- well-resourced languages. However, these methods rely on manually curated post- correction data, which are relatively scarce compared to the non-annotated raw images that need to be digitized. In this paper, we present a semi-supervised learning method that makes it possible to utilize these raw images to improve performance, specifically through the use of self-training, a technique where a model is iteratively trained on its own outputs. In addition, to enforce consistency in the recognized vocabulary, we introduce a lexically aware decoding method that augments the neural post-correction model with a count-based language model constructed from the recognized texts, implemented using weighted finite-state automata (WFSA) for efficient and effective decoding. Results on four endangered languages demonstrate the utility of the proposed method, with relative error reductions of 15%–29%, where we find the combination of self-training and lexically aware decoding essential for achieving consistent improvements.1

## introduction there is a vast amount of textual data available in printed form @xcite . in this paper, we address the task of digitizing printed materials that contain text in endangered languages, that is, languages with small populations of first-language speakers and limited acquisition among younger speakers. printed texts in endangered languages come from various sources, including linguistic documentation and cultural and educational books. extracting text data from these documents is valuable for a multitude of reasons. automatic digitization can aid language documentation, preservation, and accessibility efforts by archiving the texts and making them searchable for language learners, teachers, and speakers, contributing to essential resources for community-based language revitalization. further, most endangered languages are under-represented in natural language processing technologies, primarily because there is little to no data available for training and evaluation @xcite . this challenge can be mitigated by converting printed materials in these languages to a machine-readable format. optical character recognition (ocr) systems can be used to produce digitized text, and recent work @xcite has demonstrated that post-correction improves the performance of existing general-purpose ocr systems on endangered languages (an example is in figure 1 ). most state-of-the-art ocr post-correction methods use neural sequence-to-sequence models and rely on considerable resources such as a large number of manual transcriptions @xcite or substantial textual data to train a language model @xcite . to adapt these methods for the less-well-resourced endangered languages setting, @xcite add translations and structural biases to the model. however, even with such methods targeted to low-resource learning, post-correction performance is still dependent on manually curated data, which are minimally available for most endangered languages. on the other hand, unannotated figure 1 : ocr post-correction on a scanned document that contains text in the endangered language kwak'wala. the goal of post-correction is to fix the recognition errors made by the first pass ocr system. raw images that need to be digitized are relatively less scarce; for many endangered languages, hundreds of printed pages exist, with only a small subset manually transcribed. in this paper, we propose a semi-supervised learning method for ocr post-correction that efficiently utilizes these unannotated pages to improve performance. the method has two key components. we first present a self-training method for ocr postcorrection (section 4) to create pseudo-training data. a baseline post-correction model is used to correct the initial ocr output on the unannotated pages, and the generated ''post-corrected'' text is then used as pseudo-training data to improve the post-correction model. the self-training process is repeated to iteratively obtain better predictions on the unannotated pages. while self-training is a straightforward way to use the unannotated data, incorrect predictions in the pseudo-training data may introduce noise into the model @xcite . to counterbalance the influence of this noise, we propose lexically aware decoding (section 5), an inference strategy that encourages the model to generate predictions that contain ''known'' words. we use the pseudo-training data to train a count-based language model, represented with a weighted finite-state automaton (wfsa). our proposed decoding method jointly uses an lstm decoder and the wfsa to make ocr postcorrection predictions. the intuition behind the joint decoding strategy is simple. as the model iteratively improves with self-training, the quality of the pseudo-training data is also likely to improve and contain an increasing number of correctly predicted words, resulting in a better count-based language model. consequently, joint decoding reinforces the prediction of more accurate words and mitigates the noise introduced by incorrect words in the pseudo-training data. we conduct experiments on four endangered languages: ainu, griko, kwak'wala, and yakkha. our proposed method reduces the character and word error rates by 15%-29% over a state-ofthe-art ocr post-correction method for endangered languages. we find that the combination of self-training and lexically aware decoding is essential for achieving consistent improvements in performance. ## self-training self-training is a semi-supervised learning method, where a trained model is used to make predictions on unlabeled data, and the model is then retrained on its own predictions @xcite . consider that we have a set of images with manually created transcriptions and a set of images without gold transcriptions. we can obtain a first pass transcription for the text contained in the images (both sets) with existing ocr tools. more formally, we have a gold-transcribed dataset @xmath0, y (i) } d @xmath1, where x (i) is the first pass transcription and y (i) is the error-free manual transcription of the ith training instance. 3 we also have a dataset for which only the first pass ocr is available (i.e., no manual transcriptions), @xmath2. for most cases in the endangered languages setting, the set without gold transcriptions is much larger, that is, u d. since self-training requires a baseline model to get an initial set of predictions on u , we first train the base model described in section 3 on the goldtranscribed set d. let the trained base model be f θ . next, we use the predictions on u from f θ to self-train the model. we follow the self-training strategy recommended in @xcite , which involves two steps: ''pseudo-training'' and ''finetuning''. we describe each step of the self-training procedure in detail below: for an instance x, let the prediction be f θ (x). ## lexically aware decoding although self-training is a simple approach that leads to improvements in post-correction performance without additional manual annotation, incorrect predictions in the pseudo-annotated data may introduce noise into the model, potentially reinforcing the errors in the next self-training iteration @xcite . such noise is more likely to occur in the endangered languages setting, where the base model is trained on minimal data and thus sometimes generates erroneous predictions. while some self-training methods use confidence scores to remove noisy predictions (such as @xcite , these are typically designed for classification tasks. designing such heuristics is challenging for ocr post-correction because the predictions are generated at the character-level; specific characters may be incorrect, but discarding the entire predicted sequence (i.e., the line or sentence) is inefficient, particularly in a lowresource scenario. to mitigate these issues, we propose lexically aware decoding, an inference strategy based on our observations of the challenges associated with the ocr post-correction task. more specifically, our preliminary experiments with self-training indicated that the errors made by the model are typically inconsistent. for a particular word, some instances may be correctly predicted by the model. for the instances of the word that are incorrect, we observe that they are likely to be erroneous in different ways, i.e., different subsets of characters in the word are incorrectly predicted. this is expected since the same word can appear in varied contexts, or the first pass ocr for the word can differ. our empirical observations on the pseudo-annotated dataset s showed that, since the errors are inconsistent, the correct form of the word is more frequent than incorrect forms. lexically aware decoding is designed to influence the ocr post-correction model to generate words that frequently occur in the set s, in the expectation that these are correct word forms. we first describe the construction of a model that accounts for word frequency in the predictions along with a character n-gram model to enable the prediction of unseen words. then, we present a joint decoding method that uses the frequencybased models in combination with the lstm decoder for improved ocr post-correction. ## experiments in this section, we present experiments with our semi-supervised post-correction method on four typologically diverse endangered languages. ## conclusion digitization at scale for documents in underrepresented languages is a promising avenue towards tackling one aspect of their marginalization, the lack of data. with this work, we take a step towards better digitization for extremely data-scarce scenarios. we develop a semi-supervised method that combines self-training with lexically aware decoding, reducing error rates by up to 29% over a state-of-the-art ocr post-correction model on four typologically diverse endangered languages. in future work, we plan to expand our method to take advantage of additional outputs of the language documentation process. for example, documentary linguists typically collect word lists (which range from lists of common words like the swadesh lists @xcite to domain-specific vocabularies). using such word lists within the lexically aware decoding framework could further improve performance and enable the application of our technique to even lower-resourced languages. additionally, the improvements we achieve through semi-supervised learning are potentially orthogonal to the improvements @xcite achieve by incorporating information from translations of the target text. as future work, we plan to investigate the combination of these two approaches in an attempt to utilize all available sources of information to improve performance. finally, while using a character-level n-gram lm improves performance on unknown words, it does not explicitly utilize morphological struc-ture to generate unseen inflections of words. in the future, we plan to incorporate morphological analysis during post-correction decoding, which will be helpful for morphologically rich endangered languages.