OCR P ost C orrection for E ndangered L anguage T exts (2020)

Abstract: There is little to no data available to build natural language processing models for most endangered languages. However, textual data in these languages often exists in formats that are not machine-readable, such as paper books and scanned images. In this work, we address the task of extracting text from these resources. We create a benchmark dataset of transcriptions for scanned books in three critically endangered languages and present a systematic analysis of how general-purpose OCR tools are not robust to the data-scarce setting of endangered languages. We develop an OCR post-correction method tailored to ease training in this data-scarce setting, reducing the recognition error rate by 34% on average across the three languages.

## introduction natural language processing (nlp) systems exist for a small fraction of the world's over 6,000 living languages, the primary reason being the lack of resources required to train and evaluate models. technological advances are concentrated on languages that have readily available data, and most other languages are left behind @xcite . this is particularly notable in the case of endangered languages, i.e., languages that are in danger of becoming extinct due to dwindling numbers of native speakers and the younger generations shifting to using other languages. for most endangered languages, finding any data at all is challenging. in many cases, natural language text in these languages does exist. however, it is locked away in formats that are not machine-readable -paper books, scanned images, and unstructured web pages. these include books from local publishing †: work done at carnegie mellon university. 1 code and data are available at https://shrutirij. github.io/ocr-el/ . houses within the communities that speak endangered languages, such as educational or cultural materials. additionally, linguists documenting these languages also create data such as word lists and interlinear glosses, often in the form of handwritten notes. examples from such scanned documents are shown in figure 1 . digitizing the textual data from these sources will not only enable nlp for endangered languages but also aid linguistic documentation, preservation, and accessibility efforts. in this work, we create a benchmark dataset and propose a suite of methods to extract data from these resources, focusing on scanned images of paper books containing endangered language text. typically, this sort of digitization requires an optical character recognition (ocr) system. however, the large amounts of textual data and transcribed images needed to train state-of-the-art ocr models from scratch are unavailable in the endangered language setting. instead, we focus on post-correcting the output of an off-the-shelf ocr tool that can handle a variety of scripts. we show that targeted methods for post-correction can significantly improve performance on endangered languages. although ocr post-correction is relatively wellstudied, most existing methods rely on considerable resources in the target language, including a substantial amount of textual data to train a language model @xcite @xcite or to create synthetic data @xcite . while readily available for high-resource languages, these resources are severely limited in endangered languages, preventing the direct application of existing post-correction methods in our setting. as an alternative, we present a method that compounds on previous models for ocr postcorrection, making three improvements tailored to the data-scarce setting. first, we use a multisource model to incorporate information from the high-resource translations that commonly appear in endangered language books. these translations are usually in the lingua franca of the region (e.g., figure 1 (a , b , c )) or the documentary linguist's primary language (e.g., figure 1 (d ) from @xcite ). next, we introduce structural biases to ease learning from small amounts of data. finally, we add pretraining methods to utilize the little unannotated data that exists in endangered languages. we summarize our main contributions as follows: ## problem setting in this section, we first define the task of ocr post-correction and introduce how we incorporate translations into the correction model. next, we discuss the sources from which we obtain scanned documents containing endangered language texts. ## benchmark dataset from the sources described above, we select documents from three critically endangered languages 5 for annotation -ainu, griko, and yakkha. these languages were chosen in an effort to create a geographically, typologically, and orthographically diverse benchmark. we focus this initial study on scanned images of printed books as opposed to handwritten notes, which are a relatively more challenging domain for ocr. we manually transcribed the text corresponding to the endangered language content. the text corresponding to the translations is not manually transcribed. we also aligned the endangered language text to the ocr output on the translations, per the formulation in section 2.1. we describe the annotated documents below and example images from our dataset are in figure 1 (a), (b), (c). ainu is a severely endangered indigenous language from northern japan, typically considered a language isolate. in our dataset, we use a book of ainu epic poetry (yukara), with the "kutune shirka" yukara @xcite in ainu transcribed in latin script. 6 each page in the book has a two-column structure -the left column has the ainu text, and the right has its japanese translation already aligned at the line-level, removing the need for manual alignment (see figure 1 (a) ). the book has 338 pages in total. given the effort involved in annotation, we transcribe the ainu text from 33 pages, totaling 816 lines. griko is an endangered greek dialect spoken in southern italy. the language uses a combination of the latin alphabet and the greek alphabet as its writing system. the document we use is a book of griko folk tales compiled by @xcite . the book is structured such that in each fold of two pages, the left page has griko text, and the right page has the corresponding translation in italian. of the 175 pages in the book, we annotate the griko text from 33 pages and manually align it at the sentence-level to the italian translation. this results in 807 annotated griko sentences. yakkha is an endangered sino-tibetan language spoken in nepal. it uses the devanagari writing system. we use scanned images of three children's books, each of which has a story written in yakkha along with its translation in nepali and english @xcite . we manually transcribe the yakkha text from all three books. we also align the yakkha text to both the nepali and the english ocr at the sentence level with the help of an existing yakkha dictionary @xcite . in total, we have 159 annotated yakkha sentences. ## ocr systems: promises and pitfalls as briefly alluded to in the introduction, training an ocr model for each endangered language is challenging, given the limited available data. instead, we use the general-purpose ocr system from the google vision ai toolkit foot_5 to get the first pass ocr transcription on our data. the google vision ocr system @xcite is highly performant and supports 60 major languages in 29 scripts. it can transcribe a wide range of higher resource languages with high accuracy, ideal for our proposed method of incorporating high-resource translations into the post-correction model. moreover, it is particularly well-suited to our task because it provides script-specific ocr models in addition to languagespecific ones. per-script models are more robust to unknown languages because they are trained on data from multiple languages and can act as a general character recognizer without relying on a single language's model. since most endangered languages adopt standard scripts (often from the region's dominant language) as their writing systems, the per-script recognition models can provide a stable starting point for post-correction. the metrics we use for evaluating performance are character error rate (cer) and word error rate (wer), representing the ratio of erroneous characters or words in the ocr prediction to the total number in the annotated transcription. more details are in section 6. the cer and wer using the google vision ocr on our dataset are in ## ocr post-correction model in this section, we describe our proposed ocr post-correction model. the base architecture of the model is a multi-source sequence-to-sequence framework @xcite that uses an lstm encoder-decoder model with attention @xcite . we propose improvements to training and modeling for the multi-source architecture, specifically tailored to ease learning in data-scarce settings. ## experiments this section discusses our experimental setup and the post-correction performance on the three endangered languages on our dataset. ## related work post-correction for ocr is well-studied for highresource languages. early approaches include lexical methods and weighted finite-state methods (see @xcite for an overview). recent work has primarily focused on using neural sequence-to-sequence models. @xcite use a bilstm encoder-decoder with attention for historical english post-correction. similar to our base model, @xcite use a multi-source model to combine the first pass ocr from duplicate documents in english. there has been little work on lower-resourced languages. @xcite present a probabilistic edit distance based post-correction model applied to @xcite show improvements on romanized sanksrit ocr by adding a copy mechanism to a neural sequence-to-sequence model. multi-source encoder-decoder models have been used for various tasks including machine translation @xcite and morphological inflection @xcite . perhaps most relevant to our work is the multi-source model presented by @xcite , which uses high-resource translations to improve speech transcription of lower-resourced languages. finally, bustamante et al. (2020) construct corpora for four endangered languages from textbased pdfs using rule-based heuristics. data creation from such unstructured text files is an important research direction, complementing our method of extracting data from scanned images. ## conclusion this work presents a first step towards extracting textual data in endangered languages from scanned images of paper books. we create a benchmark dataset with transcribed images in three endangered languages: ainu, griko, and yakkha. we propose an ocr post-correction method that facilitates learning from small amounts of data, which results in a 34% average relative error reduction in both the character and word recognition rates. as future work, we plan to investigate the effect of using other available data for the three languages (for example, word lists collected by documentary linguists or the additional griko folk tales collected by @xcite ). additionally, it would be valuable to examine whether our method can improve the ocr on highresource languages, which typically have much better recognition rates in the first pass transcription than the endangered languages in our dataset. further, we note our use of the google vision ocr system to obtain the first pass ocr for our experiments, primarily because it provides script-specific models as opposed to other generalpurpose ocr systems that rely on languagespecific models (as discussed in section 4). future work that focuses on overcoming the challenges of applying language-specific models to endangered language texts would be needed to confirm our method's applicability to post-correcting the first pass transcriptions from different ocr systems. lastly, given the annotation effort involved, this paper explores only a small fraction of the endangered language data available in linguistic and general-purpose archives. future work will focus on large-scale digitization of scanned documents, aiming to expand our ocr benchmark on as many endangered languages as possible, in the hope of both easing linguistic documentation and preservation efforts and collecting enough data for nlp system development in under-represented languages.