OCR Improves Machine Translation for Low-Resource Languages (2022)

Abstract: We aim to investigate the performance of current OCR systems on low resource languages and low resource scripts. We introduce and make publicly available a novel benchmark, OCR4MT, consisting of real and synthetic data, enriched with noise, for 60 low-resource languages in low resource scripts. We evaluate state-of-the-art OCR systems on our benchmark and analyse most common errors. We show that OCR monolingual data is a valuable resource that can increase performance of Machine Translation models, when used in backtranslation. We then perform an ablation study to investigate how OCR errors impact Machine Translation performance and determine what is the minimum level of OCR quality needed for the monolingual data to be useful for Machine Translation.

## introduction despite many recent successes, machine translation still lacks support or fails to achieve good performance for most low-resource languages, which represent a very large fraction of the languages spoken by the world's population @xcite @xcite . the poor performance in these settings can largely be attributed to the lack of training data. many techniques for improving machine translation, such as backtranslation @xcite @xcite and approaches which make use of pre-trained language models @xcite @xcite , rely heavily on high quality monolingual data, which is not readily available for lowresource languages. fortunately, many books and other resources in these languages have been digitized and made available online. however, this textual data is "locked" away in formats such as pdfs and images, which are not readily accessible. as a result, there are large unexplored collections of data in many languages which could be used as a source for monolingual data. for example, one nepali books corpus foot_0 , contains around 342m tokens, which would potentially make it one of the largest sources of monolingual data for this language. a solution to this problem is to rely on modern optical character recognition (ocr) tools to extract the text. unfortunately however, most of the ocr models have only been evaluated on a handful of languages, and public benchmarks for lowresource scripts and languages are lacking @xcite . as a result, a comprehensive evaluation of ocr tools, particularly for low-resource languages and scripts, is still an open problem. moreover, there is little-to-no understanding of the downstream effect that recognition errors will have on the data augmentation techniques that make use of high-quality monolingual data, such as the methods that low-resource language translation typically relies upon. in this paper, we pose the question of what is the minimum level of ocr quality needed for ocrextracted monolingual text to be useful for machine translation, particularly in low-resource scenarios. to this end, in this work: (i) we create and release an ocr benchmark, ocr4mt, first of its kind, based on real and synthetic data, enriched with noise, for 60 low-resource languages in low resource scripts; (ii) we evaluate commercial and research state-of-the-art ocr models on our benchmark, analyse their performance and extract their common errors for many languages; and (iii) we investigate how the most frequent ocr errors impact machine translation performance and determine what is the minimum level of ocr quality needed for monolingual data to be useful for machine translation. from our results, we observe that the best avail-able ocr systems work well on latin scripts and perform significantly worse on non-latin and non-european scripts (e.g., perso-arabic, khmer). our findings also show that monolingual data from ocr is a valuable source of data for improving machine translation for low resource languages, paving the way for future research on data augmentation for machine translation based on monolingual data extracted from ocr-ed documents. ## related work despite extensive progress, machine translation for low resource languages is still an unsolved problem. this is mainly due to two different aspects: model architecture and lack of training data. in our work we focus on addressing the latter aspect. one effective method to increase training data is to augment the parallel training corpus with backtranslations of target language sentences @xcite . there are large collections of unexplored scanned documents (i.e., pdfs) and images in low resource languages, that can be used as monolingual data for backtranslation, such as online repositories of books 2 or online archives 3 . works like @xcite or @xcite also acknowledge that textual data for most lowresource languages often exists in formats that are not machine-readable, such as paper books and scanned images. they address the task of extracting text from these resources and create benchmark datasets of transcriptions for several endangered languages: ainu, griko, yakkha @xcite and shipibo-konibo, ashaninka, yanesha, yine @xcite . a summary of current benchmarks and data resources for lowresource languages in our research can be applied on large data resources of endangered and low resource languages, such as ailla 4 or elar 5 . @xcite find that endangered language linguistic archives contain thousands of scanned documents -the archive of the indigenous languages of latin america (ailla) contains around 10,000 such documents and the endangered languages archive (elar) has around 7,000. @xcite find that endangered language documents often contain a translation into another (usually highresource) language. multilingual documents represent the majority in the archives they examined: ailla contains 4,383 scanned documents with bilingual text and 1,246 scanned documents with trilingual text, while elar contains around 5,000 multilingual documents. this monolingual data can be collected using optical character recognition (ocr) tools. however, we don't know what is the quality of ocr tools, particularly for low-resource languages and low resource scripts. we aim to address this problem, by building a benchmark of 60 low resource languages with the goal of testing ocr systems and analyse how their errors impact backtranslation performance. rijhwani et al. 2020a also show how generalpurpose ocr tools such as @xcite are not robust to the data-scarce setting of endangered languages. they address this problem, by developing an ocr post-correction method tailored to ease the training in this datascarce setting. the work most similar to ours is the recent research by @xcite . they also built a pipeline to generate analog synthetic documents on which they run a commercial ocr model and analyse the ocr errors. unlike our work, however, their focus is on improving named entity recognition (ner) accuracy and on only 4 different languages: (english, german) from conll 2003 @xcite and (english, chinese and arabic) from conll 2012 @xcite . our work's novelty consists in providing the first large-scale benchmark of 60 low resource languages and low resource scripts, with the purpose of evaluating ocr performance on each language and it's downstream impact on machine translation. ## ocr4mt benchmark to build a benchmark useful for multiple lowresource languages and low resource scripts, we proposed the use of texts that are freely-available in multiple languages. to this end, we chose the universal declaration of human rights (udhr) database 6 which represents a legal domain, and the flores 101 dataset @xcite which is based on wikipedia. moreover, we chose these datasets because they provide data in many languages, and have plain text we can evaluate ocr models on. our benchmark contains real and artificially-created pdfs 7 . udhr is composed of articles on fundamental human rights to be universally protected and it has been translated into over 500 languages. for each language, udhr contains documents in different formats: plain text, pdf, xml and html. there are currently 460 translations fully converted to unicode and available as text. each document is composed of 30 short articles, on average 3 sentences each. we used the plain text and corresponding pdf files as validation data for the ocr systems. the flores 101 dataset consists of text data: 3,001 sentences extracted from english wikipedia, for 101 languages, covering a variety of different topics and domains. we artificially created pdfs from the text documents by saving/exporting the text documents as pdf. language selection. we select 60 languages which are both in flores 101 and the udhr datasets. we prioritize low resource languages, with low resource scripts. the scripts, together with the corresponding languages present in our benchmark can be seen in annotation process. the udhr data is composed of one document image per language (pdf), and each document contains a preface and around 30 articles. in addition, each document has an accompanying text version. to build the benchmark, we first manually annotate the bounding boxes for each of the 30 pdf documents. using the bounding boxes, we split each document image into individ-6 https://www.unicode.org/udhr/translations.html 7 we call real, the pdfs that had this format originally and we call artificially-created, the pdfs that were originally text documents and were converted into pdf format. we artificially created pdfs in order to increase our benchmark data size, as by applying augmentation techniques (i.e., adding noise) they can resemble the real pdfs. ## ocr evaluation to estimate the impact of recognition errors in downstream tasks, namely machine translation, we perform a black-box evaluation of two sota ocr systems, one commercial and one research. these represent reasonable choices for an non-ocr expert, such as mt practitioners. below, we describe our experimental setup in detail. ## ocr impact in machine translation monolingual data is a valuable resource for machine translation, particularly for data augmentation techniques such as backtranslation. while there is plenty of monolingual data available for a few languages, there is a lack of data for very low resource languages. fortunately, we have observed that there exist collections of monolingual data for low resource languages available as pdfs and images. however, we still do not know whether the quality of the ocr-ed data is good enough to be used for training and improve the performance of a machine translation (mt) model. in this section, we explore the performance of an mt model after being trained on backtranslated ocr-ed (ocr+bt) data. in particular, we explore the setup in which a pre-trained multilingual model is fine-tuned on backtranslated data obtained from ocr-ed monolingual data. we use this setup to understand the cases in which ocr data improves or hurts performance. ## conclusion in this paper, we proposed a new benchmark with real and synthetic data, enriched with noise, for 60 low-resource languages in low resource scripts. we group the 60 languages into groups according to their scripts and location, evaluate sota ocr models on our benchmark and extract their most common errors. we use the sota ocr errors to measure their impact on machine translation models by comparing the mt models fine-tuned with ocr-ed data with pre-trained mt models and mt models fine-tuned with initial/non ocr-ed data. our most important takeaway is that ocr-ed monolingual data improves machine translation (mt) through backtranslation. this augmentation is robust to most types of errors, except replacements, and in general most current ocr models produce good enough recognition to be able to train mt models, with the exception of a few scripts like perso arabic. our work paves the way for future research on data augmentation for machine translation based on ocr documents. the scripts to download and process the benchmark introduced in this paper are available at https://github.com/facebookresearch/flores .