E fficient OCR : An Extensible, Open-Source Package for Efficiently Digitizing World Knowledge (2023)

Abstract: Billions of public domain documents remain trapped in hard copy or lack an accurate digitization. Modern natural language processing methods cannot be used to index, retrieve, and summarize their texts; conduct computational textual analyses; or extract information for statistical analyses, and these texts cannot be incorporated into language model training. Given the diversity and sheer quantity of public domain texts, liberating them at scale requires optical character recognition (OCR) that is accurate, extremely cheap to deploy, and sample-efficient to customize to novel collections, languages, and character sets. Existing OCR engines, largely designed for small-scale commercial applications in high resource languages, often fall short of these requirements. EffOCR (EfficientOCR), a novel open-source OCR package, meets both the computational and sample efficiency requirements for liberating texts at scale by abandoning the sequence-to-sequence architecture typically used for OCR, which takes representations from a learned vision model as inputs to a learned language model. Instead, EffOCR models OCR as a character or word-level image retrieval problem. EffOCR is cheap and sample efficient to train, as the model only needs to learn characters’ visual appearance and not how they are used in sequence to form language. Models in the EffOCR model zoo can be deployed off-the-shelf with only a few lines of code and include lightweight models designed for mobile phones that are extremely cheap to deploy. Importantly, EffOCR also allows for easy, sample efficient customization with a simple model training interface and minimal labeling requirements due to its sample efficiency. We illustrate the utility of EffOCR by cheaply and accurately digitizing 20 million historical U.S. newspaper scans, evaluating zero-shot performance on randomly selected documents from the U.S. National Archives, and accurately digitizing a Japanese document collection for which all other OCR solutions failed.

## introduction vast document collections remain trapped in hard copy or lack accurately digitized texts. for example, the u.s. national archives holds approximately 13.28 billion pages of textual records, most of which are in the public domain. 1 these documents are preserved because they are central to the workings of the u.s. government, have long-term research value, or provide valuable information for the public, but working with most of them is costly and time-consuming. the u.s. national archives are not unique: many other countries have national archives with public domain collections numbering in the billions of pages, not to mention state and local archives and libraries. without accurate machine-readable data, modern natural language processing (nlp) tools cannot be used to index, retrieve, and summarize materials; conduct computational textual analyses; or extract information for statistical investigations. public domain texts, if accurately digitized, could also provide massive scale information for training large language models, with no risks of copyright infringement. using optical character recognition (ocr) to digitize public domain collections on a large scale entails several challenges. cost: first, the ocr solution must be cheap to deploy, given document collections whose size numbers in the millions or even billions of pages. commercial engines -as well as large open-source ocr models -fall well short of this requirement. using them to digitize large-scale collections would require astronomical budgets. accuracy: second, digitized texts need to be sufficiently accurate for end users' objectives, which are highly diverse. accuracy can be particularly central for quantitative applications, for which small errors can create major statistical outliers. models for lower resource languages, if they exist, tend to perform much worse than models for high resource settings like english. sample efficient, easy training: documents are highly heterogeneous in terms of their fonts or handwritings, languages, scripts, backgrounds, and artifacts from scanning and aging. there are a diversity of documents for which no existing ocr solution works zero-shot, particularly in low resource languages. yet stakeholders who would like to digitize these documents rarely have familiarity with deep learning frameworks. bringing high quality ocr to low resource settings requires a simple api for training and a sample efficient architecture, with an accessible compute and annotation burden. a diversity of pre-trained and tuneable models: users have diverse accuracy needs, scaling requirements, and budgets. a comprehensive ocr solution would make it easy to compare the accuracy and deployment costs of models of varying sizes so that users can choose the one that best suits their needs for a particular application. to meet these objectives, we developed effocr, an open-source ocr package designed for researchers, libraries, and archives seeking a computationally and sample efficient ocr solution for digitizing diverse document collections. effocr has two key ingredients: 1) a novel ocr architecture and 2) a carefully designed interface to facilitate off-the-shelf ocr usage, customization via model training if necessary, and easy sharing of ocr models. the novel effocr model architecture is treated in detail in @xcite , where we compare accuracy, sample efficiency, and deployment costs to a range of popular ocr engines. in short, ocr predominantly models text recognition as a sequence-to-sequence (seq2seq) problem, in which learned representations from a vision model are taken as inputs to a learned language model. learning how vision embeddings are used in sequence to form language requires large amounts of data. for example, the predominant transformer sequenceto-sequence ocr package @xcite was trained on 684 million text lines using 32 32gb v100 gpu cards. state-of-the-art seq2seq ocr is sample-inefficient to tune and infeasible for users to extend to low resource languages, which may not even have a transformer large language model (llm) that can be used to initialize the model, as language modeling advances are concentrated in less than two dozen languages @xcite . the typical stakeholder working with low resource documents has a minimal budget for training and limited experience with deep learning frameworks, underscoring the need for a much more sample efficient framework with an easy-to-use api. additionally, seq2seq ocr requires autoregressive decoding, which makes inference slower than it would be, all else equal, with parallel decoding. effocr abandons the seq2seq ocr model that predominates in the literature, instead modeling ocr as a word or character level image retrieval problem. effocr first localizes words using highly accurate, scalable object detection methods (ultalytics, 2023; @xcite . recognition is then modeled as a contrastively trained image retrieval problem, where image embeddings of the same character or word have similar representations, regardless of their style. effocr is trained primarily on digital fonts, combined with a modest number of character and word crops from real-world documents. at inference time, characters/words are recognized by computing their nearest neighbor in an offline dictionary of exemplar embeddings created with a digital font. @xcite show, using english, japanese, and polytonic greek benchmarks, that the effocr architecture is accurate, highly sample efficient, cheap to train, and extremely fast to deploy when using backbones designed for mobile phones. to meet the challenges of digitizing largescale and low-resource document collections, the effocr package contains the following components: effocr has been extensively tested. for example, we have used it to cheaply digitize 20 million pages of historical public domain u.s. newspaper scans that are extremely heterogeneous, posting the massive-scale output to hugging face. 2 creating this dataset within our modest budget while meeting accuracy requirements would have been impossible without effocr. we have also examined performance in settings where no existing ocr solutions provide usable output, and tested zeroshot performance on a random selection of u.s. national archive documents, with a model that did not see any similar content during training. tutorials are available at https://effocr.github.io/ . effocr has a gnu general public license. it is being actively maintained and crowd-sourcing of annotations to expand the pre-trained model zoo to other languages and settings, including handwriting, is underway. the rest of this paper is organized as follows. section 2 briefly compares effocr to existing, popular ocr solutions. section 3 describes the key features of the ocr package, and section 4 examines several use cases: using effocr to digitize 20 million historical newspaper scans, using effocr zero-shot on randomly selected collections from the u.s. national archives, and using effocr to digitize a historical japanese publication for which all existing ocr solutions fail. finally, section 5 discusses the limitations of the effocr package. ## comparisons to other ocr engines there is a vast literature on ocr. of primary interest here are widely used ocr softwares, which are the most plausible alternatives to effocr. effocr-as the name suggests -is tailored towards applications requiring computational or sample efficiency. @xcite conduct detailed experiments comparing the effocr architecture to other widely used solutions, considering accuracy, sample efficiency, and computational efficiency. we refer the interested reader to that paper for details, summarizing the two key themes that emerge here. customization is highly relevant: as the preponderance of researchers still using data entry firms suggests, sometimes no existing ocr solution provides acceptable accuracy. for typewritten japanese documents from the mid-20th century, that are of considerable relevance to studying japan's remarkable 20th century growth performance, @xcite show that the best performing engine (baidu, the leading commercial ocr for asian languages) gets over half of characters wrong. the widespread failure of ocr to provide acceptable results is also evidenced by a large post-ocr error correction literature (e.g., lyu et al. (2021); nguyen et al. (2021); van strien. et al. (2020 ## applications scalability: we have tested the utility of effocr with various real-world applications. in the first application, we cheaply and accurately digitized 20 million newspaper page scans from library of congress's chronicling america collection @xcite . the resulting dataset, american stories, is available for download on hugging face. 6 . figure 1 illustrates why this is a challenging task: newspapers are extremely heterogeneous in their fonts and image quality. @xcite provide a detailed analysis of the quality of the resulting text dataset. we first trained character effocr using synthetic data plus a labeled set of 291 newspaper lines @xcite , created in a couple of hours. we then bootsrapped word level annotations by creating them with the character level effocr model, filtering out lines with a high non-word rate. with effocr, combined with layout analysis using layout parser, we could digitize the dataset with a @xmath023 million usd. trocr base, the most accurate open-source ocr, would have exceeded our budget by a factor of nearly 50. zero-shot performance: second, we show that our english lightweight word-level model has strong zero-shot performance on randomly selected document collections from the u.s. national archives. this model saw only newspapers in training, to test true zero shot performance. we selected a single textline from each of 300 random documents from separate national archive record groups. effocr achieved a 11.2% cer on the diverse collection, compared with a 11.8% cer from tesseract (best), a 12.1% cer from easy-ocr, and a 51% cer from trocr (small), which appeared to struggle with blurry and partially obscured text. we suspect the results could be significantly improved by including a random sample of documents from the national archives in training, to broaden the set of real world documents that the model is exposed to. all open-source models performed significantly worse than gcv (1.2% cer), but as discussed earlier cost concerns presently preclude its use at scale. despite being engineered for low-resource, few-shot contexts, effocr remains competitive in high-resource, zero-shot situations. low resource settings: finally, we use effocr to digitize historical japanese firm level records for vertically written japanese documents (teikoku @xcite , where the best available solution (from baidu ocr) mispredicts over half of characters. we use the evaluation set in @xcite , which consists of randomly selected segments that were double labeled. using a training set of 898 labeled table cells, we achieve a cer of 0.7%, 80 times more accurate than the best existing solution. as a result, we are able to study a variety of questions about japan's remarkable growth performance that would have been impossible to examine without effocr. to further examine the limits of sample efficiency, we calculate the character classification error when the (character) model only sees one (or up to 5) labeled character(s) for each of the characters that appear in the training set, which comprise 77% of the characters in the test set. this results in character classification errors of 13.4% and 2.0% respectively. while the model does clearly benefit from seeing multiple crops of characters that appear frequently, this illustrates viable few shot performance. currently, the effocr model zoo has pre-trained models supporting typewritten english, japanese, and polytonic greek. over the coming months, we will be crowd-sourcing annotations (including handwriting) from package users and colleagues. we will use them, along with digital fonts, to pretrain additional models. in addition, users are encouraged to contribute their models. effocr does not currently support handwriting. we started with typewritten documents because there are billions of public domain typeface documents that are of considerable interest to researchers and the general public. we are planning to expand the model zoo to include handwriting and users have already offered to contribute annotations. synthetic handwriting generators, e.g. @xcite , can provide extensive data for pre-training for scripts that they support, analogous to the use of digital fonts for typeface documents. we will make synthetic handwriting datasets available so that package users can also use them for training their own custom models. as with the recognizer, other standard training parameters are exposed. in particular, adjusting the image input shape may be valuable for particularly long or short lines. hyperparameters and training procedure used to generate models listed in the model zoo (table s -1) are listed in @xcite .