Efficient OCR for Building a Diverse Digital History (2024)

Abstract: Many users consult digital archives daily, but the information they can access is unrepresentative of the diversity of documentary history. The sequence-to-sequence architecture typically used for optical character recognition (OCR) – which jointly learns a vision and language model – is poorly extensible to low-resource document collections, as learning a language-vision model requires extensive labeled sequences and compute. This study models OCR as a character level image retrieval problem, using a contrastively trained vision encoder. Because the model only learns characters’ visual features, it is more sample efficient and extensible than existing architectures, enabling accurate OCR in settings where existing solutions fail. Crucially, it opens new avenues for community engagement in making digital history more representative of documentary history.

## introduction digital texts are central to the study, dissemination, and preservation of human knowledge. tens of thousands of users consult digital archives daily in europe alone @xcite , yet billions of documents remain trapped in hard copy in libraries and archives around the world. these documents contain extremely diverse character sets, languages, fonts or handwriting, printing technologies, and artifacts from scanning and aging. converting them into machine-readable data that can power indexing and search, computational textual analyses, and statistical analyses -and be more easily consumed by the public -requires highly extensible, accurate, efficient tools for optical character recognition (ocr). current predominant ocr technology -developed largely for small-scale commercial applications in high resource languages -falls short of these requirements. ocr is typically modeled as a sequence-to-sequence (seq2seq) problem, with learned embeddings from a neural vision model taken as inputs to a learned neural language model. the seq2seq architecture is challenging to extend and customize to novel, lower resource settings @xcite , because training a visionlanguage model requires a vast collection of labeled image-text pairs and significant compute. this study shows that on printed japanese documents from the 1950s, the best performing existing ocr mis-predicts over half of characters. poor performance is widespread, spurring a large post-ocr error correction literature @xcite @xcite and skewing digital history towards limited settings that are not representative of the diversity of documentary history. this study develops a novel, open source ocr architecture, effocr (efficientocr), designed for researchers and archives seeking a sampleefficient, customizable, scalable ocr solution for diverse documents. effocr combines the simplicity of early ocr systems, such as tauschek's 1920s reading machine, with deep learning, bringing ocr back to its roots: the optical recognition of characters. deep learning-based object detection methods are used to localize individual characters or words in the document image. character (word) recognition is modeled as an image retrieval problem, using a vision encoder contrastively trained on character (word) crops. effocr performs accurately, even when using lightweight models designed for mobile phones that are cheap to train and deploy. using documents that are fundamental to studying japan's remarkable 20th century economic growth, the study shows effocr can provide a sample efficient, highly accurate ocr architecture for contexts where all current solutions fail. effocr's blend of accuracy and efficient runtime also makes it attractive for digitizing massive-scale collections in high resource languages, which the study illustrates with library of congress's collection of historical u.s. newspapers (library of congress, 2022). effocr has been used to cheaply and accurately digitize the over 20 million page scans in this collection @xcite . in principle, contextual understanding could be extremely valuable to ocr, but in practice state-ofthe-art transformer seq2seq models are extremely costly to train, expensive to deploy, and do not exist for lower resource languages, with advances concentrated in a handful of languages. this study shows that taking a step back from seq2seq models unlocks massive gains in sample efficiency. researchers, with a modest number of annotations and modest compute, can train their own ocr for settings where all existing solutions fail, using our user-friendly effocr open-source package. new characters specific to a setting can also be added at inference time -since they don't need to be seen in sequence during training -important for contexts such as archaeology and certain historical applications where new characters are regularly encountered. these features facilitate making digital history more representative of documentary history. ## methods modern ocr overwhelmingly uses deep neural networks -either a convolutional neural network (cnn) or vision transformer (vit) -to encode images. the representations created by passing an input image through a neural encoder are then decoded to the associated text. figure 1 underscores two fundamental differences between effocr and seq2seq. first, sequence-to-sequence architectures typically require line level inputs, and individual characters or words are not localized; rather, images or their representations are divided into fixed size patches. in contrast, effocr localizes characters and words using modern object detection methods @xcite via the "localizer" module. second, seq2seq sequentially decodes the learned image representations into text using a learned language model that takes the image representations as inputs. in contrast, effocr recognizes text by using contrastive training @xcite to learn a meaningful metric space for character or word-level ocr. a vision encoder, the "recognizer" module, projects crops of the same character (word) -regardless of style -nearby, whereas crops of different characters (words) are projected further apart. effocr thus generates full lines of text in the following way: (1) the localizer produces bounding boxes for characters (words) in the input image; (2) these localized character (word) images are embedded with the recognizer; (3) the character (word) embeddings are decoded to machine-readable text in parallel by retrieving the label of their nearest neighbor in an offline index of exemplar character (word) embeddings, created by rendering labeled character (word) images with a digital font; and (4) the bounding boxes from the localizer are re-used to robustly infer the order of the machine-readable characters (words) and the presence of white spaces. embedding distances are computed using cosine similarity with a facebook artificial intelligence similarly search (faiss) backend @xcite . the vision embeddings alone are sufficient to infer text since they represent characters -not text lines like in seq2seq -and hence decoding them does not require a language model with learned parameters. this study develops both character and word level ocr models, with the former being more suitable for character-based languages and the latter more suitable for alphabet-based languages. when modeling ocr as a word level problem, effocr defaults to character level recognition if the distance between a word crop embedding and the nearest embedding in the offline dictionary of word embeddings is below a threshold cosine similarity. this is important, as hyphenated words at the end of lines, acronyms, proper nouns, and antiquated terms often make it infeasible to construct a comprehensive word dictionary. effocr is trained on digital font renders, along with a modest number of labeled crops from target datasets. the recognizer is trained using the supervised contrastive ("supcon") loss function @xcite , a generalization of the in-fonce loss @xcite that allows for multiple positive and negative pairs for a given anchor. we use the "outside" supcon loss formulation, as implemented in pytorch metric learning (musgrave et al., 2020), where τ is the temperature, i indexes a sample in a "multiviewed" batch (in this case multiple fonts/augmentations of characters with the same identity), p (i) is the set of indices of all positives in the multiviewed batch that are distinct from i, a(i) is the set of all indices excluding i, and z is an embedding of a sample in the batch. to create training batches for the recognizer, ef-focr uses a custom m per class sampling algorithm without replacement. this metric learning batch sampling algorithm also implements batching and training with hard negatives, where the negative samples in a batch are selected to be se-mantically close to one another, and thus contrasts made between anchors and hard negatives may be especially informative. different vision encoders can be used interchangeably for the effocr character localizerwhich locates the character/word crops -and recognizer -which learns a metric space for these crops. three models are considered for character level effocr: a vision transformer model (effocr-t base) with xcit (small) @xcite for both the localizer and recognizer, a convolutional base model (effocr-c base) with convnext (tiny) @xcite for both the localizer and recognizer, and a convolutional small model (effocr-c small), which uses lightweight architectures designed for mobile phones -yolov5 (small) @xcite for the localizer and mo-bilenetv3 (small) for the recognizer. for word level ocr, we develop effocr-word (small), which uses the same lightweight architectures as effocr-c (small). effocr-word (small) defaults to effocr-c (small) when the cosine similarity between a word crop embedding and the nearest embedding in the offline word embedding dictionary is below 0.82, a hyperparameter that is (like all model hyperparameters) tuned on the validation set. the base models use a two-stage object detector for character localization, specifically a cascade r-cnn @xcite , whereas the small models use one-stage object detection for faster speed @xcite . the supplementary materials describe the effocr architecture and training recipes with no detail spared and evaluate models using alternative vision transformer encoders. ## related literature effocr's architecture draws inspiration from metric learning methods for efficient image retrieval @xcite , joining a recent literature on self-supervision through simple data augmentation for image encoders @xcite @xcite . the closest frameworks to effocr in their overall design are the original ocr conceptualizations, such as tauschek's 1920s reading machine, which used human engineered features to recognize localized characters. more recently, charnet @xcite , developed for scene text (not documents), uses separate convolutional networks for dense classification and regression at a single scale, outputting a character class and bounding box at every spatial location, and then aggregates this information with confidence scores to make final predictions. effocr in contrast deploys widely used, highly optimized object detection methods to localize characters and then feeds character crops to a contrastively trained recognizer. 1 other ocr frameworks -that are widely used, have state-of-the-art performance, or provide an instructive architectural contrast with effocr -are described in section 5, which introduces the comparisons that we will make. ## training and evaluation datasets evaluating effocr requires benchmark datasets that are representative of the diversity of documentary history. traditional ocr benchmarks focus on commercial applications like receipts @xcite -and sota ocr systems evaluate on these data -which are not relevant to digital history. instead, the study draws on the literature on historical image datasets @xcite . first, it uses documents from historical japan that can elucidate fundamental questions that have been understudied due to a lack of digital data, such as the drivers of japan's rapid transformation from a poor agrarian economy to a wealthy industrialized nation. horizontally and vertically written tabular data -providing rich information on japanese firms and their personnel -are drawn from two 1950s publications (jinji @xcite . a 1930s prose publication providing detailed biographies of tens of thousands of individuals (jinji @xcite ) is also examined. these texts could use over 13,000 kanji characters. the second context is library of congress's chronicling america (locca) collection, which contains over 19 million historical public domain newspaper page scans. this collection is highly diverse, as shown in figure 2 . library of congress provides an ocr, but the quality is low @xcite . there is a large literature studying historical newspapers at scale, which overwhelmingly uses keyword search and does not unlock the power of large language models due to poor quality digitization @xcite . locca elucidates how effocr: 1) performs in the highest resource setting, english; 2) extensibility across latin and kanji characters, lected because it contains both line-level and word transcriptions. polytonic greek uses five diacritics to notate older greek texts. it is challenging because the diacritics have a similar appearance. the supplemental materials show example documents from all benchmarks. ## measurement and comparisons ocr accuracy is measured using the character error rate (cer), the levenshtein distance between the ocr'ed string and the ground truth, normalized by the length of the ground truth. a cer of 0.5, for instance, translates to mispredicting approximately half of characters. the most widely used ocr engines are commercial products that do not currently support finetuning and have proprietary architectures. the study compares effocr to google cloud vision (gcv) and baidu ocr (popular for asian languages). we include these comparisons because they are relevant to practitioners. we also consider four open source architectures: easyocr's convolutional recurrent neural network (crnn) framework @xcite ), trocr's sequence-to-sequence encoder-decoder transformer (base and small) @xcite ), tesseract's bi-directional lstm, and paddleocr's single vision text recognition (svtr), which also abandons seq2seq, dividing text images into small (non-character) patches, using mixing blocks to perceive inter-and intra-character patterns, and recognizing text by linear prediction @xcite . a large literature has examined a variety of custom-designed ocr systems. we focus on those that either (1) make similar architectural choices (svtr), (2) are considered sota, regardless of architectural choices (trocr), or (3) are very popular (tesseract and easyocr). the pre-trained easyocr, paddleocr, and trocr models are fine-tuned on the same target data as effocr. considerable resources have been devoted to pre-training these models. for example, trocr was pre-trained on 684 million english synthetic text lines. hence, these comparisons elucidate performance when these pre-trained models are further tuned on the target datasets. for a more apples-to-apples comparison, the study examines the accuracy of these architectures when trained from scratch (using a pre-trained checkpoint not trained for ocr, when supported by the architecture) on 8,000 synthetic text lines (like ef-focr) and the same target crops. easyocr and paddleocr do not support vertical japanese, and trocr does not support any japanese. tesseract offered little support for fine-tuning until recently and hence most of its applications have been offthe-shelf, which is this study's focus. all results come from a single model run, with training details provided in the supplemental materials. ## results effocr provides a highly accurate ocr with minimal training data, in contexts where current solutions fail. for vertical japanese tables, the best effocr cer is 0.7% ( the cer (uncased) for the locca newspapers is 1.5%. gcv has the best performance (0.5%), followed by fine-tuned trocr (base) (1.3% cer). the advantage of effocr on englishthe quintessential high resource setting -is its opensource codebase and fast runtime. gcv makes significant layout errors when fed full newspaper page scans, which have complex layouts @xcite , and hence the performance in figure 3 provides representative examples of errors, showing the target crop, the localized crop, and its five nearest neighbors, with the correct prediction highlighted in green. errors tend to occur when the character is illegible or homoglyphic to another character (e.g., o and 0). for example, a 0 in one font can occasionally be indistinguishable from an o in another, an error that would be straightforward to correct in post-processing. the supplementary materials report results from additional encoders, and examine how different architecture and design choices for effocr contribute to its performance. in particular, we notice little difference between the best performing cnn encoders and vision transformer encoders in terms of cer, regardless of language, when holding approximately constant the number of model parameters. this is consistent with an existing literature on the convergent performances of (appropriately modernized) cnns and vision transformers @xcite . effocr outperforms all other architectures that support polytonic greek, including google cloud vision. this illustrates the versatility of the architecture. effocr's parsimonious architecture allows it to learn efficiently. to quantify this, we train different ocr models from scratch using varying amounts of annotated data. all architectures are pre-trained from scratch on 8,000 synthetic text lines, starting from pre-trained checkpoints not customized for ocr when supported by the framework. they are then fine-tuned on the study's benchmark datasets, with varying train splits: 70%, 50%, 20%, 5%, and 0% (using only synthetic data). these exercises are performed for chronicling america and horizontal japanese, as vertical japanese is not supported by the comparison architectures. figure 4 plots the percentage of the benchmark dataset used in training on the x-axis and the cer on the y-axis. on just 99 labeled table cells for japanese and 21 labeled rows for locca (the 5% train split), effocr's cer is around 4%, showing viable few shot performance. the other architectures remain unusable. effocr performs nearly as well using 20% of the training data as using 70%, where it continues to outperform all other alternatives. here, our focus is on the design of bespoke, efficient models for low-resource contexts. one might wish to assess how effocr performs on completely out-of-domain texts. elsewhere, researchers have used the effocr package and effocr-word (small) model trained only on newspapers to process randomly selected, highly diverse documents from the u.s. national archives @xcite . effocr performs similarly to other opensource ocr engines, achieving a cer of 11.2% as compared with a 11.8% cer from tesseract (best), a 12.1% cer from easyocr, and a 51% cer from trocr (small). the sample efficiency of effocr suggests it could be trained to perform well off-the-shelf on diverse archival documents by labeling a small number of samples across a wide range of common historical document types, an effort that could be crowd-sourced. ## discussion indexing, analyzing, disseminating, and preserving diverse documentary history requires community engagement of stakeholders with the requisite fine-grained knowledge of the relevant settings. effocr facilitates this engagement because it is highly extensible to low-resource settings, sampleefficient to customize, and simple and cheap to train and deploy. in contrast, seq2seq is more aligned with the commercial objective of designing a product that is difficult for competitors to imitate. for example, effocr can be trained in the cloud with free student compute credits, whereas trocr required training on a multi-million dollar cluster with 32 32gb v100 cards. lower resource languages may lack the pre-trained language models required to initialize a transformer seq2seq model, and sufficient compute resources are also unlikely to be available. effocr encourages community engagement by integrating the follow features: character/word level: effocr creates semantically rich visual embeddings of individual characters (words), a parsimonious problem. annotators can select which of the most probable predictions from the pre-trained recognizer are correct, potentially using a simple mobile interface, or line level labels can be mapped to the character (word) level once a localizer has been developed. language extensibility: language modeling advances have concentrated around less than two dozen modern languages, out of many thousands @xcite . omitting the language model makes effocr extensible and easy-to-train. to extend effocr to a new language, all one needs are renders for the appropriate character set. additionally, characters do not need to be seen in sequence during training, so new characters can be added at inference time, valuable for archaeological contexts where new characters are regularly discovered. omitting the language model makes it easy to mix scripts, necessary for some languages. the recognizer can also be exposed to characters in training using any desired sequencing. this is not true of multilingual seq2seq training, which leads to many ocr errors with endangered languages @xcite . decoupling localization and recognition: theoretically, localization and recognition (akin to classification) may rely on different features of the image, suggesting modularity @xcite . practically, decoupling allows localization and recognition to use different training sets, economizing on annotation costs since these tasks can require very different numbers of labels depending on the script. it also encourages community innovation and future-proofness, because it simplifies training recipes and makes it straightforward to swap in new localizers or recognizers -including zero-shot models such as @xcite -as the literature advances. scalable: the small effocr models achieve fast cpu inference that can scale cheaply to hundreds of millions of documents. ## reproducibility we release all code and training data used to create effocr. scripts in the public repository exactly reproduce the figures cited above. all other material needed to reproduce these results is detailed in the supplemental materials, including training hyperparmeters. the models in this paper can also be deployed through the open-source effocr python package (cc-by 4.0 license). ## limitations this study does not focus on handwriting due to space constraints, but the approach would be analogous. synthetic handwriting generators, e.g., @xcite , could provide extensive data for pre-training, analogous to this study's use of digital fonts. there are some settings where effocr's framework is not suitable. if large portions of a document are illegible, context is necessary. moreover, the heavy use of ligatures and/or slanting in some character sets and handwriting could lead to more challenging character localization. this challenge is mitigated with the word-level effocr model. ## ethical considerations effocr presents no major ethical concerns. its methods are entirely open source, and its training data are entirely in the public domain. its core functionality, accurately transcribing texts in lowresource settings, is ethically sound. by making it easier to digitize scanned document texts in lowresource settings, it can promote the inclusion of more diverse groups in nlp, social science, and humanities research. its sample and computational efficiency minimizes environmental harm by reducing compute requirements at training and inference time. some applications of effocr could raise ethical flags. we discourage users from applying effocr to copyrighted documents unless the application is protected by fair use. while effocr is a potentially useful tool for studying bias, e.g., through analyses of historical documents, potentially harmful or offensive content transcribed by effocr should not be shared without proper context.