Lacuna Reconstruction: Self-Supervised Pre-Training for Low-Resource Historical Document Transcription (2022)

Abstract: We present a self-supervised pre-training approach for learning rich visual language representations for both handwritten and printed historical document transcription. After supervised fine-tuning of our pre-trained encoder representations for low-resource document transcription on two languages, (1) a heterogeneous set of handwritten Islamicate manuscript images and (2) early modern English printed documents, we show a meaningful improvement in recognition accuracy over the same supervised model trained from scratch with as few as 30 line image transcriptions for training. Our masked language model-style pre-training strategy, where the model is trained to be able to identify the true masked visual representation from distractors sampled from within the same line, encourages learning robust contextualized language representations invariant to scribal writing style and printing noise present across documents.

## introduction document transcription is the task of converting images of handwritten or printed text into a symbolic form suitable for indexing, searching, and computational analysis. 1 historical documents, whether they were (re)produced via handwriting or the early printing press, confound current statistical document transcription models due to (1) extremely varied style and content across domains, 1 we use the generic term document transcription to refer to both the task of optical character recognition (ocr), which is typically reserved for printed documents, and handwritten text recognition (htr) for manuscripts. figure 1 : example page image crops from an islamicate manuscript dated to 1842 (top, ref: leiden or. 669), showcasing its dense, visual complexity with extensive marginalia, and printed proceedings of london's old bailey courthouse (bottom, c. 18 th century) @xcite . first, historical printed documents, such as books produced from early modern england (c. 16 th -18 th centuries; bottom of fig. 1 ), use nonstandardized spacing and fonts @xcite and can contain code-switching that confuses language models @xcite . however, this variation pales in comparison to their handwritten counterparts. for instance, pre-modern islamicate manuscripts (i.e., persian and arabic handwritten documents from c. 7 th -19 th centuries; top of fig. 1 ), differ in script family, scribal handwriting style, and symbol inventory/vocabulary. as a result, a large degradation in performance is observed when evaluating htr models on unseen manuscripts @xcite . production and imaging noise also present a problem for historical document transcription models. whether it be uneven inking from a printing press, inconsistent text baselines, or holes resulting from insect damage to ancient pages, techniques must be designed to cope with the noise @xcite . while neural networks have a demonstrated capability to model complex data distributions, they typically require large amounts of supervised training data to do so, which is infeasible for historical documents. unsupervised, non-neural transcription models with fewer parameters alleviate the need to create labeled data @xcite , but struggle with complex handwriting variation. for islamicate manuscripts, ground truth transcription often requires paleography experts to decipher the ancient writing systems as they appear in each scribal writing style. in this paper, we propose a self-supervised learning framework designed to overcome these three challenges presented by historical documents. inspired by the astounding success of self-supervised pre-training techniques for masked language modeling (mlm) in nlp @xcite , visual models @xcite , and speech recognition @xcite , our approach pre-trains a neural text line-image encoder by learning to distinguish masked regions of unlabeled line images from other distractor regions. specifically, our contribution is the following: in doing so, we argue that our approach to pretraining implicitly incentivizes the model to discover and encode discrete character classes in its internal representations, while ignoring style differences occurring in lines using different fonts or languages, or authored by other scribes. islamicate htr while machine recognition of handwritten, historic english/german documents can range from 5-12% character error rate (cer) on a sufficient amount of in-sample manuscript training data @xcite , performance on arabic-script languages is much more challenging, leading to substantially higher 2 207 cer. pre-modern islamicate manuscripts (i.e., persian and arabic handwritten documents from c. 7 th -19 th centuries), often differ in script family, scribal handwriting style, and symbol inventory/vocabulary. in the top of figure 1 , we present an extreme example of some of the problematic visual variation that can be observed. even a model trained in a supervised fashion on such a complex document sees a large degradation in performance when evaluating htr models on unseen manuscripts @xcite . until recently, ocr performance on arabic-script printed texts was still poor, typically above 25% cer (alghamdi and teahan, 2017), which is too high for downstream users (i.e., researchers and librarians). recent studies involving islamicate manuscripts found that state-of-the-art systems are only able to achieve 40 to mid-20% cer using proprietary software (e.g., google cloud vision, rdi, transkribus) @xcite @xcite . however, results from these studies only report in-domain performancean unrealistic scenario where considerable amounts of labeled data can be obtained to enable both training and testing on the same manuscript. in contrast, out-of-domain performance tends to suffer considerably, supported by @xcite 's study of neural ocr for printed arabic-script documents. our work aims to address such performance issues for both in-domain and out-of-domain islamicate htr settings by learning general, content-rich pre-trained language representations from large amounts of heterogeneous unlabeled data. historical ocr closely related to manuscript transcription, ocr is another task involving language recognition from images. however, ocr operates on documents that have been printed by a machine with regular, re-used character fonts exhibiting much less superficial glyph variation than human handwriting. ocr is far from a solved problem in the case of documents printed on early modern (c. 16 th -18 th centuries; see bottom of fig. 1 ), movable-type printing presses, where humans would manually set metal type casts with non-standard spacing and fonts @xcite . in this setting, inking noise and historical font shapes confuse ocr models trained on modern, computer-generated documents @xcite . @xcite 's ocular explicitly uses a generative probabilistic model inspired by historical printing processes to model such noise. later work has extended it to handle more typesetting noise @xcite , and produce both diplomatic and normalized transcriptions @xcite . separately, ocr post-correction models have been proposed to resolve ocr outputs in historical documents @xcite and other low-resource settings @xcite . in contrast, our approach pre-trains the visual language recognition model's encoder, which produces better contextualized representations in order to reduce the amount of errors the model itself makes. unlike ocular, our proposed method does not use a language model and is not fully unsupervised as we require 1-3 pages of transcribed data for learning to transcribe during fine-tuning. ## approach when human readers encounter a lacuna, a blank information gap in a portion of a book or manuscript, they must infer its latent meaning using nearby context like in a cloze test @xcite . we argue that the most useful information for inference lies in the ability to reason about the identities of the missing characters in the lacuna using the identities of the surrounding characters. indeed, mlm-style pre-training techniques are also motivated by the idea of the cloze test, and recent research indicates that language representations learned through the prediction of missing content using surrounding sentential context are useful for many downstream tasks @xcite @xcite . our approach combines the ideas of lacuna inference and masked pre-training to provide a useful learning signal for downstream historical document transcription, a setting with massive digitized collections but few transcribed examples. specifically, we introduce a self-supervised pretraining method that randomly masks lacuna-like regions of document line images and learns to reconstruct them by distinguishing them from nearby line image segments, or foils. while lacuna can be reconstructed in a generative way, we find that a discriminative contrastive loss works better in practice. by leveraging a diverse set of unlabeled data 3 208 figure 2: our proposed two-stage approach for low-resource document transcription first pre-trains a line image encoder using a self-supervised contrastive loss on unlabeled data (left), followed by a fine-tuning phase, in which the pre-trained encoder learns to transcribe 1-3 pages of supervised data using a ctc loss (right). for pre-training, the model is forced to infer the identities of masked text regions in the presence of scribal writing variation or typesetting noise ubiquitous in historical documents. in the next sections, we describe our model/masking strategy in detail. ## datasets in this section, we describe the unlabeled pretraining and labeled fine-tuning/testing datasets used in our experiments. representative line images from five of the datasets are exhibited in figure 3 . ## experiments in this section, we describe our experimental setup, including architectural details and hyperparameters for the neural line image encoder, pre-train/finetune specifics, dataset splits, and the baseline systems we compare against. ## results in this section, we present document transcription results for both islamicate manuscripts and early modern english works introduced in section 4. we compare performance against supervised and unsupervised prior work, and investigate the impact of pre-training/fine-tuning dataset sizes. ## conclusion in this paper, we proposed a two-phase pretrain/fine-tune approach for document transcrip-tion and applied it to historical documents in low-resource settings. our pre-training strategy, inspired by reconstructing missing information, or lacuna, in documents uses hundreds of thousands of unlabeled line images to learn rich visual language representations. after supervised fine-tuning on tens of transcribed line images, we showed large character error rate reduction on islamicate manuscripts exhibiting major script and style variation and we improved over several stateof-the-art ocr systems on early modern english printed works. we estimate that our approach could save human annotators significant amounts of time and enable more distant readings of library collections.