PHD : Pixel-Based Language Modeling of Historical Documents (2023)

Abstract: The digitisation of historical documents has provided historians with unprecedented research opportunities. Yet, the conventional approach to analysing historical documents involves converting them from images to text using OCR, a process that overlooks the potential benefits of treating them as images and introduces high levels of noise. To bridge this gap, we take advantage of recent advancements in pixel-based language models trained to reconstruct masked patches of pixels instead of predicting token distributions. Due to the scarcity of real historical scans, we propose a novel method for generating synthetic scans to resemble real historical documents. We then pre-train our model, PHD, on a combination of synthetic scans and real historical newspapers from the 1700-1900 period. Through our experiments, we demonstrate that PHD exhibits high proficiency in reconstructing masked image patches and provide evidence of our model’s noteworthy language understanding capabilities. Notably, we successfully apply our model to a historical QA task, highlighting its usefulness in this domain.

## introduction recent years have seen a boom in efforts to digitise historical documents in numerous languages and sources @xcite @xcite , leading to a transformation in the way historians work. researchers are now able to expedite the analysis process of vast historical corpora using nlp tools, thereby enabling them to focus on interpretation instead of the arduous task of evidence collection @xcite . the primary step in most nlp tools tailored for historical analysis involves optical character recognition (ocr). however, this approach poses several challenges and drawbacks. first, ocr *this paper shows dataset samples that are racist in nature strips away any valuable contextual meaning embedded within non-textual elements, such as page layout, fonts, and figures. 1 moreover, historical documents present numerous challenges to ocr systems. this can range from deteriorated pages, archaic fonts and language, the presence of nontextual elements, and occasional deficiencies in scan quality (e.g., blurriness), all of which contribute to the introduction of additional noise. consequently, the extracted text is often riddled with errors at the character level @xcite , which most large language models (llms) are not tuned to process. token-based llms are especially sensitive to this, as the discrete structure of their input space cannot handle well the abundance of out-of-vocabulary words that characterise ocred historical documents @xcite . therefore, while llms have proven remarkably successful in modern domains, their performance is considerably weaker when applied to historical texts @xcite . finally, for many languages, ocr systems either do not exist or perform particularly poorly. as training new ocr models is laborious and expensive @xcite , the application of nlp tools to historical documents in these languages is limited. this work addresses these limitations by taking advantage of recent advancements in pixel-based language modelling, with the goal of constructing a general-purpose, image-based and ocr-free language encoder of historical documents. specifically, we adapt pixel @xcite , a language model that renders text as images and is trained to reconstruct masked patches instead of predicting a distribution over tokens. pixel's training methodology is highly suitable for the historical domain, as (unlike other pixel-based language models) it does not rely on a pretraining dataset given the paucity of large, high-quality datasets comprising historical scans, we pretrain our model using a combination of 1) synthetic scans designed to resemble historical documents faithfully, produced using a novel method we propose for synthetic scan generation; and 2) real historical english newspapers published in the caribbeans in the 18th and 19th centuries. the resulting pixelbased language encoder, phd (pixel-based model for historical documents), is subsequently evaluated based on its comprehension of natural language and its effectiveness in performing question answering from historical documents. we discover that phd displays impressive reconstruction capabilities, being able to correctly predict both the form and content of masked patches of historical newspapers ( §4.4). we also note the challenges concerning quantitatively evaluating these predictions. we provide evidence of our model's noteworthy language understanding capabilities while exhibiting an impressive resilience to noise. finally, we demonstrate the usefulness of the model when applied to the historical qa task ( §5.4). to facilitate future research, we provide the dataset, models, and code at https://gith ub.com/nadavborenstein/pixel-bw . ## training a pixel-based historical lm we design phd to serve as a general-purpose, pixel-based language encoder of historical documents. ideally, phd should be pretrained on a large dataset of scanned documents from various historical periods and different locations. however, large, high-quality datasets of historical scans are not easily obtainable. therefore, we propose a novel method for generating historical-looking artificial data from modern corpora (see subsection 4.1). we adapt our model to the historical domain by continuously pretraining it on a medium-sized corpus of real historical documents. below, we describe the datasets and the pretraining process of the model. ## conclusion in this study, we introduce phd, an ocr-free language encoder specifically designed for analysing historical documents at the pixel level. we present a novel pretraining method involving a combination of synthetic scans that closely resemble historical documents, as well as real historical newspapers published in the caribbeans during the 18th and 19th centuries. through our experiments, we observe that phd exhibits high proficiency in reconstructing masked image patches, and provide evidence of our model's noteworthy language understanding capabilities. notably, we successfully apply our model to a historical qa task, achieving a binary accuracy score of nearly 75%, highlighting its usefulness in this domain. finally, we note that better evaluation methods are needed to further drive progress in this domain.