Cleaning Dirty Books: Post- OCR Processing for Previously Scanned Texts (2021)

Abstract: Substantial amounts of work are required to clean large collections of digitized books for NLP analysis, both because of the presence of errors in the scanned text and the presence of duplicate volumes in the corpora. In this paper, we consider the issue of deduplication in the presence of optical character recognition (OCR) errors. We present methods to handle these errors, evaluated on a collection of 19,347 texts from the Project Gutenberg dataset and 96,635 texts from the HathiTrust Library. We demonstrate that improvements in language models now enable the detection and correction of OCR errors without consideration of the scanning image itself. The inconsistencies found by aligning pairs of scans of the same underlying work provides training data to build models for detecting and correcting errors. We identify the canonical version for each of 17,136 repeatedly-scanned books from 58,808 scans. Finally, we investigate methods to detect and correct errors in single-copy texts. We show that on average, our method corrects over six times as many errors as it introduces. We also provide interesting analysis on the relation between scanning quality and other factors such as location and publication year.

## introduction the hathitrust and gutenberg corpora are critical resources for literary analysis and nlp research, providing legal access to tens of thousands of texts for research purposes. both were constructed from scanned texts, with manual correction in the case of the gutenberg corpus. these efforts for the gutenberg project have begun as early as the 1970s when there was a foreseeable need to digitize open-domain books. furthermore, the hathitrust dataset also was constructed from a compilation of books from multiple libraries from universities and states, and adds a substantial amount of extra content. ## alignment methods we focus on a collection of books from the hathitrust dataset of which we have 96,635. our first task was to find duplicate books and to align the content such that we could find the differences. ## ocr errors in single-copy texts we now consider ocr errors for single copy texts. in this setting, we cannot use any alignment technique as the books live in isolation. for this case, we train models for both ocr error detection and correction using the 17,136 sets of duplicate books and their alignments. all models were run on a compute server with 2.30 ghz cpu and teslav100 gpu. no hyperparameter tuning was done on any models; default values were run for all models. ## conclusion in this paper, we demonstrated how to improve the quality of an important corpus of digitized books, by correcting transcription errors that generally occur due to ocr. our key idea to provide ground truth was to identify thousands of duplicate books (titles scanned in different locations and of uncertain quality). we aligned them at the token level to find where the differences occur, and used modern language models to determine which book copy is of higher quality. additionally, we used this alignment as training data to train a model for correcting ocr errors in singleton books (books without any duplicates). we showed that our methods correct over six times as many errors as it introduces, and also demonstrate that our errors tend to be semantically sensible. through our efforts, we produced a substantially better version of over 50,000 distinct titles from the hathitrust and guttenberg as a foundation for future nlp research as well as show some interesting analysis from post-ocr processing.