Bootstrapping AI : Interdisciplinary Approaches to Assessing OCR Quality in E nglish-Language Historical Documents (2025)

Abstract: New LLM-based OCR and post-OCR correction methods promise to transform computational historical research, yet their efficacy remains contested. We compare multiple correction approaches, including methods for “bootstrapping” fine-tuning with LLM-generated data, and measure their effect on downstream tasks. Our results suggest that standard OCR metrics often underestimate performance gains for historical research, underscoring the need for discipline-driven evaluations that can better reflect the needs of computational humanists.

## introduction optical character recognition (ocr) has long posed challenges for large-scale computational analysis of historical documents, particularly those with difficult-to-parse text. while vision-to-text models such as chatgpt-4o or llama 3.2 increasingly outperform conventional methods like tesseract, they remain financially or technically out of reach for many humanities institutions if used at scale. an alternative is to use generative ai for correcting baseline ocr output. while conventional nlp metrics show mixed levels of improvement from large language model (llm)based correction, historians report dramatic quality gains in anecdotal tests @xcite . this discrepancy prompted two central questions: (1) how can we measure post-ocr correction in ways aligned with historians' needs? (2) do these methods substantially improve ocr sufficiently to alter downstream tasks? through experiments on multiple post-ocr correction strategies, including fine-tuning with both human and llm-generated transcription data, we evaluate standard metrics and explore discipline-specific alternatives. ## similar work improving ocr accuracy remains a widelyacknowledged critical task for historical text anal-ysis @xcite @xcite . recent developments in large language models have spurred varied approaches to this issue @xcite . one body of work has focused on the possibility of using nonspecialized instruct models for improving ocr quality. testing a wide variety of such models, @xcite find little improvement and even mild degradation in output quality. however, studies such as @xcite ; @xcite ; bang (2024) challenge this conclusion, reporting substantive benefits, especially for printed, english-language texts from the last several centuries. the source of this discrepancy remains unclear-potential explanations include changes within the models or differences in the application of evaluation metrics. @xcite expand on such research by incorporating extensive approaches for error identification and analysis in relation to historical spanish, while @xcite experiments with prompt context, laying out a useful approach to evaluating downstream impact. another approach, including work by @xcite , @xcite , @xcite , and @xcite , has focused on developing methods for fine-tuning local models. such research, which is often applied to lower-resourced languages or older historical texts, frequently grapples with a paucity of the gold-standard transcriptions necessary for training, requiring engagement with synthetic data that our materials enabled us to avoid. finally, a set of more recent papers, including @xcite , @xcite and @xcite have explored the potential efficacy of vision-to-text models for historical ocr on both typed and hand-written documents. ## conclusion while these experiments show the efficacy of some forms of llm-based ocr correction for improving ocr accuracy, our methods of fine-tuning bart and byt5 models produced results close to, but not quite as good as, those of current stateof-the-art models. they did, however, demonstrate that fine-tuning with silver-quality ocr data produces results comparable to results with smaller quantities of human-generated text. this finding presents a path forward for lower-resourced researchers and institutions, while casting doubt on the continued necessity of large-scale human transcription through crowdsourcing. trade-offs exist between computational cost and accuracy, but the choice over those trade-offs should be made by informed researchers with a clear sense of their specific downstream tasks. with more optimized training techniques, we hope our results can be further refined, closing the gap between our low-cost method and the flagship multimodal large language models. in addition, our comparison between the traditional metrics of cer/wer and a "historically specific" focus on precision indicates the importance of multiple, discipline-specific (or at least task-specific) frameworks for evaluating ocr quality. according to edit-distance-based metrics, even the best of our fine-tuned models underperformed the tesseract baseline. however, when considering the impact of these same changes on downstream tasks, both ner precision and embedding similarity showed improvements, suggesting a significant semantic change not adequately captured by cer/wer. ultimately, we believe that this work demonstrates the importance of a robust interdisciplinary conversation on how-and for what-nlp is being used within the humanities.