Post-Correction of Historical Text Transcripts with Large Language Models: An Exploratory Study (2024)

Abstract: The quality of automatic transcription of heritage documents, whether from printed, manuscripts or audio sources, has a decisive impact on the ability to search and process historical texts. Although significant progress has been made in text recognition (OCR, HTR, ASR), textual materials derived from library and archive collections remain largely erroneous and noisy. Effective post-transcription correction methods are therefore necessary and have been intensively researched for many years. As large language models (LLMs) have recently shown exceptional performances in a variety of text-related tasks, we investigate their ability to amend poor historical transcriptions. We evaluate fourteen foundation language models against various post-correction benchmarks comprising different languages, time periods and document types, as well as different transcription quality and origins. We compare the performance of different model sizes and different prompts of increasing complexity in zero and few-shot settings. Our evaluation shows that LLMs are anything but efficient at this task. Quantitative and qualitative analyses of results allow us to share valuable insights for future work on post-correcting historical texts with LLMs.

## introduction over the last two decades, heritage institutions have digitised their collections on a massive scale, yielding millions of historical documents in digital format along with their machine-readable text @xcite . whether obtained through optical character recognition (ocr, for printed documents), handwritten text recognition (htr, for manuscripts) or automatic speech recognition techniques (asr, for audio documents), the availability of textual transcriptions has not only improved the accessibility of historical documents, but has also opened up the possibility of applying machinereading techniques to their content. increasingly, research and initiatives are being undertaken to process and mine the rich information contained in unstructured heritage text data @xcite , or to develop computational approaches to its analysis @xcite . while highly promising, these efforts face a persistent challenge that considerably impacts their effectiveness: the suboptimal quality of transcriptions. most text data derived from digitised historical documents contains transcription errors, for two reasons. firstly, despite recent advances in the accuracy of text recognition -thanks to the adoption of neural approaches and robust transcription frameworks @xcite @xcite ) -the quality of digital documents and the diversity of document types, languages, scripts, fonts and handwriting still poses significant challenges to ocr, htr and asr approaches. secondly, even though the latest transcription engines are much better than their predecessors, collections digitised long ago are rarely reprocessed, often for budgetary reasons. the impact of noisy transcriptions on downstream processes is well-documented, with a detrimental effect on search capacities @xcite , named entity processing @xcite , language modeling @xcite , and most natural language processing (nlp) tasks @xcite . possible answers to this situation lie in the development of post-correction methods aimed at producing a better, corrected version of a transcription with respect to the corresponding original text and, more recently, in the targeted re-transcription of identified faulty text sections-a complex process that requires robust pipelines @xcite . latest approaches to post-correction use sequence-to-sequence neural networks, with character-based translation models @xcite as well as lstms or transformer-based models @xcite . despite significant progress, performances vary greatly across the variety of historical texts @xcite , and systems still have difficulties dealing with extreme noise @xcite , generalising @xcite ) and avoiding undesired changes to error-free text, a particularly important requirement with historical sources @xcite . at the core of many approaches to nlp, language representations have evolved from auxiliaries to machine learning systems, such as n-gram models and word vectors, to specialised experts fine-tuned for specific tasks, such as transformerbased pre-trained language models. current efforts are aimed at more versatile systems, training autoregressive generative models on ever larger amounts of data and model sizes. this results in the emergence of large language models (llms) with exceptional robustness and generalisability capabilities, even in zero and few-shot settings @xcite @xcite , to cite but a few). since the launch of the gpt (generative pre-trained) model series by openai in 2018, and in particular the chatgpt conversational robot released in november 2022 based on the gpt-3.5 or gpt-4 models, the llm race is on. more and more models are being released, with impressive performance on various text-related tasks and results close to the state of the art, e.g. in question answering @xcite , machine translation @xcite and stance detection @xcite . of all the ways in which chatgpt can be used, its ability to revise and improve the quality of texts was quickly noticed. having been trained on hundreds of billions of tokens to predict the next word in a sequence, such text editing ability is hardly surprising. while evaluations have been carried out on tasks involving language editing capacities such as text summarisation and grammatical error correction-with fairly good results for some metrics @xcite @xcite -the ability of llms to amend historical transcriptions has not yet been systematically studied. can large language models that have powerful capabilities for generating and understanding language in different contexts also help to correctly rewrite texts that have been poorly transcribed by automated processes? capitalising on the recent rise of generative llms, this study aims to investigate, beyond anecdotal observations, the effectiveness of large language models to post-correct historical transcripts. in particular, we want to gain a better understanding of whether, to what extent, and under what conditions llms can produce good corrections of historical transcriptions, and seek to answer the following questions: -ability to correct. when prompted to correct the transcription of a given historical document, do llms improve, degrade, or leave the input text intact? to this end, we evaluate fourteen foundation language models from four model series against various post-correction benchmarks comprising different languages, time periods, document types, and different transcription quality and origins. we compare the performance of different prompts of increasing complexity in zero and few-shot settings and provide quantitative and qualitative analyses of the results. ## background we briefly highlight some key facets of llms and refer to @xcite for a detailed survey. llms are text generators that are trained on massive plain text data. based on well-established technology -deep neural networks and self-supervised learning -their success is mainly due to two key factors: scaling up model size and the amount of training data @xcite . the former was made possible by the transformer architecture @xcite and the latter by 20 years' worth of internet text data. coupled with optimisation frameworks @xcite , this has led to the rapid emergence of large language models with hundreds of billions of parameters, with increasing capacity to learn, generalise and act as general-purpose task solvers as more (clean) data is input. quickly identified as a paradigm shift in artificial intelligence systems, llms, and more generally foundation models, possess the crucial ability to learn in context, i.e. to perform a task with only a few instructions and task demonstrations, generating the expected output without updating model parameters @xcite . their behaviour is implicitly deduced and not explicitly constructed. following openai's initial releases, numerous models are being published by various commercial and academic players, sparking a global debate on the opportunities and risks from a scientific and societal point of view @xcite . llms's text editing capabilities have particularly been assessed in the context of grammatical error correction (gec). several studies have compared state-of-the-art gec systems with gpt models against various benchmarks and found that they can perform gec tasks effectively (especially on sentence input) and that they are slightly better at error detection than at correction due to overcorrection @xcite , a tendency that can be controlled by optimising prompt and example selection @xcite @xcite . few attempts have been made to post-correct transcriptions with llms, including llm-based selection of the best post-corrections for contemporary documents @xcite , and fine-tuning the bart autoregressive transformer to post-correct historical newspapers @xcite . to our knowledge, no previous study has explored llmbased post-correction of historical transcripts. ## approach we aim to assess the ability of generative llms to correct machine transcriptions of historical documents and to provide insights into what works best. using post-correction transcription benchmarks, our approach is, essentially, to compare the similarities between original automated transcriptions and their ground truth (gt), and between llm-corrected versions of the original transcriptions and the same gt, and to observe their variation, i.e. whether the llm-corrected version is a degradation or an improvement of the original transcriptions. to better reflect the diversity of archival collections and obtain results that are, to some extent, generalisable, we consider various post-correction benchmarks covering different document types, languages and periods, as well as different transcription qualities and origins. their different formats are standardised to ensure consistent handling of the data, especially as far as different levels of text segmentation (line, sentence region) are concerned. the selection of llms is based on model size, training data, resource requirements and accessibility. we consider fourteen models from four series and craft five prompts to guide their text generation. various post-processing heuristics are applied to clean up the output. despite their astonishing performance, we can hypothesise that post-correcting historical transcripts may pose several challenges for llms. first, unlike typical text generation tasks where multiple answers can be valid (summarisation, translation, dialogue systems), transcription correction requires a single correct answer that exactly matches the gt. this, of course, runs counter to llm's tendency to hallucinate. second, llms are not specifically trained for post-correction but learn in context through natural language instructions that they must understand -an ability that is unequal between models. third, the level and nature of noise in historical transcripts vary considerably from document to document, i.e. texts can range from minimally to extremely noisy, with different forms of noise. while this challenge affects all approaches, llms may not have encountered many noisy historical transcripts during their training. ## results and discussion with eight datasets, three text unit levels, multiple quality bands, fourteen models, five prompts and zero-and few-shot settings, experiments cover many dimensions and yield many results. we first present and discuss metric-based results by iteratively removing dimensions before manually exploring performance factors. due to to space constraints, some figures are included in the appendix. ## conclusions and future work this exploratory study shows that llms are not good at correcting transcriptions of historical documents of any kind, at least in the applied experimental setting. not only do they not improve the original transcriptions, they usually degrade them, making llm-based post-correction of historical transcripts a rather distant prospect. nevertheless, we have found that instructing models about the nature of the input and guiding their output format leads to better results, and that large open-access and multilingual models from the bloom(z) and llama series can compete with commercial gpt models. on the basis of these findings, future work should investigate in more detail some of the elements that could not be studied due to the scale of the experiments presented here. these include, among others: testing prompts that are even more tailored to the specifics of each document type, distinguishing between error detection and correction prompts (in a chain-of-thought fashion), searching for the temperature hyperparameter, attempting model fine-tuning and model self-evaluation, and consolidating the error taxonomy and error analysis on a few datasets.