Prompting the Past: Exploring Zero-Shot Learning for Named Entity Recognition in Historical Texts Using Prompt-Answering LLM s (2025)

Abstract: This paper investigates the application of prompt-answering Large Language Models (LLMs) for the task of Named Entity Recognition (NER) in historical texts. Historical NER presents unique challenges due to language change through time, spelling variation, limited availability of digitized data (and, in particular, labeled data), and errors introduced by Optical Character Recognition (OCR) and Handwritten Text Recognition (HTR) processes. Leveraging the zero-shot capabilities of prompt-answering LLMs, we address these challenges by prompting the model to extract entities such as persons, locations, organizations, and dates from historical documents. We then conduct an extensive error analysis of the model output in order to identify and address potential weaknesses in the entity recognition process. The results show that, while such models display ability for extracting named entities, their overall performance is lackluster. Our analysis reveals that model performance is significantly affected by hallucinations in the model output, as well as by challenges imposed by the evaluation of NER output.

## introduction named entity recognition (ner), oftentimes also referred to as named entity recognition and classification (nerc), is in essence a token classification task that aims to extract various types of named entities from a given written source. the choice of how fine-grained we want our analysis to be dictates the number of different labels we want to extract; a coarse-grained analysis would only look at names of people, locations and organizations, for example, while a more fine-grained approach would include dates, events, artifacts, monetary values etc. while ner is by no means a solved problem in nlp, there have been numerous efforts made to provide tools for modern languages. however, such tools have significant gaps in terms of ner resources (e.g. @xcite al. (2020); hvingelby et al. (2020) ), and many are still ongoing @xcite , which only highlights the importance of further research in this domain. at the same time, ner for historical texts faces several unique challenges in its own right. ocr errors are common due to the poor quality of old prints, leading to misrecognized characters and words @xcite . the evolution of language over time, with outdated vocabulary, spelling variations, and different grammar rules, complicates entity recognition, especially since historical texts often lack labeled datasets, making supervised learning difficult. models trained on modern data struggle with domain transfer to text from antiquated sources, as historical contexts and naming conventions differ significantly. a common example of this phenomenon is toponyms changing through time (e.g. byzantium, istanbul, constantinople); so while we refer to the same geographical location, the name differs, and such changes are oftentimes not linked to each other in databases in order to indicate equivalence. nonstandardized naming, ambiguity in references, and the need for contextual understanding further hinder accurate recognition. additionally, historical texts are often multilingual, requiring models to handle archaic language variants from several languages at the same time. these factors, combined with cultural and diachronic variations in entity references, make ner for historical texts a complex and challenging task. this study is motivated by the proven benefits of prompt-based learning @xcite . the goal of this paper is to further the development of nerc systems for historical texts. specifically, we want to explore the potential of promptanswering llms for extracting nes from historical text in a zero-shot scenario, using historical newspaper data in english, german and french. we investigate this research avenue in order to counteract the costly nature of creating manually annotated ner datasets from scratch, while also leveraging the potential of prompt-answering llms in low resource settings. in our exploration, we aim to address the following research questions: at the same time, we identify several potential benefits of this work for future research. by enabling the creation of historical social networks, for example, we can uncover and analyze relationships and interactions among individuals across time periods. additionally, enhancing archival annotation improves the accessibility and usability of historical documents, allowing researchers to extract meaningful insights more efficiently. such methods facilitate cultural and historical research by automating large-scale annotation, significantly reducing the time and cost associated with manual processes, thereby enabling access to diverse historical narratives. ## background earlier work on historical ner has primarily been conducted on monolingual language models and various choices of model architecture and data sources. moreover, transformer-based models have been gaining significantly more traction. here, the trend leans towards using off-the-shelf modern lms, which are later fine-tuned with historical labeled data for the task of ner @xcite , but there are also studies experimenting with data sourced entirely from historical text, and fine-tuned on modern labeled data @xcite . moreover, the trend has been to branch out towards multilingual models in order to take advantage of their transfer learning capabilities @xcite . the biggest hurdle in the way of designing accurate and high-performing ner systems seems to be the lack of annotated quality data. ideally, we would want to have large amounts of manually annotated datasets which are curated using expert knowledge. the process of obtaining such data is, however, expensive both in terms of time and resources needed for such endeavors. furthermore, enormous amounts of data that could be used for annotation reside in libraries and archives, and have yet to be digitized -which is another time-consuming and costly process. while there are significant efforts being made to contribute to this gap in the field, the vast majority are focused around texts from modern sources. such examples include the icelandic ner corpus @xcite , its norwegian counterpart @xcite , the swedish suc @xcite , or the danish dane @xcite . naturally, new research directions have come forth, aiming to circumvent the data scarcity issue. the expensive nature of supervised learning prompts for exploration into the capabilities of few-shot learning for lm architectures @xcite . with the recent emergence of promptanswering models and their impressive few-shot learning abilities (schick and schütze, 2021), several studies have attempted to explore their performance on ner @xcite . moreover, while schick and schütze (2021) explore true fewshot learning where there is no development set available for hyperparameter tuning and additional prompt engineering, and highlights its potential for future applications, new research on prompt engineering for few-shot ner is quick to emerge @xcite . a similar exploration to the one we show in the present paper has been conducted by @xcite for dutch historical text. their dataset was created based on letters from the dutch east india company dating from the 17th and 18th century. in their paper, they compare the performance of monolingual (bertje, robbert) and multilingual (mbert, xlm-r) language models. the study finds that multilingual models outperform monolingual ones in handling the language variations and cross-lingual transfer needed for historical texts. overall, both model types benefit from combining historical texts and editorial notes, with multilingual models showing more robustness across various text types. more recently, gonzález-gallardo et al. (2023) investigate how language models like gpt-3.5 handle entity recognition in historical documents, high-lighting also code-switching between french and ancient greek. the study points out that while gpt-3.5 is trained in over 100 languages, it struggles with unrepresented languages such as ancient greek. the paper discusses challenges such as the model's difficulty understanding mixed-language texts and the limitations of historical archives that remain inaccessible to models, impacting their performance in recognizing historical entities. the expensive nature of labeled data for training and evaluation makes the prospect of zero-shot and few-shot learning significantly more appealing for ner research. the basis of our exploration lies in a study conducted by @xcite . the paper uses labeled data from the clef-hipe 2020 dataset @xcite , which is an openaccess ocr-ed newspaper corpus annotated for ner. the dataset contains swiss and luxembourgish newspapers from 1790 to 2010 in english, german, and french. the authors focus on zeroshot ner using t0++ @xcite , and only use data up to 1950 at the latest in order to keep the focus on the historical aspect of their exploration. their study shows that, while the model shows some capacity of extracting nes from the given dataset, dealing with historical text poses additional challenges through spelling variation and ocr errors. they also prompt for further investigation of the capabilities of generative llms in this given context. ## method our exploration can be seen as a three-step process. the first phase is to run all of our chosen models on the same dataset as the original study described in @xcite , which we describe in section 3.2. the second step is to evaluate and assess the kind of errors that the models are prone to by doing a manual examination of the output of each model. third and last, we aim to address some of the more common causes of errors in the model output and re-evaluate in order to see how that affects model performance. ## results we apply each model to our ner task in a zeroshot setup to assess their baseline performance without extensive customization. we used prompts designed to extract named entities across multiple languages, testing the models' ability to handle common entity types. a manual analysis of the output of each model reveals several systematic types of errors that take a toll on overall model performance. a common case is models retaining parts of the prompt and regurgitating them as output, instead of outputting parts of the actual input text. for example, out of 50,495 potential entities annotated by t5 3b, over 80% of them contain the words "input" or "in input". the same phenomenon is observed in t5 11b, but to a lesser degree -only 56% of the extracted entities keep the word "input". when looking at its multilingual counterpart, we notice that mt5 displays the same anomaly. out of all output nes from mt5 3b, 51% contain at least one occurrence of the word "input", which drops to 49% in the case of mt5 13b. this carries over in the case of both versions of the bloomz model as well, but to a different extent. instead of just keeping parts of the prompt text, the model takes the entire content of the prompt, including the input sentence, and splits it into segments using commas as delimiters. we believe that this could be the case due to the model not properly capturing sentence boundaries, which has been known to cause problems for this particular model family @xcite . in light of these observations, we are unable to calculate reliable performance scores for these models (@xmath0, and we therefore no longer include these 6 models in the rest of our analysis. we focus instead on t0 and aya, and more specifically t0++ and aya 101, as larger model versions seem to lead to slight improvements in performance. ## discussion our results reveal that, while prompt-answering models are able to extract named entities in a zero- (a) original prompt (b) original prompt, filtered (c) modified prompt (d) modified prompt, filtered figure 2: results for aya 101, using the original prompt and our modified version, both before and after filtering hallucinated entities. frequent ocr errors introduce unpredictable variations in the spelling of "gold" words, including inconsistencies in spacing, letter placement, and diacritics. t0 automatically corrects these during its predictions, which hinders our ability to match its answers accurately with the corresponding tokens in the sentence. this is exemplified in sentence d) in another hurdle in the way of effective ne extraction and evaluation is the frequent occurrence of hallucinations in the model output. filtering out hallucinated entities does lead to an increase of around 5% in overall f1 score for t0++ (see figure 1 ), and to a lesser extent in aya 101 as well (see figure 2 ). however, the overall results are still around the same ranges as before, which only highlights the difficulty of evaluating ner spans accurately, as well as the model's tendency to overgenerate rather than not provide an output at all. this is made evident by examples a) and b) in moreover, the relatively uniform distribution of hallucinations among labels supports the assump-tion that t0 models tend to produce non-empty outputs, and therefore over-generate rather than provide a blank answer or no answer at all @xcite . the same phenomenon has been observed across all investigated model families, including t5, mt5 and bloomz. it is also important to note that aya 101 achieves higher recall scores than t0++ for french and german, likely due to the fact that it was trained on multilingual data as opposed to english exclusively. therefore, while the model might not be able to label the entities correctly, it is more likely to extract entities in languages other than english. the overall effect of prompt engineering and filtering of hallucinations is not to be overlooked either. both of these approaches lead to small improvements in model performance, which prompts for further exploration in this direction. ## conclusions and future work in this paper, we explore the zero-shot capabilities of prompt-answering llms for ner on historical text. our study shows that, while prompt-answering llms display some capacity to automatically extract nes, they do not reach satisfactory enough results for further use (e.g. reliable automatic annotation of archival text). moreover, we also highlight the models' tendency to produce output even in scenarios where it generates false positive results, and we draw attention to the extensive amount of hallucinations produced by the models. lastly, we attempt to explore the effect that hallucinations have on model performance by conducting a comparative evaluation after filtering them from model output. the main contribution resulting from this approach is enhancing the understanding of llms' limitations and capabilities in historical ner tasks, providing valuable insights for improving model reliability. our findings advance historical ner research by broadening the model comparison, extensive error analysis, testing prompt modifications, and addressing hallucination issues. in future work, we would be keen to investigate the effects of prompt engineering on few-shot ner for historical text, with the hope of benefiting from the proven advantages of prompt-based learning @xcite . adjusting the way we feed our prompts into the model can also affect the overall model performance, as previously shown in @xcite . since the model has the tendency to over-generate, and at times it provides an answer extracted form the prompt rather than the input text itself, it could potentially be more beneficial to treat prompting as a two-step process, where we first provide the model with the prompt, and then input the text we want to work with as a secondary step. another possible avenue for research is to look into what would be the minimum amount of data or examples required for few-shot or zero-shot learning in historical ner tasks using llms without having to compromise on performance. lastly, since it is common practice for current state-of-theart models to be released in "families" consisting of various sizes of the same ground architecture, it could also be relevant to experiment with how more variation in parameter size affects the capabilities of such prompt-answering llms -including, but not limited to, the model families already mentioned in this paper. a final way forward would be to ensure that the llm used has seen sufficient amounts of historical text and, if possible, ner examples in historical texts during training. this study highlights the potential of generative models in improving access to and the analysis of historical texts, aiding in digital humanities efforts, as well as in archival and historical research, while also drawing attention to some of their potential pitfalls.