Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0 (2022)

Abstract: In this work, we explore whether the recently demonstrated zero-shot abilities of the T0 model extend to Named Entity Recognition for out-of-distribution languages and time periods. Using a historical newspaper corpus in 3 languages as test-bed, we use prompts to extract possible named entities. Our results show that a naive approach for prompt-based zero-shot multilingual Named Entity Recognition is error-prone, but highlights the potential of such an approach for historical languages lacking labeled datasets. Moreover, we also find that T0-like models can be probed to predict the publication date and language of a document, which could be very relevant for the study of historical texts.

## introduction this paper lies at the focal point of three orthogonal advances. first, the recent surge in glam 1 -led digitisation efforts @xcite , open citizen science @xcite and the expansive commodification of data @xcite , have enabled a new mode of historical inquiry that capitalises on the 'big data of the past' @xcite . second, the 2017 breakthrough that was the transformer architecture @xcite has led to the so-called imagenet moment of natural language processing @xcite and brought about unprecedented progress in transfer-learning @xcite , few-shot learning @xcite , zero-shot learning @xcite , and prompt-based learning (le scao and rush, 2021) for natural language. third, the growing popularity of promptbased methods @xcite has resulted in a new paradigm for training and fine-tuning large language models (llm) as well as novel applications in named entity recognition (ner) @xcite . ner for historical texts has been the focus of a growing body of research, most recently surveyed by @xcite . both ner and the related task of entity linking can enhance our ability to search and navigate digitised historical materials @xcite . however, applying ner to historical texts poses a number of challenges, including those due to errors in optical character recognition (ocr) @xcite @xcite and domain transfer @xcite . to advance research in this area, an increasing number of datasets have been created to support the development and evaluation of ner approaches in historical text @xcite @xcite in this paper, we examine the zero-shot abilities of t0-a prompt-based llm developed as part of the bigscience project for open research @xcite -on the challenging task of historical ner foot_0 . this endeavour had two main hurdles: ## prompt-based factual probing in addition to our main experiment on ner, we run two further experiments to assess t0's ability to do inference in a multilingual setting and to identify historical variation in textual corpora. probing for language to gauge t0's ability to reason in a multilingual setting, we test the model's language identification ability. to that end, we use a trilingual 14 subset of the wili-2018 -wikipedia language identification dataset @xcite and prompt the model on language ( probing for publication date to assess t0's treatment of historical text, we study how well it predicts the likely date of publication for a piece of text from our test dataset by prompting on publication date ( ## conclusion we have presented our experiment to evaluate t0 for zero-shot historical ner, as well as on the pre14 french, german, and english; 1000 sentences each. diction of language and publication date of historical texts. our results show that historical texts present additional challenges for zero-shot ner (especially because historical datasets often include noisy ocr), but that t0 can however be used as is for language and date prediction. next steps will be experimenting on different prompts and matching methods, as well as testing few-shot ner.