Explainable Publication Year Prediction of Eighteenth Century Texts with the BERT Model (2022)

Abstract: In this paper, we describe a BERT model trained on the Eighteenth Century Collections Online (ECCO) dataset of digitized documents. The ECCO dataset poses unique modelling challenges due to the presence of Optical Character Recognition (OCR) artifacts. We establish the performance of the BERT model on a publication year prediction task against linear baseline models and human judgement, finding the BERT model to be superior to both and able to date the works, on average, with less than 7 years absolute error. We also explore how language change over time affects the model by analyzing the features the model uses for publication year predictions as given by the Integrated Gradients model explanation method.

## introduction collections of historical language, such as ecco which comprises over 180,000 titles published in the eighteenth century, are at the focus of a growing interest in the nlp community. the large quantities of raw textual data in these collections, which may cover whole centuries worth of published works, are suitable for language modelling research, a popular and highly relevant topic in nlp. the historical language itself poses new and interesting challenges, especially due to the fact that the collections span over a time frame long enough to be affected by natural language change and evolution. furthermore, artifacts relating to the technical process -namely the ocr qualityof the works pose a whole new set of challenges rarely met in modern nlp which mostly deals with born-digital texts, for the most part devoid of such artifacts. these new developments in nlp are crucial also for historians and other humanists apply-ing them to new research questions and ways to produce historical evidence. the transformer model @xcite ) and especially the bert @xcite bidirectional encoder based on the transformer, form the foundation of present-day practical nlp research and are naturally also applied in the historical language domain. bert models have already been trained with various historical data sets and languages, including at least english, german, french, latin and classical chinese @xcite @xcite . the range of tasks to which it has been used in the domain is already diverse, covering at least named entity recognition, construction of word embeddings, event detection, stance detection, word sense disambiguation and the study of the animacy of target expressions @xcite @xcite @xcite . issues particular to the historical language domain have also produced new challenges for bert appliers to adress, like the effect of ocr quality @xcite . in this work, we will follow two directions. firstly, we set out to train from scratch and release a dedicated bert model specifically on and for the ecco dataset. then, we establish whether such a targeted bert model provides an advantage over other existing historical english bert models, or even the modern english bert. to this end we pursue a benchmark task whereby the model is trained to predict the year of publication based on the text itself. we find that the model performs much better on this task than we intuitively expected, and therefore we carry out and report on a more extensive analysis of the task including a comparison to hu-man performance, and provide aggregated feature attributions to the bert model predictions using the integrated gradients model explanation method of @xcite . for the purposes of this work, it is additionally useful to know the following information about ecco. first, ecco is temporally skewed toward the end of the eighteenth century, with many more works being published particularly in the final two decades of the century than in earlier ones. second, while some non-english works are included in the collection, 94% of the documents in it are in english (the other languages with more than 1% representation are french, with 2.7% and latin with 2.5%). third, the print quality and thus ocr quality of the documents in ecco correlates both with their format (pamphlet vs. book) as well as publication date, with more recent publications having a significantly better average ocr quality. further, ocr quality also differs between ecco1 and ecco2, which were scanned and ocr'd using different processes. finally, there may often be multiple editions of a single work within ecco, and while they have been printed in the eighteenth century, they may well have originated from e.g. antiquity. further, when the year of publication of a work has not been printed on its title page, the year has often been estimated. on the level of the whole ecco data, this manifests itself as frequency spikes on every round fifth, tenth and fiftieth year. @xcite for the purposes of the year regression experi-ments in this work, we have dealt with the last two problems by limiting the subset of ecco we are experimenting on to only those where the year of publication is certain, as well as only to the first editions of works that first appear in the eighteenth century. the size of this subset is approximately 40,000 documents. ## methods in this section, we describe the models and methods used in this work: the pre-trained bert model, the bert-based year regression and feature attribution, and finally the linear baseline. ## results there are 39,429 ecco works that have a verified year of publication during the 18th century and that constitute the earliest publication of the given work. all results are reported on the same test set of 1971 randomly selected works, which contain a total of about 225,000 text segments. a development set of the same size was reserved for hyperparameter selection, with the remaining 35,487 documents being used for training. ## conclusions the contributions of the paper are two-fold. firstly, we pre-trained and openly distribute a bert model specifically focusing on the historical english language in the eighteenth century collections online (ecco) that is widely used in the humanities. to benchmark the model and gain understanding of its performance on historical english, we use the task of publication year prediction, in other words given the text, the task is to regress its publication year. our findings and analysis of the model's performance on this task then form the second contribution of the paper. we establish that the accuracy with which the model is able to predict the year of publication is well above our baseline models on full documents and also well above human performance on text snippets. we also carried out an initial qualitative analysis of predictive features, both for our simple linear baselines and for the bert models. we observe a degree of a useful signal among these features, intuitively understandable to a human, demonstrating the applicability of model explanation techniques also to the complex bert model. nevertheless, it is clear that numerous challenges still remain. this initial study has several natural future work directions. firstly, a further, more detailed analysis of the predictive features, and there-fore of the model's predictions is clearly called for. secondly, a more detailed comparison between human and model decisions will be carried out. and finally, as we have not specifically taken into account the ocr noise when pre-training the bert models, more noise-aware variants of the transformer model, e.g. characterbased models, will be tested on the ecco data. the ecco-bert model is freely available as turkunlp/eccobert-base-cased-v1 in the hugging face model repository.