Survey and reproduction of computational approaches to dating of historical texts (2021)

Abstract: Finding the year of writing for a historical text is of crucial importance to historical research. However, the year of original creation is rarely explicitly stated and must be inferred from the text content, historical records, and codicological clues. Given a transcribed text, machine learning has successfully been used to estimate the year of production. In this paper, we present an overview of several estimation approaches for historical text archives spanning from the 12th century until today.

## introduction knowing when a text was written is of crucial importance for relating its content to a historical context. with the increasing digitization of historical archives, many new research opportunities have emerged for studying how languages have evolved. however, such studies rely on digitized corpora explicitly stating when the texts were originally written. this information is often not given by the original scribe, although educated guesses from later owners can sometimes be found in manuscripts. additionally, improved dating of historical manuscripts can help historians to better understand the chronology of their sources. the premise for our paper is an imagined scenario where a historian or philologist needs help with a transcribed collection. we imagine being given a partially annotated set of documents (given either as specific years or as intervals) and employing a computer model to determine the production years of the un-labelled documents. although there is literature describing different ways of solving the problem of the above scenario, there is little work done on comparing the different modelling approaches. in this paper, we will survey and evaluate computational approaches to the problem of estimating the production dates of text in digitalized historical archives. we have reimplemented several methods for estimation and feature extraction proposed in the literature. our experimental setup allows us to evaluate combinations of different methods on datasets representing different times, text lengths, and genres. our reimplementation is available as open source foot_0 . our primary historical datasets were two medieval archives containing legal documents from denmark and sweden. comparing results on these collections is of special interest, as they are similar with respect to content, but differ in the number of documents, temporal distributions and detail of annotation. to assess the generalizability of the methods we also include two modern collections. these modern collections, that have previously been treated in the literature, are a collection of english news items, from the semeval 2015 shared task on diachronic text evaluation @xcite , and colonia, a corpus of historical portuguese @xcite . an overview of the relevant literature is presented in section 2, section 3 contains a description of the datasets, our experimental setup is described in section 4, and, finally, results and discussion are presented in section 5. ## previous work the problem of automatic text dating has been treated using various methods and applied to a wide range of different types of corpora. to the best of our knowledge, the task of assigning a date to documents was first introduced in the information retrieval community with the main goal to query document collections based on temporal relevance. @xcite treat the problem as a text classification task in which documents are dated by comparing them to temporal query profiles. they refer to such profiles as temporal language models, which essentially capture the distribution of term or concept usage over time. the same idea is found in the work of @xcite , in which word frequencies across time are used to infer temporal association rules. the work by @xcite was later expanded by @xcite who improved the temporal language models by applying various steps of pre-processing, including filtering words based on tf-idf scores and pos tags, applying stemming, and collocation extraction. the above works were made on corpora of newspaper articles, which have remained to be an object of research within the field of automatic dating, most recently in the se-meval 2015 shared task on diachronic text evaluation (dte) on english news snippets @xcite . following the work on temporal language modelling, @xcite introduced using support vector machines (svm) for the task of dating, in which documents are represented by feature vectors of word and character counts, in addition to other handcrafted features. whereas in the temporal language modelling approach the sole problem is to learn the distribution of words in a set of documents belonging to a specific time span, the goal of mapping a document to a date is now part of the learning objective. later work on news corpora has focused on how the extraction of temporal references, such as expressions for time and events, can facilitate the task of dating, which was also the research question in two of the three subtasks of the dte shared task @xcite . aside from news, scholars have studied a wide range of different historical corpora, ranging from broad collections such as google ngrams @xcite to more narrow collections as in the dadoe-val2020 shared task @xcite , which introduced a diachronic corpus of political work by alcide de gasperi. while news items naturally contain explicit temporal references for when the text was written, this is often not the case when working with other genres. for example, if a philologist were to date a piece of literature, their work may solely rely on features such as lexicon, grammar, topic, or style, as the contemporary context is often implicit. thus, work outside the news genre has generally put less emphasis on extracting temporal references, and instead explore how the language in a text can be represented. one of the first studies to extend the work beyond the news genre was @xcite , who use language modelling to predict the date of a collection of short stories published between 1798 to 2008 from project gutenberg 2 . subsequently, language modelling has not been applied to the problem of dating. work has been done to identify temporal trends in historical corpora @xcite @xcite , by using language modelling to measure the distance between time periods, but models were not explicitly applied to the task of dating. instead, studies have focused on creating vector representations of documents and then using those for classification. the raw text has been used directly as input to create bag-of-words and/or characters, with ngram sizes ranging from one to three words @xcite @xcite , and one to five characters @xcite @xcite . other features may be extracted, such as syntactic features using pos annotations @xcite @xcite and stylistic measures such as lexical diversity @xcite . most commonly, the problem of dating a text is defined as a classification problem in which classes are treated as bins corresponding to different time spans. several estimators have been applied, including logistic regression @xcite , support vector machines @xcite @xcite and multinomial naive bayes @xcite . the size of the bins depend on the problem and the data available. for dating of contemporary news items, scholars have worked with granularities down to a yearly basis @xcite . this is typically not possible when working with historical text, as data is sparse and may in turn not have such a precise date of production. here, scholars have instead worked on dating documents within a century @xcite or a decade @xcite . compared to classification, regression methods have not been extensively explored. in regression, samples are mapped to a date directly instead of a bin, thus circumventing the obstacle of deciding on a specific bin size. also, regression preserves the ordinal nature of the problem, which classification ignores. @xcite propose to use ordinal regression. in this approach, the task of dating is considered as a ranking problem, where reference documents are placed on a timeline, which is then used to estimate the most probable time spans for query documents. another attempt using regression comes from the field of image processing, where @xcite applied gaussian processes (gp) to the problem of estimating the date of medieval manuscripts using visual features extracted from the facsimile together with the transcribed text. whether classification or regression is best suited for the problem of dating text -and what pitfalls such approaches have -are still open questions. as for feature extraction, neural methods have over the last decades undermined the use of manual feature extraction for a wide range of problems, including text classification. @xcite applied graph convolutional networks to the problem of dating, utilizing syntactic information and temporal reference extraction in addition to the words of the text. for smaller corpora, neural approaches are yet to be tested, which is out of the scope of this paper. instead, we seek to describe and compare the methods that have already been established for the dating of historical text corpora. ## experimental setup in our experimental setup, we have implemented a number of ways of doing feature extraction. we then evaluated all combinations between those feature spaces and a number of ways of doing the mapping to years on the timeline. in the documents of several datasets, clearly stated years can be found. in order not to let the estimators simply learn to find this information (especially in the charter datasets, where roman numerals are frequently encountered) we have removed all numerals from the text as a part of the preprocessing. some of the methods we have evaluated were quite demanding of the hardware. hence, we randomised the training, validation, and test sets while preprocessing the datasets, using the same sets for all evaluations. it should be noted that this is not standard for several of our approaches (e.g., naive bayes) which are normally evaluated using cross-validation. however, we saw this as the only way of making a fair comparison and not risk giving some estimators more or different data. ## results and discussion the results from the experiments can be found in all classifiers perform above baseline for at least one feature set. considering the best performing feature sets per classifier (highlighted with blue), character models perform the best across classifiers except for gaussian naive bayes. aside from being able to capture features such as morphology and spelling, character models have the advantage that the feature space is smaller than for word models, which in turn increases the number of examples that estimators consider. whether it is the features or simply the data size that is at play is difficult to read from these numbers. when working with vector representation of words and higher level character n-grams, the feature set easily becomes larger than the number of samples used for training a model. in these cases, one could argue that it is unlikely for the estimators using these representations (svms, naive bayes) not to find something in the training set that correlates with the timeline, even though the feature might not necessarily be related to language change. the problem is compounded by that the training, validation, and test data were all drawn from the same data generating process and, hence, might have the same spurious correlations in relation to the target labels. if we compare the linear svm with the nonlinear svm, the linear version has the advantage of being more qualitatively interpretable due to the lack of warping of the feature space. however, if we compare the models in terms of accuracy, using a non-linear kernel yields slightly better results. when we compare the test set predictions of the different estimators, they do tend to correlate. as is revealed in figure 4 , there is a strong relationship between the predictions made using different svm estimators (linear and non-linear), especially on similar feature sets. if we consider the predictions using the non-linear svm on character unigrams, we see a slightly stronger correlation with the predictions of the linear svm when using higher orders, which suggests that a more complex model is able to utilize its nonlinear combination of features on the problem. however, in terms of accuracy results, this advantage is not widely outspoken. thus, we argue that choosing a linear kernel may still be preferable, as its predictions are more easily explained to a community of philologists or historians. despite not appearing in more recent research, the temporal language model outperforms other models on several datasets using character features. all estimators that we have evaluated describe language as a distribution of words or characters. what distin- while performing well on character feature sets, the temporal language model struggles when it comes to word representations with accuracies below 10%. this suggests that the temporal language model is sensitive to larger feature spaces, in which smoothing might not be sufficient. furthermore, it performs poorly on the colonia dataset. whether this is due to the number of samples, document length, or dataset distribution is difficult to say, and it calls for further analysis of the models with respect to dataset statistics. finally, we wish to discuss the performance of regression to classification methods. most previous work has preferred to use classification instead of regression, treating the timeline as discrete and with temporally independent labels. that labels are independent is reflected in the use of categorical accuracy as the evaluation metric. if we look at figure 3 , this is illustrated by the inner dashed lines, outside which predictions are considered incorrect, even though they are close to the target temporally. in this respect, regression methods should have an advantage, however, this is not reflected in our results. it would be interesting to further compare what advantages there are -if any -to choosing regression over the classification. ## conclusion and future work in this paper, we present a survey of several methods found in the literature for estimating the production years of transcribed historical documents. we have reproduced the methods used in a number of papers, including different n-gram/word/pos-tag feature spaces and several linear (naive bayes, linear svm) and non-linear (gaussian process, svm with rbf kernel) estimators. our results show that several of the combinations of estimators and feature models work well, but that character n-gram features provide the best results overall. in particular, the temporal language model with character features surpasses more recently proposed models. whether this is due to the linguistic features (e.g., suffixes or phonetic changes leading to changes in spelling) that they potentially capture or simply due to a reduced feature space giving better model parameter estimates, we cannot conclude from our results. therefore, we call for further analysis of the estimators, preferably favouring more interpretable approaches (e.g., linear svm). our experiments show that combinations of estimators and feature transforms that worked well on younger materials were often also successful on older materials, and vice versa. as the datasets that we compare not only differ in age, but also in number and size of samples. for future work, it would be interesting to investigate the robustness of the methods from the literature with respect to such dataset statistics. in this respect, it would also be relevant to include recent work on neural models such as using word embeddings and convolutional networks, which have been shown to work well for dating on large corpora. however, these have yet to be trialed on smaller corpora.