M ac BERT h: Development and Evaluation of a Historically Pre-trained Language Model for E nglish (1450-1950) (2021)

Abstract: The new pre-train-then-fine-tune paradigm in Natural made important performance gains accessible to a wider audience. Once pre-trained, deploying a large language model presents comparatively small infrastructure requirements, and offers robust performance in many NLP tasks. The Digital Humanities community has been an early adapter of this paradigm. Yet, a large part of this community is concerned with the application of NLP algorithms to historical texts, for which large models pre-trained on contemporary text may not provide optimal results. In the present paper, we present “MacBERTh”—a transformer-based language model pre-trained on historical English—and exhaustively assess its benefits on a large set of relevant downstream tasks. Our experiments highlight that, despite some differences across target time periods, pre-training on historical language from scratch outperforms models pre-trained on present-day language and later adapted to historical language.

## introduction & related work social scientists and humanities scholars have long been interested in describing cultural systems and understanding the way in which these change across time. traditionally, such shifts were documented with 'manual' interpretative methods, but more recently researchers in dh have begun applying machine learning techniques to support their interpretation. in the case of researchers working with historical text, current work has been occupied with developing and evaluating nlp algorithms with the goal of modeling the way in which concepts, categories and discourses (e.g. of class, gender) change over time along with their linguistic representations. in this context, applications include data-driven approaches to conceptual change @xcite @xcite , historical word sense disambiguation @xcite @xcite , named-entity recognition in historical text @xcite @xcite @xcite or unsupervised semantic change @xcite . in view of the growing weight of the new nlp paradigm of "pre-train-and-fine-tune"-which leverages large language models in order to produce strong feature extractors @xcite @xcite -, the question arises as to whether similar performance boosts can be gained for current nlp-oriented research dealing with historical text. due to the heavy domain-shift along grammatical, semantic and orthographic language layers, models pre-trained on contemporary data are less helpful when applied on historical data. previous work has experimented with adapting contemporary models to historical data on a per-task basis @xcite , although it is unclear whether this approach can yield a general purpose historical model. an alternative approach would be to adapt historical text to the modern standards using a historical text normalization system @xcite . while this could indeed help tackling orthographic shifts, grammatical and semantic shifts would be left un-adapted. moreover, error percolation from the automatic normalization system would still be an issue. arguably, the main advantage of pre-training this type of models is their ability to exploit very large datasets. in the case of historical linguistic resources, the known scarcity of digitized texteven for commonly high-resource languages like english-turns such enterprise problematic. however, ongoing efforts towards making historically relevant book collections digitally accessible @xcite @xcite have kick-started experimentation in this respect. for example, @xcite pre-trained a model on latin text spanning several centuries. @xcite and @xcite have employed contextualized character-level models, @xcite has released historical german and french models trained on historical newspaper data, and @xcite and @xcite trained and released models on an english corpus spanning the 18th to 20th centuries. several questions appear in this context. for example, it remains unclear whether all target periods can benefit equally from a "historically" pre-trained model or whether the performance benefits of these models vary across periods depending on the available amount and type of documents. moreover, it is unclear what the advantages are of the two current alternative pre-training approaches. in some cases, pre-existing models pre-trained on contemporary datasets are first "historically fine-tuned" before being applied on downstream tasks. this approachmotivated by the promise to leverage a larger outof-domain contemporary dataset-has been shown to outperform their non-adapted counterparts (although, see german bert vs. europeana bert in @xcite , but it remains unclear how these fine-tuned models compare to models pre-trained "historically" from scratch, and, more generally, whether the presence of modern data in the training process diminishes model performance (as suggested by @xcite . contributions in this paper, we introduce a model pre-trained on a large span of historical english (1450-1900), and show its advantages with respect to present-day models, as well as models adapted from present-day to historical english on an exhaustive set of ad-hoc downstream tasks. moreover, we show how model performance strongly depends on the time period of the target application. ## experimental setup we rely on the large language model known as "bert"-a stack of transformer layers with a selfattention mechanism @xcite , optimizing a masked language model (mlm) objective @xcite . despite the existence of several mlm alternatives, bert remains a good choice, considering that (i) it is well-established and most thoroughly studied, and (ii) on-going evaluation of alternative choices-mostly focus on natural language understanding (nlu) tasks-has not yielded a clearly superior architecture. we rely on the seminal implementation, foot_2 with the hyper-parameterization corresponding to the "bert-base uncased" architecture. 3 pre-training is done with default parameters, except for the maximum sequence length (set to 128 subtokens) for 1,000,000 training steps. ## results we now describe the benchmarking tasks and the results of the competing models in detail. ## discussion the exhaustive set of benchmarking experiments allows us to assess the impact of pre-training mlm architectures on historical data for diachronic tasks. as expected, historical pre-training helps to improve performance on diachronically relevant tasks. accordingly, both turingbert and macberth generally outperformed the models pre-trained on contemporary data only-with the exception of both wsd tasks, where multibert (in word-incontext) and bert (in the parameter-free wsd) outperformed turingbert. the historically pretrained macberth outperformed all competing models across tasks on partitions of the test data stemming from earlier periods. furthermore, based on the fact that macberth displays an advantage across tasks and conditions, we can conclude that enlarging the historical span and coverage of pre-training data is advantageous. 15 importantly, the period to which only macberth had access during pre-training coincides roughly with the beginning of late modern english (around the 1700s) and the consolidation of the modern standard. therefore, if variety in the pre-training data sources results in more powerful feature extractors, the impact for diachronic downstream tasks of the pre-training data stemming from before the 1700s is even so larger. interestingly, in many tasks, the advantage shown by macberth was not restricted to the earlier periods. for example, in part-of-speech tagging, the increase in accuracy appeared to be correlated with the total amount of data available for training and testing. this can be interpreted as a sign of better sample efficiency that pre-training on varied datasets confers the model. finally, a question that arises from the present experiments concerns those aspects of the experimental setting that may be responsible for the observed disadvantage of turingbert on diachronic tasks-even on time spans to which turingbert had access during pre-training. the fact that the tokenizer is restricted to the specific domain of contemporary english may force the model to aggregate over odd subword tokenizations in order to extract word-level feature representations, putting it in a weakened position. adapting a model originally pre-trained on contemporary english may also import too strong an inductive bias when the model is later fine-tuned on historical english. in any case, pre-training from scratch on historical data may be a more robust strategy than adapting a pre-trained model. ## conclusion & future work our experiments have shown the potential of historical pre-training for diachronically-relevant tasks. historical pre-training, however, did not benefit the processing of historical texts from all different time spans to the same extent. a more balanced pre-training dataset could help alleviate these issues. still, since collecting new data for certain time spans and genres is hindered by the scarcity of such material, researchers are left with the only option of up-sampling the available resources-c.f. @xcite . the benefits of upsampling for ranges of the diachrony that are lesser sourced should thus be explored. moreover, we have gained insight on the relative merit of different approaches to historical pre-training (pre-training from scratch vs. adapting a pre-existing model). this insight suggests a further experiment in which the "bert-based uncased" architecture is fine-tuned on the same dataset as macberth, and the resulting model is put to test alongisde macberth in order to see whether the claim holds true. however, considering the elevated cost of experimenting with mlm architectures, future research may want to refrain from costly practices like ablation studies and, instead, look at statistical modeling in order to find out the effect of particular design choices-e.g. can excessive sub-word tokenization be responsible for the drop in performance? finally, some of the benchmark tasks we imple-mented were designed ad-hoc to test the capabilities of mlms at handling historical text. future work should look into the deployment and evaluation of mlms in real-word humanities and dh scenarios in order to scale up the automated retrieval of otherwise difficult to access pieces of information. besides fine-tuning on appropriate downstream tasks, current nlp research points towards "prompt engineering" (see @xcite for a recent survey) as a promising approach. a appendix left quotation right quotation example he lov'd his country with too unskilful a tenderness. i love it to be grieved when he hideth his smiles. input he [tgt] lov'd [tgt] his country with too unskilful a tenderness.