How to age BERT Well: Continuous Training for Historical Language Adaptation (2025)

Abstract: As the application of computational tools increases to digitalize historical archives, automatic annotation challenges persist due to distinct linguistic and morphological features of historical languages like Old English (OE). Existing tools struggle with the historical language varieties due to insufficient training. Previous research has focused on adapting pre-trained language models to new languages or domains but has rarely explored the modeling of language variety across time. Hence, we investigate the effectiveness of continuous language model training for adapting language models to OE on domain-specific data. We compare the continuous training of an English model (EN) and a multilingual model (ML), and use POS tagging for downstream evaluation. Results show that continuous pre-training substantially improves performance. We retrain a modern English (EN) model and a Multi-lingual (ML) BERT model for OE. We confirmed the effectiveness of continuous pre-training for language adaptation and downstream evaluation utilizing part-of-speech (POS) tagging, advancing the potential to understand the unique grammatical structures of historical OE archives. More concretely, EN BERT initially outperformed ML BERT with an accuracy of 83% during the language modeling phase. However, on the POS tagging task, ML BERT surpassed EN BERT, achieving an accuracy of 94%, which suggests effective performance to the historical language varieties.

## introduction applying natural language processing (nlp) techniques to historical archives is a complex undertaking exacerbated by data scarcity @xcite . the limited availability of historical training data has impeded the advancement of nlp applications in archives such as oe due to the laborintensive task required for manual annotation, leaving this domain relatively underexplored @xcite . efforts to reduce the cost and human labor in sequence labeling tasks, such as pos tagging through semi-automation, have fallen short of capturing the full complexity of morphosyntactic alignment, highlighting the need for manually annotated corpora to obtain meaningful insights in nlp tasks involving historical archives @xcite . despite the capabilities of automated techniques in handling different levels of linguistic annotation @xcite @xcite , manual annotation, though tedious, is an effective method to handle the complexities of varying dialects and the intricate linguistic phenomena of historical language @xcite . furthermore, orthographic inconsistencies in historical archives pose significant challenges to corpusbased analytical linguistic techniques, including automated tagging, which can sometimes diminish the effectiveness and reliability of the analytical outcome @xcite . one approach to overcome this issue is to normalize the oe data to modern english, thereby enhancing the accuracy of pos tagging @xcite , with manual normalization shown to improve performance across the nuanced historical linguistic features and spelling variations of ancient text @xcite . however, normalization models require annotated training data, which is not available for all varieties of historical languages. in this paper, we focus on re-training a discriminative language model (i.e. bert) on oe, a west germanic language related to old frisian and old saxon @xcite , and demonstrate the refinement of historical archives with the iswoc corpus (iswoc, 2014), complete corpus of anglo-saxon poetry @xcite , and the plaintext wikipedia dump 2018 @xcite . our paper focuses on oe, an earlier stage (mid-fifth century), of the language with unique morphological patterns and features @xcite . we use pos tagging as a downstream evaluation, to evaluate the effectiveness of the re-training procedure. an example of sentences annotated with pos tags can be seen in figure 1 . our contributions can be summarized as follows: ## old english historical oe is a west-germanic language connected to old frisian and old saxon within the ingvaeonic language used in england following the settlement of the angles, saxons, jutes, and frisians from britain @xcite . during the mid-fifth century, english-speaking settlers known as the anglo-saxons established themselves in britain until the norman conquest. oe was inflected across various pos to denote first, second, and third person, singular and plural forms, and for mood, indicative, subjunctive, and imperative. @xcite the oe alphabet (figure 2 ) consists of 24 letters @xcite . these dialects played a critical role in shaping the development of the english language. american regional dialects also have origins in oe dialects, with standard modern english primarily influenced by the mercian dialect @xcite . ## related work research on historical text processing has spanned various low-resourced languages, with efforts dedicated to refining nlp methodologies for better handling ancient and historical data. previous studies have concentrated on overcoming the unique challenges posed by historical archives, such as developing tools and techniques to improve pos tagging accuracy. the preliminary efforts have paved the way for more effective nlp applications in historical linguistics, offering new opportunities for studying and preserving invaluable linguistic resources @xcite . prior work included (rögnvaldsson and helgadottir, 2008) study on morphosyntactic tagging for old norse texts. sanchez-marco et al. (2011) also adapted methods for old spanish by enhancing dictionaries with word variants and retraining taggers with limited annotated data, demonstrating some applied nlp techniques. sukhareva and chiarcos (2014) mapped annotations from english to ancient germanic languages highlighting the potential to advance our understanding of ancient texts (yang and eisenstein, 2016). the qiu and xu (2022) study concluded that incorporating historical data during training improved the capacity of bert for diachronic semantic analysis. in our experiments, we rely on the domainspecific pre-training technique with unlabeled data using masked language modeling (mlm) to enable bert and mbert to learn general language patterns from the unannotated oe archives for the digitization of important works like the oe beowulf @xcite poem to preserve historical records @xcite . ## method language modeling in the first stage of the experiment, the bert models underwent training to predict masked tokens (figure 3 ) using an unlabeled oe corpus following the original procedure proposed by @xcite . the unsupervised learning process enabled the models to learn underlying patterns from the raw oe data without the constraints of pre-existing labels @xcite . the goal of the pre-training phase was to provide the models with a foundational understanding of oe language patterns and morphologimodel pos lm-pos en bert 86.37 92.16 mbert 88.79 93.70 ## results language modeling before the unsupervised task, en bert and mbert demonstrated closely 3 ). during language modeling, en bert exhibited stable accuracy across various configurations, with minor deviations suggesting consistent learning and effective convergence on the linguistic structures within oe (figure 3 ). the stability underscored the capacity of en bert to adapt to historical linguistic patterns during the unsupervised phase, forming a robust basis for subsequent tasks. although mbert started with a higher accuracy, the model was quickly outperformed by en bert when training on more data, suggesting differing adaptation capabilities (figure 3 ). ## analysis performance the personal pronouns (pp) label attained the highest f1 scores, with mbert recording 0.98, closely followed by en bert achieving 0.97 ( tagging discrepancies in some instances, although the bert models indicated a high confidence level in predicting the pos label for some tokens, the predictions were incorrect, while in a few cases, lower confidence levels aligned with correct classifications (appendix f). misclassifications tagging discrepancies observed throughout the corpus showed the predicted frequency for adjectives (a-) indicated an overprediction, manual inspection revealed that this is mainly due to contextual ambiguities in the aelfric's lives of saints archive. other notable discrepancies included challenges predicting the conjunctions (c-) label, misclassifications for subjunctions (g-) and pronouns (pp), and underpredictions for adverbs (df) and possessive pronouns (ps) (appendix g). low predictions underrepresentation of labels in the west saxon gospels, particularly for foreign words (f-) and interjections (i-), recorded zero predictions in a few instances despite having actual labels, indicating the challenges of the models to recognize less common pos categories (appendix h). the en bert model also failed to make any predictions for interrogative adverbs (du) despite 53 representations of the label throughout the biblical archive (appendix g). ## conclusion and future work in this paper, we introduced, to the best of our knowledge, the first historical language model specifically developed for oe. we demonstrated that retraining on limited data can lead to substantial improvements in performance, as evidenced by state-of-the-art scores in part-of-speech (pos) tagging @xcite . the pretraining of the bert models on raw historical oe archives enhanced the pos tagging performance. the fine-tuning of the bert models on a manually annotated oe corpus allowed the models to refine predictions to achieve high accuracy (figure 3 ). the findings underscored the value of combining unsupervised and supervised training techniques to enhance pos tagging for historical languages. nevertheless, our analysis highlighted that employing nlp techniques on historical oe archives is a difficult task. future research should address the misclassification errors while developing strategies to enhance the generalization of the unique grammatical structures inherent in oe, including testing different models to optimize performance. a hyperparameters model dropout rate learning rate weight decay batch size steps / epochs optimizer lm 0.1 1e-3 0.01 8, 16, 32 10k adamw pos 0.1 2e-5 0.1 8, 16, 32 10 adamw