GH is BERT – Training BERT from scratch for lexical semantic investigations across historical G erman language stages (2023)

Abstract: While static embeddings have dominated computational approaches to lexical semantic change for quite some time, recent approaches try to leverage the contextualized embeddings generated by the language model BERT for identifying semantic shifts in historical texts. However, despite their usability for detecting changes in the more recent past, it remains unclear how well language models scale to investigations going back further in time, where the language differs substantially from the training data underlying the models. In this paper, we present GHisBERT, a BERT-based language model trained from scratch on historical data covering all attested stages of German (going back to Old High German, c. 750 CE). Given a lack of ground truth data for investigating lexical semantic change across historical German language stages, we evaluate our model via a lexical similarity analysis of ten stable concepts. We show that, in comparison with an unmodified and a fine-tuned German BERT-base model, our model performs best in terms of assessing inter-concept similarity as well as intra-concept similarity over time. This in turn argues for the necessity of pre-training historical language models from scratch when working with historical linguistic data.

## introduction in historical linguistics, studying semantic change and the evolution of word senses has a longstanding tradition (e.g., @xcite @xcite @xcite . however, in nlp and computational linguistics, researchers only recently began to take an interest in the topic, focusing on the task of 'shift detection' (cf. @xcite , i.e., the identification of changes in word meaning over time. the task has been taken up in a semeval challenge on identifying lexical semantic change in english, german, swedish and latin (semeval-2020; @xcite , whose success has inspired several follow-up challenges focusing on different sets of languages, e.g., @xcite , russian @xcite , and spanish @xcite . the interest in the topic is fueled by the possibility to address the task of identifying lexical semantic change via pre-trained neural language models. in particular, recent work addresses the task via methodologies based on contextualized embeddings as generated by the state-of-the-art language model bert @xcite , exploring methodologies for how to measure, quantify and evaluate semantic change on the basis of these embeddings (see, e.g., @xcite @xcite . despite this recent surge of computational methodologies developed for lexical semantic change detection (lscd), there are still many historical linguistic research questions related to lscd which have not yet been touched upon computationally. from a historical linguistic perspective, one of the major shortcomings is the lack of temporal depth. that is, most computational studies focus on identifying change in the more recent past, within one language stage, e.g., comparing english data from the 19th century with data from the 20th century ce. while this renders feasible the application of pre-trained language models such as bert, which have been trained on contemporary data, and might be of interest for information retrieval applications, this is in general not what is of interest to the historical linguist. in historical linguistics, change is usually investigated across longer periods of time of more temporal depth, with change being assessed across language stages, e.g., from old english (5th-11th century ce) to middle english (12th-15th century ce), in order to be able to track sense evolutions in more detail (cf. @xcite . yet, given that prototypically, the language use as well as the orthography in the historical language stages deviate strongly from the contemporary language, this casts doubt on the applicability of the readily available pre-trained language models to research questions related to significantly older language stages. in this paper, we address this methodological gap by developing our own historical bert-based language model for german: ghisbert. ghis-bert is trained from scratch on corpus data covering all attested stages of historical german, i.e., old high german (c. 750-1050 ce, ohg), middle high german (c. 1050-1350 ce, mhg), early new high german (c. 1350-1650, enhg), and new high german (from 1650 onwards, nhg) (see, e.g., @xcite on the german periodization scheme). we illustrate the usability of our model for research questions related to lexical semantics in historical german by conducting a lexical similarity experiment across three language stages, mhg, enhg, and nhg. our experiment is based on measuring the cosine similarity between bert embeddings produced for ten concepts extracted from the swadesh (1955) list, i.e., culturally stable concepts which should occur frequently in each of the language stages. to test our model, we assess both, the intra-concept similarity over time as well as inter-concept similarites at each of the investigated time periods. in addition, we compare ghisbert's performance with a fine-tuned german bert-base model using the same training data and use the unmodified german model for baseline comparisons. we show that ghisbert performs better than the other models with respect to capturing intra-concept similarities over time as well as capturing lexical semantic interrelations between the investigated concepts. this highlights the usability of bert-based models for historical linguistic research questions related to lexical semantics, while at the same time emphasizing the necessity of pre-training language models with the relevant historical data. ## ghisbert: a historical german language model in this paper, we present ghisbert (german historical bert), a bert-based model trained from scratch on historical german data, covering all attested stages of the language, i.e., ohg, mhg, enhg, and nhg, with data going back to 750 ce. foot_3 ## lexical similarity and stability across language stages to test the applicability of our model to investigations of lexical semantic change in historical language stages, we conduct a case study which investigates whether the lexical semantic stability of ten swadesh concepts is captured adequately over time, i.e., across three consecutive historical language stages: mhg, enhg, and nhg. to do so, we compare ghisbert with bert-fine and bert-german via a lexical similarity analysis, assessing the inter-concept similarity at each time stage as well as the intra-concept similarity of each concept across time. ## results inter-concept similarity the inter-concept similarity at each time stage shows how similar the average embedding of each concept is to the average embeddings of all other concepts. in terms of the inter-concept similarity at each stage, ghisbert provides for the best results, presenting similarities that range between 0.18 and 1, representing low and high similarities between concepts adequately, while bert-fine and bert-german provide much higher similarities, see the heatmaps in figure 1 . in particular, bert-german produces very high simliarity values between concepts at each stage, i.e., values ranging between 0.87 and 1, not being able to capture the differences between the concepts. overall, ghisbert gives the most pronounced representation of synchronic inter-concept similarities. at the mhg stage, ei 'egg' shows the lowest similarity to all other concepts with all three models. this is an interesting text effect which is borne out in particularly by the ghisbert embeddings: in the mhg proportion of the data, ei only occurs in latin texts, referring to the 3rd person masculine pronoun ei 'he', and not to 'egg'. as such, it is no surprise that it differs from all other concepts. other lexical semantic similarities which are neatly captured by ghisbert at all stages are the relationship between animal concepts, e.g., fisch 'fish', vogel 'bird', and hund 'dog' show high simliarities to one another, and the interrelation between body parts, e.g., kopf 'head' and fuss 'foot'. in addition, frau 'woman', which is the only human, sentient concept, shows lower similarities than the other concepts to one another (with ei being an exception here). intra-concept similarity across time yet, what is striking, is that ghisbert adequately estimates a larger difference, i.e., a lower similarity, between mhg and enhg than between enhg and nhg, while this is not the case for the other two models, with a significant difference between the cos m e distributions of ghisbert to the other models (@xmath1.001). ghisbert's results are in line with broader linguistic developments in historical german (see, e.g., @xcite , for which mhg can be characterized as a major period of change, with a considerably freer word order and several strong phonological and morphological changes (e.g., vowel reduction), leading to the enhg period, thus reflecting a stronger change between mhg and enhg than between enhg and nhg. in addition, several of our qualitative observations fit well with the results produced by ghis-bert. for one, concepts which show lower similarities with respect to both cos m e and cos en , i.e., fisch 'fish' and fuss 'foot', show polysemy in the corpora from all three language stages, with fuss 'foot' referring to the body part, the 'foot' (bottom) of a mountain, and its usage as a measure of length. fisch 'fish' in turn is found in its biological as well as astrological usage and additionally occurs often in biblical contexts. for another, kopf 'head', which shows a comparably low cos m e but a large cos en similarity, seems to be undergoing change between mhg and enhg: in mhg, kopf is still mainly found in its historically older use as 'drinking vessel, cup' (cf. @xcite , which differs strikingly from the usage in enhg and nhg as 'head', and is no longer found in modern german. while this development stands out with ghisbert, it is less evident with the other models, see in sum, our lexical similarity analysis shows that ghisbert provides for the best results in terms of capturing the lexical semantic relationships between our ten target concepts in the historical language stages. the results produced by ghisbert present a more nuanced picture of synchronic as well as diachronic interrelations between target con-cepts than the results achieved via the unmodified and the fine-tuned bert-german models. overall, these findings are in line with our manual qualitative cross-checks of the underlying data. ## conclusion this paper provides evidence for the usability of bert-based models for investigations of lexical semantic change going beyond the contemporary language stage. more precisely, we show via a lexical similarity analysis that bert embeddings can be used for assessing inter-and intra-concept similarities across three historical german language stages, middle high german, early new high german, and new high german. in a systematic comparison, we show that pre-training a bert-based model from scratch with the relevant historical data provides for more adequate results than fine-tuning alone. this in turn highlights the relevance of pretraining neural language models with languagespecific data for lexical semantic investigations.