Lexical Semantic Change Annotation with Large Language Models (2025)

Abstract: This paper explores the application of state-of-the-art large language models (LLMs) to the task of lexical semantic change annotation (LSCA) using the historical German DURel dataset. We evaluate five LLMs, and investigate whether retrieval-augmented generation (RAG) with historical encyclopedic knowledge enhances results. Our findings show that the Llama3.3 model achieves comparable performance to GPT-4o despite significant parameter differences, while RAG marginally improves predictions for smaller models but hampers performance for larger ones. Further analysis suggests that our additional context benefits nouns more than verbs and adjectives, demonstrating the nuances of integrating external knowledge for semantic tasks.

## introduction the increasing application of natural language processing (nlp) methods to the humanities presents a range of challenges, particularly when working with historical or non-standard language data. one such challenge is the detection of lexical semantic change (lsc) @xcite , i.e. how words have shifted in meaning due to cultural, social, or linguistic contexts. although large language models (llms) have demonstrated considerable success in modern language tasks, their ability to effectively interpret historical texts is still limited. many works show how language models struggle with such longtail knowledge @xcite . linguistic limitations in particular then influence other computational research fields, such as the analysis of literary texts, where subtle shifts in meaning must be captured to correctly interpret a historical text via the lens of a historical reader. llms are predominantly trained on contemporary data, which may lack the necessary historical linguistic context to accurately process and interpret older texts. many tasks in the humanities that employ llms are directly or indirectly dependent on historical knowledge, including the recognition of historical facts, events, and discourse, as well as changes in word meaning over time (e.g., gay, awful, computer). when working with older text collections, this impacts applications such as historical sentiment analysis, the classification of historical texts, the analysis of narrative and character descriptions, and even machine translation of older documents. understanding how different llms represent historical semantics is crucial for researchers who work with historical texts, as it informs their choice of model for specific tasks. this paper investigates the performance of multiple state-of-the-art llms on the task of lsc annotation for historical german. the goal is to evaluate the models' ability to detect semantic shifts, and therefore to potentially infer which of the models would be the best to represent historical semantics via this proxy task. the research questions of this paper are the following: rq1: how well can the current state-of-the-art llms annotate lexical semantic change given two contexts and a target word? rq2: can historical, referential knowledge increase the performance of the lexical semantic change annotation task? ## lexical semantic change detection lexical semantic change detection (lscd) is a well-established subfield of computational linguistics and nlp. given a large diachronic text corpus (i.e. a corpus is that is divided into two or more time slices), the goal is to automatically detect which words have changed in meaning. these results are then compared to manually annotated gold datasets of word meaning shift, where annotations are either on a binary or graded (1-4) scale of relatedness @xcite . typically, word embeddings are used to detect lsc; both contextualized approaches relying on transformer-based models or approaches based on static word vectors are possible. with static approaches, word embedding models of each corpus slice are created individually and are then aligned (e.g. through orthogonal procrustes) @xcite . contextualized approaches first calculate sets of token embeddings for every word in question, taking the surrounding context into account. here, only one model is used, and one time slice corresponds to one set of token embeddings instead. the cosine distance of one word to itself in different time slices is then used to gauge its semantic "stableness", indicating whether the word has changed meaning or not. contextualized approaches could either average embeddings beforehand or calculate average pairwise distances instead @xcite . to evaluate lscd approaches, some datasets were already manually created for different languages. here, multiple contexts are strategically paired so that they contain the same word. human annotators evaluate these pairs, assessing how stable the word meaning within these two contexts appears to be. repeated annotation of these context pairs then results in a word usage graph, from which individual word senses or a single category of meaning shift can be inferred @xcite @xcite . in the remainder of the paper, this process will be referred to as lexical semantic change annotation (lsca). with the emergence of large decoder-only language models, such as chatgpt, the view on lscd has changed. previously, embeddings were the only reliable way of detecting lsc, simply because not enough training data are available to fine-tune a model on the task. so, lscd is currently a strictly unsupervised task. now, lscd can rely on the vast amount of knowledge that has been pre-trained into llms, which already show huge popularity with zero-shot (unsupervised) approaches. so far however, the results of using llms for lscd have been mixed. @xcite compared bert with gpt-3.5 for lscd by having the models rank 37 target words by degree of change, finding comparable performance between the two. in contrast, @xcite reported better performance when prompting gpt-4 to rate context pairs, outperforming the bert-based embedding approach. however, their study was limited to the short-term change dataset, tempowic. in this paper, the focus lies on lsca, where we assess meaning change at the instance level by prompting llms to annotate context pairs, and we evaluate through correct classification, not ranking. by applying this approach to the long-range change, german dataset durel, we aim to extend previous findings on the overall representation of semantic change in llms. ## summary of findings we can conclude that the open-domain llama3.3 model performs on par with the closed-domain gpt-4o model for the lsca task (though larger models tend to perform better overall), suggesting that both models contain similar knowledge of historical semantics. providing additional context through a historical encyclopedia yields mixed results: the augmentation only slightly positively impacts the smaller model, and the performance is highly dependent on the target word. overall, we found that llms may process semantics differently than humans would, as the models put a larger emphasis on the domain in which a word is used. future work will need to address this challenge, uncovering the reasoning for the models behavior as well as steering models more towards a human intuition of lexical semantics.