Definition generation for lexical semantic change detection (2024)

Abstract: We use contextualized word definitions generated by large language models as semantic representations in the task of diachronic lexical semantic change detection (LSCD). In short, generated definitions are used as ‘senses’, and the change score of a target word is retrieved by comparing their distributions in two time periods under comparison. On the material of five datasets and three languages, we show that generated definitions are indeed specific and general enough to convey a signal sufficient to rank sets of words by the degree of their semantic change over time. Our approach is on par with or outperforms prior non-supervised sense-based LSCD methods. At the same time, it preserves interpretability and allows to inspect the reasons behind a specific shift in terms of discrete definitions-as-senses. This is another step in the direction of explainable semantic change modeling.

## introduction and related work lexical semantic change detection (lscd) methods up to now have mostly been based on token embeddings produced by large language models. while efficient, when measured on the existing benchmarks like diachronic word usage graphs @xcite , these methods are largely non-interpretable and produce rather abstract 'change scores'. on the other hand, historical linguistics usually deals with semantic change in terms of discrete and interpretable senses being lost or gained (or changing their frequency). recently, a number of works were published which made an attempt to bridge this gap. in particular, @xcite proposed a sense distribution based lscd method. basically, they perform word sense disambiguation (wsd) on every occurrence of a target word in two diachronic corpora, using pre-trained sense embeddings (based on wordnet and babelnet). once all the occurrences are assigned a sense, the sense frequency distributions are compared between two time periods to quantify the semantic change. this approach preserves the possibility to interpret these shifts, e.g., by analyzing which sense is 'responsible' for the shift. we argue that while such methods constitute a significant advance for lscd, they are inherently limited by their reliance on a pre-defined sense inventory. even the best ontologies like babelnet can miss important senses, especially when dealing with chronologically recent text data. for many languages, good ontologies simply do not exist. thus, we propose to replace retrieving a fitting sense for a given target word usage from an external ontology by generating a dictionary-like contextualized definition for this specific occurrence, using a large language model (llm). these definitions serve as semantic representations of target word usages in the lscd pipeline. the usage of generated definitions as semantic representations in lscd was first proposed by @xcite and further developed by @xcite , but they did not conduct comprehensive empirical evaluations for semantic change detection per se. in this paper, we fill in this gap and actually test definitions as representations on the existing diachronic semantic change benchmarks. we show that our method yields competitive results, often outperforming @xcite , without relying on any manually created lexical database, but at the same time preserves interpretability via human-readable definitions of senses. the contributions of this paper are as follows: ## data we experiment on english, norwegian and russian benchmarks, since for these languages, definition generators were already created by @xcite . however, scaling to other languages is comparatively easy and requires only a small dataset of contextualized definitions (see §3). to evaluate the performance of a semantic change detection system, we used existing lscd datasets (diachronic corpora and gold scores for the target words): the english part of the sem-eval'20 task 1 foot_0 @xcite , nor-diachange foot_1 @xcite for norwegian, and rushifteval foot_2 @xcite for russian. nordiachange actually contains two datasets and rushifteval contains three datasets, with different time period pairs under comparison (for norwegian, the sets of target words are also different). the russian datasets feature the highest number of target words (99, as compared to 37 in english and norwegian datasets). note that semeval'20 task 1 included two subtasks: binary classification of words (changed or not changed) and ranking the words by the degree of their change. in this work, we focus only on the ranking task: 1) because the russian dataset does not include binary labels, and 2) because even in the english and norwegian datasets the binary labels are in many ways derivatives of the change scores. 4 it is also important to note that the rushifteval dataset was used in a shared task of the same name @xcite . however, the scores in its leaderboard or in @xcite are not directly comparable to the scores in this work, since in the shared task, the dataset was split into development and test parts, so that the participants were able to tune their systems on the development set. in this paper, we focus on unsupervised approaches, aiming to avoid the necessity of tuning hyperparameters and leaving this for future work. ## definition generation methods our general pipeline of generating definitions from an llm ('defgen') is similar to @xcite . we use the definition generation models presented by @xcite . they are based on mt0-xl @xcite and were fine-tuned on wordnet @xcite , oxford @xcite and cod-woe @xcite for english, codwoe for russian and bokmålsordboka 6 for norwegian. the models are extensively described and evaluated in @xcite al. (2024 all codwoe datasets originally come from wiktionary, so it is straightforward to extend this method to any major language. as a prompt for the llm, we used the original example usage followed by the question 'what is the definition of target-word?' (in english, norwegian or russian). additionally, we conducted a series of experiments with different generation strategies. prior work used only basic greedy decoding, while we experimented with alternative strategies such as multinomial sampling, beam search, and diverse beam search @xcite . note that these experiments do not deal with lscd -they only evaluate the capability of the models to generate definitions similar to the gold ones. the results of our experiments for english, russian and norwegian, evaluated with bleu, rougel and bertscore, are shown in in the experiments below, we use definitions generated with all three approaches: greedy decoding, beam search and diverse beam search, to explore to what extent the definition generation performance translates to lscd performance. ## semantic change detection with generated definitions in this section, we describe the methods we used in our experiments, starting with the xlm-r and sense embeddings baselines from prior work. next, we present our own approach based on definition embeddings and the standard lscd aggregation algorithms: prt (prototype embeddings), apd (average pairwise distance), and their arithmetic mean (am) @xcite . finally, we move on to our interpretable 'definitions-as-senses' method which employs definitions directly (as text strings) and merges similar definitions together using levenshtein edit distance. ## qualitative analysis when predicting semantic change on the basis of the distribution of senses (or definitions-as-senses), it becomes possible to analyze and interpret this change, by simply looking at the distribution of entries (senses) which contribute most to the difference. let's consider the top performing set of english definitions (generated with greedy decoding and merged in the minimalist approach with the edit distance threshold of 50). for the word 'ball' in the semeval'20 time periods, the jsd metric yields a high change score of 0.83. after looking at the list of top frequent definitions-as-senses for this word, it becomes clear that its dominant sense has changed: while in time period 1 (19th century), more than 82% of all usage examples were given the definition 'a spherical object especially one that is round in shape' with 'a party' being the next most frequent sense, in time period 2 (20th century), 80% of 'ball' usages were defined as 'the object hit in a game', with similar definitions following this one in the top-frequent list. this is a clear evidence of the 'dancing party' sense for the word 'ball' becoming obsolete in the 20th century, with the sports-related sense taking the dominant position. for the noun 'attack', the system predicts a medium change degree of 0.34. again, it is straightforward to find the reasons. while in both periods the dominant sense is the same ('an instance of military action against an enemy'), in the time period 2 its ratio drops down from 87% to 80%, and we observe the appearance of a new rare but not unique sense of 'an instance of sudden violent activity of a bodily organ or system especially the heart'. this is a linguistically plausible explanation of a semantic shift, much more useful to a lexicographer than a raw change score. see appendix b for more examples in english, norwegian and russian. ## conclusion in this paper, we showed how contextualized dictionary-like definitions generated by a finetuned large language model can be used for the practical downstream task of semantic change detection (in particular, ranking words by the degree of their diachronic semantic change). following @xcite , we treat generated definitions as semantic representations of the target words. these definitions (and their frequency distributions) can be used 'as is', following @xcite 's method, or after embedding them in a dense vector space using any available sentence embedding model. the second method yields results which are empirically better (con-sidering existing benchmarks for three different languages), but the first method makes it much easier to interpret and explain semantic change, since it operates directly on generated definitions in their textual forms. we consider this study a small step towards more explainable semantic change modeling, which can be closer to linguistically plausible discrete 'senses', while still retaining empirical performance. in the future, we plan to explore to what extent it is possible to improve our results by tuning hyperparameters on development sets (where available). another direction for future research is using more advanced string distance metrics like weighted levenshtein distance, longest common subsequence ratio, or word mover's distance @xcite , in the hope that it will allow to handle more nuanced similarities and dissimilarities between generated definitions.