A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change (2024)

Abstract: Contextualized embeddings are the preferred tool for modeling Lexical Semantic Change (LSC). Current evaluations typically focus on a specific task known as Graded Change Detection (GCD). However, performance comparison across work are often misleading due to their reliance on diverse settings. In this paper, we evaluate state-of-the-art models and approaches for GCD under equal conditions. We further break the LSC problem into Word-in-Context (WiC) and Word Sense Induction (WSI) tasks, and compare models across these different levels. Our evaluation is performed across different languages on eight available benchmarks for LSC, and shows that (i) APD outperforms other approaches for GCD; (ii) XL-LEXEME outperforms other contextualized models for WiC, WSI, and GCD, while being comparable to GPT-4; (iii) there is a clear need for improving the modeling of word meanings, as well as focus on *how*, *when*, and *why* these meanings change, rather than solely focusing on the extent of semantic change.

## introduction lexical semantic change (lsc) is the problem of automatically identifying words that change their meaning over time @xcite @xcite . the interest in this problem has been significantly fueled by the advent of word embeddings and modern language models. after more than a decade of ad hoc evaluation, a new evaluation framework was recently introduced, aimed at assessing and comparing the performance of different models and approaches @xcite . this framework was adopted to create benchmarks in different languages. each benchmark includes a diachronic corpus spanning two time periods, along with a list of target words and tasks aimed at detecting word meaning change over time. the most popular task, known as graded change detection (gcd), consists of ranking a list of target words based on their degree of change. the initial excitement for word embeddings prompted researchers and practitioners to solve the gcd task by using static embedding models @xcite . however, the shift towards more advanced transformer architectures has established the use of contextualized embedding models as the preferred tool for addressing gcd @xcite . on one hand, these models distinguish the different meanings of a word by contextualizing each occurrence with a different embedding. on the other hand, the generation and processing of contextualized embeddings across entire corpora pose scalability challenges, both in terms of time and memory consumption @xcite . different strategies have been adopted to tackle these challenges, leading to a proliferation of evaluations across diverse settings (e.g., limited samples of benchmarks) and conditions (e.g., pre-trained vs. fine-tuned models). as a result, these evaluations on gcd hinder a fair comparison among the performance of different models and approaches, thereby deviating from the original goal of the framework. moreover, while the gcd task is attracting more and more evaluations, it addresses only a partial complexity inherent to the established framework. notably, the framework includes three distinct aspects also discussed in @xcite : as a matter of fact, when contextualized embedding models are used to address gcd, cosine similarities among word embeddings serve as surrogate for (i), without evaluation focused on this aspect. additionally, most approaches to gcd, pass from (i) to (iii), sidestepping the intermediate aspect (ii). that is, they quantify semantic change as overall proximity variation, without inducing word senses. consequently, while these approaches can be evaluated through gdc, they preclude the interpretation of which meaning(s) have changed. we argue that (i) and (ii) are equally relevant aspects as (iii), constituting a fundamental aspect of the lsc problem. their evaluation can provide valuable insights into the current state of lsc modeling, while offering a broader perspective on contextualized embedding models in natural language processing (nlp). 1 original contribution of our work our evaluation reveals that gpt-4 obtains comparable performance to xl-lexeme. in contrast to the limited accessibility foot_1 and high associated cost foot_2 of gpt-4, xl-lexeme is a considerably smaller, open-source model. thus, we argue that the use of gpt-4 is not justified for modeling the lsc problem. 2 background and related work the established lsc framework adheres to the novel annotation paradigm for word senses and encompasses (i-iii) @xcite . (i) human annotators provide semantic proximity judgments for pairs of word usages sampled from a diachronic corpus spanning two time periods. (ii) word usages and judgments are represented as nodes and edges in a weighted, diachronic graph, known as diachronic word usage graph (dwug). this graph is then clustered with a graph clustering algorithm and the resulting clusters are interpreted as word senses (see figure 1 ), thus sidestepping the need for explicit word sense definitions. finally, (iii) given a word, a ground truth score of semantic change is computed by comparing the probability distributions of clusters in different time periods, e.g., a cluster with most of its usages from one time period indicates a substantial semantic change. originally, the framework was proposed in a shared task at semeval-2020, including benchmarks for four languages, namely english (en), german (de), swedish (sw), and latin (la) @xcite . benchmarks for italian @xcite , russian (ru) @xcite , spanish (sp) @xcite , norwegian (no) @xcite , and chinese (zh) @xcite have recently been introduced. each benchmark foot_3 consists of a diachronic corpus and a set of target words over which the human annotation was conducted. the evaluation over a benchmark is typically conducted through the gcd task where the goal is to rank the targets by degree of semantic change across the corpus. the spearman correlation between predicted and ground truth scores is used to evaluate models and approaches. ## evaluation setup we consider benchmarks for eight different languages: en, la, de, sv, es, ru, no, and zh (see aligning with the unsupervised nature of the lsc framework, we compare pre-trained models without performing additional fine-tuning (see ## comparing approaches for gcd we evaluate different approaches for gcd using spearman correlation @xcite between computational predictions and ground truth scores. specifically, we process the embeddings of each target using the following approaches. ## computational annotation we evaluate different models on reproducing human judgments (i), the inferred word senses (ii), and the resulting change scores (iii). we leverage models as annotators, hence the term computational annotator, using the same procedure employed for benchmark construction @xcite @xcite @xcite . however, we cannot evaluate la as the benchmark was developed differently nor (ii) for the ru benchmark since no word senses were provided @xcite . ## concluding remarks we have performed a first-ever evaluation of models and approaches for modeling lsc under equal settings and conditions, over eight different languages. first, we evaluated different models combined with standard approaches to the popular gcd task. in particular, we consider bert, mbert, xlm-r, xl-lexeme as pre-trained models, apd and prt as form-based approaches, and ap+jsd and widid as sense-based approaches. we find that the xl-lexeme consistently outperforms other models across all approaches, and thus should be used as the defacto standard. we also find that form-based approaches significantly outperform sense-based approaches, with apd as the best approach for gcd. among the sense-based approaches, we find that evolutionary clustering is advantageous in contrast to static clustering and should be a focus of future work. we additionally extended the evaluation to includes the wic and wsi tasks, both inherently crucial to solve the complex task of lsc. we compare gpt-4 to the previous models and find that gpt-4 and xl-lexeme both perform close to human-level while the other models obtain only low-moderate performance. due to the costs associated with using gpt-4, it is not affordable to evaluate it on the remaining languages. since xl-lexeme obtains results close those of gpt-4, even beating it for the wic task, we argue that xl-lexeme can be used for lsc tasks as a affordable, scalable solution. all in all, considering the current state of the lsc modeling, we argue that only obtaining state-of-the-art performance on gcd does not solve the lsc problem, as there is a clear need to distinguish the different senses of a word and how these evolve over time @xcite @xcite . gcd maintains relevance for identifying words that have changed across multiple time periods in need of further sense-based modeling. gcd also serves to quantify the change on the level of vocabulary. in conclusion, we offer a first comparable evaluation of contextualized word embeddings for lsc and establish clear settings that should be used for future comparison and evaluation. with this work, we want to raise awareness of the current trend of the community in modeling only the gcd task. our aim is to shift the focus from merely assessing how much to how (de @xcite , when, and why, prompting the development of both unsupervised and supervised approaches for addressing the full spectrum of lsc.