Analyzing Semantic Change through Lexical Replacements (2024)

Abstract: Modern language models are capable of contextualizing words based on their surrounding context. However, this capability is often compromised due to semantic change that leads to words being used in new, unexpected contexts not encountered during pre-training. In this paper, we model semantic change by studying the effect of unexpected contexts introduced by lexical replacements. We propose a replacement schema where a target word is substituted with lexical replacements of varying relatedness, thus simulating different kinds of semantic change. Furthermore, we leverage the replacement schema as a basis for a novel interpretable model for semantic change. We are also the first to evaluate the use of LLaMa for semantic change detection.

## introduction the major advancement that novel language models (lms) have brought is the ability to dynamically generate contextualized representations (i.e., embeddings) based on specific usage context. when words are used in contexts similar to those encountered during training, lms can easily differentiate, in a computational way, between word meanings. like in the case of rock in the sentences sitting on a rock and listening to rock. however, when an existing word in our vocabulary gains a new meaning through semantic change, lms' ability to differentiate that meaning can be affected. this stems from the fact that semantic change is evidenced through new contexts that were previously unknown for the word. sometimes, the new meaning is novel to the dictionary, for example, the metaphorical web-meaning of surfing. other times, the meaning is already in existence and get the word as a new referent. this is, for example, the case for happy. it used to mean exclusively to be lucky and then gained the meaning of happiness. in an inverse process, the word gay lost its meaning of happiness and began to refer exclusively to homosexuality. one can think of this process of semantic change to be a lexical replacement of the word happy into the context of gay, like in the following sentence 1 the heart is sportive, light, and gay, life seems a long glad summer's day 2 when using lms, the representation of a word w is based on (i) the pre-trained knowledge that the model has about w given its position in the context, and (ii) the context c in which w is used. thus, when this replacement happens, lms experience a tension between the existing sense/s of happy (which do not include happiness) and the meaning of the new context (which does indicate happiness). due to semantic change, lms do not known the relationship between the new context c and the replacement word r. as a consequence, the representation of r (i.e., happy in the sense to be lucky) and the representation of c (i.e., the context of gay in the sense of happiness) pull in different directions challenging the lms' ability to contextualize @xcite . the tension increases as the gap between the data used for training the model, and the data on which the model is applied grows larger. indeed, the lms we use serve as the lens through which we view the studied texts: if our texts are contemporary with the pre-training, the gap is likely to be minimal. if, however, we intend to study historical or other outof-domain corpora through lms trained on modern text, this gap can be arbitrarily large and have major effects on follow-up studies. thus, using lms for modeling relationships beyond their pre-trained knowledge will likely result in an underestimation of semantic change. our contributions: in this paper, we propose a replacement schema to study the tension experienced by lms when words undergo semantic change. such schema involves replacing a word w in the context c with a replacement r to analyze how the representation of r differs from the original representation of w. given a word w, our experiments systematically show that lms (i.e., bert, mbert, xlm-r) experience a tension between the pre-trained knowledge of w and the new context of a gained meaning. this tension differs across linguistic relations, namely synonymy, antonymy, and hypernymy. we then use the introduced schema for detecting semantic change. our experiments show that, when random replacements are used to simulate synthetic semantic change, the use of a clustering algorithm (i.e, affinity propagation) falls short to differentiate meanings and detect such change. furthermore, we use the replacement schema to introduce a new interpretable model for semantic change detection, while being comparable with state-of-the-art for english. finally, to further investigate the tension of lms and semantic change, we compare the use of a predefined set of replacements with word substitutes generated by lms (i.e., bert, llama 2). our experiments show that smaller lms are less able to provide substitutes that handle changing contexts, while llama 2 significantly outperform the lms. notably, to the best of our knowledge, this represents the first experiment in the current literature to use llama 2 to model semantic change and only the third paper to use any generative model @xcite . ## related work for this paper, relevant work pertains both to contextualization of modern lms and the field of lexical semantic change. modern contextualized lms leverage the transformer architecture to capture the semantics of words @xcite . their success in solving nlp tasks has prompted numerous studies to explore the nature and characteristics of their contextualization ability. @xcite ; coenen et al. (2019); cai et al. (2021 ); jawahar et al. (2019) shed light on the geometry of the embedding space. serrano and smith (2019); bai et al. (2021); guan et al. (2020) investigate the interpretability of the attention mechanism. yenicelik et al. (2020); garí soler and apidianaki (2021); kalinowski and an (2021); haber and poesio (2021) examine the clusterability of word representations. abdou et al. (2022); hessel and schofield (2021); mickus et al. (2020); wang et al. (2021) analyze the impact of word position in the embeddings generation. coenen et al. (2019); levine et al. (2020); pedinotti and lenci (2020) study how word meaning are represented in the embedding space. recent work have focused on a related aspect, namely adapting lms to improve their temporal contextualization. this challenge has been addressed across various applications such as named entity recognition @xcite , fake news detection @xcite ), text summarization @xcite , and lexical semantic change @xcite @xcite . nonetheless, while temporal domain adaptation can improve performance across various tasks, @xcite demonstrated that temporal contextualization may not always be a concern. in our work we complement existing research by using lexical replacements as a proxy to analyze how language models contextualize words that have undergone semantic change. lexical semantic change (lsc) is the task of automatically identifying words that change their meaning over time @xcite . the task is recently gaining more and more attention, acting as evaluation task to assess the capability of lms in capturing word meanings across diachronic corpora @xcite @xcite . current benchmarks consist of a diachronic corpus spanning two time periods t 1 and t 2 and a reference set of target words t annotated with a degree of semantic change between t 1 and t 2 @xcite @xcite . 3 the main goal is to rank the target words in t according to their degree of semantic change. currently, contextualized embeddings represents the state-of-the-art solution for addressing lsc. approaches to lsc relying on contextualized embeddings are typically distinguished into two main categories: form-based approaches and sensebased approaches @xcite . the former captures semantic change by solely relying on similarities among raw embeddings without depending on sense disambiguation and representation. given a word w, a common strategy involves averaging all the embeddings of w from t 1 and all the embeddings of w from t 2 , and modeling the change as the cosine similarity of the average representations (prt) @xcite . the latter generally use clustering algorithm like affinity propagation to identify senses and subsequently model the change as divergence of cluster distributions (jsd) @xcite . however, when form-based approaches are employed, interpreting the detected change is not supported as they do not model each individual sense of a word. in contrast, when sense-based approaches are employed, they typically represent clusters of word usages rather than word meanings @xcite . as a result, although a new powerful lms has recently been introduced for modeling the semantics of words @xcite , the interpretation of which meaning of a word has changed and in which way, remains an open challenge. in this regard, our work is related to the novel substitute-based approaches to lsc, which interpret word meaning by generating substitutes of words in context @xcite @xcite . on one hand, word substitutes represents relevant keywords to aid the interpretation of senses. on the other hand, the generation process can only provide substitutes according to training data, and as we show in section 5.3, lms lack the knowledge to adapt to semantic changes. to this end, we propose a novel interpretable approach based on a pre-defined set of lexical replacements rather than generated substitutions. ## methodology in our experiments, we leverage a replacement schema to investigate the tension experienced by pre-trained lms due to semantic change. this involves analyzing the variations in embedding representations when a target replacement is introduced. for instance, by replacing a target like cat with a replacement like chair in a specific context like: the cat target ← chair replacement was purring loudly . ## tension caused by semantic change we analyze the tension experienced by lms by comparing the embedding of a target word w in the original sentence c to the embedding of the replacement word r in the same sentence c. to perform this comparison, we rely on the cosine distance between the embeddings of w and r. we refer to this as the self-embedding distance (sed). concretely, if w and r are split into multiple sub-words by the model, we calculate the average embeddings of the corresponding sub-words. this approach ensures the preservation of the same number of tokens in the original and synthetic sentences and enables accurate distance calculations. the less plausible the relationship between the context c and the replacement word r for lms, the higher the sed, leading them to rely on the pretrained knowledge of r to contextualize r in context. when there is a large mismatch between the meanings of the replacement word r and the context c, as is the case with the random replacement, then the sed is the highest. ## semantic change we argue that our findings in section 4 regarding the tension between a word and its context has important implications when pre-trained lms are used for modeling semantic change as we will show in this section. ## discussions and conclusions in this paper, we study semantic change using lexical replacement. from the point of view of the replaced word, a semantic change takes place as the word gains contexts which it has not encountered previously. when the replacement is closely related to the target word, for example by synonymy, the novelty of the context for the replacement word should be low. however, novelty will increase as the relation between the target and replacement becomes more distant. we are assuming that the replacements based on synchronic relations will offer insights into semantic change diachronically. to test this hypothesis, we used self-embedding distance (sed) when the context stays the same, using all layers of bert, mbert and xlm-r across four part of speech. not surprising, we found that the self-embedding distance is smallest for synonym replacements and highest for the random replacements. and like @xcite , we found that more contextualization happens across the last layers. for the different models, we also find slightly different behaviors. however, consistently, adverbs and adjective have lower sed scores than verbs and nouns. we show that hypernymy is a more distant relation for lms than antonymy and synonymy we then employ replacements for measuring the degree of semantic change. for this, we generate synonym replacements using wordnet, for each word in the english portion of the semeval-2020 task 1 benchmark. we assume that if a word has not experienced semantic change, the sed between the replacements and the target word are similar across time. if however, a word has experienced semantic change resulting in context changes, sed scores will be different over time as the replacements will be more distantly related to the contexts. this method offers a novel interpretable semantic change detection. finally, we ask the lms themselves to generate substitutions for a target word in the english semeval data.