Can Word Sense Distribution Detect Semantic Changes of Words? (2023)

Abstract: Semantic Change Detection of words is an important task for various NLP applications that must make time-sensitive predictions. Some words are used over time in novel ways to express new meanings, and these new meanings establish themselves as novel senses of existing words. On the other hand, Word Sense Disambiguation (WSD) methods associate ambiguous words with sense ids, depending on the context in which they occur. Given this relationship between WSD and SCD, we explore the possibility of predicting whether a target word has its meaning changed between two corpora collected at different time steps, by comparing the distributions of senses of that word in each corpora. For this purpose, we use pretrained static sense embeddings to automatically annotate each occurrence of the target word in a corpus with a sense id. Next, we compute the distribution of sense ids of a target word in a given corpus. Finally, we use different divergence or distance measures to quantify the semantic change of the target word across the two given corpora. Our experimental results on SemEval 2020 Task 1 dataset show that word sense distributions can be accurately used to predict semantic changes of words in English, German, Swedish and Latin.

## introduction scd of words over time has provided important insights for diverse fields such as linguistics, lexicography, sociology, and information retrieval (ir) @xcite @xcite . for example, in ir one must know the seasonal association of keywords used in user queries to provide relevant results pertaining to a particular time period. moreover, it has been shown that the performance of publicly available pretrained llms declines over time when applied to emerging data @xcite @xcite because they are trained using a static snapshot. moreover, @xcite showed that the temporal generalisation of llms is closely related to their ability to detect semantic variations of words. a word is often associated with multiple senses as listed in dictionaries, corresponding to its different meanings. polysemy (i.e. coexistence of several possible meanings for one word) has been shown to statistically correlate with the rate of semantic change in prior work @xcite @xcite . for example, consider the word cell, which has the following three noun senses according to the wordnet foot_0 : (a) cell%1:03:00 -the basic structural and functional unit of all organisms, (b) cell%1:06:04 -a handheld mobile radiotelephone for use in an area divided into small sections, and (c) cell%1:06:01 -a room where a prisoner is kept. here, the wordnet sense ids for each word sense are shown in boldface font. mobile phones were first produced in the late 1970s and came into wider circulation after 1990. therefore, the sense (b) is considered as a more recent association compared to (a) and (c). given two sets of documents, one sampled before 1970 and one after, we would expect to encounter (b) more frequently in the latter set. likewise, articles on biology are likely to contain (a). as seen from this example the sense distributions of a word in two corpora provide useful information about its possible meaning changes over time. given this relationship between the two tasks scd and wsd, a natural question arises -is the word sense distribution indicative of semantic changes of words? to answer this question, we design and evaluate an unsupervised scd method that uses only the word sense distributions to predict whether the meaning associated with a target word w has changed from one text corpora c 1 to another c 2 . for the ease of future references, we name this method sense-based semantic change score (sscs). given a target word w, we first disambiguate each occurrence of w in each corpus. for this purpose, we measure the similarity between the contextualised word embedding of w, obtained from a pre-trained masked language model (mlm), from the contexts containing w with each of the pre-trained static sense embeddings corresponding to the different senses of w. next, we compute the sense distribution of w in each corpus separately. finally, we use multiple distance/divergence measures to compare the two sense distributions of w to determine whether its meaning has changed from c 1 to c 2 . to evaluate the possibility of using word senses for scd, we compare the performance of sscs against previously proposed scd methods using the semeval-2020 task 1 (unsupervised lexical scd) @xcite benchmark dataset. this task has two subtasks: (1) a binary classification task, where for a set of target words, we must decide which words lost or gained sense(s) from c 1 to c 2 , and (2) a ranking task, where we must rank target words according to their degree of lexical semantic change from c 1 to c 2 . we apply sscs on two pre-trained static sense embeddings, and six distance/divergence measures. despite the computationally lightweight and unsupervised nature of sscs, our experimental results show that it surprisingly outperforms most previously proposed scd methods for english, demonstrating the effectiveness of word sense distributions for scd. moreover, evaluations on german, latin and swedish show that this effectiveness holds in other languages as well, although not to the same levels as in english. we hope our findings will motivate future methods for scd to explicitly incorporate word sense related information. source code implementation for reproducing our experimental results is publicly available. foot_1 2 related work semantic change detection: scd is modelled in the literature as the unsupervised task of detecting words whose meanings change between two given time-specific corpora @xcite . in recent years, several shared tasks have been held @xcite @xcite , where participants are required to predict the degree or presence of semantic changes for a given target word between two given corpora, sampled from different time periods. various methods have been proposed to map vector spaces from different time periods, such as initialisation @xcite , alignment @xcite , and joint learning @xcite @xcite . existing scd methods can be broadly categorised into two groups: (a) methods that compare word/context clusters @xcite @xcite , and (b) methods that compare embeddings of the target words computed from different corpora sampled at different time periods @xcite @xcite . @xcite recently proposed a temporal attention mechanism, which achieves sota performance for scd. however, their method requires additional training of the entire mlm with temporal attention, which is computationally expensive for large mlms and corpora. the change of the grammatical profile @xcite of a word, created using its universal dependencies obtained from udpipe @xcite , has shown to correlate with the semantic change of that word. however, the accuracy of the grammatical profile depends on the accuracy of the parser, which can be low for resource poor languages and noisy texts. sabina uban et al. (2022) used polysemy as a feature for detecting lexical semantic change discovery. the distribution of the contextualised embeddings of a word over its occurrences (aka. sibling embeddings) in a corpus has shown to be an accurate representation of the meaning of that word in a corpus, which can be used to compute various semantic change detection scores @xcite . xl-lexeme @xcite is a supervised scd method where a bi-encoder model is trained using wic (pilehvar and camacho-collados, 2019) dataset to discriminate whether a target word appears in different senses in a pair of sentences. xl-lexeme reports sota scd results for english, german, swedish and russian. sense embeddings: sense embedding learning methods represent different senses of an ambiguous word with different vectors. the concept of multiprototype embeddings to represent word senses was introduced by @xcite . this idea was further extended by @xcite , who combined both local and global contexts in their approach. clustering is used in both works to categorise contexts of a word that belong to the same meaning. although the number of senses a word can take depends on that word, both approaches assign a predefined fixed number of senses to all words. to address this limitation, @xcite introduced a non-parametric model, which is able to dynamically estimate the number of senses for each word. although clustering-based approaches can allocate multi-prototype embeddings to a word, they still suffer from the fact that the embeddings generated this way are not linked to any sense inventories @xcite . on the other hand, knowledge-based methods obtain sense embeddings by extracting sense-specific information from external sense inventories, such as the wordnet @xcite or the babelnet 3 @xcite . @xcite extended word2vec @xcite to learn sense embeddings using wordnet synsets. @xcite made use of the semantic relationships in wordnet to embed words into a shared vector space. @xcite used the definitions of word senses in babelnet and conducted wsd to extract contextual information that is unique to each sense. recently, contextualised embeddings produced by mlms have been used to create sense embeddings. to achieve this, @xcite created lmms sense embeddings by averaging over the contextualised embeddings of the sense annotated tokens from semcor @xcite . @xcite proposed senseembert (sense embedded bert), which makes use of the lexical-semantic information in babelnet to create sense embeddings without relying on senseannotated data. ares (context-aware embed-dins) @xcite ) is a knowledge-based method for generating bert-based embeddings of senses by means of the lexical-semantic information available in babelnet and wikipedia. ares and lmms embeddings are the current sota sense embeddings. ## experiments data and evaluation metrics: we use the semeval-2020 task 1 dataset @xcite to evaluate scd of words over time for english, german, swedish and latin in two subtasks: binary classification and ranking. in the classification subtask, the words in the evaluation set must be classified as to whether they have semantically changed over time. classification accuracy (i.e. percentage of the correctly predicted words in the set of test target words) is used as the evaluation metric for this task. to predict whether a target word w has its meaning changed, we use bayesian optimisation to find a threshold on the distance, d(p(z w |w, c 1 ), p(z w |w, c 2 )), between the sense distributions of w computed from c 1 and c 2 . specifically, we use the adaptive experimentation platform foot_2 to find the threshold that maximises the classification accuracy on a randomly selected heldout portion of words from the semeval dataset, reserved for validation purposes. we found that using bayesian optimisation is more efficient than conducting a linear search over the parameter space. we repeat this threshold estimation process five times and use the averaged parameter values in the remainder of the experiments. in the ranking subtask, the words in the evaluation set must be sorted according to the degree of semantic change. spearman's rank correlation coefficient (ρ ∈ [-1, 1]) between the human-rated gold scores and the induced ranking scores is used as the evaluation metric for this subtask. higher ρ values indicate better scd methods. statistics of the data used in our experiments are shown in sense embeddings: we use two pretrained sense embeddings in our experiments: lmms (loureiro and jorge, 2019) and ares @xcite . for english monolingual experiments we use the 2048-dimensional lmms 5 and ares 6 embeddings computed using bert-large-cased, 7 which use wordnet sense ids. for the multilingual experiments, we use the 768-dimensional ares embeddings computed using multilingual-bert, 8 which uses babelnet sense ids. as a baseline method that does not use sense embeddings, we use the english wsd implementation in nltk 9 to predict wordnet sense-ids for the target words, when computing the sense distributions, p(z w |s) in (1). hardware and hyperparameters: we used a single nvidia rtx a6000 and 64 gb ram in our experiments. it takes ca. 7 hours to compute semantic change scores for all target words in the 15.3m sentences in the semeval datasets for all languages we consider. the only hyperparameter in sscs is the threshold used in the binary classification task, which is tuned using bayesian optimisation. the obtained thresholds for the different distance measures are shown in ## conclusion we proposed, sscs, a sense distribution-based method for predicting the semantic change of a word from one corpus to another. sscs obtains good performance among the unsuupervised methods for both binary classification and ranking subtasks for the english unsupervised scd on the se-meval 2020 task 1 dataset. the experimental results highlight the effectiveness of using word sense distribution to detect semantic changes of words in different languages.