Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings (2023)

Abstract: Languages are dynamic entities, where the meanings associated with words constantly change with time. Detecting the semantic variation of words is an important task for various NLP applications that must make time-sensitive predictions. Existing work on semantic variation prediction have predominantly focused on comparing some form of an averaged contextualised representation of a target word computed from a given corpus. However, some of the previously associated meanings of a target word can become obsolete over time (e.g. meaning of gay as happy), while novel usages of existing words are observed (e.g. meaning of cell as a mobile phone).We argue that mean representations alone cannot accurately capture such semantic variations and propose a method that uses the entire cohort of the contextualised embeddings of the target word, which we refer to as the sibling distribution. Experimental results on SemEval-2020 Task 1 benchmark dataset for semantic variation prediction show that our method outperforms prior work that consider only the mean embeddings, and is comparable to the current state-of-the-art. Moreover, a qualitative analysis shows that our method detects important semantic changes in words that are not captured by the existing methods.

## introduction the meaning of words evolves over time, and even in everyday life, technological innovations and cultural aspects can cause a word to have a different meaning than in the past. for example, the meaning of the word gay has completely changed from happy to homosexual (figure 1a ), and cell has added cell phone to its previous meanings of prison and biology (figure 1b ). in the semantic change detection task, the goal is to detect the words whose meanings have changed across time-specific corpora @xcite . 1 source code is available at https://github.com/ a1da4/svp-gauss . as illustrated in figure 1 , we can identify two types of semantic changes associated with words -(a) the word gay obtains a new meaning by replacing its past meaning (figure 1a ), whereas (b) the word cell obtains a new meaning, while preserving its past meanings (figure 1b ). on the other hand, much prior work have resort to a scheme where they first individually represent the meaning of a target word in a given time-specific corpora using a single embedding, such as the mean of the non-contextualised @xcite @xcite @xcite or contextualised @xcite @xcite @xcite embeddings of the target word taken over all of its occurring contexts in the corpus. next, various distance measures are used to compare those embeddings to quantify the semantic variation of the target word across corpora. however, as seen from figure 1 , using the mean embedding of a target word alone for predicting semantic variations of words can be misleading especially when the variance of the embedding distribution is large. to address the above-mentioned limitations, we use the distribution of contextualised embeddings of a target word w in all of its occurrence contexts s(w) in a given corpus, which we refer to as the sibling distribution @xcite of w. we then approximate the sibling distribution of a word using a multivariate gaussian, which has shown to accurately capture the uncertainty in word embedding spaces @xcite @xcite . we can then use a broad range of distance and divergence measures defined over gaussian distributions to quantify the semantic variation of a target word across multiple time-specific corpora. experimental results on semeval-2020 task 1 benchmark dataset show that our proposed method outperforms several prior methods, and achieves comparable performance to the current state-ofthe-art (sota) @xcite . more importantly, our proposal to model both the mean and variance of sibling embeddings consistently outperforms methods that use only the mean contextualised embedding from the same masked language model (mlm) @xcite . moreover, for computational convenience, prior work had assumed the covariance matrix of sibling embeddings to be diagonal @xcite , but we show that further performance improvements can be obtained by using the full covariance matrix. ## related work historically, the diachronic semantic changes of words have been studied by linguists @xcite , which has also received much attention lately within the nlp community. automatic detection of words whose meanings change over time has provided important insights for diverse fields such as linguistics, lexicology, sociology, and information retrieval (ir) @xcite @xcite . for example, in ir one must know the seasonal association of keywords used in user queries to provide relevant results pertaining to a particular time period. moreover, it has been shown that the performance of publicly available pretrained foundation models @xcite declines over time when applied to emerging data @xcite because they are trained using a static snapshot. @xcite showed that the temporal generalisation of foundation models is closely related to their ability to detect semantic variations of words. semantic change detection is modelled in the literature as an unsupervised task of detecting words whose meanings change between two given timespecific corpora @xcite . in recent years, several shared tasks have been held @xcite @xcite , where participants are required to predict the degree or presence of semantic changes for a given target word between two given corpora sampled from different time periods. for this purpose, much prior work have used non-contextualised or contextualised word embeddings to represent the meaning of the target word in each corpus. unlike noncontextualised word embeddings, which represent a word by the same vector in all of its contexts, contextualised word embeddings represent the same target word with different vectors in different contexts. various methods have been proposed to map vector spaces from different time periods, such as initialisation @xcite , alignment @xcite , and joint learning @xcite @xcite . the existing methods that have been proposed for the semantic variation detection of words can be broadly categorised into two groups: (a) methods that compare word/context clusters @xcite @xcite , and (b) methods that compare embeddings of the target words computed from different corpora sampled at different time periods @xcite @xcite . recently, it has been reported that adding time-specific attention mechanisms @xcite achieves sota performance. however, this model requires additional training of the entire mlm including the time-specific mechanisms, which is computationally costly for largescale mlms. despite the recent success of using word embeddings for the semantic change detection task, many of these methods struggle to detect meaning changes of words which have a wide range of usages because they use only the mean embedding to represent a target word @xcite . although methods that use point estimates in the embedding space, such as using non-contextualised word embeddings or comparing the average of contextualised word embeddings, are able to detect semantic variations that result in a loss of a prior meaning (e.g. gay in figure 1a ), they are inadequate when detecting semantic variations due to novel usages of words, while preserving their former meanings (e.g. cell in figure 1b ). to alleviate this problem, some studies have used gaussian embeddings @xcite for semantic change detection @xcite . they used the mean and the diagonal approximation of the covariance matrix computed using non-contextualised word embeddings. however, as argued previously, contextualised embeddings provide useful clues regarding the meaning of a word as used in a context. therefore, in our proposed method, we consider the entire cohort of contextualised word embeddings of a target word taken across all of its occurring contexts (i.e. siblings) obtained from an mlm. as confirmed later by the evaluations presented in § 4.4, our proposed method consistently outperforms the methods proposed by @xcite and @xcite that use non-contextualised embeddings. ## semantic variation prediction let us consider a target word w that occurs in two given corpora c 1 and c 2 . for example, c 1 and c 2 could have been sampled at two distinct time slots, respectively t 1 and t 2 , reflecting any temporal semantic variations of words, or alternatively sampled at similar periods in time but from distinct domains (e.g. biology vs. law) expressing semantic variations of words due to the differences in the domains. our goal in this paper is to propose a method that can accurately predict whether w is used in the same meaning in both c 1 and c 2 (i.e. w is semantically invariant across the two corpora) or otherwise (i.e. its meaning is different in the two corpora). although we consider two corpora in the subsequent description for simplicity of the disposition, our proposed method can be easily extended to measure the semantic variation of a word over multiple corpora. according to the distributional hypothesis @xcite , the context in which a word occurs provides useful clues regarding its meaning. contextualised word embeddings such as the ones produced by mlms have shown to concisely and accurately encode contextual information related to a target word in a given context. for example, zhou and bollegala (2021) showed that contextualised word embeddings can be used to induce word-sense embeddings that represent the distinct senses of an ambiguous word with different vectors. inspired by such prior work using contextualised word embeddings as a proxy for accessing contextual information related to a target word, we propose a method to detect the semantic variations of a target word using its multiple occurrences in a corpus. to describe our proposed method in detail, let us denote the set of contexts containing w in corpus c i by s(w, c i ). the scope of the context of w could be limited to a predefined fixed token window or extended to the entire sentence containing w as we do in our experiments. let us denote the contextualised (token) embedding of w in a context @xmath0, c i ) produced by an mlm m by f m (w, s) ∈ r d , where d is the dimensionality of the token embeddings produced by m . following the terminology introduced by @xcite , we refer to type embedding f m (w, s) as the sibling embeddings of w in context s. the number of siblings of w in c i is denoted by n w @xmath1, c i )|. moreover, let the set of sibling embeddings of w created from its occurrences in c i to be d(w, c i ) = {f m (w, s)|@xmath2, c i )}. as we later see, the distribution of sibling embeddings of a word w encodes information about the usage of w in a corpus, which is useful for predicting any semantic variations of w across different corpora. we can obtain a context-independent embedding, µ w @xmath3. although much prior work has used µ w i as a proxy for the usage of w in c i for numerous tasks such as studying the properties of contextualised embeddings @xcite and predicting semantic variation of words @xcite @xcite @xcite , the mean of the sibling embedding distribution is insensitive to the rare yet important usages of the target word. in particular, when the sibling embedding distribution is not uniformly distributed around its mean, the mean embedding can be misleading as a representation of the distribution. to overcome this limitation, in addition to µ w i , we also use the covariance matrix v w @xmath4. we approximate the distribution of sibling embeddings of w using a gaussian, n (µ w i , v w i ) with mean and variance given respectively by ( 1 ) and ( 2 ). gaussian distribution is the maximum entropy distribution over the real values given a finite mean and covariance and no further information @xcite . moreover, by approximating the sibling distribution as a gaussian, we can use a broad range of distance and divergence measures for quantifying the semantic variation of w across corpora. in the field of information theory, mlms have been shown to store the information of a given sentence in a vector @xcite . there is a strong correlation between the word frequency n w i and the rank of its covariance matrix v w i (figure 2 in appendix a), which indicates that covariance matrix also retains important information regarding sibling embedding distribution. this observation further supports our proposal to represent target words by µ w i and v w i . ## conclusion we proposed a method to detect semantic variations of words using sibling embeddings. experimental results on semeval-2020 task 1 english dataset show that the proposed method consistently outperforms methods that use only the mean embedding vectors, and reports results comparable to the current sota. furthermore, a qualitative analysis shows that the proposed method correctly detects semantic variation of words, which are either over/underestimated by the existing methods.