Swap and Predict – Predicting the Semantic Changes in Words across Corpora by Context Swapping (2023)

Abstract: Meanings of words change over time and across domains. Detecting the semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions. We consider the problem of predicting whether a given target word, w , changes its meaning between two different text corpora, \mathcal{C}_1 and \mathcal{C}_2 . For this purpose, we propose \textit{Swapping-based Semantic Change Detection} (SSCD), an unsupervised method that randomly swaps contexts between \mathcal{C}_1 and \mathcal{C}_2 where w occurs. We then look at the distribution of contextualised word embeddings of w , obtained from a pretrained masked language model (MLM), representing the meaning of w in its occurrence contexts in \mathcal{C}_1 and \mathcal{C}_2 . Intuitively, if the meaning of w does not change between \mathcal{C}_1 and \mathcal{C}_2 , we would expect the distributions of contextualised word embeddings of w to remain the same before and after this random swapping process. Despite its simplicity, we demonstrate that even by using pretrained MLMs without any fine-tuning, our proposed context swapping method accurately predicts the semantic changes of words in four languages (English, German, Swedish, and Latin) and across different time spans (over 50 years and about five years). Moreover, our method achieves significant performance improvements compared to strong baselines for the English semantic change prediction task. Source code is available at https://github.com/a1da4/svp-swap .

## introduction meaning of a word is a dynamic concept that evolves over time @xcite . for example, the meaning of the word gay has transformed from happy to homosexual, whereas cell has included cell phone to its previous meanings of prison and biology. automatic detection of words whose meanings change over time has provided important insights for diverse fields such as 1 source code is available at https://github.com/ a1da4/svp-swap . figure 1 : overview of sscd. left: the contextualised word embedding distributions, d 1 and d 2 of a word which has not changed its meaning between the two corpora. two random samples of equal number of sentences containing the target word, s 1 and s 2 , are taken respectively from d 1 and d 2 and swapped between the corpora. here, we see that the contextualised word embedding distributions after swapping (i.e. d 1,swap and d 2,swap ) are similar to those before, thus preserving the distance between distributions. right: for a word that has different meanings in the two corpora, swapping process pushes both distributions to become similar, thus reducing the distance between the swapped versions smaller to that between the original ones. linguistics, lexicography, sociology, and information retrieval @xcite @xcite . for example, in e-commerce, a user might use the same keyword (e.g. scarf ) to search for different types of products in different seasons (e.g. silk scarves in spring vs. woollen scarves in winter). the performance of pretrained large language models (llms) is shown to decline over time @xcite because they are trained on static snapshots. if we can detect which words have their meanings changed, we can efficiently fine-tune llms to reflect only those changes @xcite . detecting whether a word has its meaning changed between two given corpora, sampled at different points in time, is a challenging task due to several reasons. first, a single (polysemous) word can take different meanings in different contexts even within the same corpus. therefore, creating a representation for the meaning of a word across an entire corpus is a challenging task compared to that in a single sentence or a document. prior work have averaged static @xcite @xcite @xcite or contextualised @xcite @xcite @xcite word embeddings for this purpose, which is suboptimal because averaging conflates multiple meanings of a word into a single vector. second, large corpora or word lists labelled for semantic changes of words do not exist, thus requiring semantic change detection (scd) to be approached as an unsupervised task @xcite . to address the above-mentioned challenges, we propose swapping-based semantic change detection (sscd). to explain sscd further, let us assume that we are interested in detecting whether a target word w has changed its meaning from a corpus c 1 to another corpus c 2 . first, to represent the meaning of w in a corpus, sscd uses the set of contextualised word embeddings (aka. sibling embeddings) of w in all of its contexts in the corpus, computed using a pre-trained masked language model (mlm). according to the distributional hypothesis @xcite , if w has not changed its meaning from c 1 to c 2 , w will be represented by similar distributions in both c 1 and c 2 . prior work has shown that contextualised word embedding of a word encodes word-sense related information that is useful for representing the meaning of the word in its occurring context @xcite . unlike the point estimates made in prior work @xcite by averaging word embeddings across a corpus (thus conflating different meanings), we follow @xcite and represent a word by a multivariate gaussian distribution that captures both mean and variance of the sibling distribution. various distance/divergence measures can then be used to measure the distance between the two sibling distributions of w, computed independently from c 1 and c 2 . an important limitation of the above-described approach is that it provides only a single estimate of the semantic change from a given pair of corpora, which is likely to be unreliable. this is especially problematic when the corpora are small and noisy. to overcome this limitation, sscd randomly samples sentences s 1 and s 2 that contain w, respectively from c 1 and c 2 and swaps the two samples between c 1 and c 2 . the intuition behind this swapping step is visually explained in figure 1 . on average, a random sample of distribution will be similar to the sampling distribution @xcite . therefore, if w has similar meanings in c 1 and c 2 , the distance between sibling distributions before and after the swapping step will be similar. on the other hand, if w's meaning has changed between the corpora, the distributions after swapping will be different from the original ones, thus having different distances between them. sscd conducts this sampling and swapping process multiple times to obtain a more reliable estimate of the semantic changes for a target word. we evaluate sscd against previously proposed scd methods on two datasets: semeval-2020 task 1 @xcite and liverpool fc @xcite , which cover four languages (english, german, swedish, and latin) and two time periods (some spanning longer than fifty years to as less than ten years). experimental results show that sscd achieved significant performance improvements compared to strong baselines. moreover, sscd outperforms the permutation test proposed by @xcite on both datasets and in three languages, showing the generalisability of sscd across datasets and languages. moreover, we show that there exists a trade-off between the percentage of sentences that can be swapped between two corpora (i.e. swap rate) and the performance of sscd, and propose a fully unsupervised method that does not require labelled data to determine the optimal swap rate. ## related work the phenomenon of diachronic semantic change of words has been extensively studied in linguistics @xcite but has recently attracted interest in the nlp community as well. unsupervised scd is mainly conducted using word embeddings, and various methods have been proposed that use static word embeddings such as initialisation @xcite , alignment @xcite , and joint learning @xcite @xcite . in recent years, with the advent of pretrained mlms, word embeddings can be obtained per context rather than per corpus, and the set of contextualised word embeddings (sibling embeddings) can be used for the semantic change analysis @xcite and detection @xcite @xcite @xcite @xcite . recent research has mainly focused on mlms, which embed useful information related to the meaning of words in their embeddings @xcite . while prior work use pretrained or fine-tuned mlms without considering temporal information, @xcite proposed a finetuning method by adding temporal tokens (such @xmath0. in the finetuning step, mlms optimise two masked language modelling objectives: 1) predicting the masked time tokens from given contexts, and 2) predicting the masked tokens from given contexts with time tokens. this method has been shown to outperform the previously unbeatable static word embeddings in the scd benchmark, semeval 2020 task 1 @xcite . moreover, @xcite proposed adding a time-aware attention mechanism within the mlms. in the training step, they conduct additional training on the mlm and the temporal attention mechanism on the target data. this model also achieves significant performance improvements in unsupervised scd benchmarks. yu @xcite applied scd to the temporal generalisation of pretrained mlms. pretrained mlms perform worse the further away in time from the trained timestamps, and require additional training @xcite . they show that an scd method can effectively improve the performance of pretrained mlms, because the additional training can be limited to those words whose meanings change over time. @xcite introduced a method that represents sibling embeddings at each time period using multivariate gaussians, thus enabling various divergence and distance metrics to be used to compute semantic change scores. experimental results show that instead of using only the mean of the sibling embeddings, it is important to consider also its variance, to obtain performance comparable to the previous sota. consequently, we use sibling embeddings to represent the distribution of a word in a corpus in sscd. however, existing methods make predictions only once for a given target word. such point estimates of semantic change scores are unreliable especially when a target word is rare in a corpus. to overcome this unreliability, @xcite use context swapping to make multiple predictions for the semantic change of a target word. first, the degree of semantic change is calculated by the cosine distance of the average sibling embeddings between time periods, and the reliability of the prediction is validated by context swapping-based tests (permutation test and false discovery rate). next, words with low reliability (p ≧ 0.005) are excluded from the prediction results, and only the words with high reliability are evaluated. however, this means that it is not possible to assess the semantic change of less reliable words, especially those that are less frequent. in particular, it is desirable to be able to utilise pretrained models without additional training and still be able to detect semantic changes appropriately. as discussed later in § 4.1, sscd successfully overcomes those limitations in @xcite , and consistently outperforms the latter in multiple scd tasks. recently, a supervised scd method called xl-lexeme @xcite , which finetunes sentence embeddings produced from an mlm on the word-in-context (wic) (pilehvar and camacho-collados, 2019) dataset has achieved state-of-the-art (sota) performance for scd. during training, they fine-tune the pretrained mlm to minimise the contrastive loss calculated from the sentence pairs in wic instances such that the distinct senses of a target word can be correctly discriminated. however, it has been shown that these external data-dependent methods are difficult to apply to languages that are not presented in the fine-tuning data (e.g. latin). our focus in this paper is on unsupsevised scd methods that do not require such sense-labelled resources. therefore, we do not consider such supervised scd methods. meaning modelling. given a sentence s and a target word w that occurs in s, we obtain the contextualised token embedding m (w, s) of w in s, produced by an mlm m . we refer to m (w, s) as a sibling embedding of w. given a set s w of sentences containing w, we define the set of sibling embeddings by d @xmath1, s) | @xmath2. let the mean and the covariance of d w be respectively µ w and σ w . 2 based on prior work that show contextual word embeddings to encode useful information that determine the meaning of a target word @xcite , we form the following working assumption. assumption. the set of sibling embeddings d w represents the meaning of a word w in a corpus c. we can reformulate this assumption as a null hypothesis -the meaning of w has not changed from c 1 to c 2 , (unless the corresponding sibling distributions have changed), for the purpose of computing the probability that w has its meaning changed from c 1 to c 2 . to validate this hypothesis, sscd uses the context swapping process described in algorithm 1. we first find the set of sentences that contain w algorithm 2 sscd input: target word w, swap rate r, time-specific corpora c 1 , c 2 , masked language model m , divergence/distance function d output: semantic change score 1: in c 1 and c 2 , denoted respectively by s w 1 and s w 2 . we then randomly select subsets s w 1 ∈ s w 1 and s w 2 ∈ s w 2 , containing exactly n w swap number of sentences, determined by the number of sentences in the smaller set between s w 1 and s w 1 , and the swap rate r(∈ [0, 1]) (lines 2-4). next, we exchange s w 1 and s w 2 between c 1 and c 2 and obtain swapped sets of sentences s w 1,swap and s w 2,swap (lines 5 and 6). the context swapped corpora are next used to compute a semantic change score for w by sscd as described in algorithm 2. if the sets of the mean and the covariance are different between the two time periods (i.e. on the other hand, if the sets of the mean and the covariance remain similar between the two target time periods (i.e. µ w 1 ≈ µ w 2 and σ w 1 ≈ σ w 2 ), the original semantic distance e original (line 5 in algorithm 2) will be close to zero. moreover, the distance between e original and e swap (line 10) will also be close to zero because swapping would not change the shape of the sibling distributions (i.e. e original ≈ e swap ). therefore, |e originale swap | will be smaller in this case, hence the null hypothesis cannot be rejected for w, and sscd will predict w to be semantically invariant between c 1 and c 2 . similar to @xcite , sscd can also be extended to calculate the confidence interval for rejecting the null hypothesis using boostrapping @xcite . this is particularly useful for the binary classification subtask in the semeval 2020 task 1 for unsupervised scd, where we must classify a given word as to whether its meaning has changed between the two given corpora. however, for our evaluation on the ranking subtask we require only the semantic change score. ## conclusion we proposed sscd, a method that swaps contexts between two corpora for predicting semantic changes of words. experimental results show that sscd outperforms a previous method that uses context swapping for improving the reliability of scd. moreover, sscd even with using only pretrained models, and not requiring fine-tuning achieves significant performance improvements compared to strong baselines for the unsupervised english scd. we also discussed two perspectives on our sscd: 1) we showed that modifying the context swapping method improves performance compared to random context swapping. 2) we proposed an unsupervised method to find the optimal context swapping-rate, and showed that it is relatively independent of the distance/divergence measure used in sscd. as in future work, we will apply our method for time-dependent tasks such as temporal generalisation @xcite and temporal question answering @xcite . furthermore, as our method and supervised methods are independent of each other, we are considering applying our method to the supervised scd method.