Substitution-based Semantic Change Detection using Contextual Embeddings (2023)

Abstract: Measuring semantic change has thus far remained a task where methods using contextual embeddings have struggled to improve upon simpler techniques relying only on static word vectors. Moreover, many of the previously proposed approaches suffer from downsides related to scalability and ease of interpretation. We present a simplified approach to measuring semantic change using contextual embeddings, relying only on the most probable substitutes for masked terms. Not only is this approach directly interpretable, it is also far more efficient in terms of storage, achieves superior average performance across the most frequently cited datasets for this task, and allows for more nuanced investigation of change than is possible with static word vectors.

## introduction measuring semantic change is one of the few areas of nlp where contextual embeddings have not yet led to a definitive improvement over previous methods. in particular, the commonly used approach of aligning static embeddings trained on different time periods @xcite continues to be a surprisingly hard to beat baseline. given that contextual embeddings provide a representation for each occurrence of a word in context, they would seem to be ideally suited to a more nuanced investigation of semantic change. most attempts to leverage them for this purpose, however, produce quantitatively worse results, while being less interpretable and requiring more resources. here, we present a simplified and improved approach to scalable, interpretable, semantic change detection using contextual embeddings. inspired by @xcite , we work only with the most probable replacements for masked words, and measure semantic change in terms of the distributions of replacements in each time period. not only does this better match human judgements, it is highly space efficient, works seamlessly for out-of-vocabulary words, and helps intuitively characterize meaning change and variation. ## background measuring semantic change involves a set of tasks related to determining if and how a term's meaning has changed over time. here, we focus on the task of measuring the amount of change that has occurred from one time period to another @xcite . 1 existing approaches to this task are mostly of two types. the first is associating each term with a single vector per time period and measuring the distance between vectors, of which we take @xcite to be representative. as a variation on this, several authors have proposed averaging the output of contextual embedding models to get a single vector per term in each time period, but this has generally not led to an improvement over using static vectors @xcite @xcite . a related approach is to represent words in terms of their nearest neighbors using static word vectors @xcite , but this does not show a clear improvement over other static embedding methods @xcite . a second type of approach begins with various methods for word sense induction, then measures change in terms of the relative prevalence of a term's different senses @xcite @xcite . in some cases, authors simply cluster contextual representations for each term, and measure differences in the distributions of clusters between two time periods, rather than dealing with explicit word senses @xcite @xcite . despite the additional information provided by contextual embedding models, methods using type embeddings (as opposed to token), continue to be competitive. for example, on the recent semeval multilingual semantic change detection task, none of the top four systems used token embeddings @xcite . methods using contextual embeddings have done better on some more recent mono-lingual shared tasks @xcite , but have not yet been evaluated with a consistent setup across multiple languages. ## methods building on @xcite , we represent each token in the corpus (or a sufficiently large sample of them) by a small set of probable replacement terms from a contextual embedding model. however, whereas @xcite did this for the purpose of word sense disambiguation, we do so for the purpose of measuring semantic change. for each sampled occurrence of each term, we mask the term of interest, feed the masked context through a model, and obtain the predicted token probabilities corresponding to the mask token. 2 from these, we save only the top-k most probable words (excluding stopwords and partial word pieces), and discard the rest. for a given term in a particular time period, we then count how many times each word in the model vocabulary has appeared as a top-k replacement for that term, and normalize this by its sum, giving us a distribution over replacements. to obtain a raw score of semantic change between two time periods, we compute the jensen-shannon divergence (jsd) between the two distributions representing the same term in different time periods. however, as we show below, the raw jsd scores are strongly correlated with term frequency. thus, to obtain a scaled metric, we convert the raw jsd scores into a quantile, comparing the raw score for a term of interest to other terms with similar frequency. compared to saving the full output vector per token, this approach only requires a miniscule amount of storage per token, and thus does not require the kind of heuristic dropping of tokens employed by @xcite . in addition, the dominant meanings of a word in each context can be summarized by the terms which occur most fre-quently among the top-k replacements. although such replacements are limited to the terms which exist in the model vocabulary, in practice this is sufficient to represent a nuanced set of meanings, and works even for words which get tokenized into multiple word pieces, as we show below. more formally, given two corpora c1 and c2, let the count of token v as a top-k replacement for term t in corpus c be: finally, we correct for frequency effects by rescaling the raw jsd scores against the scores for terms with similar frequency as the target term, giving us a quantile scaled in [0, 1]: ## discussion and related work as noted by others, new and larger datasets for rigorously evaluating semantic change are badly needed @xcite . existing datasets are relatively small, and are mostly based on inspecting a limited number of examples per term. unfortunately, determining ground truth for semantic change is challenging, and producing such resources is costly. ideally, future datasets for evaluation should be larger, both to allow for more robust evaluation, and to have sufficient targets for both hyperparameter tuning and evaluation. in addition to the dataset we have used in this paper, two others are available from shared tasks on spanish and russian, respectively @xcite . both of these are comparable in size to the gems dataset used here. unfortunately, they are less useful for evaluation because most submissions to these shared tasks only evaluated on the task data, and not on other datasets. as shown by the replication of @xcite in @xcite , a method can sometimes perform well on one language but fail to generalize to others. as such, we have based our evaluation on datasets for which there has been a consistent evaluation of methods across multiple languages. as future work, a careful replication study of all methods from each competition on all available datasets, including an assessment of sensitivity to hyperparameters, would be highly informative. besides @xcite , the closest prior work to ours is @xcite , who use dynamic patterns to generate many variations on example usages sampled from the given corpora. these variations are then used to generate hundreds of replacement terms from a masked language model with associated probabilities. these probabilities are averaged (heuristically combining replacements with differing numbers of word pieces) to obtain a mean vector for each sampled instance. finally, semantic change is computed as the average cosine distance between all pairs of vectors across corpora. this method was evaluated as part of the lscdiscovery shared task on spanish @xcite . preliminary work on this method was described in @xcite , where a slightly different version of it was evaluated on the rushifteval shared task on russian @xcite . compared to @xcite , our approach is considerably simpler, and better suited to storing representations of a complete corpus for subsequent analysis and exploration. in particular, we only consider a small number of substitutes for each example (storing only the top-k most probable terms, without the associated probabilities). we do not use dynamic patterns, and only consider terms in the model vocabulary as potential substitutes. we also associate each term with a single distribution over the model vocabulary per time period (not per mention), and use jensen-shannon divergence to more naturally measure the distance between distributions. importantly, we also correct for frequency effects, as described above. although our approach avoids the onerous storage requirements of methods which save full contextual vectors, it still requires considerable processing time to obtain the top-k replacements for all tokens. future work could explore smaller or more efficient models for this purpose. 6 finally, despite its simplicity, measuring the cosine distance between aligned static vectors remains a strong and efficient baseline @xcite . more work is needed to determine where contextual embeddings can offer sufficient advantage in measuring semantic change to justify their greater computational cost. compared to static embeddings, our approach is weakest on the german and swedish datasets, which could relate to the quality of the pretrained models that are available for those languages, the data used for pretraining, or perhaps issues that arise in tokenization of the reference corpora. for a tentative exploration of some possible factors, please refer to appendix c. ## conclusion we have presented a simplified and improved approach to measuring semantic change using contextual embeddings, based on the jensen-shannon divergence between the distributions of the most probable replacements for masked tokens in different time periods, corrected for frequency effects. this approach achieves superior performance on average, while remaining directly interpretable, with vastly reduced storage requirements.