Statistically Significant Detection of Semantic Shifts using Contextual Word Embeddings (2021)

Abstract: Detecting lexical semantic change in smaller data sets, e.g. in historical linguistics and digital humanities, is challenging due to a lack of statistical power. This issue is exacerbated by non-contextual embedding models that produce one embedding per word and, therefore, mask the variability present in the data. In this article, we propose an approach to estimate semantic shift by combining contextual word embeddings with permutation-based statistical tests. We use the false discovery rate procedure to address the large number of hypothesis tests being conducted simultaneously. We demonstrate the performance of this approach in simulation where it achieves consistently high precision by suppressing false positives. We additionally analyze real-world data from SemEval-2020 Task 1 and the Liverpool FC subreddit corpus. We show that by taking sample variation into account, we can improve the robustness of individual semantic shift estimates without degrading overall performance.

## introduction semantic change detection methods are used in historical linguistics and digital humanities to study the evolution of word meaning over time and in different domains @xcite . while semantic shift estimates have been shown to correlate with simulated @xcite and manual @xcite annotations of meaning change, to our knowledge, no existing methods attempt to characterize the uncertainty of the estimated semantic shift for each individual word. this is especially problematic because semantic change detection is usually based on word embeddings @xcite and recently it has been observed that their stability can vary widely across term frequencies @xcite , implying that many semantic shift estimates are erroneously inflated or underestimated. prior studies have addressed this issue by filtering out words that fall below a term frequency threshold. while this approach can remove a majority of false positives, it risks the introduction of false negatives. a more robust approach would take into account the sample variation to determine whether there is evidence that an estimated semantic shift is sufficiently different from zero to be considered statistically significant. unfortunately, non-contextual word embeddings lose this information, flattening all instances of the same term into a single word embedding. in this paper, we focus on the problem of robust semantic change detection in scenarios where there is either limited data or low frequency terms of interest (e.g. del @xcite ). our approach is based on contextual word embeddings, such as those produced by bert @xcite , and permutation-based statistical tests. contextual word embeddings have several advantages over non-contextual embeddings for inferring semantic shift when there is limited data. first, we can leverage pre-trained models that were trained on large-scale data, encoding prior knowledge of the language. second, as contextual word embeddings are generated for every instance of a given word, there is an opportunity to characterize the strength of evidence for each semantic shift using statistical testing. the approach we propose in this paper focuses on the application and evaluation of statistical testing in semantic change detection. our contributions are as follows: ## related work computational methods for semantic change detection are used to compare corpora spanning decades or even centuries. they have been used, for example, to analyze historical word usage @xcite and to identify statistical laws of language change @xcite . more recently, there has been increased interest in detecting short-term meaning change, such as novel slang terms, in amazon reviews @xcite , twitter data @xcite and specialist online communities @xcite . prior to the wide-spread use of word embeddings, numerous methods were developed to detect semantic change, including dynamic topic models @xcite , word co-occurrence statistics @xcite and graphbased methods @xcite . methods for semantic change detection based on word embeddings exploit their distributional properties to identify words whose relative position in the embedding space has changed over time, implying a concordant change in meaning @xcite . the earliest work in this area was based on continuous training, initializing each embedding model with embeddings from the previous time step @xcite . subsequent methods improved performance by training independent embedding models for each corpus @xcite . embeddings are invariant under rotation and therefore need to be aligned by solving the orthogonal procrustes problem @xcite . this alignment step can be avoided altogether by, for example, comparing word neighborhoods @xcite or using temporal referencing @xcite . semantic shift detection with contextual word embeddings is becoming increasingly popular. @xcite used bert embeddings to define exemplar representations for pre-defined word senses to track usage over time. several meth-ods have side-stepped the need for known word senses by clustering bert embeddings @xcite and shown that clustering-based approaches can scale to the whole vocabulary @xcite . another benefit of using contextual word embeddings is that pre-trained models are widely available for many different languages. these models can be used for fine-tuning to perform semantic change detection using more limited data @xcite . lastly, researchers have started to experiment with ensembling multiple types of word embeddings and distance metrics to improve overall performance @xcite . recently, there have been several benchmarking studies of semantic change detection methods using simulated data @xcite and manually annotated data sets @xcite . these studies found that variations on the method proposed by @xcite performed best, however, methods based on contextual word embeddings were either absent or the study was based on data where contextual information was partially lost due to shuffling the order of sentences in the corpus @xcite . in this paper, we use contextual word embeddings to identify statistically significant semantic shifts. statistical significance has not been the subject of much investigation in the semantic shift literature. indeed, the only approach we are aware of was proposed by @xcite , which used bootstrapping to perform change-point detection in time series. however, their method is not applicable to scenarios where there is only two time points nor does it take into account the sample variance within each corpus. our approach addresses both of these issues and can be applied to any semantic change detection method based on contextual word embeddings. ## data ## methodology our approach uses contextual word embeddings and permutation-based statistical testing to detect semantic shifts in scenarios where data is limited. while our approach can be applied to any method based on contextual embeddings, we used the method proposed by @xcite because of its conceptual simplicity and faster runtime compared to other methods. we generated all contextual word embeddings with bert @xcite using the implementation from huggingface's transformers library @xcite . in all experiments, we used a base version of bert (the exact pre-trained models used are specified in later sections) with 12 attention layers and a hidden layer size of 768. all parameters were set to the default values used in the transformers library ver. 2.5.0, unless stated otherwise. ## synthetic evaluation we created multiple simulated data sets to highlight the importance of significance testing when performing semantic change detection across the whole vocabulary. previous work also used simulated data to compare methods @xcite , however, they focused on simulating time series, whereas we only simulate two time periods. ## empirical evaluation we used five manually annotated data sets (see ## conclusions we presented an approach to identify statistically significant semantic change using contextual word embeddings, permutation tests and false discovery rate. our work was motivated by the fact that while there are many methods for semantic change detection, they only estimate the magnitude of a given shift and ignore the uncertainty in their estimates. as a result, existing semantic change detection methods are problematic to apply if (i) the data set is of limited size or (ii) you need to estimate the semantic shift for words with low term frequencies. in simulation, we demonstrated that using a combination of permutation tests and false discovery rate allows us to scale semantic shift estimation to every word in the vocabulary while avoiding false positives and achieving high precision (figure 2 ). in our analysis of the semeval-2020 data sets, we showed that significance testing has a generally positive impact on spearman correlation ( in these examples, the false discovery rate procedure made less of a difference due to the limited number of significance tests performed. in the liverpool fc corpus, less than a third of target words achieved significance with fdr correction, causing outliers to have a greater influence on correlation than they did on permutation tests ( in this paper, we used a semantic shift method which did not exploit the full potential of contextual word embeddings, i.e. to disambiguate between word senses. in future work, we are going to investigate whether more scalable techniques (e.g. @xcite ) could be applied at the word sense level, where even large data sets could potentially start to suffer from the small data problems we focused on in this article.