Scalable and Interpretable Semantic Change Detection (2021)

Abstract: Several cluster-based methods for semantic change detection with contextual embeddings emerged recently. They allow a fine-grained analysis of word use change by aggregating embeddings into clusters that reflect the different usages of the word. However, these methods are unscalable in terms of memory consumption and computation time. Therefore, they require a limited set of target words to be picked in advance. This drastically limits the usability of these methods in open exploratory tasks, where each word from the vocabulary can be considered as a potential target. We propose a novel scalable method for word usage-change detection that offers large gains in processing time and significant memory savings while offering the same interpretability and better performance than unscalable methods. We demonstrate the applicability of the proposed method by analysing a large corpus of news articles about COVID-19.

## introduction studying language evolution is important for many applications, since it can reflect changes in the political and social sphere. in the literature, the study of language evolution either focuses on long-term changes in the meaning of a word, or on more common short-term evolutionary phenomena, such as the word suddenly appearing in a new context, while keeping its meaning unchanged in a lexicographic sense. we refer to all types of language evolution-short-or long-term, with or without meaning change-as word usage change, a broad category that includes semantic change, but also any shifts in the context in which a word appears. recent studies @xcite show that clustering of contextual embeddings could be a proxy for word usage change: if clusters, which in theory capture distinct word usages, are distributed differently across time periods, it indicates a possible change in word's context or even loss or gain of a word sense. thus, the cluster-based approach offers a more intuitive interpretation of word usage change than alternative methods, which look at the neighborhood of a word in each time period to interpret the change @xcite and ignore the fact that a word can have more than one meaning. the main limitation of the cluster-based methods is the scalability in terms of memory consumption and time: clustering is applied to each word in the corpus separately and all occurrences of a word need to be aggregated into clusters. for large corpora with large vocabularies, where some words can appear millions of times, the use of these methods is severely limited. to avoid the scalability issue, cluster-based methods are generally applied to a small set of less than a hundred manually pre-selected words @xcite . this drastically limits the application of the methods in scenarios such as identification of the most changed words in a large corpus or measuring of usage change of extremely frequent words, since clustering of all of word's contextual embeddings requires large computational resources. one way to solve the scalability problem using contextual embeddings is to average a set of contextual representations for each word into a single static representation @xcite . averaging, while scalable, loses a lot on the interpretability aspect, since word usages are merged into a single representation. the method we propose in this paper tackles scalability and interpretability at the same time. the main contributions of the paper are the following: • a scalable method for contextual embeddings clustering that generates interpretable representations and outperforms other cluster-based methods. ## related work diachronic word embedding models have undergone a surge of interest in the last two years with the successive publications of three articles dedicated to a literature review of the domain @xcite @xcite . most approaches build static embedding models for each time slice of the corpus and then make these representations comparable by either employing incremental updating @xcite or vector space alignment @xcite . the alignment method has proved superior on a set of synthetic semantic drifts @xcite and has been extensively used @xcite and improved @xcite in the literature. the recent semeval task on unsupervised lexical semantic change detection has shown that this method is most stable and yields the best averaged performance across four semeval corpora @xcite ). yet another approach @xcite is based on comparison of neighbors of a target word in different time periods. this approach has been recently used to tackle the scalability problem @xcite . in all these methods, each word has only one representation within a time slice, which limits the sensitivity and interpretability of these techniques. the recent rise of contextual embeddings such as bert @xcite and elmo @xcite introduced significant changes to word representations. contextual embeddings can be used for usage change detection by aggregating the information from the set of token embeddings. this can be done either through averaging of all vectors within a time slice and then computing averaged vector similarity @xcite , by computing a pairwise distance between vectors from different time slices @xcite , or by clustering all token representations to approximate its set of senses @xcite . the analysis in this paper derives from this last set of methods, which demonstrate a higher performance than static embeddings methods at least on some datasets @xcite . automatic semantic shift detection has been used for text stream monitoring tasks, such as event detection @xcite viewpoint analysis @xcite or monitoring of rapid discourse changes during crisis events @xcite . none of these applications use clustering techniques and, as far as we are aware, only @xcite uses contextual embeddings for news stream analysis. in this paper we demonstrate the large potential of contextual embeddings for the interpretable tracking of shortterm changes in word usage, which has a practical application for crisis-related news monitoring. ## scalability and interpretability limitations of previous methods the main motivation for this research are the scalability or interpretability issues of previous methods for word usage change detection. the ones using contextual embeddings are either interpretable but unscalable @xcite or scalable but uninterpretable @xcite . the scalability issues of interpretable methods can be divided into two problems. ## methodology our word usage change detection pipeline follows the procedure proposed in the previous work @xcite : for each word, we generate a set of contextual embeddings using bert @xcite . these representations are clustered using k-means or affinity propagation and the derived cluster distributions are compared across time slices by either using jensen-shannon divergence (jsd) @xcite or the wasserstein distance (wd) @xcite . finally, words are ranked according to the distance measure, assuming that the ranking resembles a relative degree of usage shift. the primary contributions of this work lay in the embedding generation step, which improves the scalability of the method, and in leveraging wd to compute the distance between clusters. we also propose post-processing steps, which domain experts could use for the interpretation of results. we now describe the pipeline in more details. ## evaluation we use six existing manually annotated datasets for evaluation. the first dataset, proposed by @xcite , consists of 100 english words labelled by five annotators according to the level of semantic change between the 1960s and 1990s foot_7 . to build the dataset, the annotators evaluated semantic change using their intuition, without looking at the context. this procedure is problematic since an annotator may forget or not be aware of a particular sense of the word. the organizers of the recent semeval-2020 task 1-unsupervised lexical semantic change detection @xcite )-employed another approach: the annotators had to decide whether a pair of sentences from different time periods convey the same meaning of the word @xcite . for each of the four languages-german, english, latin and swedishsenses were manually annotated by labeling word senses in a pair of sentences drawn from different time periods. all semeval-2020 task 1 corpora contain only two periods and the sentences are shuffled and lemmatized. the lexical semantic change score is defined as the difference between word sense frequency distributions in the two time periods and measured by the jensen-shannon distance @xcite . the durel dataset @xcite is composed of 22 german words, ranked by semantic change by five annotators between two time periods, 1750-1799 and 1850-1899. similarly to semeval, the ranking was build by evaluating the relatedness of pairs of sentences from two periods. in order to conduct usage change detection on the target words proposed by @xcite , we fine-tune the english bert-baseuncased model and generate contextual embeddings on the corpus of historical american english (coha) 9 . we only use data from the 1960s to the 1990s (1960s has around 2.8m and 1990s 3.3m words), to match the manually annotated data. for the semeval task 1 evaluation set, we fine-tune the bert models and generate contextual embeddings on the four corpora provided by the organizers of the task, english (about 13.4m words), german (142m words), swedish (182m words) and latin (11.2m words). finally, we fine-tune bert and generate embeddings on the german dta corpus (1750-1799 period has about 25m and 1850-1899 has 38m tokens) 10 . the results are shown in for each period, a word is represented by its top nearest neighbors (nn) according to cd. semantic change is measured as the size of the intersection between the nn lists of two periods. on average, the proposed scalable clustering with filtering and merging of clusters leads to a higher correlation with gold standard than the standard non-scalable clustering methods: the best method (aff-prop wd) achieving a spearman correlation with the gold standard of 0.474 compared to the best non-scalable k-means 5 jsd achieving the spearman correlation of 0.391. the method also outperforms averaging and nn, though it is outperformed by a large margin by the sgns+op+cd, achieving the score of 0.533. the best performing clustering algorithm differs for different datasets. on average, affinity propagation only outperforms k-means when filtering and merging of clusters is employed. the effect of the filtering on k-means is positive on average but the difference is thin, as the number of clusters is low. wd leads to better results than jsd on most of the corpora where averaging outperforms clustering, the only exception is durel. an extreme example is the swedish semeval dataset, where the clustering with jsd performs particularly poorly: using the wd, which takes into account the average embeddings on top of cluster distributions, greatly increases the correlation with the gold standard. on the contrary, on coha where averaging performs poorly in comparison to clustering, wd is under-performing. ## use case: aylien covid-19 corpus the combination of scalable clustering with the interpretation pipeline opens new opportunities for diachronic corpus exploration. in this section, we demonstrate how it could be used to analyze the aylien coronavirus news dataset. the corpus contains about 500k news articles related to covid-19 from january to april 2020 11 , unevenly distributed over the months (160m words in march, 41m in february, 35m in april and 10m in january). we split the corpus into monthly chunks and apply our scalable word usage change detection method. ## conclusion we proposed a scalable and interpretable method for word usage change detection, which outperforms the non-scalable contextual embeddingsbased methods by a large margin. the new method also allows completely data-driven analysis of word sense dynamic in large corpora, which was impossible to conduct with unscalable methods. this opens new opportunities in both language change studies and text stream monitoring tasks. in this paper we focused on the latter application by analysing a large corpus of covid-19 related news. the method is outperformed by the state-of-theart sgns+op+cd method. we hypothesise that this can be connected with the fact that the sentences in all but one evaluation corpus (coha) are shuffled, meaning that bert models cannot leverage the usual sequence of 512 tokens as a context, but are limited to the number of tokens in the sentence. we will explore this hypothesis in the future. despite achieving lower performance than the sgns+op+cd method, we nevertheless argue that our method offers a more fine-grained interpretation than methods based on non-contextual embeddings, since it accounts for the fact that words can have multiple meanings. the cluster-based technique returns a degree of change and a set of sentence clusters for each word in the corpus, roughly corresponding to word senses or particular usages. for this reason, the approach can be used for detection of new word usages and for tracing how these usages disappear, as we have shown in section 6. even more, word usages and their distributions over time could be linked with real-word events by labeling sentence clusters with a set of clusterspecific keywords. overall, we observe a large disparity between results on different evaluation corpora. this is in line with the results of the semeval 2020 task 1 @xcite , where none of the best-performing methods was able to achieve the best result on all corpora. in practice, different methods focus on different aspects of word usage change: averaging and sgns+op+cd focus on average variation of word usage, hiding the intra-period diversity. when it comes to clustering, jsd-based method detects the appearance or disappearance of a given usage, even a minor one. the wd-based method, using information from both the cluster distribution and the embeddings vectors, represents a compromise between the averaging and the jsd-based methods. in this paper we follow the general approach in semantic shift detection literature and apply our analysis on the raw text. however, our results demonstrate that at least news monitoring applications would benefit from the application of the traditional text processing pipeline, in particular the extraction of named entities and dates. this will be addressed in the future work.