What is Done is Done: an Incremental Approach to Semantic Shift Detection (2022)

Abstract: Contextual word embedding techniques for semantic shift detection are receiving more and more attention. In this paper, we present What is Done is Done (WiDiD), an incremental approach to semantic shift detection based on incremental clustering techniques and contextual embedding methods to capture the changes over the meanings of a target word along a diachronic corpus. In WiDiD, the word contexts observed in the past are consolidated as a set of clusters that constitute the “memory” of the word meanings observed so far. Such a memory is exploited as a basis for subsequent word observations, so that the meanings observed in the present are stratified over the past ones.

## introduction the use of contextual embedding techniques is receiving more and more attention in the field of semantic shift detection. in particular, pre-trained models like bert @xcite , elmo @xcite , and xlm-r (cuba @xcite , are being proposed as promising solutions to capture the different meanings of a target word according to the different contexts in which the word appears throughout a considered diachronic corpus. such solutions generally employ clustering techniques to aggregate embeddings of a specific word into clusters @xcite . the idea is that each cluster denotes a specific word meaning that can be recognized in the considered documents. in this way, it is possible to analyze the shift of a word meaning/sense by exploiting the evolution of a cluster over time. for instance, an increasing number of elements in a cluster denotes that the associated word meaning is getting frequently adopted. on the opposite, a cluster with a decreasing number of elements over time refers to a word meaning that is getting obsolete. usually, the corpus is static, meaning that all the documents of the considered time periods are available as one whole, and a single clustering activity is performed over the entire corpus, generating clusters of word meaning with documents of different time periods @xcite @xcite . as a result, the time period in which a document is added to the corpus is not taken into account for cluster composition, and this is not completely satisfactory for an appropriate recognition of meaning changes over time. when a dynamic corpus is considered, namely time periods and documents can be progressively added, scalability issues also arise, since the clusters of word meanings need to be re-calculated or updated. as a possible solution, some recent works propose to perform clustering separately for each time period. in this case, the resulting clusters need to be aligned in order to recognize similar word meanings in different, consecutive time periods @xcite . however, solutions based on clustering alignment are not satisfactory as well, since they do not capture the possible evolution pattern of a meaning across different time periods. a recent work proposes an average-based approach to track semantic shift via continuously evolving embeddings @xcite computed as a weighted running average @xcite of embeddings generated by a contextual model. this method is suitable to be applied on stream data and it is far more scalable than typically cluster-based methods. nevertheless, it does not allow to analyse which meanings are actually changed. in this paper, we present what is done is done (widid), an incremental approach to semantic shift detection based on incremental clustering techniques and contextual embeddings to capture the changes over the meanings of a target word along a diachronic corpus. in widid, we work under the assumption that the documents of the corpus become available as a stream and they are segmented in a sequence of time periods. the word contexts observed in past time periods are consolidated as a set of clusters that constitute the "memory" of the word meanings observed so far. such a memory is then exploited as a basis for subsequent word observations in the current time period. the idea of widid is that the clusters of word meanings previously created cannot be changed (what is done is done), and the word meanings that are observed in the present must be stratified/integrated over the past ones. to enforce scalability, incremental clustering techniques are employed in widid, so that the word embeddings extracted from the documents of the current time period are compared and assimilated into the set of consolidated clusters coming from the past time periods. a comparative evaluation of the proposed widid approach against a reference benchmark is discussed in the paper according to multiple configurations characterized by different clustering algorithms and embedding methods. in particular, we present experiments based on a pretrained bert model as well as results obtained from a trained doc2vec model, which has been adapted to provide pseudo-contextual word embeddings to extend the conventional static word representations of context-free embedding techniques. as a further contribution of widid, different metrics for semantic shift evaluation of word meanings are defined in the paper and experimental results are provided to discuss their effectiveness. the paper is organized as follows. in section 2, the relevant literature is discussed. in section 3, we present the widid approach. incremental clustering techniques and semantic shift measures of widid are illustrated in sections 4 and 5, respectively. experimental results are discussed in section 6. section 7 finally provides our concluding remarks. ## related work works related to widid are about the use of word embeddings for semantic shift detection by leveraging the idea that semantically-related words are close to each other in the embedding space @xcite . in approaches relying on context-free embeddings, independent word vectors defined over different "temporal" vector spaces can be compared after applying an alignment mechanism @xcite such as the procrustes @xcite . moreover, recent contextualised architectures are proposed, like elmo @xcite , bert @xcite , and xlm-r @xcite , which generate dynamic word embeddings according to the use of the words in the input sequences, thus enabling the recognition of different meanings by comparing the context in which words are used throughout the text. the solution proposed in @xcite is one of the first examples based on bert embeddings to track changes in word meanings and it requires lexicographic supervision, like the use of a reference dictionary (e.g., the oxford dictionary for the english language) to list the possible word meanings beforehand, thus it is hardly applicable to low-resource languages. a number of unsupervised approaches based on contextual embeddings are proposed to sidestep the need of lexicographic resources @xcite . in general, these kinds of approaches follow a three-step scheme: i) extraction of embeddings for each occurrence of a target word from a contextual model such as bert @xcite , elmo @xcite , or xlm-r @xcite ; ii) aggregation of the embeddings with a clustering algorithm like k-means @xcite , affinity propagation @xcite , or dbscan @xcite ; iii) comparison of the vector distribution over clusters according to time by using a semantic distance measure, like jensen-shannon divergence @xcite , entropy difference @xcite , or wasserstein distance @xcite . the main limitation of applying clustering to word embeddings is the scalability issues about memory consumption and time. as a recent contribution, in @xcite , a scalable and interpretable method is proposed based on merging of similar embeddings to reduce the number of representations to consider for a given word and time slice. further solutions to overcome scalability issues are provided by @xcite and @xcite . in particular, they propose to limit the number of embeddings by randomly sampling sentences from each period. the intrinsic time-complexity issues of applying clustering algorithms to embeddings are also addressed in @xcite by reducing the embedding dimensionality. in @xcite , the contextual embeddings of a word are averaged to generate a single word representation for each time period. in @xcite , the average pairwise distance between embeddings of different time periods is calculated. even if these solutions are more efficient and scalable than clustering, they provide uninterpretable results since multiple word occurrences are collapsed into a single representation, like in context-free embeddings. most of the cluster-and average-based approaches estimate the magnitude of semantic shifts ignoring the uncertainty of their estimations. as a result, estimations can be erroneously inflated since the irregularities of word frequencies over time can negatively affect the stability of word embeddings @xcite . in this respect, @xcite propose a solution based on the combination of bert embeddings with permutation-based statistical test and term-frequency thresholding. original contribution of widid. with respect to the above solutions, the widid approach is based on incremental clustering techniques applied to contextual word embeddings. in widid, the "memory" of word meanings observed in the past is consolidated in a set of clusters that is not re-calculated in subsequent time periods. as a result, only the word embeddings of the current time period are analyzed with the aim to measure the change with respect to the clusters of past word meanings. this way, it is possible to compare specific word meanings also from a qualitative point of view (i.e., interpretable results) without requiring any alignment mechanisim across time periods. in other words, the stratified layers of clusters over time allow to reconstruct not only the quantity of semantic shift but also the evolution of a word meaning. ## the widid approach consider a diachronic document corpus @xmath0. given a target word w, the goal of semantic shift detection is to measure how much the meaning(s) of w is changed from c 1 to c 2 . the widid approach relies on a contextual embedding model to represent each occurrence of the target word w in a corpus c j (either c 1 or c 2 ). we keep track of the word embedding representations collected for w over time by relying on the embedding model e 1 that contains the word vectors computed over c 1 . given this input, we process the new documents in c 2 as follows (see figure 1 ). ## incremental clustering in widid, we rely on incremental clustering to aggregate contextual embedding vectors that represent similar word meanings into the same cluster. we propose an incremental extension of affinity propagation (ap) @xcite , called affinity propagation a posteriori (app) (see algorithm 1). let's call x and x 1 , and l and l 1 the embeddings and the cluster labels at time t and t-1, respectively. at time @xmath1. at each time @xmath2, for each existing cluster computed at time t -1, the data points x @xmath3, i.e. the centroid µ of each cluster. the set of the centroids for x 1 is denoted µx 1 . then, the standard ap algorithm is executed on µx 1 ∪ x, with the aim to obtain a new set of temporary labels l 2 , i.e., the new assignment of data points to algorithm 1 the app algorithm input t: time step x: data at time step t x1: data at time step t -1 l1: labels at time step t -1 γ: trim factor output l, x: at time step t 1: if @xmath4: l ← ap (x) 3: l, x ← t rim(l, x, γ) 4: ## semantic shift measuring clustering contextual word embeddings for a word w at time t results in a set of k clusters k 2 = ϕ w,1 , ..., ϕ w,k where ϕ w,i ⊆ φ 1 w ∪ φ 2 w . in particular, we denote as ϕ 1 w,i , ϕ 2 w,i the set of embeddings from φ 1 w , and φ 2 w respectively, enclosed in the i-th cluster; formally we define ϕ 1 w,@xmath5,i ∩ φ 1 w and ϕ 2 w,@xmath6,i ∩ φ 2 w . according to this, in widid, we propose three different aggregation measures to estimate semantic change. borrowing from giulianelli (2019), we employ the jensen-shannon divergence to measure semantic change leveraging cluster distributions. in addition, we adapt the methods of @xcite and @xcite for scenarios where embeddings are clustered. ## concluding remarks in this paper, we presented the widid approach characterized by incremental clustering techniques and contextual word embedding methods. ongoing work is about the fine-tuning of adopted embedding models to further improve the quality of results. moreover, we are working on defining cluster analysis techniques. the idea is to exploit the results of semantic shift measures to interpret possible trend patterns over clusters along the time, such as a broad meaning that forks into narrower ones, or a meaning that increases its popularity and vice versa. further work is about the specification of aging policies to manage the memory of aged embeddings in the cluster evolution.