Graph-based Clustering for Detecting Semantic Change Across Time and Languages (2024)

Abstract: Despite the predominance of contextualized embeddings in NLP, approaches to detect semantic change relying on these embeddings and clustering methods underperform simpler counterparts based on static word embeddings. This stems from the poor quality of the clustering methods to produce sense clusters—which struggle to capture word senses, especially those with low frequency. This issue hinders the next step in examining how changes in word senses in one language influence another. To address this issue, we propose a graph-based clustering approach to capture nuanced changes in both high- and low-frequency word senses across time and languages, including the acquisition and loss of these senses over time. Our experimental results show that our approach substantially surpasses previous approaches in the SemEval2020 binary classification task across four languages. Moreover, we showcase the ability of our approach as a versatile visualization tool to detect semantic changes in both intra-language and inter-language setups. We make our code and data publicly available.

## introduction since the 19th century, language change has been of ongoing scholarly interest in historical linguistics, stemming from a curiosity to understand intricate genealogy of languages through the comparison of linguistic patterns across earlier and later text corpora @xcite @xcite . up to now, many more curiosities have emerged, including the establishment of empirical principles of language change @xcite @xcite @xcite , the justification of hypothetical pathways of language change @xcite @xcite , the investigation of ancestral relationships among hundreds of languages @xcite @xcite , the discovery of linguistic and extralinguistic factors driving language change (blaxter, 2015), etc. a seminal work by @xcite outlined the characteristics of language change along five dimensions: time, geographical places, medium, registers and social contexts. this has inspired many works to date that leverage these dimensions as a lens to examine changes in grammatical meaning @xcite , syntax @xcite and many more. in computational linguistics, there has been a surge of interest in leveraging machine learning methods, as cost-efficient alternatives to labor-intensive human inspection. a special focus has been given to detect lexical meaning change, aiming to track changes in word meanings through the analysis of word usages across different time periods @xcite @xcite @xcite @xcite @xcite . for instance, @xcite and @xcite leverage static word embeddings to represent target words across time periods, and then identify the presence of semantic change in each target word by assessing the similarity between its word embeddings from different time periods. @xcite and cuba gyllensten et al. (2020) employ contextualized word embeddings and a clustering method to detect changes in each word sense over time. although contextualized embeddings excel in many nlp tasks, the performance of these embeddings coupled with clustering methods falls under static counterparts in detecting semantic change @xcite . in this work, we identify two major limitations of previous works relying on contextualized embeddings and clustering methods, namely (a) they struggle to capture word senses, especially those with low frequency, leading to poor semantic representation of word senses and (b) they produce time-independent sense clusters and use them to represent word senses varying over time-which is particularly problematic in the presence of a big time gap (e.g., 100 years) between earlier and later time periods. moreover, these works are limited in scope to detect only intra-language semantic changes. to address these issues, we introduce a graph-based clustering approach that leverages contextualized embeddings to capture the evolution of each word sense across both time and languages. as a result, our approach allows for comparing changes in each word sense across languages over time. this allows for a detailed study of interlanguage semantic change, especially to determine if the meanings of word translations across languages remain consistent or diverge over time. we comparably evaluate our approach in the se-meval2020 binary classification and ranking tasks @xcite for detecting semantic change across english, german, latin and swedish, and investigate the potential of our graph-based approach, as a visualization tool, to detect semantic changes in both intra-language and inter-language setups. our findings are summarized below: intra-language semantic change detection. recently, there has been a growing interest towards detecting meaning changes of target words within each language through a corpus-based study on word usage across time periods @xcite @xcite @xcite @xcite . many approaches have been proposed in the semeval2020 shared tasks @xcite . most approaches fall under two categories, based on the choice of word embeddings. for static embeddings, approaches, such as @xcite and @xcite , begin by refining pre-trained static word embeddings of target words on two corpora from different time periods, resulting in a separate embedding space for each time period. they then employ alignment techniques @xcite to adjust these word embeddings from different time periods. lastly, a distance measure is applied to these adjusted word embeddings to detect semantic change. for contextualized word embeddings, approaches like @xcite and cuba gyllensten et al. (2020) employ the bert and xlm-r encoders to produce contextualized word embeddings of each target word. they then employ k-means @xcite to partition embeddings of the target word from different time periods into multiple (time-independent) sense clusters. lastly, a frequency-based criterion is applied to these sense clusters to detect semantic change. @xcite show that careful data preprocessing can further improve the performance of semantic change detection, and suggest encoding lemmatized target words instead of their original word forms. @xcite propose to ensemble two top-performing approaches for detecting semantic changes. @xcite and @xcite propose to detect semantic change by comparing two frequency distributions of a target word across time periods. each distribution represents the frequencies of vocabulary words predicted as top substitutes for the target word using a masked language model. our work differs from others in several aspects: first, we leverage temporal and spatial dynamic graphs, which are derived from bert embeddings, to represent changes in word meanings across time and space (languages) 2 . this allows for detecting nuanced changes in each word sense, including both the acquisition and loss of meanings over time. second, we introduce our clustering method and strategy, and our distance metric, designed to produce sense clusters that excel in capturing word senses, especially for low-frequency senses. moreover, we compute the similarity between sense clusters over time to detect semantic change, while previous approaches do so by applying a frequencybased criterion foot_2 . ## experiments we evaluate our approach in semeval2020 task 1 ( §4.1), and showcase its ability as a visualization tool to detect semantic changes in both intra-algorithm 1 our clustering method require: @xmath0, tsc as the maximum distance between similar clusters, t low as the minimum cluster size for a low-frequency sense cluster. 1: initial centroids of clusters: @xmath1: ## conclusions we proposed a graph-based clustering approach to capture changes within each word sense across both time and languages. we addressed an intriguing concern that contextualized embeddings coupled with clustering methods seem not suitable for detecting semantic change, as they underperform their static embedding counterparts. we identified a crucial reason for this: previous approaches rely on low-quality clustering methods to handle contextualized embeddings. our results demonstrated that, when equipped with an appropriate clustering method, strategy, and distance metric, contextualized embeddings can produce high-quality sense clusters that effectively capture word senses, even low-frequency ones. these factors attribute to our approach's superiority over the shared task winner, ulb, which relies on static embedding. further, the use of our approach as a visualization tool highlights its value in conducting exploratory studies on both intra-and inter-language semantic changes.