A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection (2024)

Abstract: Detecting temporal semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions.Lexical Semantic Change Detection (SCD) task involves predicting whether a given target word, w , changes its meaning between two different text corpora, C_1 and C_2 .For this purpose, we propose a supervised two-staged SCD method that uses existing Word-in-Context (WiC) datasets.In the first stage, for a target word w , we learn two sense-aware encoders that represent the meaning of w in a given sentence selected from a corpus.Next, in the second stage, we learn a sense-aware distance metric that compares the semantic representations of a target word across all of its occurrences in C_1 and C_2 .Experimental results on multiple benchmark datasets for SCD show that our proposed method achieves strong performance in multiple languages.Additionally, our method achieves significant improvements on WiC benchmarks compared to a sense-aware encoder with conventional distance functions.

## introduction the notion of word meaning is a dynamic one, and evolves over time as noted by @xcite . for example, the meaning of the word cell has changed over time to include cell phone to its previous meanings of prison and related to biology. detection of such semantic changes of words over time remains a challenging, yet an important task for lexicography, sociology, and information retrieval @xcite @xcite . for example, in e-commerce, a user might use the same keyword, such as scarf, to search for different types of products based on seasonal variations, such as silk scarves in 1 source code is available at https://github.com/ livnlp/svp-sdml . spring versus woollen scarves in winter. recently, a decline in the performance of pretrained large language models (llms) over time has been observed, and attributed to their training on static snapshots @xcite . recognising the words with changed meanings enables efficient fine-tuning of llms to incorporate only those with specific semantic shifts @xcite . detecting whether a word has its meaning changed between two given corpora, sampled at different points in time, requires overcoming two important challenges, which we name as the representational and measurement challenges. representational challenge: a word can take different meanings in different contexts even within the same corpus. therefore, creating a representation for the meaning of a word across an entire corpus is a challenging task compared to that in a single sentence or a document. prior work has averaged static @xcite @xcite @xcite or contextualised @xcite @xcite @xcite word embeddings for this purpose, which is suboptimal because averaging conflates multiple meanings of a word into a single vector. ## related work the diachronic semantic changes of words have been extensively investigated by linguists to explore how meanings evolve over time @xcite . in recent years, the advent of diachronic corpora and advancements in representing word meanings have paved the way for automated scd in nlp @xcite . to detect semantically changed words in time-separated corpora, unsupervised scd methods compare word embeddings trained on the target corpora. many methods exist that align vector spaces over time spanned by static word embeddings, such as initialisation @xcite , alignment @xcite , and joint learning @xcite @xcite . likewise, methods for comparing sets of target word representations computed as contextualised word embeddings have also been proposed. such methods include comparing the average @xcite @xcite @xcite @xcite or each pair of embeddings @xcite . additionally, @xcite and @xcite have proposed methods that consider the variance in the sets of embeddings. these unsupervised scd methods have been evaluated using unsupervised scd tasks @xcite @xcite . due to the lack of manually-labelled data for lexical semantic change, prior work on scd have been limited to unsupervised approaches. a notable exception is xl-lexeme @xcite , a supervised scd method that uses senselevel supervision. they focused on the wic task (pilehvar and camacho-collados, 2019) de-signed to detect semantic differences of a target word in a given pair of sentences. xl-lexeme fine-tunes a pretrained multilingual masked language model (mlm) across multilingual wic datasets, and achieves strong performance in some scd tasks. recent work show scd is a challenging task for llms such as gpt-3.5 @xcite , reporting low performance even with carefully designed prompts. ## semantic distance metric learning given a target word w and two corpora c 1 and c 2 sampled at distinct time points, the goal in scd is to predict a score for w that indicates the degree of semantic change undergone by w between c 1 and c 2 . for this purpose, we propose, sdml, which overcomes the two challenges introduced in § 1. specifically, to address the representational challenge, we first learn a sense-aware encoder in § 3.1. next, to overcome the measurement challenge, we learn a sense-aware distance metric between two sense-aware representations of a target word computed from c 1 and c 2 , as described in § 3.2. ## ablation study motivated by the superior performance reported in § 4.2 by sdml, we conduct an ablation study to understand the importance of (i) using senseaware distance metric, and (ii) considering crossdimensional correlations by mahalanobis matrix. firstly, we conduct a quantitative comparison between our sdml (sense-aware sentence encoder and sense-aware distance metric using full components of mahalanobis matrix, sdml full ) and two variants; (i) sense-aware sentence encoder only (baseline), and (ii) sense-aware sentence encoder and sense-aware distance metric using diagonal components of mahalanobis matrix (sdml diag ). results on scd tasks are shown in secondly, we perform a qualitative analysis using gold labels. following @xcite , we pick up the (i) top-8 semantically changed words, and (ii) top-8 semantically stable words in the semeval en. additionally, due to the sense-aware method, we also evaluated the relationship between the scd performance and two factors; polysemy and frequency as described in @xcite . we count the number of senses (synsets defined in wordnet) and frequency of the evaluation set of words in given corpora. from much prior work on scd use the euclidean distance (or cosine similarity) for measuring semantic change scores of words @xcite @xcite @xcite , which (a) weights all dimensions equally, and (b) does not consider cross-dimensional correlations. however, our experimental results show that euclidean distance is a suboptimal choice for this purpose and learning a mahalanobis distance met-ric is more appropriate. ## conclusion we proposed a supervised two-staged scd method to address two challenges in the scd tasks: 1) models must obtain sense-aware embeddings of target words over time, and 2) due to the lack of labelled training data, scd is often an unsupervised task using conventional distance metrics. for the first challenge, we propose to learn a sense-aware encoder. next, we address the second challenge by learning a sense-aware distance metric to compare sense-aware embeddings. in both stages, we used wic datasets to provide senselevel supervision. experimental results show that our proposed method, sdml, achieves strong performance in four scd benchmarks. moreover, sdml achieves significant performance improvement in wic benchmarks. our findings highlight the importance of learning both a sense-aware encoder and a sense-aware distance metric.