UA lberta at LSCD iscovery: Lexical Semantic Change Detection via Word Sense Disambiguation (2022)

Abstract: We describe our two systems for the shared task on Lexical Semantic Change Discovery in Spanish. For binary change detection, we frame the task as a word sense disambiguation (WSD) problem. We derive sense frequency distributions for target words in both old and modern corpora. We assume that the word semantics have changed if a sense is observed in only one of the two corpora, or the relative change for any sense exceeds a tuned threshold. For graded change discovery, we follow the design of CIRCE (Pömsl and Lyapin, 2020) by combining both static and contextual embeddings. For contextual embeddings, we use XLM-RoBERTa instead of BERT, and train the model to predict a masked token instead of the time period. Our language-independent methods achieve results that are close to the best-performing systems in the shared task.

## introduction lexical semantic change discovery is a task with growing interest and applications in various areas, such as natural language processing and lexicography @xcite . the shared task on semantic change discovery and detection in spanish (lscdiscovery) consists of two phases: 1) graded change discovery, and 2) binary change detection @xcite . we adopt different approaches for both phases. the two sub-tasks consider different aspects of lexical semantic change (lsc). the definition for graded change discovery follows @xcite : given a diachronic corpus pair c1 and c2, rank the intersection of their (content-word) vocabularies according to their degree of change between c1 and c2. for binary lsc detection, the definition is the same as used in the semeval-2020 task 1: unsupervised lexical semantic change detection @xcite : given a target word *w* and two sets of its usages u1 and u2, decide whether *w* lost or gained senses from u1 to u2, or not. the direction of change is not important in either task. the inputs to the tasks consist of a list of target words and a pair of corpora from different time periods, annotated for the semantic relationship between word usages. the gold labels on a set of target words are inferred from sense frequency distributions derived by clustering the manual annotations @xcite . the output in phase 1 is a list of the target words ranked by the amount of change. the output in phase 2 is a list of binary change detection labels per word. for graded change discovery, our approach is similar to circe (pömsl and lyapin, 2020), the top performing system in the semeval 2020 task for graded change. we use embeddings and euclidean distance to obtain rankings. however, we obtain contextual embeddings from token prediction instead of time period prediction. in addition, we use xlm-roberta instead of bert, because it performs well across a variety of tasks in spanish @xcite , and models based on this architecture produce effective contextual embeddings @xcite . for binary change detection, we propose a novel approach based on framing lsc discovery as word sense disambiguation (wsd) problem, which is the task of determining the meaning of words in context given a sense inventory @xcite . using a recently-proposed wsd system, amuse @xcite , we identify the sense of each target word in context to determine if senses were lost or gained over time. following the theory of @xcite , we posit that wordnettype sense inventories match the intuitions of the annotators of the shared task data. our approach has the advantage of being interpretable, providing interesting insights into the nature of lexical semantic change by identifying specific senses that appear or disappear in texts over time. our systems are highly competitive. for phase 1, our system achieves 0.5731 correlation between the ranked words and ground truth on the test set, which would put it in third place, based on our own evaluation performed after the submission deadline. for phase 2, our system obtains f-score of 88% on the development set, and 71% on the evaluation set, which ranks it as second according to the main metric. ## related work in the semeval 2020 task for graded change discovery, the circe system performed the best @xcite . the system ensembles static and contextual embeddings. static embeddings with skip-gram with negative sampling @xcite are obtained for each corpus. these embeddings are then aligned using orthogonal procrustes analysis @xcite , and the euclidean distance is found between aligned embeddings. contextual embeddings from the masked language model bert @xcite are used to classify the time period of sentences, as time specific features are useful to learn. then, embeddings are extracted from the last hidden layer for each target word. to obtain a distance, the euclidean distance is computed pairwise between the embeddings from the two corpora and the distances are averaged. the target words are then ranked for both types of embeddings. finally, the rankings are combined by a weighted average to obtain the final ranking. we modify the circe approach for our graded change discovery system by using xlm-roberta @xcite to obtain contextual embed-dings. xlm-roberta is a multilingual masked language model. it uses the same bidirectional transformer architecture as bert @xcite and roberta @xcite , and has the same number of layers and size of layers as roberta. however, compared to bert and roberta it has a larger vocabulary of 250,000 tokens, and employs the sentencepiece tokenizer @xcite . additionally, it is trained on 100 languages, instead of just english @xcite . systems for binary change detection commonly use embeddings for semantic representations, with type embeddings often outperforming token embeddings @xcite . some approaches ensemble models @xcite or use a topic model @xcite . @xcite detect change by considering the relationship between nodes in a semantic network graph. orthogonal procrustes analysis (schönemann, 1966) and vector initialization @xcite are techniques that can be used to align the semantic representations. as a distance metric between embeddings, cosine and euclidean distance are commonly used. our approach based on applying wsd for binary change detection is novel. a previous work has performed word epoch disambiguation for determining changes in word usages overtime, but this task predicts the time period (epoch) for instances @xcite . some previous work considers changes of senses overtime; however, rather than wsd, they apply sense induction @xcite , topic modelling @xcite , or bayesian models @xcite . ## methods in this section, we describe our methods separately for each of the two phases. our code is available for public use. foot_0 ## evaluation we test our methods on the development set, and report the results on the evaluation set. some results were obtained after the submission deadline. ## conclusion we presented systems for both graded and binary change discovery in the context of the shared task on lexical semantic change discovery in spanish. for the former, we proposed a system based on circe, with the modification of tuning the weights between the static and contextual embeddings, and training the model to predict a masked token rather than the time period. for the latter, we demonstrated that a wsd system can be effective in detecting word meaning changes. future work could include combining rankings from more than two different models for graded lsc discovery. we would also like to investigate if either of our two methods could be applied to the other of the two subtasks.