S ense C luster at S em E val-2020 Task 1: Unsupervised Lexical Semantic Change Detection (2020)

Abstract: We (Team Skurt) propose a simple method to detect lexical semantic change by clustering contextualized embeddings produced by XLM-R, using K-Means++. The basic idea is that contextualized embeddings that encode the same sense are located in close proximity in the embedding space. Our approach is both simple and generic, but yet performs relatively good in both sub-tasks of SemEval-2020 Task 1. We hypothesize that the main shortcoming of our method lies in the simplicity of the clustering method used.

## introduction the meaning of a word can vary not only with context, but also with time. the former phenomenon is commonly known as context-sensitivity, or, if the variation is of a more categorical nature, polysemy, whereas the latter phenomenon is captured by the term diachronic semantic drift @xcite , or alternatively lexical semantic change @xcite . as an example, a term such as "beautiful" has one main meaning, but will nonetheless imply slightly different things depending on its context of use; "suit" on the other hand has several distinct meanings (e.g. as a verb or as a noun), while "mouse" has acquired a completely new meaning with the introduction of computer hardware. of course, the distinction between context-sensitivity and polysemy is anything but clear-cut; this is a slippery theoretical slope, on which it is best to tread lightly. even so, enabling the detection of such diachronic lexical semantic change across time could accelerate research in historical linguistics @xcite , and also initiate the development of decision-making systems that exploit diachronically shifting information @xcite . the backbone of a lexical semantic change detection system is word embeddings, which represent the meaning (or at least the use) of words. different systems rely on various types of language models, nowadays predominantly distributional in nature. there is a comparably rich literature on distributional approaches to modeling diachronic semantic drift; examples include @xcite , and @xcite . more complete overviews of existing diachronic semantic shift detection techniques is provided in @xcite and @xcite . contextualized language models constitute a recent breakthrough in the field of nlp @xcite , by virtue of their ability to provide embeddings that are sensitive to a specific context of use, which is different from standard word embeddings that aggregate all of a word's contexts into one global representation. another way of characterizing this difference is to say that contextualized language models provide token-based representations, while standard word embeddings are type-based. motivated by the success of contextualized language models for handling polysemy and context-sensitivity, we investigate how such embeddings can be used to model diachronic semantic change; we use a contextualized language model as the basis of our proposed system for both sub-tasks of the unsupervised lexical semantic change detection task, featured in semeval 2020. more specifically, we produce contextualized embeddings for each occurrence of a term, and cluster these embeddings to arrive at a form of sense clusters. by leveraging multilingual contextualized representations, our approach is agnostic to which language is used in the input corpus, and it does not rely on any specific information about when the corpus was written. despite the simplicity of our approach, our system ranked 10 th in subtask 1 and 8 th in subtask 2. ## results the code for the experiments is made publicly available foot_4 . the best performing baseline is the cnt+ci+cd model, a co-occurrence counting method @xcite . for subtask 1 our method outperforms the best performing baseline on average, but performs worse for english and german. for subtask 2 our method outperforms the best performing baseline for all languages by a wide margin except for latin, where the performance is only slightly better. ## discussion & conclusion we chose xlm-r because it is a pre-trained, performant, single, contextualized model trained on all the languages in the task. as such, the method easily extends to all other languages xlm-r has been trained on. however, the choice of xlm-r has certain drawbacks. it is trained on commoncrawl data, which we assume is heavily skewed towards contemporary language. this might have a negative impact on model performance, since the task is dominated by historical data. more importantly, we believe that the biggest drawback of using xlm-r (or similar language models) is that it is trained with minimal preprocessing. the task data, on the other hand, was heavily lemmatized, pos-tagged (for english), and had frequent ocr errors. we believe this mismatch in preprocessing methods and data quality has had a very detrimental effect on the quality of the contextualized embeddings we extract from xlm-r. our approach, while not yielding great results, outperformed the baselines and scored relatively well on task 2. we argue that it performed surprisingly well given the simplicity of the approach: our method can be condensed into: based on the simplicity and shortcomings of the clustering approach we believe the relatively good performance can be attributed to the use of contextualized language models, and by extension that (i) is at least partly true. we believe that by improving the second step, i.e. the grouping of sets of contextualized embeddings into appropriate clusters, we could improve performance significantly.