HSE at LSCD iscovery in S panish: Clustering and Profiling for Lexical Semantic Change Discovery (2022)

Abstract: This paper describes the methods used for lexical semantic change discovery in Spanish. We tried the method based on BERT embeddings with clustering, the method based on grammatical profiles and the grammatical profiles method enhanced with permutation tests. BERT embeddings with clustering turned out to show the best results for both graded and binary semantic change detection outperforming the baseline. Our best submission for graded discovery was the 3rd best result, while for binary detection it was the 2nd place (precision) and the 7th place (both F1-score and recall). Our highest precision for binary detection was 0.75 and it was achieved due to improving grammatical profiling with permutation tests.

## introduction lexical semantic change detection (lscd) aims to identify which words and how change their meaning over time. lscd is usually divided into two subtasks: graded change and binary change detection. graded lscd is a subtask of ranking the intersection of (content-word) vocabularies according to their degree of change between a diachronic corpus pair c1 and c2 @xcite . in this shared task, the participants were asked to rank the set of content words in the lemma vocabulary intersection of c1 and c2 according to their degree of semantic change between c1 to c2. submissions were scored against 60 hidden words from the full target word list which were annotated for semantic change. the total number of target words were more than 4,000 (d. @xcite , and, as it was a discovery task, the target words were not preselected, balanced or cleaned. due to that, discovery is more problematic for models in comparison with semantic change detection, but it is an important task for lexicography. binary lscd is a subtask of identifying whether a target word lost or gained senses from the first set of its usage to the second, or not @xcite . previous shared tasks on lexical semantic change detection (lscd) were developed for english, german, latin, and swedish @xcite , italian @xcite , and russian @xcite . this one was in spanish (d. @xcite . spanish is a fusional romance language of the indo-european language family with rich morphology and a lot of national varieties. so far, lscd in shared tasks were developed for three romance languages, three german languages, and one slavic language. only two of them are analytical (english and swedish), while others are fusional. in this shared task we tested several methods. for graded change discovery we used bert embeddings with clustering @xcite . for binary change detection we used 3 methods. the first one was word embeddings again. two others were grammatical profiling @xcite and grammatical profiling combined with permutation tests @xcite . though grammatical profiles by themselves yield worse performance than embedding-based method, they could be significantly improved by applying of additional significance tests. ## results the submission results are presented in clustering turned out to be the best one among all our methods. in graded change discovery it was proved to be better than both baselines and took the 3rd place in the leaderboard. the clustering method was our only method that was applied to the optional gain/loss task, however, it did not show good results. while this method surpassed the baseline numbers, it proved to be significantly inferior to the other methods participating in the task. it probably happened because we approached the gain/loss task as a separate task. the better approach might have been to somehow use the results we received on the main binary task in order to calculate the gain/loss values. there is another problem with the method that we can think of. the method assigned a gain/loss label for the word if the number of clusters in two time epochs differs even by one. a better approach would probably have been to decrease the sensitivity of the method and to ignore the insignificant differences between the number of clusters. grammatical profiling demonstrated the worst results among three methods we used (see however, the results indicate that it was significantly improved by applying a permutation test. it should also be noted that grammatical profiling with a permutation test demonstrated the best precision among all participants and was only outperformed by the baseline 1. we also applied grammatical profiling for graded change discovery after the competition. the result was worse than baseline 1, but better than baseline 2 (see ## discussion ## conclusion further studies need to be carried out in order to evaluate the combination of profiling with statistical significance testing for other languages. the great advantage of grammatical profiling is that computational resources required for that method are quite low. it is helpful when the number of target words is great, like in this shared task for graded discovery. although the bert-based method demonstrated the best results, more detailed error analysis is still required.