JCT at S em E val-2020 Task 1: Combined Semantic Vector Spaces Models for Unsupervised Lexical Semantic Change Detection (2020)

Abstract: In this paper, we present our contribution in SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection, where we systematically combine existing models for unsupervised capturing of lexical semantic change across time in text corpora of German, English, Latin and Swedish. In particular, we analyze the score distribution of existing models. Then we define a general threshold, adjust it independently to each of the models and measure the models’ score reliability. Finally, using both the threshold and score reliability, we aggregate the models for the two sub- tasks: binary classification and ranking.

## introduction over the last decade, research on detection of lexical semantic change has increased. many studies were performed on various languages, corpora and periods. two years ago, two literature surveys on computational approaches to lexical semantic change (lsc) @xcite were published. last year @xcite first systematically compared a broad variety of lsc detection models on two data sets of different periods and domains and @xcite proposed a new evaluation framework for semantic change detection using word embeddings. to facilitate the comparison of different systems, semeval-2020 task 1 @xcite ) introduced a simple evaluation framework for unsupervised lexical semantic change detection in text corpora of german, english, latin and swedish. the task relies on the comparison of two time periods for each language. we participated in two sub-tasks: a classification task, where we decide which words lost or gained senses between the periods, and a ranking task, where we rank a set of target words according to their degree of lexical semantic change between the periods. given the large number of models that have already been explored, we built a system which systematically combines existing models in the unsupervised setting of the lsc detection task. since no tuning data is available, we minimized the parameters number of our system. this paper is organized as follows: first, in section 2, we describe the existing lsc detection models that we have combined. then, in section 3, we analyze the score distribution of the models in order to learn the general behaviour of words in our corpora. we aim to estimate the amount of words that changed their meaning between the periods. next, we define a general classification threshold percentile (ct) parameter and adapt it to each model separately. we use the ct parameter to measure the models' score certainty too, and filtered models with certainty low than a minimal required decision certainty. the minimal required decision certainty (mcr) is also a parameter of our system. finally, we present our aggregation methods for the two sub-tasks: classification and ranking. our system results are detailed in section 4, followed by conclusions in section 5. ## related work in this section, we shortly describe the work of @xcite which covers a wide range of lsc detection models. then, we summarize the models that we have combined in our system. this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . @xcite made a comprehensive comparison between the results of the diverse existing lsc detection models and firstly ran the models under one evaluation task and data. @xcite tested the methods for semantic change detection both across time (diachronic) and across domains (synchronic). for the diachronic task they used the evaluation framework and the durel german corpora (introduced in @xcite ). the framework was expanded for the synchronic task using the surel german corpora @xcite . existing lsc detection models are based on three methods for meaning representations: semantic vector spaces @xcite @xcite , topic distributions @xcite , and sense clusters @xcite . in semantic vector spaces, each term is represented as two vectors indicating its co-occurrence statistics at different eras. then, the semantic change is commonly measured by either similarity measures, such as cosine similarity, or contextual measures. in topic distributions, each term is modeled as a probability distribution over different topics, or term senses. then, the semantic change is measured based on the senses' frequency of use. whereas, sense clustering and topic models are similar in their mapping (uses to senses) and semantic change measures, in sense clustering, some contextual property is used to assign all uses of a term into sense clusters. @xcite focused on two meaning representations: semantic vector spaces and topic distributions. they tested a list of known lsc detection models with different combinations of semantic representations, alignment methods and detection measures. they experimented various parameter settings for comparing the models' predictions with the true results. @xcite concluded that we could use the same modelling methods for both diachronic and synchronic lsc detection. in addition, all model predictions had a strong positive correlation with the true results. the model with the best performance was skip-gram with orthogonal procrustes alignment and cosine distance (sgns+op+cd). since @xcite observed that topic distributions models (scan) @xcite have poor and unstable performance, in our system we integrated only the semantic vector spaces models (reminded at @xcite ). the combined models' representations were: raw count, positive pointwise mutual information (ppmi) @xcite , singular value decomposition (svd) @xcite , random indexing (ri) @xcite and skip-gram with negative sampling (sgns) @xcite @xcite . the integrated alignment methods were: column intersection (ci) @xcite , shared random vectors (srv) @xcite , orthogonal procrustes (op) @xcite , vector initialization (vi) @xcite and word injection (wi) @xcite . two types of measures were combined: similarity measures (cosine distance (cd) and local neighbourhood distance(lnd) @xcite ) and dispersion measures (frequency difference (fd), type difference (td) and entropy difference (hd) @xcite ). for running these models, we used the scripts for vector space representation, alignment, measuring lsc and evaluation that available at https://github.com/garrafao/lscdetection . we set the same parameters for all models' combination (@xmath0, @xmath1, @xmath2, @xmath3, @xmath4. ## system description two time-specific corpora for each of the four languages, german, english, latin and swedish, were provided by the task organizers. each line contains one sentence, where the punctuation was eliminated and each token was replaced by its lemma. within each corpus sentences were shuffled randomly. one-word sentences were removed form the latin corpus and for the other languages, sentences with less than 10 tokens were removed. due to the big size of the corpora, we removed low-frequency words for improving the efficiency of the models. the input of the system for each language is a corpus pair and a list of target words. our system scripts are publicly available on github https: //github.com/efratiamar/combinedmodelslsc . ## results given the unsupervised lsc detection task, our concept was to use a minimum number of parameters beyond the models' parameters. our system used two parameters (see section 3): general classification threshold (ct) and minimal required decision certainty (mrc). in in the evaluation phase our system achieved the highest score with configuration number 4 (@xmath5, @xmath6.5). our system was ranked 8 th in task 1 (@xmath7.636) and 13 th in task 2 (@xmath8.254). as seen in in the post-evaluation phase, we realized that low ct and mcr achieved higher accuracy, but lower spearman correlation. as seen in we also looked at what the results could have been if we were trying to fine-tune the system parameters (ct and mcr) and set for each language the configuration with the highest performance. in this case, our system obtains an average accuracy of 0.685 and average spearman correlation of 0.436. this can be deduced from to test the performance of our system, we analyzed the results of the best configurations (no.8 and no. 12). we compared the score of each model separately to the score of our system that weighs all the models' scores together. we found that there is no consistent behaviour. in other words, there was not any model that consistently outperformed all the other models or our weighted score. in the case of the german and the swedish languages, the sgns-based models produced better results than the other models. in the other languages, other models performed better. for example: in task 1, the ppmi-wi-lnd and count-td models produced the best results in english, while in the swedish language, the best results were obtained by the ppmi-wi-lnd and count-td models. in task 2, the sgns-wi-cd model produced the best results in english, while in the swedish language, it was the sgns-op-cd model. we also noticed that dispersion measures models, which strongly rely on frequency, had low performance. this could be resulted from the fact that the organizers' controlled each test set for frequency (which we could not know before they published the task description paper). ## conclusions and future work we have implemented a system that systematically incorporates existing models to identify lsc over time in text corpora of four languages. we evaluated the score distribution of existing models, suggested a general classification threshold and applied it to each of the models individually. we calculated the models' score certainty and used it to aggregate the models. in the evaluation phase of the semeval-2020 task 1, our system was ranked 8 th and 13 th in the classification and ranking sub-tasks, respectively. we plan to investigate additional aggregation methods and explore the impact of the individual models on the combined system to improve our system results. we also plan to try our system on other languages of different families, such as semitic languages @xcite and use lsc models to construct diachronic thesaurus, which bridges the lexical gap between modern and ancient language @xcite @xcite .