RIJP at S em E val-2020 Task 1: G aussian-based Embeddings for Semantic Change Detection (2020)

Abstract: This paper describes the model proposed and submitted by our RIJP team to SemEval 2020 Task1: Unsupervised Lexical Semantic Change Detection. In the model, words are represented by Gaussian distributions. For Subtask 1, the model achieved average scores of 0.51 and 0.70 in the evaluation and post-evaluation processes, respectively. The higher score in the post-evaluation process than that in the evaluation process was achieved owing to appropriate parameter tuning. The results indicate that the proposed Gaussian-based embedding model is able to express semantic shifts while having a low computational

## introduction words change their meaning over time. long-and short-term semantic shifts relate to cultural and social changes @xcite @xcite . semantic shifts have been actively studied as evidenced by recent comprehensive survey papers @xcite and the success of the workshop on lexical-semantic change held as part of the acl 2019. in particular, many teams participated in the semeval-2020 task 1: unsupervised lexical semantic change detection @xcite . in this task, participants are required to devise an automated system for finding semantic shift in four languages: english, german, swedish, and latin. it consists of two subtasks: classification and ranking. in the classification task, words are categorized by the change in their meaning in two corpora, c 1 and c 2 , for periods t 1 and t 2 . in the ranking task, words are sorted by the degree of their semantic shift between periods t 1 and t 2 . the focus of this study is on detecting semantic shifts using word embedding. to tackle this task, our rijp team represented words by gaussian distributions, taking an inspiration from gaussian embedding @xcite . in gaussian embedding, words are represented by one gaussian distribution with mean vectors and covariance matrices. the mean vectors map the words in embedding space, while the covariance matrices represent about word hierarchies. instead of obtaining the variance using a trained model as suggested by existing studies, it is derived directly from the word frequency. it particular, two embeddings are created using each of the two corpora, and the semantic change is calculated based on the kullback-leiblar (kl) divergence. the proposed model achieved scores of 0.51 and 0.14 in the evaluation period for subtasks 1 and 2, respectively. after parameter tuning in the post-evaluation process, scores of 0.70 and 0.41 were achieved in subtask 1 and 2. experimental results demonstrated that the proposed gaussian-based embedding model can successfully detect various types of semantic shifts such as narrowing and widening. the code of the model will be releasing shortly. ## related work word senses generate and disappear over time @xcite . distributed representations have made significant advances in the field of natural language processing. word embeddings such as word2vec @xcite and glove @xcite allow representing semantic patterns through vectors. @xcite applied the word embedding method to several corpora, each covering ten year, to detect semantic shifts. semantic changes of a word can be captured by the difference of collocation of other words. the majority of studies on semantic shift detection represent a word by a single vector, few studies use different embedding types such as gaussian @xcite and poincaré embeddings @xcite . semantic shifts are often measured in terms of the similarity of the word vectors between two periods with the cosine similarity and euclidean distance being the most widely used metric. only a few papers @xcite acknowledge similarity metrics suitable for gaussian embedding. ## model description this study employs an improved structure of gaussian embedding for detecting semantic shifts. according to this structure, each word is represented by a mean vector and a covariance matrix. to reduce the computational complexity, only the mean vectors are learned using the word2vec framework. the covariance matrices of words are obtained directly from their frequencies, i.e., they are not learned. ## experimental setup in the experiment, corpora c 1 and c 2 were used to create the gaussian embedding, while the kl divergence was adopted as the criterion for measuring the semantic shift. the obtained scores were sorted for subtask1: ranking and the classification task was solved with appropriate thresholds for each language. parameters allowing to achieve the highest accuracy in the post-evaluation period were employed in the proposed method. for learning mean vectors, the embedding size d was set to 300, while the window size was set to 5. the other parameters were set to the default values of the gensim library. the settings for equation 3 were α = 0.02 and β = 3. two other methods were employed for comparison with the proposed method. in the first method, only the mean vectors were used and the cosine similarity was employed to measure the semantic shift. in the second method, only covariance matrices were used, while the semantic shift was measured based on the difference in covariances between t 1 and t 2 . the same parameters were used in all methods. the threshold of subtask 1 was set to maximize the classification score. while the threshold was set for each language separately, the number or parameters can be reduced the parameters by considering moving the seed words (e.g. country names) as future work. ## results the results are shown in it can be noticed from the relationship between the frequency information and spearman's coefficient is analyzed. we focused on german and english because the proposed method achieved a higher spearman's coefficient on the german dataset but not the english dataset. for example, the obtained embedding could detect the semantic change of a noun knotenpunkt from "junction" to "hub" in german. this noun was first used to mean junction but then changed its meaning to hub, and was used with that letter meaning more often compared to the earlier meaning. thus, the frequency information is useful for identifying semantic shifts. the verb abdecken"uncover" is an example, where both the word2vec and the proposed methods were able to detect a semantic shift. the old meaning of this verb referred to taking the skin off an animal carcass, whereas now it refers to covering up. if the old meaning disappeared, it would be possible to detect the meaning using only word2vec. the spearman's coefficients obtained using the word2vec and proposed method did not change for the english dataset. there were few examples in the dataset that could be identified as changes in the hypernym-hyponym semantic shift of words. word2vec and the proposed embedding could detect semantic shifts for the noun "graft", which was first used to mean a grafting tree but later was used to mean an implant. this is an example of the development of another meaning of a word. the proposed method was found to be able to capture semantic shifts of narrowing and widening with variances. it would be desirable to find a way to reducing the number of parameters when no test data is available, which was the case in the evaluation process of this study. the majority of existing machine learning methods require parameter tuning. only a few of them play a major role in the task of detecting semantic shift, where the correct answer is unknown in advance. ## conclusion this paper described the systems submitted to semeval 2020 task1: unsupervised lexical semantic change detection. we designed to detect semantic shifts based on an improved gaussian embedding method. according to the proposed method, the mean vectors are trained using the word2vec algorithm, while the variances are obtained from word frequencies. the computational complexity and parameter sensitivity to initial values of the proposed method is reduced compared to the traditional gaussian embedding method. the proposed method that used mean vectors and variances achieved higher spearman's coefficients for several languages in subtask 2 compared to the technique that used only mean vectors (word2vec). this result indicates that variances are effective for detetcting the changes of word abstractness. as part of our future work, we plan to focus on semantic shift detection across multiple languages and detailed classification of semantic shift types.