GM - CTSC at S em E val-2020 Task 1: G aussian Mixtures Cross Temporal Similarity Clustering (2020)

Abstract: This paper describes the system proposed by the Random team for SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection. We focus our approach on the detection problem. Given the semantics of words captured by temporal word embeddings in different time periods, we investigate the use of unsupervised methods to detect when the target word has gained or lost senses. To this end, we define a new algorithm based on Gaussian Mixture Models to cluster the target similarities computed over the two periods. We compare the proposed approach with a number of similarity-based thresholds. We found that, although the performance of the detection methods varies across the word embedding algorithms, the combination of Gaussian Mixture with Temporal Referencing resulted in our best system.

## introduction the recent development in word embeddings, and their increasing capability to capture lexical semantics has inspired the application of these methods to new tasks and introduced new challenges. the diachronic analysis of language is one of these linguistic tasks which has benefited from the advantages of these new methods, i.e. the capability to build semantic representations of words by skimming through large corpora spanning multiple time periods. semeval 2020 task 1 @xcite addresses the current lack of a systematic approach for the evaluation of automatic methods for diachronic analysis by proposing a common evaluation framework that comprises two tasks and covers corpora written in four different languages, namely german @xcite , english @xcite , latin @xcite , and swedish @xcite . given two corpora c 1 and c 2 for two periods t 1 and t 2 , subtask 1 requires participants to classify a set of target words in two categories: words that have lost or gained senses from t 1 to t 2 and words that did not, while subtask 2 requires participants to rank the target words according to their degree of lexical semantic change between the two periods. we tackle the problem of automatically detecting lexical semantic changes with approaches that rely on temporal word embeddings. these approaches create a word vector representation for each time period by exploiting a shared semantic space. similarity measures can then be used to capture the extent of a word semantic change between two periods. some temporal word embedding techniques adopt a two-step approach, where they first learn separate word embeddings for each time period and then align the word vectors across multiple time periods @xcite . other dynamic approaches incorporate the alignment directly into the learning stage via the optimisation function @xcite . dynamic word embeddings can be further categorised according to the constraint imposed on the alignment. the explicit alignment adopts a conservative approach to the semantic drift that a word can undergo by posing a limit to the distance between the word vectors belonging to the two temporal spaces. in the implicit alignment, there is no need for explicit constraint since the alignment is automatically performed by sharing the same word context vectors across all the time lapses. in this work, we focus on dynamic word embeddings by exploring methods based on both explicit, such as dynamic word2vec @xcite , and implicit alignment, namely temporal random indexing @xcite and temporal referencing @xcite . we analyse the use of different similarity measures to determine the extent of a word semantic change and compare the cosine similarity with pearson correlation and the neighborhood similarity @xcite . while these similarity measures can be directly employed to generate a ranked list of words for subtask 2, their adoption in subtask 1 requires further manipulation. we introduce a new method to classify changing vs. stable words by clustering the target similarity distributions via gaussian mixture models. we describe the embedding models and the clustering algorithm in section 2, while section 3 provides details about the hyper-parameter selection. section 4 reports the results of the task evaluation followed by some concluding remarks in section 5. ## system description we model the problem of automatic detection of semantic change by exploiting temporal word embeddings e i : @xmath0. given two different time periods t 1 and t 2 , we create two embeddings e 1 and e 2 . we investigate several models to compute temporal word embeddings: dynamic word2vec (dw2v) @xcite simultaneously learns time-aware embeddings by aligning and reducing the dimensionality of time-binned positive point-wise mutual information matrices. temporal random indexing (tri) @xcite implicitly aligns co-occurrence matrices by using the same random projection for all the temporal bins. collocations extracts for each word and each time period the set of relevant collocations through the dice score. as similarity function, we measure the cosine similarity between the sets of collocations belonging to the two different time periods. more details are reported in @xcite . temporal referencing (tr) @xcite used only in the post-evaluation, it consists in a modified version of word2vec skipgram that adds a temporal referencing to target vectors, keeping context vectors unchanged. a similarity measure between vectors in the two temporal spaces is adopted to compute the extent of the semantic drift of the target words. we explored several similarity measures: cosine similarity (cs) is the cosine of the angle between two vectors. pearson correlation (pc) measures the linear correlation between two variables, in case of centred vectors (with zero means) is equivalent to the cosine similarity. neighborhood similarity (ns) computes two k-neighbour sets nbrs k (e 1 (w)) and nbrs k (e 2 (w)) and the union set two second-order vectors, one for each word representation u j , are created. the components of u i are the cosine similarity between the vector v j 1 and the i-th element of u: u j @xmath1, u(i)). the neighborhood similarity is the cosine similarity between the second-order vectors. in all the experiments we set @xmath2. ## experimental setup in all the runs, we do not pre-process data and we use a context window size of 5 while analyzing sentences. the t r model foot_3 has been adopted into its original implementation foot_4 , as the t ri foot_5 approach and dw 2v foot_6 one. for runs involving t ri, we experimented with a varying vector size from 200 to 1, 000. moreover, we investigated (1) the initialization of the count matrix at time j with the matrix at time j -1, (2) the contribution of positive-only projections, and (3) the application of ppmi weights, as explained in @xcite . for dw 2v , we use the parameter setting proposed in @xcite . we set λ = 10, τ = 50, γ = 100, ρ = 50 and experimented with a number of iterations from one to five. as vocabulary, we kept the top 50,000 most frequent tokens for both t ri and dw 2v . in the t r runs, we set the vector size to 100, and we experimented eight iterations for english and latin, and four for german and swedish. we use 20 negative samples, keeping only the tokens that occur at least 10 times. all the other parameters used for configuring the models are reported in tab. 1. ## results semeval 2020 task 1 provide three baselines, namely freq. baseline, which uses the absolute difference of the normalized frequency in the two corpora as a measure of change; count baseline, which implements the model described in @xcite ; and maj. baseline that always predicts the majority class. tab. 2 reports the main results obtained by the different models. it shows the results obtained from the official submissions at the challenge and the results obtained by the t r approach performed during the post-evaluation phase. the results obtained for subtask 1 are reported using the accuracy metric, while for subtask 2, the spearman's rank-order correlation coefficients are used. considering the results of the evaluation phase, the models show inconsistent behaviors. t ri showed the best performance when considering "all the languages" for both subtasks, although in subtask 1 it is not able to overcome count baseline and maj. baseline. focusing on subtask 1, if we consider each language in isolation, we see that dw 2v gives the best results for english foot_7 while overall p c is our best system for german language, although it is not able to overcome count baseline. collocation is the best system for latin (although outperformed by freq. baseline) while t ri is our best system for sweden language. in subtask 2, the best english score was reported by overall n s . overall cs (collocation) performed the best in german language. for latin and sweden, t ri provided the best results, and interestingly, it is one of the few systems that did not generate a negative correlation, although outperformed by countbaseline in latin language. at the end of the challenge, when the labelled test set was released, we performed more experiments reported in the post-evaluation row. in this phase, we run an additional system, t r, which outperformed best performance in one language while performing differently on the remaining others. during the post-evaluation, we decided to investigate also the role of gmms for class labeling (sec. 2). we compared gmms with semi-manual thresholds µ s , µ s -σ s , µ s + σ s and winsorizing (kokic and bell, 1994) computing µ s and σ s on data provided for subtask 1, where µ s and σ s are the mean and the standard deviation computed on the similarity set s. figure 1 reports the different accuracy scores obtained by the five methods for the t ri, collocation, dw 2v , t r approaches. the scores for the gmms strategy are close to those obtained by µ s for tri and collocation. while gmms outperforms µ s + σ s in every run, µ s -σ s seems to work better than gmms except that in t r. winsorizing works better than gmms in t ri and collocation. gmms outperforms winsorizing in dw 2v and t r. these results are not clear enough to advocate for a specific threshold. consequently, further analysis will be part of future work in order to understand what is the better threshold that could be included in the gmms process. ## conclusions we described the runs we submitted to the semeval-2020 task 1: unsupervised lexical semantic change detection. this paper has two main contributions. we reported a comparison of some of the most recent approaches to model lexical semantic change with temporal word embeddings, and we experimented with an automatic unsupervised procedure to classify changing and stable words. results show that implicit alignment works generally better in modelling the lexical semantic change. in future works we plan to carry out an analysis on unlemmatised corpora and gauge a better understanding of the impact of gaussian mixture clustering for unsupervised lexical semantic change detection.