G lo V e I nit at S em E val-2020 Task 1: Using G lo V e Vector Initialization for Unsupervised Lexical Semantic Change Detection (2020)

Abstract: This paper presents a vector initialization approach for the SemEval2020 Task 1: Unsupervised Lexical Semantic Change Detection. Given two corpora belonging to different time periods and a set of target words, this task requires us to classify whether a word gained or lost a sense over time (subtask 1) and to rank them on the basis of the changes in their word senses (subtask 2). The proposed approach is based on using Vector Initialization method to align GloVe embeddings. The idea is to consecutively train GloVe embeddings for both corpora, while using the first model to initialize the second one. This paper is based on the hypothesis that GloVe embeddings are more suited for the Vector Initialization method than SGNS embeddings. It presents an intuitive reasoning behind this hypothesis, and also talks about the impact of various factors and hyperparameters on the performance of the proposed approach. Our model ranks 12th and 10th among 33 teams in the two subtasks. The implementation has been shared publicly.

## introduction and background lexical semantic change (lsc) detection is an active research topic in the field of natural language processing, and has been applied for diachronic (across time) and synchronic (across domains) tasks @xcite . previously limited to manual "close-reading" approaches, the availability of large-scale corpora have allowed the use of computational methods for this task @xcite . this topic has found various applications in various disciplines such as improving information retrieval from historical documents (morsy and karypis, 2016), preventing cross-domain ambiguity in requirements elicitation interviews @xcite , and studying the impact of societal and cultural changes on word meanings and usage @xcite . lsc detection involves the use of two corpora c 1 and c 2 which, in the diachronic case, belong to different time periods t 1 and t 2 respectively. the various approaches found in the literature usually involve the construction of a word embedding space specific to each corpus. the embeddings can be constructed through count-based methods such as positive pointwise mutual information and random indexing, or predictive methods such as skip-gram with negative sampling (sgns). most of these models are stochastic in nature which means that separately trained embedding models live in their own space. in order to project them onto a unified space, alignment techniques such as vector initialization and orthogonal procrustes are used. the lsc of a word is then quantitatively determined by measuring the contextual dissimilarity between the word's representations @xcite . recently, there have been efforts to evaluate these various methods by comparing their results with manually-annotated data @xcite . the semeval-2020 task 1 is one such effort which is based on lsc detection in corpora of english, german, latin, and swedish languages @xcite . its aim is to provide an evaluation framework for unsupervised lsc detection systems by comparing their results against a ground truth, as annotated by native speakers or scholars. it consists of two subtasks: given two corpora c 1 and c 2 (for time periods t 1 and t 2 ), for a set of target words, 1. decide which words lost or gained senses between t 1 and t 2 , and which ones did not; as annotated by human judges. ## conclusion and future work in this paper, we reported our work in the semeval2020 task 1. we proposed a glove-based vi approach which achieved the 10th and 12th ranks out of 33 participating teams in the two subtasks. we gave a theoretical reasoning behind why glove is less sensitive towards high frequency than sgns and thus more suited for vi method, and empirically showed the magnitude of the difference between the number of updates to word embeddings in the two models. we believe that despite a lower-than-expected performance in this competition, our work presents a good case for the suitability of glove model when corpora of larger size are involved. however, proving this quantitatively is a challenging task because of the limitations associated with manual annotation-based evaluation.