Discovery Team at S em E val-2020 Task 1: Context-sensitive Embeddings Not Always Better than Static for Semantic Change Detection (2020)

Abstract: This paper describes the approaches used by the Discovery Team to solve SemEval-2020 Task 1 - Unsupervised Lexical Semantic Change Detection. The proposed method is based on clustering of BERT contextual embeddings, followed by a comparison of cluster distributions across time. The best results were obtained by an ensemble of this method and static Word2Vec embeddings. According to the official results, our approach proved the best for Latin in Subtask 2.

## introduction each word has a variety of senses and connotations, constantly evolving through usage in social interactions and changes in cultural and social practices. identifying and understanding these changes is important for linguistic research and social analysis, since it allows the detection of cultural and linguistic trends and possibly predict future changes. detecting these changes can also be used to improve many nlp tasks, such as text classification and information retrieval. the semeval-2020 task 1 -unsupervised lexical semantic change detection @xcite deals with detection of semantic change in temporal corpora containing texts from two distinct time periods in four languages: english, german, latin and swedish. the challenge defines two subtasks: subtask 1 is binary classification, i.e., to determine whether a word has changed or not; subtask 2 aims at ranking a set of target words according to their rate of semantic change. in this paper, we present the approaches used by the discovery team to tackle these two subtasks. the discovery team qualified as 11th and 5th on subtasks 1 and 2, respectively, and also proved the best for latin language in subtask 2. our systems leverage the transformer-based bert model to generate contextualised embeddings for each word usage. then these embeddings are aggregated into meaningful time-specific word representations. we explore different aggregation techniques, such as clustering (k-means and affinity propagation) and averaging. we also combine bert-based representations with static word2vec embeddings 1 . ## conclusion we present the approaches employed by the discovery team to tackle semeval-2020 task 1: unsupervised lexical semantic change detection @xcite . while our main method was based on clustering bert contextualised embeddings, the best official result was obtained by combining this technique with a method for semantic change detection based on static word2vec embeddings. the methods based on contextualised embeddings with clustering are outperformed by averaging of contextualised embeddings and static embeddings methods. other task participants, in particular the winning team of subtask 2, used similar static and contextual methods and reached the same conclusion on the adequacy of static embeddings for these specific tasks and corpora @xcite . however, the discrepancy among languages is significant and the results averaged on all four corpora can be misleading. a more thorough analysis on how different embeddings perform in different settings (short or long term semantic change, type of corpus preprocessing, etc...) and different languages will be performed in the future work.