IMS at S em E val-2020 Task 1: How Low Can You Go? Dimensionality in Lexical Semantic Change Detection (2020)

Abstract: We present the results of our system for SemEval-2020 Task 1 that exploits a commonly used lexical semantic change detection model based on Skip-Gram with Negative Sampling. Our system focuses on Vector Initialization (VI) alignment, compares VI to the currently top-ranking models for Subtask 2 and demonstrates that these can be outperformed if we optimize VI dimensionality. We demonstrate that differences in performance can largely be attributed to model-specific sources of noise, and we reveal a strong relationship between dimensionality and frequency-induced noise in VI alignment. Our results suggest that lexical semantic change models integrating vector space alignment should pay more attention to the role of the dimensionality parameter.

## introduction lexical semantic change (lsc) detection has drawn increasing attention in recent years @xcite . semeval-2020 task 1 provides a multi-lingual evaluation framework to compare the variety of proposed model architectures @xcite ). an important component of high-performance lsc detection models is an alignment method to make semantic vector spaces comparable across time. in this paper we focus on a particular alignment method for type embeddings, vector initialization (vi), and how its performance interacts with vector dimensionality. we compare vi to two further state-of-the-art alignment methods, orthogonal procrustes (op) and word injection (wi), which have shown high performance in previous studies @xcite @xcite and are also used in the top-ranking systems for subtask 2. a systematic comparison of performance across dimensionalities d reveals that the optimal d of the models on the semeval test data is lower than in standard choices, and that vi's performance strongly depends on d, showing large drops for high dimensionalities. we demonstrate that this effect is correlated with the amount of frequency noise picked up by vi, i.e., the degree to which cosine distances between vectors reflect frequency differences between words rather than semantic differences. if properly tuned regarding dimensionalities and noise, vi outperforms op and wi as alignment method. ## related work the semantic representations we test fall into the large body of work on distributional semantic vector space models (turney and pantel, 2010) and represent specific instances of type-based word embeddings @xcite . the need for vector space alignment in lsc detection is shared with bilingual lexicon induction @xcite and term extraction @xcite where corpus-specific semantic representations need to be mapped to common coordinate axes. alignment techniques introduce varying levels of noise @xcite , and the noise level (the signal-to-noise-ratio) determines the optimal dimensionality of word embeddings @xcite . we regard information contained within the semantic representation capturing anything but semantic relations between words as noise (e.g. word frequency). sources of noise include the corpora, the representation method and alignment techniques. consequently, a specific semantic representation learning algorithm (such as skip-gram with negative sampling) may have a different optimal dimensionality depending on the alignment technique it relies on. up to now, previous research on lsc detection has not paid much attention to this relationship between dimensionality and noise: models with different susceptibilities to noise have typically been tested without varying the dimensionality @xcite @xcite . ## system overview most models in lsc detection combine three sub-systems: (i) creating semantic word representations, (ii) aligning them across corpora, and (iii) measuring differences between the aligned representations @xcite . semantic representations can either be token-based, keeping one representation (e.g. a vector) per word use @xcite , or type-based, collapsing information from different uses into one representation @xcite . the alignment step is needed mostly for vector space models, which might otherwise introduce arbitrary orthogonal transformations to the vector spaces they produce @xcite . our system focuses on the type-based skip-gram negative sampling (sgns) with vector initialisation alignment (vi) and cosine distance (cd). we chose this method due to its surprisingly good performance with @xmath0, (i) orthogonal procrustes (op) and (ii) word injection (wi). for our experiments we use the code provided by @xcite . 2 ## experimental setup semeval-2020 task 1 comprises a binary classification task (subtask 1) and a ranking task (subtask 2) on data from four languages: english, german, latin and swedish @xcite . subtask 1 asks participants to decide which target words lost or gained senses between corpora from two time periods t 1 and t 2 , and which ones did not. subtask 2 asks participants to rank a set of target words according to their degree of lsc (change in sense frequency distribution) between t 1 and t 2 . the tasks are different in that it is possible for a word to show a high degree of lsc in subtask 2, while not gaining or losing a sense in subtask 1 (or vice versa). for example the german word abgebrüht is used to describe (1) the process of cooking food in water and (2) an emotionally insensitive person. both senses are present across time periods t 1 and t 2 , but sense 1 dominates period t 1 while sense 2 dominates t 2 . thus, the number of senses has not changed, but the word has undergone significant semantic change. the four languages have a list of 31 to 48 target words, each annotated with values for subtasks 1 and 2. performance of a model is measured by accuracy and spearman's rank-order correlation coefficient. for each of the four languages two corpora (corpus 1 , corpus 2 ) are provided by the organizers, containing sentences from different time periods. these corpora show strong differences in terms of size, time-period and genre (see appendix a), posing a very heterogeneous, challenging setting for evaluation and parameter tuning. 4 5 results ## conclusion our shared task system investigated vector initialization (vi) alignment in a commonly used lsc detection model based on skip-gram with negative sampling, while focusing on the role of vectorspace dimensionality. our results suggest that lsc detection models integrating vector-space alignment should pay more attention to model-specific characteristics and the dimensionality parameter in particular. current state-of-the-art models are often dominated by applying op and wi to alignment, as a wide variety of reasonable parameters yield good results, whereas vi is very susceptible to parameters like training order, dimensionality and epochs. however we demonstrate that vi is able to outperform op and wi alignment if tuned properly. due to time limitations we could not fully explore the effects of epochs on vi, which have proven to play a significant role for dimensionality-dependent performance. future work will include a closer look at the connection between dimensionality, frequency noise and training epochs.