CMCE at S em E val-2020 Task 1: Clustering on Manifolds of Contextualized Embeddings to Detect Historical Meaning Shifts (2020)

Abstract: This paper describes the system Clustering on Manifolds of Contextualized Embeddings (CMCE) submitted to the SemEval-2020 Task 1 on Unsupervised Lexical Semantic Change Detection. Subtask 1 asks to identify whether or not a word gained/lost a sense across two time periods. Subtask 2 is about computing a ranking of words according to the amount of change their senses underwent. Our system uses contextualized word embeddings from MBERT, whose dimensionality we reduce with an autoencoder and the UMAP algorithm, to be able to use a wider array of clustering algorithms that can automatically determine the number of clusters. We use Hierarchical Density Based Clustering (HDBSCAN) and compare it to Gaussian MixtureModels (GMMs) and other clustering algorithms. Remarkably, with only 10 dimensional MBERT embeddings (reduced from the original size of 768), our submitted model performs best on subtask 1 for English and ranks third in subtask 2 for English. In addition to describing our system, we discuss our hyperparameter configurations and examine why our system lags behind for the other languages involved in the shared task (German, Swedish, Latin). Our code is available at https://github.com/DavidRother/semeval2020-task1

## introduction with the advent of increasingly sophisticated distributional models, semantic change analysis has gained considerable popularity in natural language processing. earlier methods use static word embeddings to identify laws of semantic change @xcite @xcite . recently, contextualized embeddings @xcite have allowed to track more fine-grained sense changes of words over time @xcite . the semeval-2020 "unsupervised lexical semantic change detection task" @xcite ) aims at two levels of granularity to detect lexical semantic change: 1) a binary classification task to identify whether or not a word lost/gained a sense between two time periods, and 2) a ranking task, where a list of words has to be ordered by the amount of change these have undergone between the same two time periods. both subtasks have to be solved for four languages: english, german, latin and swedish. to address the tasks, we deploy a language-agnostic system with three ingredients. i) we represent words by multilingual contextualized word embeddings (mbert) @xcite . ii) we apply dimensionality reduction on the contextualized word embeddings with an autoencoder and the umap algorithm (a type of manifold learning), as it is much easier and more efficient to work with low dimensional vectors in subsequent steps, allowing the application of a wider range of clustering algorithms. iii) we employ a hierarchical clustering approach to find potential 'sense clusters' for each word. these sense clusters can then be used to find sense losses or gains and shifts in the sense distribution between two time epochs. while our system ranks in the mid-field overall, we perform best for english in subtask 1 and third in subtask 2. while we can only speculate about the approaches of the other participants, we believe this to be due to our language-agnostic approach, which uses no language-specific resources whatsoever except for large-scale contextualized embeddings, possibly limiting its success in lower-resource languages such as latin. our analysis further suggests that hyperparameter tuning would have been crucial for better performances in swedish and german. for the test data, a ranking was determined according to the jensen-shannon distance (jsd) between the distribution of senses between both epochs for each word. ultimately, the fit of the system is computed by calculating the spearman rank correlation between the model output and the gold ranking. ## system overview to build a system that accomplishes these tasks, we use a pre-trained deep language model in order to have a meaningful and rich representation of words that is sensitive to the respective context. we then employ unsupervised clustering algorithms to combine the different representations and additionally try dimensionality reduction methods to boost clustering performance. in our submission we use a pipeline that implements those steps. the language model we use is mbert from which we retrieve contextualized word embeddings of the focus words from its last layer. we subsequently perform dimensionality reduction by first training and applying an autoencoder model on the data and then apply the umap algorithm afterwards. for the purpose of finding the different sense clusters we use hierarchical density based clustering (hdbscan) as a last step. for illustration, we plot the instances for the word 'chairman' in figure 1 . our clustering finds two distinct senses which corresponds to the annotation in wordnet @xcite . the official competition scores of our submission are shown in ## conclusion our contribution to the semeval-2020 "unsupervised lexical semantic change detection task" identifies lexical semantic sense change in an unsupervised, knowledge-free, and language-agnostic manner. it utilizes contextualized word embeddings to determine implicit sense information from the context of words. these contextual word vectors are reduced in dimensionality with an autoencoder and umap to enable clustering with a hierarchical density based clustering (hdbscan), which is an appealing clustering algorithm compared with other, more complex architectures. we found that our chosen hyperparameters for english yielded good results, and further post-evaluation confirmed this. however, for the other languages, our hyperparameter configuration was not optimal, as evidenced by improved performance in post-evaluation with other hyperparameter choices. overall, we showed that, for the given task of semantic change analysis, we can achieve best or near-best performance for one of the shared task's languages even when massivly condensing the information from mbert's contextualized embeddings, yielding a very time-and resource-efficient solution to the problem (that has a memory footprint of 1% of the original embeddings after the dimensionality reduction). that our system has rather mediocre performance for the other languages may also hint at a certain randomness in the task definition (e.g., w.r.t. the parameters k and n) or as arising from the rather small sizes or selection of some of the involved corpora or their annotation process.